[2024-08-19 23:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-19 23:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 537): INFO AMP_ENABLE: true AMP_OPT_LEVEL: '' AUG: AUTO_AUGMENT: rand-m9-mstd0.5-inc1 COLOR_JITTER: 0.4 CUTMIX: 1.0 CUTMIX_MINMAX: null MIXUP: 0.8 MIXUP_MODE: batch MIXUP_PROB: 1.0 MIXUP_SWITCH_PROB: 0.5 RECOUNT: 1 REMODE: pixel REPROB: 0.25 BASE: - '' DATA: BATCH_SIZE: 64 CACHE_MODE: part DATASET: imagenet DATA_PATH: /dataset/ImageNet_ILSVRC2012 IMG_SIZE: 224 INTERPOLATION: bicubic MASK_PATCH_SIZE: 32 MASK_RATIO: 0.6 NUM_WORKERS: 8 PERSISTENT_WORKERS: true PIN_MEMORY: true ZIP_MODE: false ENABLE_AMP: false EVAL_MODE: false FUSED_LAYERNORM: false MODEL: DDP: hfai DROP_PATH_RATE: 0.4 DROP_RATE: 0.0 LABEL_SMOOTHING: 0.1 MLLA: APE: false DEPTHS: - 2 - 4 - 8 - 4 DROP_PATH_RATE: 0.1 DROP_RATE: 0.0 EMBED_DIM: 64 IMAGE_SIZE: 224 IN_CHANS: 3 MLP_RATIO: 4.0 NUM_HEADS: - 2 - 4 - 8 - 16 PATCH_SIZE: 4 SIMPLE_DOWNSAMPLE: false SIMPLE_PATCH_EMBED: false MMCKPT: false NAME: msvmambav3_small_224 NUM_CLASSES: 1000 PRETRAINED: '' RESUME: '' RMT: CHUNKWISE_RECURRENTS: - true - true - false - false DEPTHS: - 2 - 2 - 6 - 2 DROP_PATH_RATE: 0.1 EMBED_DIMS: - 64 - 128 - 256 - 512 HEADS_RANGES: - 3 - 3 - 3 - 3 INIT_VALUES: - 1 - 1 - 1 - 1 LAYERSCALES: - false - false - false - false MLP_RATIOS: - 3 - 3 - 3 - 3 NUM_HEADS: - 3 - 6 - 12 - 24 PATCH_NORM: true TYPE: vssm VMAMBA2: APE: false ATTN_TYPES: - mamba2 - mamba2 - mamba2 - mamba2 BIDIRECTION: false DEPTHS: - 2 - 4 - 8 - 4 DROP_PATH_RATE: 0.2 DROP_RATE: 0.0 D_STATE: 64 EMBED_DIM: 64 IMAGE_SIZE: 224 IN_CHANS: 3 LEPE: false LINEAR_ATTN_DUALITY: false MLP_RATIO: 4.0 NUM_HEADS: - 2 - 4 - 8 - 16 PARTIAL_WIN_SIZE: -1 PATCH_SIZE: 4 RES_SCALE: - false - false - false - false SIMPLE_DOWNSAMPLE: true SIMPLE_PATCH_EMBED: true SSD_AEXP: false SSD_CHUNK_SIZE: 256 SSD_EXPANSION: 2 SSD_NGROUPS: 1 SSD_NORM_DA: false SSD_POSITIVE_DA: false WIN_ONLY: - false - false - false - false ZACT: false VSSM: ADD_SE: true AXIS_STAGE: [] CONVFFN: true CONV_FFN_RATIO: 4 DEPTHS: - 2 - 3 - 20 - 2 DOWNSAMPLE: v3 EMBED_DIM: 96 FULL_RES_INDEX: 1 GMLP: false IN_CHANS: 3 MLP_ACT_LAYER: gelu MLP_DROP_RATE: 0.0 MLP_RATIO: 4.0 NORM_LAYER: ln2d NUM_HEADS: - 1 - 2 - 4 - 8 PATCHEMBED: v2 PATCH_NORM: true PATCH_SIZE: 4 POSEMBED: false PRE_NORM: false SSM_ACT_LAYER: silu SSM_CONV: 3 SSM_CONV_BIAS: true SSM_DROP_RATE: 0.0 SSM_DT_RANK: auto SSM_D_STATE: 1 SSM_FORWARDTYPE: vms_noz SSM_INIT: v0 SSM_RANK_RATIO: 2.0 SSM_RATIO: 1.0 OUTPUT: ./exclude/output_msv2/msvmambav3_small_224/20240819232309 PRINT_FREQ: 10 SAVE_FREQ: 1 SEED: 0 TAG: '20240819232309' TEST: CROP: true SEQUENTIAL: false SHUFFLE: false THROUGHPUT_MODE: false TRAIN: ACCUMULATION_STEPS: 1 AUTO_RESUME: true BASE_LR: 0.002 CLIP_GRAD: 5.0 EPOCHS: 300 LAYER_DECAY: 1.0 LR_SCHEDULER: DECAY_EPOCHS: 30 DECAY_RATE: 0.1 GAMMA: 0.1 MULTISTEPS: [] NAME: cosine WARMUP_PREFIX: true MIN_LR: 2.0e-05 MOE: SAVE_MASTER: false OPTIMIZER: BETAS: - 0.9 - 0.999 EPS: 1.0e-08 MOMENTUM: 0.9 NAME: adamw START_EPOCH: 0 USE_CHECKPOINT: false WARMUP_EPOCHS: 20 WARMUP_LR: 2.0e-06 WEIGHT_DECAY: 0.05 TRAINCOST_MODE: false [2024-08-19 23:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 538): INFO {"cfg": "./configs/msv2/msvmambav3_small_224.yaml", "opts": null, "batch_size": 64, "data_path": "/dataset/ImageNet_ILSVRC2012", "zip": false, "cache_mode": "part", "pretrained": null, "resume": null, "accumulation_steps": null, "use_checkpoint": false, "disable_amp": false, "output": "./exclude/output_msv2", "tag": "20240819232309", "eval": false, "throughput": false, "fused_layernorm": false, "optim": null, "model_ema": true, "model_ema_decay": 0.99984, "model_ema_force_cpu": false, "memory_limit_rate": -1, "ddp": "hfai", "enable_preload": true, "enable_persistance": true, "mesa": false, "mesa_value": 1.0, "mute_repeat": false} [2024-08-19 23:27:17 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-19 23:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 135): INFO VSSM( (patch_embed): Sequential( (0): Conv2d(3, 48, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (1): Identity() (2): LayerNorm2d((48,), eps=1e-05, elementwise_affine=True) (3): Identity() (4): GELU(approximate='none') (5): Conv2d(48, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (6): Identity() (7): LayerNorm2d((96,), eps=1e-05, elementwise_affine=True) ) (layers): ModuleList( (0): Sequential( (blocks): Sequential( (0): VSSBlock( (norm): LayerNorm2d((96,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((96,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=96, out_features=12, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=12, out_features=96, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(96, 96, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=96) (in_proj): Linear2d(in_features=96, out_features=96, bias=False) (act): SiLU() (conv2d): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=96) (out_act): Identity() (out_proj): Linear2d(in_features=96, out_features=96, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.0) (convFFN): ConvFFN( (linear1): Conv2d(96, 384, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(384, 96, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) ) (norm2): LayerNorm2d((96,), eps=1e-05, elementwise_affine=True) ) (1): VSSBlock( (norm): LayerNorm2d((96,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((96,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=96, out_features=12, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=12, out_features=96, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(96, 96, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=96) (in_proj): Linear2d(in_features=96, out_features=96, bias=False) (act): SiLU() (conv2d): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=96) (out_act): Identity() (out_proj): Linear2d(in_features=96, out_features=96, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.015384615398943424) (convFFN): ConvFFN( (linear1): Conv2d(96, 384, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(384, 96, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) ) (norm2): LayerNorm2d((96,), eps=1e-05, elementwise_affine=True) ) ) (downsample): Sequential( (0): Identity() (1): Conv2d(96, 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (2): Identity() (3): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) ) ) (1): Sequential( (blocks): Sequential( (0): VSSBlock( (norm): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=192, out_features=24, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=24, out_features=192, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(192, 192, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=192) (in_proj): Linear2d(in_features=192, out_features=192, bias=False) (act): SiLU() (conv2d): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192) (out_act): Identity() (out_proj): Linear2d(in_features=192, out_features=192, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.03076923079788685) (convFFN): ConvFFN( (linear1): Conv2d(192, 768, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(768, 768, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=768) ) (norm2): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) ) (1): VSSBlock( (norm): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=192, out_features=24, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=24, out_features=192, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(192, 192, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=192) (in_proj): Linear2d(in_features=192, out_features=192, bias=False) (act): SiLU() (conv2d): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192) (out_act): Identity() (out_proj): Linear2d(in_features=192, out_features=192, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.04615384712815285) (convFFN): ConvFFN( (linear1): Conv2d(192, 768, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(768, 768, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=768) ) (norm2): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) ) (2): VSSBlock( (norm): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=192, out_features=24, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=24, out_features=192, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(192, 192, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=192) (in_proj): Linear2d(in_features=192, out_features=192, bias=False) (act): SiLU() (conv2d): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192) (out_act): Identity() (out_proj): Linear2d(in_features=192, out_features=192, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.0615384615957737) (convFFN): ConvFFN( (linear1): Conv2d(192, 768, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(768, 768, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=768) ) (norm2): LayerNorm2d((192,), eps=1e-05, elementwise_affine=True) ) ) (downsample): Sequential( (0): Identity() (1): Conv2d(192, 384, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (2): Identity() (3): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) ) (2): Sequential( (blocks): Sequential( (0): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.07692307978868484) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (1): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.0923076942563057) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (2): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.10769230872392654) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (3): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.1230769231915474) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (4): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.13846154510974884) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (5): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.1538461595773697) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (6): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.16923077404499054) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (7): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.1846153885126114) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (8): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.20000000298023224) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (9): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.2153846174478531) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (10): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.23076923191547394) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (11): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.2461538463830948) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (12): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.26153847575187683) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (13): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.2769230902194977) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (14): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.29230770468711853) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (15): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.3076923191547394) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (16): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.32307693362236023) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (17): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.3384615480899811) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (18): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.35384616255760193) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) (19): VSSBlock( (norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=384, out_features=48, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=48, out_features=384, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(384, 384, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=384) (in_proj): Linear2d(in_features=384, out_features=384, bias=False) (act): SiLU() (conv2d): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384) (out_act): Identity() (out_proj): Linear2d(in_features=384, out_features=384, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.3692307770252228) (convFFN): ConvFFN( (linear1): Conv2d(384, 1536, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(1536, 384, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(1536, 1536, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=1536) ) (norm2): LayerNorm2d((384,), eps=1e-05, elementwise_affine=True) ) ) (downsample): Sequential( (0): Identity() (1): Conv2d(384, 768, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1)) (2): Identity() (3): LayerNorm2d((768,), eps=1e-05, elementwise_affine=True) ) ) (3): Sequential( (blocks): Sequential( (0): VSSBlock( (norm): LayerNorm2d((768,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((768,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=768, out_features=96, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=96, out_features=768, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(768, 768, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=768) (in_proj): Linear2d(in_features=768, out_features=768, bias=False) (act): SiLU() (conv2d): Conv2d(768, 768, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=768) (out_act): Identity() (out_proj): Linear2d(in_features=768, out_features=768, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.38461539149284363) (convFFN): ConvFFN( (linear1): Conv2d(768, 3072, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(3072, 768, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(3072, 3072, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=3072) ) (norm2): LayerNorm2d((768,), eps=1e-05, elementwise_affine=True) ) (1): VSSBlock( (norm): LayerNorm2d((768,), eps=1e-05, elementwise_affine=True) (op): SS2D( (out_norm): LayerNorm2d((768,), eps=1e-05, elementwise_affine=True) (se): SEModule( (avg_pool): AdaptiveAvgPool2d(output_size=1) (fc): Sequential( (0): Linear(in_features=768, out_features=96, bias=False) (1): ReLU(inplace=True) (2): Linear(in_features=96, out_features=768, bias=False) (3): Sigmoid() ) ) (conv2d_b1): Conv2d(768, 768, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), groups=768) (in_proj): Linear2d(in_features=768, out_features=768, bias=False) (act): SiLU() (conv2d): Conv2d(768, 768, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=768) (out_act): Identity() (out_proj): Linear2d(in_features=768, out_features=768, bias=False) (dropout): Identity() ) (drop_path): timm.DropPath(0.4000000059604645) (convFFN): ConvFFN( (linear1): Conv2d(768, 3072, kernel_size=(1, 1), stride=(1, 1)) (drop1): Dropout(p=0.0, inplace=True) (act): GELU(approximate='none') (linear2): Conv2d(3072, 768, kernel_size=(1, 1), stride=(1, 1)) (drop2): Dropout(p=0.0, inplace=True) (dwc): Conv2d(3072, 3072, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=3072) ) (norm2): LayerNorm2d((768,), eps=1e-05, elementwise_affine=True) ) ) (downsample): Identity() ) ) (classifier): Sequential( (norm): LayerNorm2d((768,), eps=1e-05, elementwise_affine=True) (permute): Identity() (avgpool): AdaptiveAvgPool2d(output_size=1) (flatten): Flatten(start_dim=1, end_dim=-1) (head): Linear(in_features=768, out_features=1000, bias=True) ) ) [2024-08-19 23:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 137): INFO number of params: 50216296 [2024-08-19 23:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 139): INFO number of GFLOPs: 8.771503104 [2024-08-19 23:27:42 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-19 23:27:42 msvmambav3_small_224] (optimizer.py 27): INFO No weight decay list: ['patch_embed.0.bias', 'patch_embed.2.weight', 'patch_embed.2.bias', 'patch_embed.5.bias', 'patch_embed.7.weight', 'patch_embed.7.bias', 'layers.0.blocks.0.norm.weight', 'layers.0.blocks.0.norm.bias', 'layers.0.blocks.0.op.Ds', 'layers.0.blocks.0.op.out_norm.weight', 'layers.0.blocks.0.op.out_norm.bias', 'layers.0.blocks.0.op.conv2d_b1.bias', 'layers.0.blocks.0.op.conv2d.bias', 'layers.0.blocks.0.convFFN.linear1.bias', 'layers.0.blocks.0.convFFN.linear2.bias', 'layers.0.blocks.0.convFFN.dwc.bias', 'layers.0.blocks.0.norm2.weight', 'layers.0.blocks.0.norm2.bias', 'layers.0.blocks.1.norm.weight', 'layers.0.blocks.1.norm.bias', 'layers.0.blocks.1.op.Ds', 'layers.0.blocks.1.op.out_norm.weight', 'layers.0.blocks.1.op.out_norm.bias', 'layers.0.blocks.1.op.conv2d_b1.bias', 'layers.0.blocks.1.op.conv2d.bias', 'layers.0.blocks.1.convFFN.linear1.bias', 'layers.0.blocks.1.convFFN.linear2.bias', 'layers.0.blocks.1.convFFN.dwc.bias', 'layers.0.blocks.1.norm2.weight', 'layers.0.blocks.1.norm2.bias', 'layers.0.downsample.1.bias', 'layers.0.downsample.3.weight', 'layers.0.downsample.3.bias', 'layers.1.blocks.0.norm.weight', 'layers.1.blocks.0.norm.bias', 'layers.1.blocks.0.op.Ds', 'layers.1.blocks.0.op.out_norm.weight', 'layers.1.blocks.0.op.out_norm.bias', 'layers.1.blocks.0.op.conv2d_b1.bias', 'layers.1.blocks.0.op.conv2d.bias', 'layers.1.blocks.0.convFFN.linear1.bias', 'layers.1.blocks.0.convFFN.linear2.bias', 'layers.1.blocks.0.convFFN.dwc.bias', 'layers.1.blocks.0.norm2.weight', 'layers.1.blocks.0.norm2.bias', 'layers.1.blocks.1.norm.weight', 'layers.1.blocks.1.norm.bias', 'layers.1.blocks.1.op.Ds', 'layers.1.blocks.1.op.out_norm.weight', 'layers.1.blocks.1.op.out_norm.bias', 'layers.1.blocks.1.op.conv2d_b1.bias', 'layers.1.blocks.1.op.conv2d.bias', 'layers.1.blocks.1.convFFN.linear1.bias', 'layers.1.blocks.1.convFFN.linear2.bias', 'layers.1.blocks.1.convFFN.dwc.bias', 'layers.1.blocks.1.norm2.weight', 'layers.1.blocks.1.norm2.bias', 'layers.1.blocks.2.norm.weight', 'layers.1.blocks.2.norm.bias', 'layers.1.blocks.2.op.Ds', 'layers.1.blocks.2.op.out_norm.weight', 'layers.1.blocks.2.op.out_norm.bias', 'layers.1.blocks.2.op.conv2d_b1.bias', 'layers.1.blocks.2.op.conv2d.bias', 'layers.1.blocks.2.convFFN.linear1.bias', 'layers.1.blocks.2.convFFN.linear2.bias', 'layers.1.blocks.2.convFFN.dwc.bias', 'layers.1.blocks.2.norm2.weight', 'layers.1.blocks.2.norm2.bias', 'layers.1.downsample.1.bias', 'layers.1.downsample.3.weight', 'layers.1.downsample.3.bias', 'layers.2.blocks.0.norm.weight', 'layers.2.blocks.0.norm.bias', 'layers.2.blocks.0.op.Ds', 'layers.2.blocks.0.op.out_norm.weight', 'layers.2.blocks.0.op.out_norm.bias', 'layers.2.blocks.0.op.conv2d_b1.bias', 'layers.2.blocks.0.op.conv2d.bias', 'layers.2.blocks.0.convFFN.linear1.bias', 'layers.2.blocks.0.convFFN.linear2.bias', 'layers.2.blocks.0.convFFN.dwc.bias', 'layers.2.blocks.0.norm2.weight', 'layers.2.blocks.0.norm2.bias', 'layers.2.blocks.1.norm.weight', 'layers.2.blocks.1.norm.bias', 'layers.2.blocks.1.op.Ds', 'layers.2.blocks.1.op.out_norm.weight', 'layers.2.blocks.1.op.out_norm.bias', 'layers.2.blocks.1.op.conv2d_b1.bias', 'layers.2.blocks.1.op.conv2d.bias', 'layers.2.blocks.1.convFFN.linear1.bias', 'layers.2.blocks.1.convFFN.linear2.bias', 'layers.2.blocks.1.convFFN.dwc.bias', 'layers.2.blocks.1.norm2.weight', 'layers.2.blocks.1.norm2.bias', 'layers.2.blocks.2.norm.weight', 'layers.2.blocks.2.norm.bias', 'layers.2.blocks.2.op.Ds', 'layers.2.blocks.2.op.out_norm.weight', 'layers.2.blocks.2.op.out_norm.bias', 'layers.2.blocks.2.op.conv2d_b1.bias', 'layers.2.blocks.2.op.conv2d.bias', 'layers.2.blocks.2.convFFN.linear1.bias', 'layers.2.blocks.2.convFFN.linear2.bias', 'layers.2.blocks.2.convFFN.dwc.bias', 'layers.2.blocks.2.norm2.weight', 'layers.2.blocks.2.norm2.bias', 'layers.2.blocks.3.norm.weight', 'layers.2.blocks.3.norm.bias', 'layers.2.blocks.3.op.Ds', 'layers.2.blocks.3.op.out_norm.weight', 'layers.2.blocks.3.op.out_norm.bias', 'layers.2.blocks.3.op.conv2d_b1.bias', 'layers.2.blocks.3.op.conv2d.bias', 'layers.2.blocks.3.convFFN.linear1.bias', 'layers.2.blocks.3.convFFN.linear2.bias', 'layers.2.blocks.3.convFFN.dwc.bias', 'layers.2.blocks.3.norm2.weight', 'layers.2.blocks.3.norm2.bias', 'layers.2.blocks.4.norm.weight', 'layers.2.blocks.4.norm.bias', 'layers.2.blocks.4.op.Ds', 'layers.2.blocks.4.op.out_norm.weight', 'layers.2.blocks.4.op.out_norm.bias', 'layers.2.blocks.4.op.conv2d_b1.bias', 'layers.2.blocks.4.op.conv2d.bias', 'layers.2.blocks.4.convFFN.linear1.bias', 'layers.2.blocks.4.convFFN.linear2.bias', 'layers.2.blocks.4.convFFN.dwc.bias', 'layers.2.blocks.4.norm2.weight', 'layers.2.blocks.4.norm2.bias', 'layers.2.blocks.5.norm.weight', 'layers.2.blocks.5.norm.bias', 'layers.2.blocks.5.op.Ds', 'layers.2.blocks.5.op.out_norm.weight', 'layers.2.blocks.5.op.out_norm.bias', 'layers.2.blocks.5.op.conv2d_b1.bias', 'layers.2.blocks.5.op.conv2d.bias', 'layers.2.blocks.5.convFFN.linear1.bias', 'layers.2.blocks.5.convFFN.linear2.bias', 'layers.2.blocks.5.convFFN.dwc.bias', 'layers.2.blocks.5.norm2.weight', 'layers.2.blocks.5.norm2.bias', 'layers.2.blocks.6.norm.weight', 'layers.2.blocks.6.norm.bias', 'layers.2.blocks.6.op.Ds', 'layers.2.blocks.6.op.out_norm.weight', 'layers.2.blocks.6.op.out_norm.bias', 'layers.2.blocks.6.op.conv2d_b1.bias', 'layers.2.blocks.6.op.conv2d.bias', 'layers.2.blocks.6.convFFN.linear1.bias', 'layers.2.blocks.6.convFFN.linear2.bias', 'layers.2.blocks.6.convFFN.dwc.bias', 'layers.2.blocks.6.norm2.weight', 'layers.2.blocks.6.norm2.bias', 'layers.2.blocks.7.norm.weight', 'layers.2.blocks.7.norm.bias', 'layers.2.blocks.7.op.Ds', 'layers.2.blocks.7.op.out_norm.weight', 'layers.2.blocks.7.op.out_norm.bias', 'layers.2.blocks.7.op.conv2d_b1.bias', 'layers.2.blocks.7.op.conv2d.bias', 'layers.2.blocks.7.convFFN.linear1.bias', 'layers.2.blocks.7.convFFN.linear2.bias', 'layers.2.blocks.7.convFFN.dwc.bias', 'layers.2.blocks.7.norm2.weight', 'layers.2.blocks.7.norm2.bias', 'layers.2.blocks.8.norm.weight', 'layers.2.blocks.8.norm.bias', 'layers.2.blocks.8.op.Ds', 'layers.2.blocks.8.op.out_norm.weight', 'layers.2.blocks.8.op.out_norm.bias', 'layers.2.blocks.8.op.conv2d_b1.bias', 'layers.2.blocks.8.op.conv2d.bias', 'layers.2.blocks.8.convFFN.linear1.bias', 'layers.2.blocks.8.convFFN.linear2.bias', 'layers.2.blocks.8.convFFN.dwc.bias', 'layers.2.blocks.8.norm2.weight', 'layers.2.blocks.8.norm2.bias', 'layers.2.blocks.9.norm.weight', 'layers.2.blocks.9.norm.bias', 'layers.2.blocks.9.op.Ds', 'layers.2.blocks.9.op.out_norm.weight', 'layers.2.blocks.9.op.out_norm.bias', 'layers.2.blocks.9.op.conv2d_b1.bias', 'layers.2.blocks.9.op.conv2d.bias', 'layers.2.blocks.9.convFFN.linear1.bias', 'layers.2.blocks.9.convFFN.linear2.bias', 'layers.2.blocks.9.convFFN.dwc.bias', 'layers.2.blocks.9.norm2.weight', 'layers.2.blocks.9.norm2.bias', 'layers.2.blocks.10.norm.weight', 'layers.2.blocks.10.norm.bias', 'layers.2.blocks.10.op.Ds', 'layers.2.blocks.10.op.out_norm.weight', 'layers.2.blocks.10.op.out_norm.bias', 'layers.2.blocks.10.op.conv2d_b1.bias', 'layers.2.blocks.10.op.conv2d.bias', 'layers.2.blocks.10.convFFN.linear1.bias', 'layers.2.blocks.10.convFFN.linear2.bias', 'layers.2.blocks.10.convFFN.dwc.bias', 'layers.2.blocks.10.norm2.weight', 'layers.2.blocks.10.norm2.bias', 'layers.2.blocks.11.norm.weight', 'layers.2.blocks.11.norm.bias', 'layers.2.blocks.11.op.Ds', 'layers.2.blocks.11.op.out_norm.weight', 'layers.2.blocks.11.op.out_norm.bias', 'layers.2.blocks.11.op.conv2d_b1.bias', 'layers.2.blocks.11.op.conv2d.bias', 'layers.2.blocks.11.convFFN.linear1.bias', 'layers.2.blocks.11.convFFN.linear2.bias', 'layers.2.blocks.11.convFFN.dwc.bias', 'layers.2.blocks.11.norm2.weight', 'layers.2.blocks.11.norm2.bias', 'layers.2.blocks.12.norm.weight', 'layers.2.blocks.12.norm.bias', 'layers.2.blocks.12.op.Ds', 'layers.2.blocks.12.op.out_norm.weight', 'layers.2.blocks.12.op.out_norm.bias', 'layers.2.blocks.12.op.conv2d_b1.bias', 'layers.2.blocks.12.op.conv2d.bias', 'layers.2.blocks.12.convFFN.linear1.bias', 'layers.2.blocks.12.convFFN.linear2.bias', 'layers.2.blocks.12.convFFN.dwc.bias', 'layers.2.blocks.12.norm2.weight', 'layers.2.blocks.12.norm2.bias', 'layers.2.blocks.13.norm.weight', 'layers.2.blocks.13.norm.bias', 'layers.2.blocks.13.op.Ds', 'layers.2.blocks.13.op.out_norm.weight', 'layers.2.blocks.13.op.out_norm.bias', 'layers.2.blocks.13.op.conv2d_b1.bias', 'layers.2.blocks.13.op.conv2d.bias', 'layers.2.blocks.13.convFFN.linear1.bias', 'layers.2.blocks.13.convFFN.linear2.bias', 'layers.2.blocks.13.convFFN.dwc.bias', 'layers.2.blocks.13.norm2.weight', 'layers.2.blocks.13.norm2.bias', 'layers.2.blocks.14.norm.weight', 'layers.2.blocks.14.norm.bias', 'layers.2.blocks.14.op.Ds', 'layers.2.blocks.14.op.out_norm.weight', 'layers.2.blocks.14.op.out_norm.bias', 'layers.2.blocks.14.op.conv2d_b1.bias', 'layers.2.blocks.14.op.conv2d.bias', 'layers.2.blocks.14.convFFN.linear1.bias', 'layers.2.blocks.14.convFFN.linear2.bias', 'layers.2.blocks.14.convFFN.dwc.bias', 'layers.2.blocks.14.norm2.weight', 'layers.2.blocks.14.norm2.bias', 'layers.2.blocks.15.norm.weight', 'layers.2.blocks.15.norm.bias', 'layers.2.blocks.15.op.Ds', 'layers.2.blocks.15.op.out_norm.weight', 'layers.2.blocks.15.op.out_norm.bias', 'layers.2.blocks.15.op.conv2d_b1.bias', 'layers.2.blocks.15.op.conv2d.bias', 'layers.2.blocks.15.convFFN.linear1.bias', 'layers.2.blocks.15.convFFN.linear2.bias', 'layers.2.blocks.15.convFFN.dwc.bias', 'layers.2.blocks.15.norm2.weight', 'layers.2.blocks.15.norm2.bias', 'layers.2.blocks.16.norm.weight', 'layers.2.blocks.16.norm.bias', 'layers.2.blocks.16.op.Ds', 'layers.2.blocks.16.op.out_norm.weight', 'layers.2.blocks.16.op.out_norm.bias', 'layers.2.blocks.16.op.conv2d_b1.bias', 'layers.2.blocks.16.op.conv2d.bias', 'layers.2.blocks.16.convFFN.linear1.bias', 'layers.2.blocks.16.convFFN.linear2.bias', 'layers.2.blocks.16.convFFN.dwc.bias', 'layers.2.blocks.16.norm2.weight', 'layers.2.blocks.16.norm2.bias', 'layers.2.blocks.17.norm.weight', 'layers.2.blocks.17.norm.bias', 'layers.2.blocks.17.op.Ds', 'layers.2.blocks.17.op.out_norm.weight', 'layers.2.blocks.17.op.out_norm.bias', 'layers.2.blocks.17.op.conv2d_b1.bias', 'layers.2.blocks.17.op.conv2d.bias', 'layers.2.blocks.17.convFFN.linear1.bias', 'layers.2.blocks.17.convFFN.linear2.bias', 'layers.2.blocks.17.convFFN.dwc.bias', 'layers.2.blocks.17.norm2.weight', 'layers.2.blocks.17.norm2.bias', 'layers.2.blocks.18.norm.weight', 'layers.2.blocks.18.norm.bias', 'layers.2.blocks.18.op.Ds', 'layers.2.blocks.18.op.out_norm.weight', 'layers.2.blocks.18.op.out_norm.bias', 'layers.2.blocks.18.op.conv2d_b1.bias', 'layers.2.blocks.18.op.conv2d.bias', 'layers.2.blocks.18.convFFN.linear1.bias', 'layers.2.blocks.18.convFFN.linear2.bias', 'layers.2.blocks.18.convFFN.dwc.bias', 'layers.2.blocks.18.norm2.weight', 'layers.2.blocks.18.norm2.bias', 'layers.2.blocks.19.norm.weight', 'layers.2.blocks.19.norm.bias', 'layers.2.blocks.19.op.Ds', 'layers.2.blocks.19.op.out_norm.weight', 'layers.2.blocks.19.op.out_norm.bias', 'layers.2.blocks.19.op.conv2d_b1.bias', 'layers.2.blocks.19.op.conv2d.bias', 'layers.2.blocks.19.convFFN.linear1.bias', 'layers.2.blocks.19.convFFN.linear2.bias', 'layers.2.blocks.19.convFFN.dwc.bias', 'layers.2.blocks.19.norm2.weight', 'layers.2.blocks.19.norm2.bias', 'layers.2.downsample.1.bias', 'layers.2.downsample.3.weight', 'layers.2.downsample.3.bias', 'layers.3.blocks.0.norm.weight', 'layers.3.blocks.0.norm.bias', 'layers.3.blocks.0.op.Ds', 'layers.3.blocks.0.op.out_norm.weight', 'layers.3.blocks.0.op.out_norm.bias', 'layers.3.blocks.0.op.conv2d_b1.bias', 'layers.3.blocks.0.op.conv2d.bias', 'layers.3.blocks.0.convFFN.linear1.bias', 'layers.3.blocks.0.convFFN.linear2.bias', 'layers.3.blocks.0.convFFN.dwc.bias', 'layers.3.blocks.0.norm2.weight', 'layers.3.blocks.0.norm2.bias', 'layers.3.blocks.1.norm.weight', 'layers.3.blocks.1.norm.bias', 'layers.3.blocks.1.op.Ds', 'layers.3.blocks.1.op.out_norm.weight', 'layers.3.blocks.1.op.out_norm.bias', 'layers.3.blocks.1.op.conv2d_b1.bias', 'layers.3.blocks.1.op.conv2d.bias', 'layers.3.blocks.1.convFFN.linear1.bias', 'layers.3.blocks.1.convFFN.linear2.bias', 'layers.3.blocks.1.convFFN.dwc.bias', 'layers.3.blocks.1.norm2.weight', 'layers.3.blocks.1.norm2.bias', 'classifier.norm.weight', 'classifier.norm.bias', 'classifier.head.bias'] [2024-08-19 23:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 196): INFO no checkpoint found in ./exclude/output_msv2/msvmambav3_small_224/20240819232309, ignoring auto resume [2024-08-19 23:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-19 23:28:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][0/625] eta 2:31:26 lr 0.000002 wd 0.0500 time 14.5390 (14.5390) data time 0.6776 (0.6776) model time 0.0000 (0.0000) loss 7.0483 (7.0483) grad_norm 1.5631 (1.5631) loss_scale 65536.0000 (65536.0000) mem 23532MB [2024-08-19 23:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][10/625] eta 0:17:20 lr 0.000004 wd 0.0500 time 0.3774 (1.6922) data time 0.0011 (0.0627) model time 0.0000 (0.0000) loss 6.9365 (7.0007) grad_norm 1.4538 (1.5454) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][20/625] eta 0:10:43 lr 0.000005 wd 0.0500 time 0.3708 (1.0638) data time 0.0011 (0.0334) model time 0.0000 (0.0000) loss 6.9631 (6.9896) grad_norm 1.4502 (1.5019) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][30/625] eta 0:08:19 lr 0.000007 wd 0.0500 time 0.3694 (0.8401) data time 0.0009 (0.0230) model time 0.0000 (0.0000) loss 6.9970 (6.9789) grad_norm 1.4151 (1.4811) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][40/625] eta 0:07:04 lr 0.000008 wd 0.0500 time 0.3741 (0.7261) data time 0.0011 (0.0177) model time 0.0000 (0.0000) loss 6.9846 (6.9729) grad_norm 1.3085 (1.4563) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][50/625] eta 0:06:17 lr 0.000010 wd 0.0500 time 0.3756 (0.6573) data time 0.0008 (0.0144) model time 0.0000 (0.0000) loss 6.9466 (6.9709) grad_norm 1.3636 (1.4376) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][60/625] eta 0:05:45 lr 0.000012 wd 0.0500 time 0.3792 (0.6106) data time 0.0011 (0.0123) model time 0.3781 (0.3714) loss 6.9283 (6.9664) grad_norm 1.3453 (1.4224) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][70/625] eta 0:05:20 lr 0.000013 wd 0.0500 time 0.3661 (0.5769) data time 0.0011 (0.0107) model time 0.3651 (0.3704) loss 6.9193 (6.9619) grad_norm 1.3501 (1.4081) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][80/625] eta 0:05:00 lr 0.000015 wd 0.0500 time 0.3669 (0.5515) data time 0.0011 (0.0096) model time 0.3658 (0.3702) loss 6.8947 (6.9584) grad_norm 1.3301 (1.3938) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][90/625] eta 0:04:44 lr 0.000016 wd 0.0500 time 0.3755 (0.5319) data time 0.0008 (0.0086) model time 0.3746 (0.3706) loss 6.9168 (6.9555) grad_norm 1.2958 (1.3814) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][100/625] eta 0:04:30 lr 0.000018 wd 0.0500 time 0.3696 (0.5160) data time 0.0008 (0.0079) model time 0.3688 (0.3706) loss 6.8699 (6.9525) grad_norm 1.1613 (1.3651) loss_scale 65536.0000 (65536.0000) mem 11262MB [2024-08-19 23:28:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][110/625] eta 0:04:20 lr 0.000020 wd 0.0500 time 0.3695 (0.5052) data time 0.0010 (0.0073) model time 0.3685 (0.3747) loss 6.9335 (6.9504) grad_norm 1.2253 (1.3528) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][120/625] eta 0:04:10 lr 0.000021 wd 0.0500 time 0.3788 (0.4963) data time 0.0008 (0.0068) model time 0.3780 (0.3778) loss 6.9508 (6.9479) grad_norm 1.1947 (1.3420) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:28:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][130/625] eta 0:04:00 lr 0.000023 wd 0.0500 time 0.3736 (0.4868) data time 0.0011 (0.0063) model time 0.3725 (0.3769) loss 6.9117 (6.9450) grad_norm 1.2536 (1.3322) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:28:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][140/625] eta 0:03:52 lr 0.000024 wd 0.0500 time 0.3678 (0.4786) data time 0.0009 (0.0060) model time 0.3669 (0.3762) loss 6.9127 (6.9408) grad_norm 1.0519 (1.3163) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][150/625] eta 0:03:44 lr 0.000026 wd 0.0500 time 0.3719 (0.4717) data time 0.0011 (0.0057) model time 0.3708 (0.3759) loss 6.9227 (6.9382) grad_norm 1.2743 (1.3048) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][160/625] eta 0:03:36 lr 0.000028 wd 0.0500 time 0.3635 (0.4656) data time 0.0013 (0.0054) model time 0.3623 (0.3755) loss 6.9244 (6.9360) grad_norm 1.1652 (1.2928) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][170/625] eta 0:03:29 lr 0.000029 wd 0.0500 time 0.3708 (0.4600) data time 0.0011 (0.0051) model time 0.3696 (0.3749) loss 6.9014 (6.9335) grad_norm 1.1992 (1.2843) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][180/625] eta 0:03:22 lr 0.000031 wd 0.0500 time 0.3745 (0.4551) data time 0.0011 (0.0049) model time 0.3735 (0.3745) loss 6.9101 (6.9303) grad_norm 1.0197 (1.2776) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][190/625] eta 0:03:16 lr 0.000032 wd 0.0500 time 0.3690 (0.4507) data time 0.0014 (0.0047) model time 0.3676 (0.3743) loss 6.8691 (6.9269) grad_norm 1.5729 (1.2723) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][200/625] eta 0:03:09 lr 0.000034 wd 0.0500 time 0.3752 (0.4468) data time 0.0011 (0.0045) model time 0.3741 (0.3741) loss 6.8381 (6.9236) grad_norm 1.1248 (1.2672) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][210/625] eta 0:03:03 lr 0.000036 wd 0.0500 time 0.3716 (0.4433) data time 0.0011 (0.0044) model time 0.3706 (0.3739) loss 6.8595 (6.9204) grad_norm 1.1828 (1.2685) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][220/625] eta 0:02:58 lr 0.000037 wd 0.0500 time 0.3707 (0.4402) data time 0.0008 (0.0042) model time 0.3699 (0.3739) loss 6.9160 (6.9168) grad_norm 1.1913 (1.2674) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][230/625] eta 0:02:52 lr 0.000039 wd 0.0500 time 0.3698 (0.4374) data time 0.0008 (0.0041) model time 0.3690 (0.3739) loss 6.8458 (6.9139) grad_norm 1.4424 (1.2657) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][240/625] eta 0:02:47 lr 0.000040 wd 0.0500 time 0.3824 (0.4362) data time 0.0008 (0.0040) model time 0.3816 (0.3757) loss 6.8751 (6.9106) grad_norm 1.8221 (1.2699) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][250/625] eta 0:02:42 lr 0.000042 wd 0.0500 time 0.3695 (0.4337) data time 0.0008 (0.0039) model time 0.3687 (0.3756) loss 6.7856 (6.9067) grad_norm 1.6202 (1.2864) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][260/625] eta 0:02:37 lr 0.000044 wd 0.0500 time 0.3634 (0.4314) data time 0.0008 (0.0037) model time 0.3626 (0.3754) loss 6.8052 (6.9039) grad_norm 3.2011 (1.3118) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][270/625] eta 0:02:32 lr 0.000045 wd 0.0500 time 0.3720 (0.4294) data time 0.0008 (0.0037) model time 0.3712 (0.3753) loss 6.8467 (6.8999) grad_norm 1.6778 (1.3331) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][280/625] eta 0:02:27 lr 0.000047 wd 0.0500 time 0.3706 (0.4274) data time 0.0011 (0.0036) model time 0.3695 (0.3751) loss 6.8284 (6.8972) grad_norm 1.2929 (1.3431) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][290/625] eta 0:02:22 lr 0.000048 wd 0.0500 time 0.3751 (0.4262) data time 0.0008 (0.0036) model time 0.3743 (0.3758) loss 6.6947 (6.8931) grad_norm 1.1425 (1.3604) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][300/625] eta 0:02:17 lr 0.000050 wd 0.0500 time 0.3664 (0.4245) data time 0.0010 (0.0035) model time 0.3654 (0.3758) loss 6.8514 (6.8907) grad_norm 4.3563 (1.4014) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][310/625] eta 0:02:13 lr 0.000052 wd 0.0500 time 0.3826 (0.4229) data time 0.0011 (0.0034) model time 0.3815 (0.3757) loss 6.8637 (6.8878) grad_norm 2.3087 (1.4169) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][320/625] eta 0:02:08 lr 0.000053 wd 0.0500 time 0.3709 (0.4214) data time 0.0008 (0.0033) model time 0.3701 (0.3756) loss 6.7805 (6.8858) grad_norm 2.2689 (1.4305) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][330/625] eta 0:02:03 lr 0.000055 wd 0.0500 time 0.3693 (0.4200) data time 0.0009 (0.0033) model time 0.3685 (0.3755) loss 6.6777 (6.8819) grad_norm 2.9009 (1.4531) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][340/625] eta 0:01:59 lr 0.000056 wd 0.0500 time 0.3618 (0.4186) data time 0.0011 (0.0032) model time 0.3607 (0.3754) loss 6.7887 (6.8787) grad_norm 2.3428 (1.4802) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][350/625] eta 0:01:54 lr 0.000058 wd 0.0500 time 0.3722 (0.4174) data time 0.0009 (0.0032) model time 0.3713 (0.3753) loss 6.7668 (6.8763) grad_norm 2.3645 (1.4947) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][360/625] eta 0:01:50 lr 0.000060 wd 0.0500 time 0.3762 (0.4161) data time 0.0008 (0.0031) model time 0.3754 (0.3752) loss 6.6622 (6.8735) grad_norm 2.1553 (1.5339) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][370/625] eta 0:01:45 lr 0.000061 wd 0.0500 time 0.3761 (0.4150) data time 0.0011 (0.0030) model time 0.3749 (0.3751) loss 6.7914 (6.8709) grad_norm 3.5931 (1.5742) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][380/625] eta 0:01:41 lr 0.000063 wd 0.0500 time 0.3804 (0.4139) data time 0.0008 (0.0030) model time 0.3796 (0.3751) loss 6.6717 (6.8681) grad_norm 3.1436 (1.5874) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][390/625] eta 0:01:37 lr 0.000064 wd 0.0500 time 0.3690 (0.4129) data time 0.0009 (0.0029) model time 0.3682 (0.3751) loss 6.8535 (6.8652) grad_norm 2.0034 (1.6167) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][400/625] eta 0:01:32 lr 0.000066 wd 0.0500 time 0.3679 (0.4119) data time 0.0010 (0.0029) model time 0.3669 (0.3749) loss 6.7756 (6.8632) grad_norm 1.9099 (1.6333) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][410/625] eta 0:01:28 lr 0.000068 wd 0.0500 time 0.3692 (0.4114) data time 0.0012 (0.0029) model time 0.3680 (0.3753) loss 6.8133 (6.8611) grad_norm 2.2160 (1.6588) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][420/625] eta 0:01:24 lr 0.000069 wd 0.0500 time 0.3658 (0.4105) data time 0.0009 (0.0028) model time 0.3649 (0.3752) loss 6.8061 (6.8584) grad_norm 7.2491 (1.7048) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][430/625] eta 0:01:19 lr 0.000071 wd 0.0500 time 0.3573 (0.4096) data time 0.0009 (0.0028) model time 0.3564 (0.3751) loss 6.7881 (6.8564) grad_norm 4.1554 (1.7634) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][440/625] eta 0:01:15 lr 0.000072 wd 0.0500 time 0.3641 (0.4088) data time 0.0008 (0.0027) model time 0.3633 (0.3750) loss 6.7487 (6.8543) grad_norm 2.8934 (1.7922) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][450/625] eta 0:01:11 lr 0.000074 wd 0.0500 time 0.3683 (0.4084) data time 0.0009 (0.0027) model time 0.3674 (0.3755) loss 6.5807 (6.8515) grad_norm 2.6170 (1.8356) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][460/625] eta 0:01:07 lr 0.000076 wd 0.0500 time 0.3736 (0.4077) data time 0.0010 (0.0027) model time 0.3725 (0.3754) loss 6.7353 (6.8487) grad_norm 2.2476 (1.8827) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][470/625] eta 0:01:03 lr 0.000077 wd 0.0500 time 0.3744 (0.4069) data time 0.0011 (0.0026) model time 0.3733 (0.3753) loss 6.6978 (6.8454) grad_norm 1.6242 (1.9072) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][480/625] eta 0:00:58 lr 0.000079 wd 0.0500 time 0.3675 (0.4062) data time 0.0008 (0.0026) model time 0.3667 (0.3752) loss 6.6933 (6.8423) grad_norm 4.1583 (1.9214) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][490/625] eta 0:00:54 lr 0.000080 wd 0.0500 time 0.3755 (0.4055) data time 0.0011 (0.0026) model time 0.3744 (0.3751) loss 6.7943 (6.8408) grad_norm 2.6763 (1.9605) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][500/625] eta 0:00:50 lr 0.000082 wd 0.0500 time 0.3739 (0.4049) data time 0.0010 (0.0025) model time 0.3728 (0.3751) loss 6.7945 (6.8380) grad_norm 1.5982 (1.9856) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][510/625] eta 0:00:46 lr 0.000084 wd 0.0500 time 0.3746 (0.4043) data time 0.0009 (0.0025) model time 0.3737 (0.3750) loss 6.8870 (6.8365) grad_norm 5.2893 (2.0172) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][520/625] eta 0:00:42 lr 0.000085 wd 0.0500 time 0.3874 (0.4038) data time 0.0009 (0.0025) model time 0.3865 (0.3750) loss 6.6604 (6.8343) grad_norm 1.5785 (2.0399) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][530/625] eta 0:00:38 lr 0.000087 wd 0.0500 time 0.3729 (0.4032) data time 0.0010 (0.0025) model time 0.3719 (0.3750) loss 6.7744 (6.8314) grad_norm 3.8657 (2.0553) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][540/625] eta 0:00:34 lr 0.000088 wd 0.0500 time 0.3770 (0.4026) data time 0.0011 (0.0024) model time 0.3758 (0.3749) loss 6.8198 (6.8291) grad_norm 3.1904 (2.0922) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][550/625] eta 0:00:30 lr 0.000090 wd 0.0500 time 0.3694 (0.4021) data time 0.0008 (0.0024) model time 0.3685 (0.3748) loss 6.7960 (6.8260) grad_norm 2.6608 (2.1088) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][560/625] eta 0:00:26 lr 0.000092 wd 0.0500 time 0.3717 (0.4015) data time 0.0011 (0.0024) model time 0.3706 (0.3748) loss 6.6339 (6.8239) grad_norm 3.6677 (2.1351) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][570/625] eta 0:00:22 lr 0.000093 wd 0.0500 time 0.3698 (0.4010) data time 0.0011 (0.0024) model time 0.3687 (0.3747) loss 6.7878 (6.8221) grad_norm 4.0166 (2.1637) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][580/625] eta 0:00:18 lr 0.000095 wd 0.0500 time 0.3680 (0.4005) data time 0.0008 (0.0024) model time 0.3671 (0.3746) loss 6.8223 (6.8199) grad_norm 4.6989 (2.1834) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][590/625] eta 0:00:14 lr 0.000096 wd 0.0500 time 0.3789 (0.4001) data time 0.0010 (0.0023) model time 0.3779 (0.3746) loss 6.7352 (6.8177) grad_norm 2.5722 (2.1995) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][600/625] eta 0:00:09 lr 0.000098 wd 0.0500 time 0.3769 (0.3997) data time 0.0012 (0.0023) model time 0.3758 (0.3746) loss 6.6425 (6.8151) grad_norm 3.6196 (2.2202) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][610/625] eta 0:00:05 lr 0.000100 wd 0.0500 time 0.3545 (0.3992) data time 0.0008 (0.0023) model time 0.3537 (0.3745) loss 6.5688 (6.8123) grad_norm 3.2405 (2.2399) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [0/300][620/625] eta 0:00:01 lr 0.000101 wd 0.0500 time 0.3709 (0.3989) data time 0.0006 (0.0023) model time 0.3704 (0.3746) loss 6.7832 (6.8104) grad_norm 4.2453 (2.2575) loss_scale 65536.0000 (65536.0000) mem 11218MB [2024-08-19 23:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 0 training takes 0:04:09 [2024-08-19 23:31:58 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-19 23:31:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-19 23:32:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 6.0742 (6.0742) Acc@1 0.586 (0.586) Acc@5 7.324 (7.324) Mem 11218MB [2024-08-19 23:32:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 6.4258 (6.2734) Acc@1 0.195 (1.301) Acc@5 1.367 (5.362) Mem 11218MB [2024-08-19 23:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 6.3047 (6.2757) Acc@1 1.465 (1.579) Acc@5 5.176 (5.662) Mem 11218MB [2024-08-19 23:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 2.033 Acc@5 6.946 [2024-08-19 23:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 2.0% [2024-08-19 23:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 2.03% [2024-08-19 23:32:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-19 23:32:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-19 23:32:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 6.9883 (6.9883) Acc@1 0.000 (0.000) Acc@5 0.391 (0.391) Mem 11218MB [2024-08-19 23:32:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.140) Loss 6.9766 (6.9854) Acc@1 0.293 (0.107) Acc@5 0.879 (0.533) Mem 11218MB [2024-08-19 23:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 6.9805 (6.9870) Acc@1 0.049 (0.098) Acc@5 0.342 (0.507) Mem 11218MB [2024-08-19 23:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.092 Acc@5 0.484 [2024-08-19 23:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.1% [2024-08-19 23:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.09% [2024-08-19 23:32:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-19 23:32:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-19 23:32:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][0/625] eta 0:08:34 lr 0.000102 wd 0.0500 time 0.8239 (0.8239) data time 0.4252 (0.4252) model time 0.0000 (0.0000) loss 6.5345 (6.5345) grad_norm 5.1918 (5.1918) loss_scale 65536.0000 (65536.0000) mem 11223MB [2024-08-19 23:32:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][10/625] eta 0:04:29 lr 0.000103 wd 0.0500 time 0.3806 (0.4375) data time 0.0008 (0.0397) model time 0.0000 (0.0000) loss 6.5938 (6.6492) grad_norm 3.1869 (3.3756) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][20/625] eta 0:04:06 lr 0.000105 wd 0.0500 time 0.3693 (0.4074) data time 0.0011 (0.0213) model time 0.0000 (0.0000) loss 6.8553 (6.6399) grad_norm 9.0497 (3.8952) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][30/625] eta 0:03:57 lr 0.000107 wd 0.0500 time 0.3725 (0.3990) data time 0.0013 (0.0148) model time 0.0000 (0.0000) loss 6.5868 (6.6451) grad_norm 3.3960 (3.9246) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][40/625] eta 0:03:50 lr 0.000108 wd 0.0500 time 0.3731 (0.3938) data time 0.0008 (0.0115) model time 0.0000 (0.0000) loss 6.8585 (6.6379) grad_norm 2.1553 (3.6735) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][50/625] eta 0:03:44 lr 0.000110 wd 0.0500 time 0.3767 (0.3903) data time 0.0008 (0.0094) model time 0.0000 (0.0000) loss 6.4643 (6.6506) grad_norm 4.9157 (3.5354) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][60/625] eta 0:03:39 lr 0.000111 wd 0.0500 time 0.3809 (0.3876) data time 0.0008 (0.0081) model time 0.3801 (0.3730) loss 6.4821 (6.6428) grad_norm 3.7297 (3.4440) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][70/625] eta 0:03:35 lr 0.000113 wd 0.0500 time 0.3889 (0.3879) data time 0.0008 (0.0071) model time 0.3882 (0.3808) loss 6.4384 (6.6327) grad_norm 3.1603 (3.4451) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][80/625] eta 0:03:30 lr 0.000115 wd 0.0500 time 0.3798 (0.3860) data time 0.0008 (0.0064) model time 0.3790 (0.3776) loss 6.7086 (6.6322) grad_norm 3.7883 (3.4361) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][90/625] eta 0:03:25 lr 0.000116 wd 0.0500 time 0.3699 (0.3845) data time 0.0011 (0.0058) model time 0.3688 (0.3760) loss 6.6676 (6.6314) grad_norm 2.7048 (3.4787) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][100/625] eta 0:03:21 lr 0.000118 wd 0.0500 time 0.3683 (0.3834) data time 0.0008 (0.0053) model time 0.3675 (0.3753) loss 6.6073 (6.6256) grad_norm 3.9187 (3.4431) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][110/625] eta 0:03:16 lr 0.000119 wd 0.0500 time 0.3742 (0.3823) data time 0.0008 (0.0049) model time 0.3734 (0.3744) loss 6.6888 (6.6246) grad_norm 3.5458 (3.3594) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:32:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][120/625] eta 0:03:12 lr 0.000121 wd 0.0500 time 0.3777 (0.3819) data time 0.0008 (0.0046) model time 0.3769 (0.3746) loss 6.7392 (6.6307) grad_norm 3.5395 (3.3842) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:33:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][130/625] eta 0:03:08 lr 0.000123 wd 0.0500 time 0.3902 (0.3816) data time 0.0008 (0.0044) model time 0.3894 (0.3750) loss 6.6213 (6.6315) grad_norm 1.9313 (3.3838) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:33:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][140/625] eta 0:03:04 lr 0.000124 wd 0.0500 time 0.3863 (0.3811) data time 0.0012 (0.0041) model time 0.3850 (0.3747) loss 6.7861 (6.6342) grad_norm 4.7704 (3.3670) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:33:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][150/625] eta 0:03:00 lr 0.000126 wd 0.0500 time 0.3772 (0.3806) data time 0.0010 (0.0039) model time 0.3762 (0.3745) loss 6.6741 (6.6328) grad_norm 2.9592 (3.3444) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][160/625] eta 0:02:56 lr 0.000127 wd 0.0500 time 0.3733 (0.3802) data time 0.0010 (0.0038) model time 0.3722 (0.3744) loss 6.6106 (6.6283) grad_norm 4.3235 (3.3601) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:33:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][170/625] eta 0:02:52 lr 0.000129 wd 0.0500 time 0.3729 (0.3799) data time 0.0008 (0.0036) model time 0.3721 (0.3743) loss 6.4471 (6.6241) grad_norm 3.4419 (3.3006) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:33:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][180/625] eta 0:02:48 lr 0.000131 wd 0.0500 time 0.3746 (0.3796) data time 0.0010 (0.0035) model time 0.3735 (0.3742) loss 6.4684 (6.6208) grad_norm 2.3948 (3.2756) loss_scale 65536.0000 (65536.0000) mem 11221MB [2024-08-19 23:33:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][190/625] eta 0:02:45 lr 0.000132 wd 0.0500 time 0.5932 (0.3804) data time 0.0008 (0.0034) model time 0.5924 (0.3757) loss 6.7223 (6.6165) grad_norm 2.6666 (inf) loss_scale 32768.0000 (64163.5183) mem 11221MB [2024-08-19 23:33:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][200/625] eta 0:02:41 lr 0.000134 wd 0.0500 time 0.3748 (0.3801) data time 0.0011 (0.0033) model time 0.3737 (0.3754) loss 6.3037 (6.6115) grad_norm 2.6976 (inf) loss_scale 32768.0000 (62601.5522) mem 11221MB [2024-08-19 23:33:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][210/625] eta 0:02:37 lr 0.000135 wd 0.0500 time 0.3724 (0.3798) data time 0.0009 (0.0032) model time 0.3715 (0.3752) loss 6.5997 (6.6086) grad_norm 2.4524 (inf) loss_scale 32768.0000 (61187.6398) mem 11221MB [2024-08-19 23:33:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][220/625] eta 0:02:33 lr 0.000137 wd 0.0500 time 0.3767 (0.3795) data time 0.0008 (0.0031) model time 0.3758 (0.3751) loss 6.7012 (6.6050) grad_norm 4.8895 (inf) loss_scale 32768.0000 (59901.6833) mem 11221MB [2024-08-19 23:33:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][230/625] eta 0:02:29 lr 0.000139 wd 0.0500 time 0.3704 (0.3792) data time 0.0011 (0.0030) model time 0.3693 (0.3749) loss 6.6723 (6.6056) grad_norm 2.5296 (inf) loss_scale 32768.0000 (58727.0649) mem 11221MB [2024-08-19 23:33:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][240/625] eta 0:02:25 lr 0.000140 wd 0.0500 time 0.3687 (0.3791) data time 0.0010 (0.0029) model time 0.3676 (0.3749) loss 6.7001 (6.6016) grad_norm 4.0697 (inf) loss_scale 32768.0000 (57649.9253) mem 11221MB [2024-08-19 23:33:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][250/625] eta 0:02:22 lr 0.000142 wd 0.0500 time 0.3817 (0.3789) data time 0.0011 (0.0028) model time 0.3806 (0.3748) loss 6.6231 (6.6004) grad_norm 4.7931 (inf) loss_scale 32768.0000 (56658.6135) mem 11221MB [2024-08-19 23:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][260/625] eta 0:02:18 lr 0.000143 wd 0.0500 time 0.3754 (0.3787) data time 0.0008 (0.0028) model time 0.3746 (0.3747) loss 6.5323 (6.5973) grad_norm 3.2604 (inf) loss_scale 32768.0000 (55743.2644) mem 11221MB [2024-08-19 23:33:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][270/625] eta 0:02:14 lr 0.000145 wd 0.0500 time 0.3718 (0.3785) data time 0.0011 (0.0027) model time 0.3706 (0.3746) loss 6.6961 (6.5968) grad_norm 2.8642 (inf) loss_scale 32768.0000 (54895.4686) mem 11221MB [2024-08-19 23:33:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][280/625] eta 0:02:10 lr 0.000147 wd 0.0500 time 0.3758 (0.3789) data time 0.0011 (0.0027) model time 0.3748 (0.3752) loss 6.6196 (6.5957) grad_norm 4.0436 (inf) loss_scale 32768.0000 (54108.0142) mem 11221MB [2024-08-19 23:34:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][290/625] eta 0:02:06 lr 0.000148 wd 0.0500 time 0.3746 (0.3786) data time 0.0008 (0.0026) model time 0.3738 (0.3750) loss 6.6004 (6.5933) grad_norm 4.4230 (inf) loss_scale 32768.0000 (53374.6804) mem 11221MB [2024-08-19 23:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][300/625] eta 0:02:02 lr 0.000150 wd 0.0500 time 0.3654 (0.3783) data time 0.0012 (0.0025) model time 0.3642 (0.3747) loss 6.6624 (6.5948) grad_norm 2.9336 (inf) loss_scale 32768.0000 (52690.0731) mem 11221MB [2024-08-19 23:34:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][310/625] eta 0:01:59 lr 0.000151 wd 0.0500 time 0.3729 (0.3789) data time 0.0009 (0.0025) model time 0.3720 (0.3756) loss 6.2189 (6.5941) grad_norm 2.4226 (inf) loss_scale 32768.0000 (52049.4920) mem 11221MB [2024-08-19 23:34:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][320/625] eta 0:01:55 lr 0.000153 wd 0.0500 time 0.3696 (0.3787) data time 0.0011 (0.0025) model time 0.3686 (0.3754) loss 6.7124 (6.5920) grad_norm 3.7311 (inf) loss_scale 32768.0000 (51448.8224) mem 11221MB [2024-08-19 23:34:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][330/625] eta 0:01:51 lr 0.000155 wd 0.0500 time 0.3683 (0.3786) data time 0.0008 (0.0024) model time 0.3675 (0.3753) loss 6.8316 (6.5905) grad_norm 5.8594 (inf) loss_scale 32768.0000 (50884.4471) mem 11221MB [2024-08-19 23:34:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][340/625] eta 0:01:47 lr 0.000156 wd 0.0500 time 0.3766 (0.3785) data time 0.0010 (0.0024) model time 0.3756 (0.3753) loss 6.7056 (6.5893) grad_norm 2.4570 (inf) loss_scale 32768.0000 (50353.1730) mem 11221MB [2024-08-19 23:34:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][350/625] eta 0:01:44 lr 0.000158 wd 0.0500 time 0.3715 (0.3785) data time 0.0010 (0.0023) model time 0.3705 (0.3754) loss 6.4463 (6.5894) grad_norm 1.7362 (inf) loss_scale 32768.0000 (49852.1709) mem 11221MB [2024-08-19 23:34:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][360/625] eta 0:01:40 lr 0.000159 wd 0.0500 time 0.3862 (0.3785) data time 0.0011 (0.0023) model time 0.3851 (0.3755) loss 6.2678 (6.5855) grad_norm 2.7914 (inf) loss_scale 32768.0000 (49378.9252) mem 11221MB [2024-08-19 23:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][370/625] eta 0:01:36 lr 0.000161 wd 0.0500 time 0.3810 (0.3785) data time 0.0010 (0.0023) model time 0.3801 (0.3755) loss 6.4494 (6.5851) grad_norm 3.5119 (inf) loss_scale 32768.0000 (48931.1914) mem 11221MB [2024-08-19 23:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][380/625] eta 0:01:32 lr 0.000163 wd 0.0500 time 0.3763 (0.3784) data time 0.0011 (0.0023) model time 0.3753 (0.3754) loss 6.5767 (6.5823) grad_norm 2.6393 (inf) loss_scale 32768.0000 (48506.9606) mem 11221MB [2024-08-19 23:34:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][390/625] eta 0:01:28 lr 0.000164 wd 0.0500 time 0.3773 (0.3784) data time 0.0009 (0.0022) model time 0.3764 (0.3754) loss 6.2557 (6.5830) grad_norm 3.2909 (inf) loss_scale 32768.0000 (48104.4297) mem 11221MB [2024-08-19 23:34:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][400/625] eta 0:01:25 lr 0.000166 wd 0.0500 time 0.3779 (0.3783) data time 0.0010 (0.0022) model time 0.3769 (0.3754) loss 6.5097 (6.5790) grad_norm 2.0452 (inf) loss_scale 32768.0000 (47721.9751) mem 11221MB [2024-08-19 23:34:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][410/625] eta 0:01:21 lr 0.000167 wd 0.0500 time 0.3634 (0.3782) data time 0.0009 (0.0022) model time 0.3625 (0.3753) loss 6.5661 (6.5759) grad_norm 3.2396 (inf) loss_scale 32768.0000 (47358.1314) mem 11221MB [2024-08-19 23:34:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][420/625] eta 0:01:17 lr 0.000169 wd 0.0500 time 0.3752 (0.3782) data time 0.0010 (0.0022) model time 0.3742 (0.3753) loss 6.5085 (6.5734) grad_norm 2.5829 (inf) loss_scale 32768.0000 (47011.5724) mem 11221MB [2024-08-19 23:34:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][430/625] eta 0:01:13 lr 0.000171 wd 0.0500 time 0.3713 (0.3780) data time 0.0010 (0.0021) model time 0.3702 (0.3753) loss 6.4173 (6.5728) grad_norm 3.3036 (inf) loss_scale 32768.0000 (46681.0951) mem 11221MB [2024-08-19 23:34:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][440/625] eta 0:01:09 lr 0.000172 wd 0.0500 time 0.3726 (0.3780) data time 0.0008 (0.0022) model time 0.3718 (0.3752) loss 6.3007 (6.5717) grad_norm 2.1009 (inf) loss_scale 32768.0000 (46365.6054) mem 11221MB [2024-08-19 23:35:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][450/625] eta 0:01:06 lr 0.000174 wd 0.0500 time 0.3678 (0.3780) data time 0.0011 (0.0021) model time 0.3667 (0.3752) loss 6.3950 (6.5685) grad_norm 3.4751 (inf) loss_scale 32768.0000 (46064.1064) mem 11221MB [2024-08-19 23:35:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][460/625] eta 0:01:02 lr 0.000175 wd 0.0500 time 0.3721 (0.3783) data time 0.0009 (0.0021) model time 0.3712 (0.3756) loss 6.3386 (6.5664) grad_norm 2.6414 (inf) loss_scale 32768.0000 (45775.6876) mem 11221MB [2024-08-19 23:35:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][470/625] eta 0:00:58 lr 0.000177 wd 0.0500 time 0.3652 (0.3782) data time 0.0008 (0.0021) model time 0.3644 (0.3756) loss 6.4326 (6.5628) grad_norm 1.7385 (inf) loss_scale 32768.0000 (45499.5159) mem 11221MB [2024-08-19 23:35:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][480/625] eta 0:00:54 lr 0.000179 wd 0.0500 time 0.3728 (0.3782) data time 0.0011 (0.0021) model time 0.3716 (0.3755) loss 6.6295 (6.5618) grad_norm 2.3447 (inf) loss_scale 32768.0000 (45234.8274) mem 11221MB [2024-08-19 23:35:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][490/625] eta 0:00:51 lr 0.000180 wd 0.0500 time 0.3725 (0.3781) data time 0.0010 (0.0021) model time 0.3715 (0.3755) loss 6.6915 (6.5612) grad_norm 2.9491 (inf) loss_scale 32768.0000 (44980.9206) mem 11221MB [2024-08-19 23:35:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][500/625] eta 0:00:47 lr 0.000182 wd 0.0500 time 0.3664 (0.3780) data time 0.0010 (0.0020) model time 0.3654 (0.3754) loss 6.7017 (6.5599) grad_norm 4.5135 (inf) loss_scale 32768.0000 (44737.1497) mem 11221MB [2024-08-19 23:35:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][510/625] eta 0:00:43 lr 0.000183 wd 0.0500 time 0.3770 (0.3780) data time 0.0011 (0.0020) model time 0.3759 (0.3754) loss 6.6935 (6.5583) grad_norm 3.8066 (inf) loss_scale 32768.0000 (44502.9198) mem 11221MB [2024-08-19 23:35:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][520/625] eta 0:00:39 lr 0.000185 wd 0.0500 time 0.3773 (0.3784) data time 0.0009 (0.0020) model time 0.3764 (0.3758) loss 6.0368 (6.5559) grad_norm 2.7769 (inf) loss_scale 32768.0000 (44277.6814) mem 11221MB [2024-08-19 23:35:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][530/625] eta 0:00:35 lr 0.000187 wd 0.0500 time 0.3746 (0.3783) data time 0.0008 (0.0020) model time 0.3738 (0.3758) loss 6.7022 (6.5546) grad_norm 2.8896 (inf) loss_scale 32768.0000 (44060.9266) mem 11221MB [2024-08-19 23:35:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][540/625] eta 0:00:32 lr 0.000188 wd 0.0500 time 0.3727 (0.3782) data time 0.0009 (0.0020) model time 0.3718 (0.3757) loss 6.4891 (6.5510) grad_norm 1.9686 (inf) loss_scale 32768.0000 (43852.1848) mem 11221MB [2024-08-19 23:35:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][550/625] eta 0:00:28 lr 0.000190 wd 0.0500 time 0.3810 (0.3782) data time 0.0011 (0.0020) model time 0.3798 (0.3757) loss 6.5506 (6.5504) grad_norm 2.3706 (inf) loss_scale 32768.0000 (43651.0200) mem 11221MB [2024-08-19 23:35:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][560/625] eta 0:00:24 lr 0.000191 wd 0.0500 time 0.3656 (0.3781) data time 0.0011 (0.0020) model time 0.3645 (0.3757) loss 6.5205 (6.5504) grad_norm 2.2835 (inf) loss_scale 32768.0000 (43457.0267) mem 11221MB [2024-08-19 23:35:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][570/625] eta 0:00:20 lr 0.000193 wd 0.0500 time 0.3789 (0.3781) data time 0.0010 (0.0020) model time 0.3779 (0.3756) loss 6.5541 (6.5487) grad_norm 2.8648 (inf) loss_scale 32768.0000 (43269.8284) mem 11221MB [2024-08-19 23:35:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][580/625] eta 0:00:17 lr 0.000195 wd 0.0500 time 0.3728 (0.3784) data time 0.0008 (0.0019) model time 0.3720 (0.3760) loss 6.2211 (6.5475) grad_norm 2.6575 (inf) loss_scale 32768.0000 (43089.0740) mem 11221MB [2024-08-19 23:35:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][590/625] eta 0:00:13 lr 0.000196 wd 0.0500 time 0.3779 (0.3784) data time 0.0008 (0.0019) model time 0.3771 (0.3760) loss 6.2752 (6.5455) grad_norm 2.7437 (inf) loss_scale 32768.0000 (42914.4365) mem 11221MB [2024-08-19 23:35:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][600/625] eta 0:00:09 lr 0.000198 wd 0.0500 time 0.3751 (0.3783) data time 0.0008 (0.0019) model time 0.3743 (0.3760) loss 6.1028 (6.5427) grad_norm 4.6598 (inf) loss_scale 32768.0000 (42745.6106) mem 11221MB [2024-08-19 23:36:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][610/625] eta 0:00:05 lr 0.000199 wd 0.0500 time 0.3650 (0.3783) data time 0.0006 (0.0019) model time 0.3644 (0.3759) loss 6.0647 (6.5405) grad_norm 2.6201 (inf) loss_scale 32768.0000 (42582.3110) mem 11221MB [2024-08-19 23:36:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [1/300][620/625] eta 0:00:01 lr 0.000201 wd 0.0500 time 0.3585 (0.3780) data time 0.0008 (0.0019) model time 0.3577 (0.3757) loss 6.4672 (6.5389) grad_norm 2.6081 (inf) loss_scale 32768.0000 (42424.2705) mem 11221MB [2024-08-19 23:36:07 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 1 training takes 0:03:56 [2024-08-19 23:36:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-19 23:36:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-19 23:36:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.504 (0.504) Loss 5.4062 (5.4062) Acc@1 5.176 (5.176) Acc@5 19.336 (19.336) Mem 11221MB [2024-08-19 23:36:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 5.8047 (5.5270) Acc@1 3.613 (4.887) Acc@5 12.598 (15.953) Mem 11221MB [2024-08-19 23:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.121) Loss 5.7852 (5.5826) Acc@1 2.734 (5.301) Acc@5 8.398 (15.955) Mem 11221MB [2024-08-19 23:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 6.060 Acc@5 17.618 [2024-08-19 23:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 6.1% [2024-08-19 23:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 6.06% [2024-08-19 23:36:11 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-19 23:36:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-19 23:36:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.510 (0.510) Loss 6.9805 (6.9805) Acc@1 0.000 (0.000) Acc@5 0.342 (0.342) Mem 11221MB [2024-08-19 23:36:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.148) Loss 6.9492 (6.9716) Acc@1 0.098 (0.120) Acc@5 0.781 (0.533) Mem 11221MB [2024-08-19 23:36:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 6.9531 (6.9712) Acc@1 0.146 (0.114) Acc@5 0.488 (0.502) Mem 11221MB [2024-08-19 23:36:15 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.102 Acc@5 0.472 [2024-08-19 23:36:15 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.1% [2024-08-19 23:36:15 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.10% [2024-08-19 23:36:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-19 23:36:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-19 23:36:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][0/625] eta 0:07:59 lr 0.000202 wd 0.0500 time 0.7665 (0.7665) data time 0.4217 (0.4217) model time 0.0000 (0.0000) loss 6.3680 (6.3680) grad_norm 2.5981 (2.5981) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][10/625] eta 0:04:13 lr 0.000203 wd 0.0500 time 0.3803 (0.4124) data time 0.0011 (0.0395) model time 0.0000 (0.0000) loss 6.3455 (6.4540) grad_norm 1.7676 (2.9510) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][20/625] eta 0:03:58 lr 0.000205 wd 0.0500 time 0.3770 (0.3942) data time 0.0008 (0.0212) model time 0.0000 (0.0000) loss 6.1181 (6.4727) grad_norm 2.5063 (2.9620) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][30/625] eta 0:03:51 lr 0.000207 wd 0.0500 time 0.3771 (0.3884) data time 0.0008 (0.0147) model time 0.0000 (0.0000) loss 6.4530 (6.4808) grad_norm 2.8845 (2.9960) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][40/625] eta 0:03:45 lr 0.000208 wd 0.0500 time 0.3782 (0.3852) data time 0.0008 (0.0114) model time 0.0000 (0.0000) loss 6.7463 (6.4781) grad_norm 2.5333 (3.0198) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][50/625] eta 0:03:40 lr 0.000210 wd 0.0500 time 0.3682 (0.3832) data time 0.0009 (0.0094) model time 0.0000 (0.0000) loss 6.7086 (6.4657) grad_norm 4.1305 (2.9683) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][60/625] eta 0:03:35 lr 0.000211 wd 0.0500 time 0.3749 (0.3821) data time 0.0008 (0.0080) model time 0.3741 (0.3757) loss 5.8374 (6.4428) grad_norm 2.4871 (2.9196) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][70/625] eta 0:03:31 lr 0.000213 wd 0.0500 time 0.3878 (0.3813) data time 0.0010 (0.0071) model time 0.3867 (0.3753) loss 5.9357 (6.4190) grad_norm 2.7226 (2.8480) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][80/625] eta 0:03:28 lr 0.000215 wd 0.0500 time 0.3683 (0.3825) data time 0.0008 (0.0063) model time 0.3675 (0.3803) loss 6.4157 (6.4217) grad_norm 2.6320 (2.8122) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][90/625] eta 0:03:25 lr 0.000216 wd 0.0500 time 0.3693 (0.3838) data time 0.0009 (0.0058) model time 0.3685 (0.3835) loss 6.5869 (6.4260) grad_norm 3.6029 (2.7931) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][100/625] eta 0:03:21 lr 0.000218 wd 0.0500 time 0.3727 (0.3829) data time 0.0013 (0.0053) model time 0.3714 (0.3815) loss 6.5132 (6.4261) grad_norm 2.4311 (2.8260) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:36:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][110/625] eta 0:03:16 lr 0.000219 wd 0.0500 time 0.3748 (0.3823) data time 0.0015 (0.0049) model time 0.3733 (0.3803) loss 6.5845 (6.4278) grad_norm 2.9629 (2.8339) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][120/625] eta 0:03:12 lr 0.000221 wd 0.0500 time 0.3759 (0.3816) data time 0.0010 (0.0046) model time 0.3749 (0.3793) loss 6.5651 (6.4291) grad_norm 2.1640 (2.7987) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][130/625] eta 0:03:08 lr 0.000223 wd 0.0500 time 0.3775 (0.3811) data time 0.0011 (0.0044) model time 0.3763 (0.3786) loss 6.5769 (6.4301) grad_norm 2.6704 (2.7750) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][140/625] eta 0:03:04 lr 0.000224 wd 0.0500 time 0.3740 (0.3807) data time 0.0010 (0.0041) model time 0.3730 (0.3782) loss 6.4827 (6.4244) grad_norm 3.2876 (2.7922) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][150/625] eta 0:03:00 lr 0.000226 wd 0.0500 time 0.3806 (0.3804) data time 0.0011 (0.0039) model time 0.3795 (0.3778) loss 6.5463 (6.4220) grad_norm 3.9320 (2.8279) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][160/625] eta 0:02:56 lr 0.000227 wd 0.0500 time 0.3721 (0.3801) data time 0.0008 (0.0038) model time 0.3712 (0.3775) loss 6.3252 (6.4257) grad_norm 3.5598 (2.8458) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][170/625] eta 0:02:52 lr 0.000229 wd 0.0500 time 0.3709 (0.3798) data time 0.0011 (0.0036) model time 0.3698 (0.3772) loss 6.4025 (6.4227) grad_norm 3.5221 (2.8230) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][180/625] eta 0:02:48 lr 0.000231 wd 0.0500 time 0.3744 (0.3794) data time 0.0008 (0.0035) model time 0.3736 (0.3768) loss 6.6236 (6.4190) grad_norm 2.1622 (2.8174) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][190/625] eta 0:02:45 lr 0.000232 wd 0.0500 time 0.3738 (0.3793) data time 0.0008 (0.0034) model time 0.3730 (0.3768) loss 6.3993 (6.4141) grad_norm 2.3571 (2.8144) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][200/625] eta 0:02:41 lr 0.000234 wd 0.0500 time 0.3716 (0.3792) data time 0.0008 (0.0033) model time 0.3707 (0.3767) loss 6.3038 (6.4113) grad_norm 1.9307 (2.8126) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][210/625] eta 0:02:37 lr 0.000235 wd 0.0500 time 0.3705 (0.3790) data time 0.0010 (0.0032) model time 0.3695 (0.3765) loss 6.3553 (6.4065) grad_norm 3.7284 (2.7941) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][220/625] eta 0:02:33 lr 0.000237 wd 0.0500 time 0.3655 (0.3787) data time 0.0008 (0.0031) model time 0.3647 (0.3762) loss 6.0802 (6.4035) grad_norm 5.5174 (2.8306) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][230/625] eta 0:02:29 lr 0.000239 wd 0.0500 time 0.3667 (0.3785) data time 0.0008 (0.0030) model time 0.3659 (0.3760) loss 6.3002 (6.3916) grad_norm 2.9070 (2.8331) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][240/625] eta 0:02:25 lr 0.000240 wd 0.0500 time 0.3751 (0.3785) data time 0.0011 (0.0030) model time 0.3740 (0.3760) loss 6.1979 (6.3810) grad_norm 1.8407 (2.8148) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][250/625] eta 0:02:21 lr 0.000242 wd 0.0500 time 0.3742 (0.3784) data time 0.0010 (0.0029) model time 0.3732 (0.3759) loss 6.3092 (6.3769) grad_norm 3.1319 (2.8068) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][260/625] eta 0:02:18 lr 0.000243 wd 0.0500 time 0.3712 (0.3781) data time 0.0010 (0.0028) model time 0.3702 (0.3757) loss 6.3941 (6.3783) grad_norm 2.7028 (2.8017) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:37:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][270/625] eta 0:02:14 lr 0.000245 wd 0.0500 time 0.3772 (0.3787) data time 0.0007 (0.0028) model time 0.3764 (0.3764) loss 6.3385 (6.3790) grad_norm 2.4960 (inf) loss_scale 16384.0000 (32586.6273) mem 11221MB [2024-08-19 23:38:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][280/625] eta 0:02:10 lr 0.000247 wd 0.0500 time 0.3904 (0.3794) data time 0.0008 (0.0027) model time 0.3895 (0.3774) loss 6.0727 (6.3755) grad_norm 2.1902 (inf) loss_scale 16384.0000 (32010.0214) mem 11221MB [2024-08-19 23:38:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][290/625] eta 0:02:07 lr 0.000248 wd 0.0500 time 0.3699 (0.3792) data time 0.0007 (0.0027) model time 0.3691 (0.3771) loss 5.8021 (6.3726) grad_norm 3.7811 (inf) loss_scale 16384.0000 (31473.0447) mem 11221MB [2024-08-19 23:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][300/625] eta 0:02:03 lr 0.000250 wd 0.0500 time 0.3813 (0.3791) data time 0.0010 (0.0026) model time 0.3803 (0.3771) loss 6.1703 (6.3699) grad_norm 2.7143 (inf) loss_scale 16384.0000 (30971.7475) mem 11221MB [2024-08-19 23:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][310/625] eta 0:01:59 lr 0.000251 wd 0.0500 time 0.3836 (0.3790) data time 0.0009 (0.0026) model time 0.3827 (0.3770) loss 5.8373 (6.3656) grad_norm 2.0293 (inf) loss_scale 16384.0000 (30502.6881) mem 11221MB [2024-08-19 23:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][320/625] eta 0:01:55 lr 0.000253 wd 0.0500 time 0.3746 (0.3788) data time 0.0008 (0.0025) model time 0.3738 (0.3768) loss 6.0375 (6.3596) grad_norm 1.9838 (inf) loss_scale 16384.0000 (30062.8536) mem 11221MB [2024-08-19 23:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][330/625] eta 0:01:51 lr 0.000255 wd 0.0500 time 0.3694 (0.3787) data time 0.0009 (0.0025) model time 0.3685 (0.3766) loss 6.1061 (6.3554) grad_norm 3.5303 (inf) loss_scale 16384.0000 (29649.5952) mem 11221MB [2024-08-19 23:38:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][340/625] eta 0:01:47 lr 0.000256 wd 0.0500 time 0.3671 (0.3785) data time 0.0011 (0.0024) model time 0.3660 (0.3765) loss 6.4146 (6.3548) grad_norm 2.7890 (inf) loss_scale 16384.0000 (29260.5748) mem 11221MB [2024-08-19 23:38:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][350/625] eta 0:01:44 lr 0.000258 wd 0.0500 time 0.3781 (0.3783) data time 0.0009 (0.0024) model time 0.3772 (0.3763) loss 6.3626 (6.3557) grad_norm 5.8672 (inf) loss_scale 16384.0000 (28893.7208) mem 11221MB [2024-08-19 23:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][360/625] eta 0:01:40 lr 0.000259 wd 0.0500 time 0.3691 (0.3782) data time 0.0013 (0.0024) model time 0.3678 (0.3762) loss 6.2509 (6.3537) grad_norm 2.5941 (inf) loss_scale 16384.0000 (28547.1911) mem 11221MB [2024-08-19 23:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][370/625] eta 0:01:36 lr 0.000261 wd 0.0500 time 0.3709 (0.3781) data time 0.0013 (0.0023) model time 0.3696 (0.3761) loss 5.8720 (6.3506) grad_norm 2.1356 (inf) loss_scale 16384.0000 (28219.3423) mem 11221MB [2024-08-19 23:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][380/625] eta 0:01:32 lr 0.000263 wd 0.0500 time 0.5835 (0.3785) data time 0.0011 (0.0023) model time 0.5824 (0.3766) loss 6.5865 (6.3493) grad_norm 1.9884 (inf) loss_scale 16384.0000 (27908.7034) mem 11221MB [2024-08-19 23:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][390/625] eta 0:01:29 lr 0.000264 wd 0.0500 time 0.5494 (0.3788) data time 0.0008 (0.0023) model time 0.5486 (0.3770) loss 6.0913 (6.3468) grad_norm 2.6677 (inf) loss_scale 16384.0000 (27613.9540) mem 11221MB [2024-08-19 23:38:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][400/625] eta 0:01:25 lr 0.000266 wd 0.0500 time 0.3652 (0.3787) data time 0.0011 (0.0022) model time 0.3641 (0.3768) loss 5.9373 (6.3392) grad_norm 3.3317 (inf) loss_scale 16384.0000 (27333.9052) mem 11221MB [2024-08-19 23:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][410/625] eta 0:01:21 lr 0.000267 wd 0.0500 time 0.3699 (0.3786) data time 0.0012 (0.0022) model time 0.3687 (0.3767) loss 6.0247 (6.3327) grad_norm 3.8160 (inf) loss_scale 16384.0000 (27067.4842) mem 11221MB [2024-08-19 23:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][420/625] eta 0:01:17 lr 0.000269 wd 0.0500 time 0.3702 (0.3785) data time 0.0010 (0.0022) model time 0.3692 (0.3766) loss 6.5236 (6.3317) grad_norm 1.6932 (inf) loss_scale 16384.0000 (26813.7197) mem 11221MB [2024-08-19 23:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][430/625] eta 0:01:13 lr 0.000271 wd 0.0500 time 0.3788 (0.3784) data time 0.0011 (0.0022) model time 0.3777 (0.3765) loss 6.5012 (6.3279) grad_norm 2.3377 (inf) loss_scale 16384.0000 (26571.7309) mem 11221MB [2024-08-19 23:39:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][440/625] eta 0:01:09 lr 0.000272 wd 0.0500 time 0.3763 (0.3782) data time 0.0008 (0.0021) model time 0.3755 (0.3764) loss 5.9849 (6.3265) grad_norm 2.8673 (inf) loss_scale 16384.0000 (26340.7166) mem 11221MB [2024-08-19 23:39:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][450/625] eta 0:01:06 lr 0.000274 wd 0.0500 time 0.3634 (0.3781) data time 0.0009 (0.0021) model time 0.3625 (0.3762) loss 6.4242 (6.3277) grad_norm 3.2425 (inf) loss_scale 16384.0000 (26119.9468) mem 11221MB [2024-08-19 23:39:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][460/625] eta 0:01:02 lr 0.000275 wd 0.0500 time 0.3765 (0.3780) data time 0.0008 (0.0021) model time 0.3757 (0.3762) loss 5.9211 (6.3238) grad_norm 2.3556 (inf) loss_scale 16384.0000 (25908.7549) mem 11221MB [2024-08-19 23:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][470/625] eta 0:00:58 lr 0.000277 wd 0.0500 time 0.3702 (0.3779) data time 0.0009 (0.0021) model time 0.3694 (0.3761) loss 5.8962 (6.3214) grad_norm 4.0818 (inf) loss_scale 16384.0000 (25706.5308) mem 11221MB [2024-08-19 23:39:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][480/625] eta 0:00:54 lr 0.000279 wd 0.0500 time 0.3770 (0.3779) data time 0.0011 (0.0021) model time 0.3758 (0.3760) loss 6.6041 (6.3223) grad_norm 3.3811 (inf) loss_scale 16384.0000 (25512.7152) mem 11221MB [2024-08-19 23:39:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][490/625] eta 0:00:51 lr 0.000280 wd 0.0500 time 0.3689 (0.3778) data time 0.0014 (0.0020) model time 0.3675 (0.3760) loss 6.2331 (6.3200) grad_norm 2.5354 (inf) loss_scale 16384.0000 (25326.7943) mem 11221MB [2024-08-19 23:39:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][500/625] eta 0:00:47 lr 0.000282 wd 0.0500 time 0.3701 (0.3777) data time 0.0010 (0.0020) model time 0.3691 (0.3759) loss 6.1129 (6.3166) grad_norm 2.2691 (inf) loss_scale 16384.0000 (25148.2954) mem 11221MB [2024-08-19 23:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][510/625] eta 0:00:43 lr 0.000283 wd 0.0500 time 0.3755 (0.3776) data time 0.0009 (0.0020) model time 0.3746 (0.3758) loss 6.6311 (6.3147) grad_norm 3.4310 (inf) loss_scale 16384.0000 (24976.7828) mem 11221MB [2024-08-19 23:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][520/625] eta 0:00:39 lr 0.000285 wd 0.0500 time 0.3803 (0.3776) data time 0.0010 (0.0020) model time 0.3793 (0.3758) loss 6.0209 (6.3142) grad_norm 3.0786 (inf) loss_scale 16384.0000 (24811.8541) mem 11221MB [2024-08-19 23:39:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][530/625] eta 0:00:35 lr 0.000287 wd 0.0500 time 0.3726 (0.3775) data time 0.0008 (0.0020) model time 0.3718 (0.3758) loss 6.3508 (6.3122) grad_norm 2.5878 (inf) loss_scale 16384.0000 (24653.1375) mem 11221MB [2024-08-19 23:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][540/625] eta 0:00:32 lr 0.000288 wd 0.0500 time 0.3666 (0.3775) data time 0.0008 (0.0020) model time 0.3658 (0.3757) loss 5.9697 (6.3079) grad_norm 3.2865 (inf) loss_scale 16384.0000 (24500.2884) mem 11221MB [2024-08-19 23:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][550/625] eta 0:00:28 lr 0.000290 wd 0.0500 time 0.3791 (0.3774) data time 0.0013 (0.0019) model time 0.3778 (0.3757) loss 6.0962 (6.3077) grad_norm 1.6260 (inf) loss_scale 16384.0000 (24352.9873) mem 11221MB [2024-08-19 23:39:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][560/625] eta 0:00:24 lr 0.000291 wd 0.0500 time 0.3809 (0.3774) data time 0.0009 (0.0019) model time 0.3799 (0.3756) loss 5.7052 (6.3038) grad_norm 2.6784 (inf) loss_scale 16384.0000 (24210.9376) mem 11221MB [2024-08-19 23:39:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][570/625] eta 0:00:20 lr 0.000293 wd 0.0500 time 0.3778 (0.3773) data time 0.0011 (0.0019) model time 0.3767 (0.3756) loss 6.3747 (6.3015) grad_norm 2.4619 (inf) loss_scale 16384.0000 (24073.8634) mem 11221MB [2024-08-19 23:39:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][580/625] eta 0:00:16 lr 0.000295 wd 0.0500 time 0.3701 (0.3773) data time 0.0010 (0.0019) model time 0.3691 (0.3755) loss 6.2079 (6.3001) grad_norm 1.9227 (inf) loss_scale 16384.0000 (23941.5077) mem 11221MB [2024-08-19 23:39:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][590/625] eta 0:00:13 lr 0.000296 wd 0.0500 time 0.5542 (0.3775) data time 0.0008 (0.0019) model time 0.5533 (0.3758) loss 6.0004 (6.2979) grad_norm 4.4982 (inf) loss_scale 16384.0000 (23813.6311) mem 11221MB [2024-08-19 23:40:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][600/625] eta 0:00:09 lr 0.000298 wd 0.0500 time 0.6309 (0.3779) data time 0.0008 (0.0019) model time 0.6301 (0.3762) loss 5.5047 (6.2938) grad_norm 2.3020 (inf) loss_scale 16384.0000 (23690.0100) mem 11221MB [2024-08-19 23:40:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][610/625] eta 0:00:05 lr 0.000299 wd 0.0500 time 0.3682 (0.3778) data time 0.0005 (0.0019) model time 0.3677 (0.3761) loss 6.2621 (6.2924) grad_norm 2.7833 (inf) loss_scale 16384.0000 (23570.4354) mem 11221MB [2024-08-19 23:40:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [2/300][620/625] eta 0:00:01 lr 0.000301 wd 0.0500 time 0.3622 (0.3776) data time 0.0008 (0.0019) model time 0.3614 (0.3759) loss 6.3541 (6.2877) grad_norm 2.5843 (inf) loss_scale 16384.0000 (23454.7118) mem 11221MB [2024-08-19 23:40:12 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 2 training takes 0:03:55 [2024-08-19 23:40:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-19 23:40:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-19 23:40:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.512 (0.512) Loss 4.6992 (4.6992) Acc@1 12.695 (12.695) Acc@5 31.445 (31.445) Mem 11221MB [2024-08-19 23:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 5.2773 (4.6889) Acc@1 9.131 (11.923) Acc@5 21.436 (31.259) Mem 11221MB [2024-08-19 23:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 5.2070 (4.8776) Acc@1 6.689 (11.135) Acc@5 19.141 (28.462) Mem 11221MB [2024-08-19 23:40:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 12.178 Acc@5 29.746 [2024-08-19 23:40:16 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 12.2% [2024-08-19 23:40:16 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 12.18% [2024-08-19 23:40:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-19 23:40:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-19 23:40:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 6.9727 (6.9727) Acc@1 0.098 (0.098) Acc@5 0.635 (0.635) Mem 11221MB [2024-08-19 23:40:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 6.9180 (6.9648) Acc@1 0.049 (0.138) Acc@5 0.781 (0.630) Mem 11221MB [2024-08-19 23:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 6.9492 (6.9580) Acc@1 0.146 (0.126) Acc@5 0.537 (0.593) Mem 11221MB [2024-08-19 23:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.122 Acc@5 0.606 [2024-08-19 23:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.1% [2024-08-19 23:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.12% [2024-08-19 23:40:20 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-19 23:40:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-19 23:40:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][0/625] eta 0:08:11 lr 0.000302 wd 0.0500 time 0.7859 (0.7859) data time 0.4514 (0.4514) model time 0.0000 (0.0000) loss 6.2972 (6.2972) grad_norm 2.5975 (2.5975) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][10/625] eta 0:04:13 lr 0.000303 wd 0.0500 time 0.3713 (0.4124) data time 0.0009 (0.0421) model time 0.0000 (0.0000) loss 5.9804 (6.1723) grad_norm 2.9933 (2.8767) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][20/625] eta 0:03:57 lr 0.000305 wd 0.0500 time 0.3669 (0.3932) data time 0.0012 (0.0226) model time 0.0000 (0.0000) loss 6.4557 (6.1077) grad_norm 2.7034 (2.7858) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][30/625] eta 0:03:50 lr 0.000306 wd 0.0500 time 0.3780 (0.3874) data time 0.0011 (0.0157) model time 0.0000 (0.0000) loss 6.4347 (6.1132) grad_norm 2.4166 (2.6142) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][40/625] eta 0:03:45 lr 0.000308 wd 0.0500 time 0.3901 (0.3849) data time 0.0008 (0.0121) model time 0.0000 (0.0000) loss 6.1743 (6.0950) grad_norm 2.7411 (2.5878) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][50/625] eta 0:03:40 lr 0.000310 wd 0.0500 time 0.3765 (0.3832) data time 0.0008 (0.0100) model time 0.0000 (0.0000) loss 5.4884 (6.0617) grad_norm 3.0616 (2.5670) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][60/625] eta 0:03:35 lr 0.000311 wd 0.0500 time 0.3730 (0.3815) data time 0.0008 (0.0085) model time 0.3722 (0.3719) loss 6.0265 (6.0648) grad_norm 1.9453 (2.6465) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][70/625] eta 0:03:32 lr 0.000313 wd 0.0500 time 0.3813 (0.3835) data time 0.0008 (0.0075) model time 0.3804 (0.3832) loss 6.0788 (6.0625) grad_norm 1.8446 (2.5768) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][80/625] eta 0:03:28 lr 0.000314 wd 0.0500 time 0.3806 (0.3826) data time 0.0008 (0.0067) model time 0.3798 (0.3805) loss 6.3307 (6.0581) grad_norm 2.8011 (2.6189) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:40:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][90/625] eta 0:03:24 lr 0.000316 wd 0.0500 time 0.3669 (0.3818) data time 0.0010 (0.0061) model time 0.3658 (0.3789) loss 5.8487 (6.0554) grad_norm 3.0726 (2.5975) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][100/625] eta 0:03:20 lr 0.000318 wd 0.0500 time 0.3849 (0.3811) data time 0.0010 (0.0056) model time 0.3839 (0.3778) loss 6.4624 (6.0585) grad_norm 1.7045 (2.5604) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][110/625] eta 0:03:16 lr 0.000319 wd 0.0500 time 0.3695 (0.3823) data time 0.0010 (0.0052) model time 0.3685 (0.3805) loss 5.8287 (6.0509) grad_norm 2.2269 (2.5313) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][120/625] eta 0:03:12 lr 0.000321 wd 0.0500 time 0.3791 (0.3818) data time 0.0008 (0.0049) model time 0.3784 (0.3796) loss 6.0693 (6.0443) grad_norm 2.8948 (2.5727) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][130/625] eta 0:03:08 lr 0.000322 wd 0.0500 time 0.3826 (0.3813) data time 0.0008 (0.0046) model time 0.3818 (0.3789) loss 6.2422 (6.0511) grad_norm 2.6974 (2.5752) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][140/625] eta 0:03:04 lr 0.000324 wd 0.0500 time 0.3767 (0.3809) data time 0.0010 (0.0044) model time 0.3757 (0.3785) loss 5.8206 (6.0477) grad_norm 3.6110 (2.5665) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][150/625] eta 0:03:00 lr 0.000326 wd 0.0500 time 0.3781 (0.3806) data time 0.0010 (0.0041) model time 0.3771 (0.3781) loss 6.4263 (6.0593) grad_norm 2.4898 (2.5747) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][160/625] eta 0:02:56 lr 0.000327 wd 0.0500 time 0.3842 (0.3805) data time 0.0012 (0.0040) model time 0.3831 (0.3780) loss 6.3133 (6.0654) grad_norm 2.5722 (2.5467) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][170/625] eta 0:02:52 lr 0.000329 wd 0.0500 time 0.3700 (0.3802) data time 0.0009 (0.0038) model time 0.3691 (0.3777) loss 5.4794 (6.0670) grad_norm 3.4460 (2.5840) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][180/625] eta 0:02:49 lr 0.000330 wd 0.0500 time 0.3753 (0.3800) data time 0.0007 (0.0036) model time 0.3746 (0.3776) loss 5.7520 (6.0696) grad_norm 3.1942 (2.5747) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][190/625] eta 0:02:45 lr 0.000332 wd 0.0500 time 0.3747 (0.3806) data time 0.0008 (0.0035) model time 0.3739 (0.3785) loss 6.0549 (6.0647) grad_norm 2.1556 (2.5515) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][200/625] eta 0:02:41 lr 0.000334 wd 0.0500 time 0.3659 (0.3803) data time 0.0008 (0.0034) model time 0.3652 (0.3782) loss 6.1654 (6.0688) grad_norm 2.5794 (2.5581) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][210/625] eta 0:02:37 lr 0.000335 wd 0.0500 time 0.3723 (0.3801) data time 0.0008 (0.0033) model time 0.3715 (0.3780) loss 6.2274 (6.0663) grad_norm 2.1554 (2.5404) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][220/625] eta 0:02:33 lr 0.000337 wd 0.0500 time 0.3724 (0.3798) data time 0.0008 (0.0032) model time 0.3716 (0.3776) loss 6.2573 (6.0702) grad_norm 1.8542 (2.5333) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][230/625] eta 0:02:29 lr 0.000338 wd 0.0500 time 0.3774 (0.3796) data time 0.0009 (0.0031) model time 0.3765 (0.3774) loss 6.4173 (6.0682) grad_norm 2.0140 (2.5177) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][240/625] eta 0:02:26 lr 0.000340 wd 0.0500 time 0.3731 (0.3793) data time 0.0008 (0.0030) model time 0.3724 (0.3772) loss 6.3099 (6.0673) grad_norm 2.0775 (2.5113) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:41:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][250/625] eta 0:02:22 lr 0.000342 wd 0.0500 time 0.3719 (0.3791) data time 0.0010 (0.0029) model time 0.3709 (0.3769) loss 6.3779 (6.0646) grad_norm 3.8660 (2.5139) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][260/625] eta 0:02:18 lr 0.000343 wd 0.0500 time 0.3776 (0.3790) data time 0.0010 (0.0029) model time 0.3766 (0.3769) loss 6.4936 (6.0694) grad_norm 1.7911 (2.4954) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][270/625] eta 0:02:14 lr 0.000345 wd 0.0500 time 0.3709 (0.3788) data time 0.0010 (0.0028) model time 0.3698 (0.3766) loss 6.3715 (6.0677) grad_norm 2.6342 (2.4876) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][280/625] eta 0:02:10 lr 0.000346 wd 0.0500 time 0.3838 (0.3787) data time 0.0011 (0.0028) model time 0.3827 (0.3766) loss 6.1585 (6.0659) grad_norm 1.7524 (2.4734) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][290/625] eta 0:02:06 lr 0.000348 wd 0.0500 time 0.3709 (0.3786) data time 0.0009 (0.0027) model time 0.3700 (0.3765) loss 6.2964 (6.0669) grad_norm 2.0837 (2.4874) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][300/625] eta 0:02:03 lr 0.000350 wd 0.0500 time 0.3745 (0.3786) data time 0.0011 (0.0026) model time 0.3734 (0.3765) loss 6.1635 (6.0637) grad_norm 2.1173 (2.4849) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][310/625] eta 0:01:59 lr 0.000351 wd 0.0500 time 0.3730 (0.3785) data time 0.0008 (0.0026) model time 0.3722 (0.3764) loss 6.3043 (6.0648) grad_norm 2.1232 (2.4735) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][320/625] eta 0:01:55 lr 0.000353 wd 0.0500 time 0.3847 (0.3791) data time 0.0008 (0.0026) model time 0.3840 (0.3771) loss 5.6811 (6.0653) grad_norm 2.0596 (2.4644) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][330/625] eta 0:01:51 lr 0.000354 wd 0.0500 time 0.3691 (0.3790) data time 0.0011 (0.0025) model time 0.3681 (0.3771) loss 5.6986 (6.0596) grad_norm 1.8668 (2.4589) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][340/625] eta 0:01:48 lr 0.000356 wd 0.0500 time 0.3782 (0.3791) data time 0.0011 (0.0025) model time 0.3772 (0.3771) loss 5.9366 (6.0580) grad_norm 2.3216 (2.4621) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][350/625] eta 0:01:44 lr 0.000358 wd 0.0500 time 0.3720 (0.3790) data time 0.0008 (0.0025) model time 0.3712 (0.3771) loss 6.5307 (6.0533) grad_norm 2.4672 (2.4602) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][360/625] eta 0:01:40 lr 0.000359 wd 0.0500 time 0.3806 (0.3788) data time 0.0010 (0.0024) model time 0.3796 (0.3769) loss 6.2597 (6.0493) grad_norm 2.1984 (2.4560) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][370/625] eta 0:01:36 lr 0.000361 wd 0.0500 time 0.3748 (0.3788) data time 0.0010 (0.0024) model time 0.3738 (0.3769) loss 5.9720 (6.0469) grad_norm 1.8426 (2.4558) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][380/625] eta 0:01:32 lr 0.000362 wd 0.0500 time 0.3834 (0.3787) data time 0.0008 (0.0024) model time 0.3827 (0.3768) loss 6.2924 (6.0426) grad_norm 2.2973 (2.4516) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][390/625] eta 0:01:28 lr 0.000364 wd 0.0500 time 0.3792 (0.3786) data time 0.0011 (0.0024) model time 0.3782 (0.3768) loss 5.8326 (6.0407) grad_norm 2.6795 (2.4524) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][400/625] eta 0:01:25 lr 0.000366 wd 0.0500 time 0.3836 (0.3790) data time 0.0008 (0.0023) model time 0.3827 (0.3771) loss 5.9505 (6.0411) grad_norm 2.5159 (2.4495) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:42:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][410/625] eta 0:01:21 lr 0.000367 wd 0.0500 time 0.3768 (0.3788) data time 0.0015 (0.0023) model time 0.3753 (0.3770) loss 6.0632 (6.0380) grad_norm 1.8337 (2.4421) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][420/625] eta 0:01:17 lr 0.000369 wd 0.0500 time 0.3774 (0.3788) data time 0.0011 (0.0023) model time 0.3762 (0.3769) loss 6.0973 (6.0347) grad_norm 2.2682 (2.4341) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][430/625] eta 0:01:13 lr 0.000370 wd 0.0500 time 0.3791 (0.3787) data time 0.0011 (0.0023) model time 0.3780 (0.3769) loss 6.3335 (6.0334) grad_norm 1.8806 (2.4257) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][440/625] eta 0:01:10 lr 0.000372 wd 0.0500 time 0.3763 (0.3786) data time 0.0008 (0.0022) model time 0.3755 (0.3768) loss 6.0514 (6.0318) grad_norm 1.5296 (2.4216) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][450/625] eta 0:01:06 lr 0.000374 wd 0.0500 time 0.3751 (0.3785) data time 0.0012 (0.0022) model time 0.3740 (0.3767) loss 5.9983 (6.0345) grad_norm 1.9709 (2.4251) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][460/625] eta 0:01:02 lr 0.000375 wd 0.0500 time 0.3765 (0.3785) data time 0.0009 (0.0022) model time 0.3756 (0.3767) loss 6.4052 (6.0356) grad_norm 2.8983 (2.4253) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][470/625] eta 0:00:58 lr 0.000377 wd 0.0500 time 0.3708 (0.3785) data time 0.0010 (0.0022) model time 0.3698 (0.3767) loss 5.9159 (6.0328) grad_norm 2.4218 (2.4340) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][480/625] eta 0:00:54 lr 0.000378 wd 0.0500 time 0.3691 (0.3783) data time 0.0011 (0.0021) model time 0.3680 (0.3765) loss 5.8150 (6.0301) grad_norm 1.9115 (2.4305) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][490/625] eta 0:00:51 lr 0.000380 wd 0.0500 time 0.3734 (0.3783) data time 0.0008 (0.0022) model time 0.3726 (0.3765) loss 6.4274 (6.0268) grad_norm 3.0923 (2.4248) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][500/625] eta 0:00:47 lr 0.000382 wd 0.0500 time 0.3703 (0.3787) data time 0.0009 (0.0022) model time 0.3693 (0.3769) loss 6.1920 (6.0231) grad_norm 2.1982 (2.4205) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][510/625] eta 0:00:43 lr 0.000383 wd 0.0500 time 0.3669 (0.3786) data time 0.0008 (0.0021) model time 0.3661 (0.3768) loss 5.9699 (6.0211) grad_norm 2.2609 (2.4155) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][520/625] eta 0:00:39 lr 0.000385 wd 0.0500 time 0.3699 (0.3785) data time 0.0010 (0.0021) model time 0.3689 (0.3767) loss 6.2681 (6.0177) grad_norm 2.0220 (2.4127) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][530/625] eta 0:00:35 lr 0.000386 wd 0.0500 time 0.3715 (0.3784) data time 0.0008 (0.0021) model time 0.3706 (0.3766) loss 6.3552 (6.0191) grad_norm 1.9109 (2.4130) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][540/625] eta 0:00:32 lr 0.000388 wd 0.0500 time 0.3745 (0.3784) data time 0.0008 (0.0021) model time 0.3737 (0.3766) loss 5.2165 (6.0152) grad_norm 1.8195 (2.4087) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][550/625] eta 0:00:28 lr 0.000390 wd 0.0500 time 0.3734 (0.3783) data time 0.0008 (0.0021) model time 0.3725 (0.3765) loss 5.0702 (6.0095) grad_norm 3.3283 (2.4053) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][560/625] eta 0:00:24 lr 0.000391 wd 0.0500 time 0.3672 (0.3782) data time 0.0011 (0.0020) model time 0.3661 (0.3765) loss 6.2304 (6.0112) grad_norm 2.1983 (2.4024) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:43:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][570/625] eta 0:00:20 lr 0.000393 wd 0.0500 time 0.3771 (0.3781) data time 0.0010 (0.0020) model time 0.3761 (0.3764) loss 5.6113 (6.0095) grad_norm 1.9831 (2.3898) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][580/625] eta 0:00:17 lr 0.000394 wd 0.0500 time 0.3762 (0.3783) data time 0.0009 (0.0020) model time 0.3752 (0.3766) loss 6.1611 (6.0115) grad_norm 1.7404 (2.3838) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][590/625] eta 0:00:13 lr 0.000396 wd 0.0500 time 0.3755 (0.3783) data time 0.0010 (0.0020) model time 0.3746 (0.3766) loss 5.0708 (6.0029) grad_norm 2.1283 (2.3763) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][600/625] eta 0:00:09 lr 0.000398 wd 0.0500 time 0.3697 (0.3782) data time 0.0008 (0.0020) model time 0.3689 (0.3765) loss 5.4028 (6.0020) grad_norm 1.6692 (2.3791) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][610/625] eta 0:00:05 lr 0.000399 wd 0.0500 time 0.3667 (0.3781) data time 0.0006 (0.0020) model time 0.3661 (0.3764) loss 6.4181 (6.0017) grad_norm 1.9371 (2.3764) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [3/300][620/625] eta 0:00:01 lr 0.000401 wd 0.0500 time 0.3713 (0.3782) data time 0.0008 (0.0020) model time 0.3705 (0.3766) loss 5.4490 (5.9997) grad_norm 1.6106 (2.3719) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:17 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 3 training takes 0:03:56 [2024-08-19 23:44:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-19 23:44:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-19 23:44:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.517 (0.517) Loss 3.6953 (3.6953) Acc@1 25.732 (25.732) Acc@5 52.344 (52.344) Mem 11221MB [2024-08-19 23:44:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.145) Loss 4.5586 (3.8150) Acc@1 15.234 (22.576) Acc@5 34.570 (47.989) Mem 11221MB [2024-08-19 23:44:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.121) Loss 4.4570 (4.0865) Acc@1 13.086 (20.087) Acc@5 33.984 (42.908) Mem 11221MB [2024-08-19 23:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 20.871 Acc@5 43.822 [2024-08-19 23:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 20.9% [2024-08-19 23:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 20.87% [2024-08-19 23:44:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-19 23:44:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-19 23:44:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.519 (0.519) Loss 6.9805 (6.9805) Acc@1 0.098 (0.098) Acc@5 0.830 (0.830) Mem 11221MB [2024-08-19 23:44:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 6.8945 (6.9840) Acc@1 0.146 (0.115) Acc@5 0.635 (0.528) Mem 11221MB [2024-08-19 23:44:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 6.9375 (6.9604) Acc@1 0.049 (0.130) Acc@5 0.488 (0.607) Mem 11221MB [2024-08-19 23:44:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.140 Acc@5 0.644 [2024-08-19 23:44:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.1% [2024-08-19 23:44:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.14% [2024-08-19 23:44:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-19 23:44:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-19 23:44:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][0/625] eta 0:07:38 lr 0.000402 wd 0.0500 time 0.7342 (0.7342) data time 0.3806 (0.3806) model time 0.0000 (0.0000) loss 6.2239 (6.2239) grad_norm 1.9886 (1.9886) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][10/625] eta 0:04:11 lr 0.000403 wd 0.0500 time 0.3795 (0.4083) data time 0.0008 (0.0357) model time 0.0000 (0.0000) loss 6.2707 (5.6999) grad_norm 1.5816 (1.8708) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][20/625] eta 0:03:57 lr 0.000405 wd 0.0500 time 0.3709 (0.3927) data time 0.0010 (0.0192) model time 0.0000 (0.0000) loss 5.7824 (5.8396) grad_norm 2.3039 (2.1022) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][30/625] eta 0:03:53 lr 0.000406 wd 0.0500 time 0.3762 (0.3918) data time 0.0011 (0.0134) model time 0.0000 (0.0000) loss 6.3496 (5.8809) grad_norm 2.6142 (2.2467) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][40/625] eta 0:03:47 lr 0.000408 wd 0.0500 time 0.3810 (0.3881) data time 0.0007 (0.0104) model time 0.0000 (0.0000) loss 5.5676 (5.8538) grad_norm 2.1620 (2.2608) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][50/625] eta 0:03:41 lr 0.000410 wd 0.0500 time 0.3735 (0.3858) data time 0.0011 (0.0086) model time 0.0000 (0.0000) loss 5.8947 (5.8703) grad_norm 2.0095 (2.2476) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][60/625] eta 0:03:36 lr 0.000411 wd 0.0500 time 0.3749 (0.3840) data time 0.0008 (0.0073) model time 0.3741 (0.3734) loss 6.3376 (5.9165) grad_norm 2.0962 (2.2306) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][70/625] eta 0:03:32 lr 0.000413 wd 0.0500 time 0.3796 (0.3824) data time 0.0008 (0.0065) model time 0.3788 (0.3725) loss 5.9662 (5.9275) grad_norm 1.8701 (2.2138) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:44:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][80/625] eta 0:03:27 lr 0.000414 wd 0.0500 time 0.3719 (0.3812) data time 0.0010 (0.0058) model time 0.3709 (0.3721) loss 5.7326 (5.9283) grad_norm 1.8907 (2.1904) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][90/625] eta 0:03:23 lr 0.000416 wd 0.0500 time 0.4074 (0.3807) data time 0.0008 (0.0053) model time 0.4065 (0.3729) loss 5.9921 (5.9103) grad_norm 2.0039 (2.1640) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][100/625] eta 0:03:19 lr 0.000418 wd 0.0500 time 0.3660 (0.3798) data time 0.0008 (0.0049) model time 0.3652 (0.3724) loss 5.8743 (5.9035) grad_norm 2.4131 (2.1525) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][110/625] eta 0:03:15 lr 0.000419 wd 0.0500 time 0.3754 (0.3791) data time 0.0008 (0.0046) model time 0.3746 (0.3723) loss 5.9157 (5.8803) grad_norm 2.1113 (2.1608) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][120/625] eta 0:03:11 lr 0.000421 wd 0.0500 time 0.3746 (0.3787) data time 0.0008 (0.0043) model time 0.3738 (0.3723) loss 5.4397 (5.8790) grad_norm 2.0253 (2.1689) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][130/625] eta 0:03:07 lr 0.000422 wd 0.0500 time 0.3707 (0.3786) data time 0.0011 (0.0041) model time 0.3696 (0.3729) loss 6.0465 (5.8680) grad_norm 2.5295 (2.1650) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][140/625] eta 0:03:03 lr 0.000424 wd 0.0500 time 0.3846 (0.3785) data time 0.0009 (0.0039) model time 0.3837 (0.3731) loss 6.4064 (5.8753) grad_norm 2.1204 (2.1751) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][150/625] eta 0:02:59 lr 0.000426 wd 0.0500 time 0.3935 (0.3784) data time 0.0011 (0.0037) model time 0.3924 (0.3735) loss 5.2943 (5.8755) grad_norm 1.7337 (2.1690) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][160/625] eta 0:02:55 lr 0.000427 wd 0.0500 time 0.3760 (0.3783) data time 0.0011 (0.0035) model time 0.3749 (0.3735) loss 5.5300 (5.8767) grad_norm 2.1008 (2.1732) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][170/625] eta 0:02:52 lr 0.000429 wd 0.0500 time 0.3805 (0.3783) data time 0.0012 (0.0034) model time 0.3793 (0.3739) loss 6.1194 (5.8784) grad_norm 2.0299 (2.1767) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][180/625] eta 0:02:48 lr 0.000430 wd 0.0500 time 0.3769 (0.3781) data time 0.0008 (0.0033) model time 0.3761 (0.3739) loss 5.9762 (5.8852) grad_norm 2.2531 (2.1902) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][190/625] eta 0:02:44 lr 0.000432 wd 0.0500 time 0.3767 (0.3779) data time 0.0010 (0.0032) model time 0.3757 (0.3738) loss 5.9629 (5.8811) grad_norm 2.4580 (2.1958) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][200/625] eta 0:02:41 lr 0.000434 wd 0.0500 time 0.3757 (0.3798) data time 0.0011 (0.0031) model time 0.3745 (0.3765) loss 5.8634 (5.8779) grad_norm 1.8399 (2.1818) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][210/625] eta 0:02:37 lr 0.000435 wd 0.0500 time 0.3709 (0.3795) data time 0.0011 (0.0030) model time 0.3697 (0.3763) loss 5.5850 (5.8821) grad_norm 3.3095 (2.1808) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][220/625] eta 0:02:33 lr 0.000437 wd 0.0500 time 0.3715 (0.3792) data time 0.0011 (0.0029) model time 0.3704 (0.3760) loss 5.7163 (5.8784) grad_norm 2.0625 (2.1708) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][230/625] eta 0:02:29 lr 0.000438 wd 0.0500 time 0.3737 (0.3790) data time 0.0010 (0.0028) model time 0.3726 (0.3758) loss 5.8544 (5.8808) grad_norm 1.7100 (2.1762) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:45:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][240/625] eta 0:02:26 lr 0.000440 wd 0.0500 time 0.3797 (0.3796) data time 0.0011 (0.0027) model time 0.3787 (0.3767) loss 5.1768 (5.8653) grad_norm 3.8626 (2.1782) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][250/625] eta 0:02:22 lr 0.000442 wd 0.0500 time 0.3691 (0.3794) data time 0.0010 (0.0027) model time 0.3681 (0.3766) loss 5.8790 (5.8643) grad_norm 2.6190 (2.1980) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][260/625] eta 0:02:18 lr 0.000443 wd 0.0500 time 0.3702 (0.3792) data time 0.0012 (0.0026) model time 0.3691 (0.3764) loss 5.4281 (5.8551) grad_norm 2.9338 (2.2031) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][270/625] eta 0:02:14 lr 0.000445 wd 0.0500 time 0.3824 (0.3790) data time 0.0010 (0.0026) model time 0.3814 (0.3762) loss 5.9396 (5.8618) grad_norm 2.3617 (2.1986) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][280/625] eta 0:02:10 lr 0.000446 wd 0.0500 time 0.3761 (0.3788) data time 0.0011 (0.0025) model time 0.3750 (0.3761) loss 6.1803 (5.8559) grad_norm 2.1148 (2.1983) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][290/625] eta 0:02:06 lr 0.000448 wd 0.0500 time 0.3650 (0.3787) data time 0.0012 (0.0025) model time 0.3638 (0.3760) loss 5.9928 (5.8480) grad_norm 1.7032 (2.1857) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][300/625] eta 0:02:02 lr 0.000450 wd 0.0500 time 0.3698 (0.3785) data time 0.0010 (0.0024) model time 0.3688 (0.3758) loss 5.9974 (5.8431) grad_norm 3.0178 (2.1857) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][310/625] eta 0:01:59 lr 0.000451 wd 0.0500 time 0.3775 (0.3783) data time 0.0008 (0.0024) model time 0.3767 (0.3757) loss 6.0338 (5.8426) grad_norm 1.9165 (2.1824) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][320/625] eta 0:01:55 lr 0.000453 wd 0.0500 time 0.3747 (0.3782) data time 0.0009 (0.0024) model time 0.3738 (0.3756) loss 5.5262 (5.8394) grad_norm 2.3437 (2.1758) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][330/625] eta 0:01:51 lr 0.000454 wd 0.0500 time 0.3765 (0.3781) data time 0.0007 (0.0023) model time 0.3758 (0.3755) loss 5.2641 (5.8301) grad_norm 3.0853 (2.1795) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][340/625] eta 0:01:47 lr 0.000456 wd 0.0500 time 0.3748 (0.3780) data time 0.0011 (0.0023) model time 0.3737 (0.3754) loss 5.0984 (5.8252) grad_norm 2.6353 (2.1896) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][350/625] eta 0:01:43 lr 0.000458 wd 0.0500 time 0.3785 (0.3779) data time 0.0011 (0.0023) model time 0.3774 (0.3754) loss 5.3894 (5.8147) grad_norm 1.5686 (2.1920) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][360/625] eta 0:01:40 lr 0.000459 wd 0.0500 time 0.3682 (0.3778) data time 0.0008 (0.0022) model time 0.3673 (0.3752) loss 4.8013 (5.7990) grad_norm 1.9015 (2.1838) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][370/625] eta 0:01:36 lr 0.000461 wd 0.0500 time 0.3679 (0.3776) data time 0.0007 (0.0022) model time 0.3671 (0.3752) loss 5.3914 (5.7947) grad_norm 1.8004 (2.1833) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][380/625] eta 0:01:32 lr 0.000462 wd 0.0500 time 0.5538 (0.3780) data time 0.0010 (0.0022) model time 0.5528 (0.3756) loss 5.6617 (5.7908) grad_norm 1.4213 (2.1765) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][390/625] eta 0:01:28 lr 0.000464 wd 0.0500 time 0.3901 (0.3785) data time 0.0012 (0.0021) model time 0.3890 (0.3762) loss 6.0577 (5.7827) grad_norm 1.5846 (2.1715) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:46:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][400/625] eta 0:01:25 lr 0.000466 wd 0.0500 time 0.3790 (0.3784) data time 0.0011 (0.0021) model time 0.3779 (0.3762) loss 5.4957 (5.7834) grad_norm 1.9545 (2.1618) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][410/625] eta 0:01:21 lr 0.000467 wd 0.0500 time 0.3771 (0.3783) data time 0.0012 (0.0021) model time 0.3759 (0.3761) loss 5.9286 (5.7760) grad_norm 1.9819 (2.1530) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][420/625] eta 0:01:17 lr 0.000469 wd 0.0500 time 0.3794 (0.3786) data time 0.0009 (0.0021) model time 0.3786 (0.3764) loss 5.5435 (5.7753) grad_norm 2.5210 (2.1575) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][430/625] eta 0:01:13 lr 0.000470 wd 0.0500 time 0.3839 (0.3786) data time 0.0009 (0.0021) model time 0.3830 (0.3764) loss 5.7856 (5.7775) grad_norm 2.1102 (2.1635) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][440/625] eta 0:01:10 lr 0.000472 wd 0.0500 time 0.3719 (0.3785) data time 0.0008 (0.0020) model time 0.3711 (0.3764) loss 5.8003 (5.7804) grad_norm 1.8323 (2.1606) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][450/625] eta 0:01:06 lr 0.000474 wd 0.0500 time 0.3755 (0.3784) data time 0.0008 (0.0020) model time 0.3746 (0.3763) loss 5.5966 (5.7766) grad_norm 2.6756 (2.1571) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][460/625] eta 0:01:02 lr 0.000475 wd 0.0500 time 0.3707 (0.3783) data time 0.0011 (0.0020) model time 0.3696 (0.3762) loss 5.5066 (5.7742) grad_norm 2.1110 (2.1549) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][470/625] eta 0:00:58 lr 0.000477 wd 0.0500 time 0.3665 (0.3782) data time 0.0011 (0.0020) model time 0.3654 (0.3761) loss 5.9289 (5.7741) grad_norm 2.2532 (2.1541) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][480/625] eta 0:00:54 lr 0.000478 wd 0.0500 time 0.3688 (0.3781) data time 0.0009 (0.0020) model time 0.3680 (0.3760) loss 5.0895 (5.7733) grad_norm 1.6605 (2.1527) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][490/625] eta 0:00:51 lr 0.000480 wd 0.0500 time 0.3737 (0.3781) data time 0.0011 (0.0019) model time 0.3726 (0.3760) loss 6.0518 (5.7718) grad_norm 1.9283 (2.1475) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][500/625] eta 0:00:47 lr 0.000482 wd 0.0500 time 0.5937 (0.3788) data time 0.0009 (0.0019) model time 0.5928 (0.3768) loss 5.4390 (5.7694) grad_norm 1.9592 (2.1471) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][510/625] eta 0:00:43 lr 0.000483 wd 0.0500 time 0.3730 (0.3787) data time 0.0009 (0.0019) model time 0.3720 (0.3767) loss 6.0614 (5.7665) grad_norm 2.3241 (2.1525) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][520/625] eta 0:00:39 lr 0.000485 wd 0.0500 time 0.3703 (0.3786) data time 0.0009 (0.0019) model time 0.3694 (0.3766) loss 5.7536 (5.7650) grad_norm 1.9474 (2.1537) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][530/625] eta 0:00:35 lr 0.000486 wd 0.0500 time 0.3695 (0.3785) data time 0.0010 (0.0019) model time 0.3685 (0.3766) loss 5.6450 (5.7584) grad_norm 1.9148 (2.1504) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][540/625] eta 0:00:32 lr 0.000488 wd 0.0500 time 0.3692 (0.3788) data time 0.0009 (0.0019) model time 0.3683 (0.3769) loss 4.7837 (5.7584) grad_norm 1.7753 (2.1498) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][550/625] eta 0:00:28 lr 0.000490 wd 0.0500 time 0.3695 (0.3788) data time 0.0011 (0.0019) model time 0.3683 (0.3769) loss 6.1385 (5.7577) grad_norm 1.9811 (2.1457) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][560/625] eta 0:00:24 lr 0.000491 wd 0.0500 time 0.3727 (0.3787) data time 0.0009 (0.0019) model time 0.3717 (0.3768) loss 6.0338 (5.7548) grad_norm 2.0074 (2.1458) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][570/625] eta 0:00:20 lr 0.000493 wd 0.0500 time 0.3784 (0.3786) data time 0.0010 (0.0018) model time 0.3774 (0.3767) loss 5.6315 (5.7514) grad_norm 3.3204 (2.1457) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][580/625] eta 0:00:17 lr 0.000494 wd 0.0500 time 0.3829 (0.3785) data time 0.0007 (0.0018) model time 0.3822 (0.3766) loss 5.2271 (5.7476) grad_norm 1.8478 (2.1439) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][590/625] eta 0:00:13 lr 0.000496 wd 0.0500 time 0.3810 (0.3784) data time 0.0011 (0.0019) model time 0.3799 (0.3765) loss 5.6366 (5.7446) grad_norm 1.8175 (2.1399) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][600/625] eta 0:00:09 lr 0.000498 wd 0.0500 time 0.3742 (0.3784) data time 0.0008 (0.0018) model time 0.3733 (0.3765) loss 5.8694 (5.7458) grad_norm 1.8221 (2.1392) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][610/625] eta 0:00:05 lr 0.000499 wd 0.0500 time 0.3599 (0.3783) data time 0.0008 (0.0018) model time 0.3592 (0.3764) loss 5.6678 (5.7439) grad_norm 1.8317 (2.1396) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [4/300][620/625] eta 0:00:01 lr 0.000501 wd 0.0500 time 0.3671 (0.3782) data time 0.0005 (0.0018) model time 0.3665 (0.3762) loss 5.6469 (5.7427) grad_norm 1.8532 (2.1398) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 4 training takes 0:03:56 [2024-08-19 23:48:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-19 23:48:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-19 23:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.523 (0.523) Loss 2.8789 (2.8789) Acc@1 38.623 (38.623) Acc@5 66.309 (66.309) Mem 11221MB [2024-08-19 23:48:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.147) Loss 3.8477 (3.1898) Acc@1 24.805 (31.982) Acc@5 47.461 (59.970) Mem 11221MB [2024-08-19 23:48:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.097 (0.123) Loss 4.0039 (3.4962) Acc@1 18.701 (28.353) Acc@5 42.920 (54.120) Mem 11221MB [2024-08-19 23:48:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 29.059 Acc@5 54.696 [2024-08-19 23:48:27 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 29.1% [2024-08-19 23:48:27 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 29.06% [2024-08-19 23:48:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-19 23:48:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-19 23:48:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 6.9688 (6.9688) Acc@1 0.049 (0.049) Acc@5 0.879 (0.879) Mem 11221MB [2024-08-19 23:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 6.9453 (7.0227) Acc@1 0.146 (0.062) Acc@5 0.391 (0.488) Mem 11221MB [2024-08-19 23:48:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 6.9688 (6.9927) Acc@1 0.244 (0.107) Acc@5 1.074 (0.644) Mem 11221MB [2024-08-19 23:48:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.130 Acc@5 0.722 [2024-08-19 23:48:31 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.1% [2024-08-19 23:48:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][0/625] eta 0:12:39 lr 0.000502 wd 0.0500 time 1.2155 (1.2155) data time 0.5989 (0.5989) model time 0.0000 (0.0000) loss 4.8980 (4.8980) grad_norm 1.7677 (1.7677) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][10/625] eta 0:04:37 lr 0.000503 wd 0.0500 time 0.3705 (0.4509) data time 0.0010 (0.0556) model time 0.0000 (0.0000) loss 5.6833 (5.5280) grad_norm 2.9057 (2.0658) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][20/625] eta 0:04:10 lr 0.000505 wd 0.0500 time 0.3731 (0.4143) data time 0.0009 (0.0296) model time 0.0000 (0.0000) loss 5.4740 (5.4372) grad_norm 1.9208 (2.0880) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][30/625] eta 0:03:58 lr 0.000506 wd 0.0500 time 0.3751 (0.4007) data time 0.0008 (0.0204) model time 0.0000 (0.0000) loss 6.1176 (5.5172) grad_norm 1.4778 (1.9467) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][40/625] eta 0:03:50 lr 0.000508 wd 0.0500 time 0.3727 (0.3938) data time 0.0009 (0.0158) model time 0.0000 (0.0000) loss 5.4409 (5.5345) grad_norm 2.3149 (1.9924) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][50/625] eta 0:03:46 lr 0.000509 wd 0.0500 time 0.3716 (0.3945) data time 0.0009 (0.0129) model time 0.0000 (0.0000) loss 5.5499 (5.5173) grad_norm 2.4681 (2.0035) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][60/625] eta 0:03:42 lr 0.000511 wd 0.0500 time 0.3698 (0.3934) data time 0.0009 (0.0110) model time 0.3689 (0.3864) loss 6.4821 (5.5439) grad_norm 1.9042 (2.0595) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:48:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][70/625] eta 0:03:36 lr 0.000513 wd 0.0500 time 0.3745 (0.3908) data time 0.0008 (0.0096) model time 0.3737 (0.3802) loss 5.1683 (5.5317) grad_norm 1.5318 (2.0646) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][80/625] eta 0:03:31 lr 0.000514 wd 0.0500 time 0.3740 (0.3889) data time 0.0008 (0.0086) model time 0.3731 (0.3783) loss 6.0354 (5.5640) grad_norm 1.8418 (2.0807) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][90/625] eta 0:03:27 lr 0.000516 wd 0.0500 time 0.3785 (0.3874) data time 0.0010 (0.0078) model time 0.3775 (0.3771) loss 5.0059 (5.5563) grad_norm 1.7803 (2.0508) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][100/625] eta 0:03:22 lr 0.000517 wd 0.0500 time 0.3942 (0.3864) data time 0.0008 (0.0071) model time 0.3934 (0.3770) loss 6.2397 (5.5591) grad_norm 1.4908 (2.0401) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][110/625] eta 0:03:18 lr 0.000519 wd 0.0500 time 0.3697 (0.3853) data time 0.0008 (0.0066) model time 0.3689 (0.3763) loss 5.8337 (5.5886) grad_norm 1.8377 (2.0480) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][120/625] eta 0:03:14 lr 0.000521 wd 0.0500 time 0.3738 (0.3847) data time 0.0008 (0.0062) model time 0.3730 (0.3763) loss 4.7230 (5.5940) grad_norm 1.8781 (2.0375) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][130/625] eta 0:03:10 lr 0.000522 wd 0.0500 time 0.3784 (0.3840) data time 0.0010 (0.0058) model time 0.3774 (0.3761) loss 5.4975 (5.5992) grad_norm 1.8467 (2.0366) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][140/625] eta 0:03:05 lr 0.000524 wd 0.0500 time 0.3705 (0.3833) data time 0.0009 (0.0054) model time 0.3696 (0.3758) loss 5.6885 (5.5962) grad_norm 2.1122 (2.0221) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][150/625] eta 0:03:01 lr 0.000525 wd 0.0500 time 0.3774 (0.3828) data time 0.0008 (0.0052) model time 0.3766 (0.3757) loss 5.4374 (5.5850) grad_norm 2.5118 (2.0130) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][160/625] eta 0:02:57 lr 0.000527 wd 0.0500 time 0.3728 (0.3823) data time 0.0008 (0.0049) model time 0.3720 (0.3755) loss 5.7188 (5.5734) grad_norm 2.7237 (2.0077) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][170/625] eta 0:02:53 lr 0.000529 wd 0.0500 time 0.3665 (0.3820) data time 0.0008 (0.0047) model time 0.3658 (0.3755) loss 5.8171 (5.5829) grad_norm 1.5353 (2.0020) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][180/625] eta 0:02:49 lr 0.000530 wd 0.0500 time 0.3682 (0.3815) data time 0.0011 (0.0045) model time 0.3671 (0.3752) loss 6.0942 (5.5817) grad_norm 3.0131 (2.0141) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][190/625] eta 0:02:45 lr 0.000532 wd 0.0500 time 0.3762 (0.3811) data time 0.0008 (0.0043) model time 0.3754 (0.3750) loss 4.8589 (5.5834) grad_norm 3.2261 (2.0258) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][200/625] eta 0:02:41 lr 0.000533 wd 0.0500 time 0.3747 (0.3809) data time 0.0010 (0.0041) model time 0.3737 (0.3751) loss 5.0897 (5.5766) grad_norm 2.1606 (2.0404) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][210/625] eta 0:02:37 lr 0.000535 wd 0.0500 time 0.3725 (0.3805) data time 0.0011 (0.0040) model time 0.3714 (0.3749) loss 5.6007 (5.5696) grad_norm 1.8224 (2.0511) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][220/625] eta 0:02:33 lr 0.000537 wd 0.0500 time 0.3792 (0.3802) data time 0.0008 (0.0039) model time 0.3783 (0.3748) loss 4.9289 (5.5653) grad_norm 1.8642 (2.0620) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:49:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][230/625] eta 0:02:30 lr 0.000538 wd 0.0500 time 0.3699 (0.3800) data time 0.0012 (0.0038) model time 0.3688 (0.3747) loss 5.0852 (5.5591) grad_norm 1.6355 (2.0512) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][240/625] eta 0:02:26 lr 0.000540 wd 0.0500 time 0.3694 (0.3804) data time 0.0011 (0.0036) model time 0.3683 (0.3755) loss 5.9433 (5.5613) grad_norm 1.6662 (2.0465) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][250/625] eta 0:02:22 lr 0.000541 wd 0.0500 time 0.3744 (0.3802) data time 0.0010 (0.0035) model time 0.3734 (0.3754) loss 6.0070 (5.5629) grad_norm 1.6430 (2.0366) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][260/625] eta 0:02:19 lr 0.000543 wd 0.0500 time 0.3753 (0.3809) data time 0.0011 (0.0035) model time 0.3742 (0.3764) loss 6.2293 (5.5671) grad_norm 2.2167 (2.0293) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][270/625] eta 0:02:15 lr 0.000545 wd 0.0500 time 0.3669 (0.3806) data time 0.0011 (0.0034) model time 0.3658 (0.3763) loss 5.2155 (5.5683) grad_norm 2.0546 (2.0252) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][280/625] eta 0:02:11 lr 0.000546 wd 0.0500 time 0.3715 (0.3805) data time 0.0010 (0.0033) model time 0.3704 (0.3762) loss 5.8694 (5.5642) grad_norm 1.9690 (2.0251) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][290/625] eta 0:02:07 lr 0.000548 wd 0.0500 time 0.3786 (0.3804) data time 0.0014 (0.0032) model time 0.3772 (0.3763) loss 5.7594 (5.5658) grad_norm 2.0365 (2.0266) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][300/625] eta 0:02:03 lr 0.000549 wd 0.0500 time 0.3762 (0.3802) data time 0.0009 (0.0031) model time 0.3753 (0.3762) loss 5.1998 (5.5617) grad_norm 2.3817 (2.0245) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][310/625] eta 0:01:59 lr 0.000551 wd 0.0500 time 0.3704 (0.3800) data time 0.0009 (0.0031) model time 0.3695 (0.3760) loss 5.0391 (5.5537) grad_norm 1.6766 (2.0330) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][320/625] eta 0:01:55 lr 0.000553 wd 0.0500 time 0.3809 (0.3798) data time 0.0012 (0.0030) model time 0.3797 (0.3759) loss 5.4726 (5.5507) grad_norm 1.8583 (2.0374) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][330/625] eta 0:01:52 lr 0.000554 wd 0.0500 time 0.3773 (0.3798) data time 0.0008 (0.0030) model time 0.3764 (0.3760) loss 6.3766 (5.5506) grad_norm 2.0550 (2.0388) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][340/625] eta 0:01:48 lr 0.000556 wd 0.0500 time 0.3773 (0.3799) data time 0.0011 (0.0029) model time 0.3762 (0.3761) loss 4.8506 (5.5435) grad_norm 1.9160 (2.0415) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][350/625] eta 0:01:44 lr 0.000557 wd 0.0500 time 0.3776 (0.3797) data time 0.0011 (0.0029) model time 0.3765 (0.3761) loss 5.6653 (5.5399) grad_norm 2.1922 (2.0417) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][360/625] eta 0:01:40 lr 0.000559 wd 0.0500 time 0.3933 (0.3800) data time 0.0012 (0.0028) model time 0.3921 (0.3765) loss 5.2297 (5.5305) grad_norm 1.7268 (2.0385) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][370/625] eta 0:01:36 lr 0.000561 wd 0.0500 time 0.3747 (0.3800) data time 0.0011 (0.0028) model time 0.3737 (0.3765) loss 5.9813 (5.5295) grad_norm 1.6292 (2.0348) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:50:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][380/625] eta 0:01:33 lr 0.000562 wd 0.0500 time 0.3597 (0.3799) data time 0.0009 (0.0027) model time 0.3588 (0.3765) loss 6.2282 (5.5278) grad_norm 1.6313 (2.0317) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:51:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][390/625] eta 0:01:29 lr 0.000564 wd 0.0500 time 0.3680 (0.3797) data time 0.0008 (0.0027) model time 0.3671 (0.3763) loss 5.7858 (5.5259) grad_norm 1.6728 (2.0265) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-19 23:51:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][400/625] eta 0:01:25 lr 0.000565 wd 0.0500 time 0.3704 (0.3796) data time 0.0008 (0.0027) model time 0.3697 (0.3762) loss 4.9741 (5.5207) grad_norm 1.7983 (2.0309) loss_scale 32768.0000 (16710.8628) mem 11221MB [2024-08-19 23:51:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][410/625] eta 0:01:21 lr 0.000567 wd 0.0500 time 0.3741 (0.3795) data time 0.0009 (0.0026) model time 0.3732 (0.3762) loss 5.3248 (5.5196) grad_norm 2.0037 (2.0328) loss_scale 32768.0000 (17101.5474) mem 11221MB [2024-08-19 23:51:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][420/625] eta 0:01:17 lr 0.000569 wd 0.0500 time 0.3737 (0.3794) data time 0.0011 (0.0026) model time 0.3726 (0.3762) loss 5.2659 (5.5132) grad_norm 4.1655 (2.0384) loss_scale 32768.0000 (17473.6722) mem 11221MB [2024-08-19 23:51:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][430/625] eta 0:01:13 lr 0.000570 wd 0.0500 time 0.3816 (0.3793) data time 0.0010 (0.0025) model time 0.3806 (0.3761) loss 5.9883 (5.5135) grad_norm 1.9507 (2.0450) loss_scale 32768.0000 (17828.5290) mem 11221MB [2024-08-19 23:51:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][440/625] eta 0:01:10 lr 0.000572 wd 0.0500 time 0.3673 (0.3797) data time 0.0009 (0.0025) model time 0.3664 (0.3766) loss 4.5887 (5.5190) grad_norm 1.6622 (2.0422) loss_scale 32768.0000 (18167.2925) mem 11221MB [2024-08-19 23:51:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][450/625] eta 0:01:06 lr 0.000573 wd 0.0500 time 0.3741 (0.3795) data time 0.0010 (0.0025) model time 0.3731 (0.3765) loss 5.7011 (5.5192) grad_norm 1.8018 (2.0349) loss_scale 32768.0000 (18491.0333) mem 11221MB [2024-08-19 23:51:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][460/625] eta 0:01:02 lr 0.000575 wd 0.0500 time 0.3894 (0.3795) data time 0.0011 (0.0025) model time 0.3882 (0.3765) loss 5.7886 (5.5229) grad_norm 1.9600 (2.0289) loss_scale 32768.0000 (18800.7289) mem 11221MB [2024-08-19 23:51:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][470/625] eta 0:00:58 lr 0.000577 wd 0.0500 time 0.3799 (0.3794) data time 0.0008 (0.0024) model time 0.3791 (0.3764) loss 5.0925 (5.5227) grad_norm 1.7670 (2.0266) loss_scale 32768.0000 (19097.2739) mem 11221MB [2024-08-19 23:51:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][480/625] eta 0:00:54 lr 0.000578 wd 0.0500 time 0.3812 (0.3793) data time 0.0008 (0.0024) model time 0.3804 (0.3763) loss 5.5590 (5.5191) grad_norm 1.6647 (2.0275) loss_scale 32768.0000 (19381.4886) mem 11221MB [2024-08-19 23:51:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][490/625] eta 0:00:51 lr 0.000580 wd 0.0500 time 0.3793 (0.3792) data time 0.0011 (0.0024) model time 0.3782 (0.3762) loss 6.0260 (5.5232) grad_norm 5.0445 (2.0339) loss_scale 32768.0000 (19654.1263) mem 11221MB [2024-08-19 23:51:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][500/625] eta 0:00:47 lr 0.000581 wd 0.0500 time 0.3744 (0.3791) data time 0.0008 (0.0023) model time 0.3736 (0.3762) loss 6.0532 (5.5251) grad_norm 2.1889 (2.0389) loss_scale 32768.0000 (19915.8802) mem 11221MB [2024-08-19 23:51:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][510/625] eta 0:00:43 lr 0.000583 wd 0.0500 time 0.3658 (0.3790) data time 0.0012 (0.0023) model time 0.3647 (0.3761) loss 5.0119 (5.5270) grad_norm 1.6854 (2.0326) loss_scale 32768.0000 (20167.3894) mem 11221MB [2024-08-19 23:51:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][520/625] eta 0:00:39 lr 0.000585 wd 0.0500 time 0.3805 (0.3789) data time 0.0011 (0.0023) model time 0.3794 (0.3761) loss 5.5617 (5.5245) grad_norm 2.4105 (2.0317) loss_scale 32768.0000 (20409.2438) mem 11221MB [2024-08-19 23:51:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][530/625] eta 0:00:35 lr 0.000586 wd 0.0500 time 0.3733 (0.3789) data time 0.0013 (0.0023) model time 0.3720 (0.3761) loss 5.8594 (5.5230) grad_norm 1.9792 (2.0297) loss_scale 32768.0000 (20641.9887) mem 11221MB [2024-08-19 23:51:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][540/625] eta 0:00:32 lr 0.000588 wd 0.0500 time 0.3707 (0.3788) data time 0.0008 (0.0023) model time 0.3698 (0.3760) loss 5.7931 (5.5228) grad_norm 2.0484 (2.0316) loss_scale 32768.0000 (20866.1294) mem 11221MB [2024-08-19 23:52:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][550/625] eta 0:00:28 lr 0.000589 wd 0.0500 time 0.3735 (0.3787) data time 0.0008 (0.0022) model time 0.3726 (0.3760) loss 5.4763 (5.5238) grad_norm 1.9075 (2.0317) loss_scale 32768.0000 (21082.1343) mem 11221MB [2024-08-19 23:52:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][560/625] eta 0:00:24 lr 0.000591 wd 0.0500 time 0.3746 (0.3790) data time 0.0010 (0.0022) model time 0.3736 (0.3763) loss 4.9135 (5.5211) grad_norm 1.5326 (2.0312) loss_scale 32768.0000 (21290.4385) mem 11221MB [2024-08-19 23:52:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][570/625] eta 0:00:20 lr 0.000593 wd 0.0500 time 0.3698 (0.3792) data time 0.0009 (0.0022) model time 0.3689 (0.3766) loss 4.5968 (5.5203) grad_norm 2.2252 (2.0311) loss_scale 32768.0000 (21491.4466) mem 11221MB [2024-08-19 23:52:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][580/625] eta 0:00:17 lr 0.000594 wd 0.0500 time 0.3745 (0.3791) data time 0.0008 (0.0022) model time 0.3737 (0.3765) loss 5.0509 (5.5157) grad_norm 1.7782 (2.0353) loss_scale 32768.0000 (21685.5353) mem 11221MB [2024-08-19 23:52:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][590/625] eta 0:00:13 lr 0.000596 wd 0.0500 time 0.3776 (0.3791) data time 0.0011 (0.0022) model time 0.3766 (0.3765) loss 5.7511 (5.5172) grad_norm 1.9250 (2.0359) loss_scale 32768.0000 (21873.0558) mem 11221MB [2024-08-19 23:52:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][600/625] eta 0:00:09 lr 0.000597 wd 0.0500 time 0.3716 (0.3790) data time 0.0011 (0.0021) model time 0.3705 (0.3764) loss 5.5739 (5.5172) grad_norm 2.0473 (2.0332) loss_scale 32768.0000 (22054.3361) mem 11221MB [2024-08-19 23:52:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][610/625] eta 0:00:05 lr 0.000599 wd 0.0500 time 0.3566 (0.3789) data time 0.0008 (0.0021) model time 0.3558 (0.3763) loss 5.1029 (5.5164) grad_norm 1.9246 (2.0331) loss_scale 32768.0000 (22229.6825) mem 11221MB [2024-08-19 23:52:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [5/300][620/625] eta 0:00:01 lr 0.000601 wd 0.0500 time 0.3636 (0.3786) data time 0.0008 (0.0021) model time 0.3628 (0.3761) loss 4.5283 (5.5168) grad_norm 2.0513 (2.0287) loss_scale 32768.0000 (22399.3816) mem 11221MB [2024-08-19 23:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 5 training takes 0:03:56 [2024-08-19 23:52:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-19 23:52:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-19 23:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.512 (0.512) Loss 2.2695 (2.2695) Acc@1 50.732 (50.732) Acc@5 75.195 (75.195) Mem 11221MB [2024-08-19 23:52:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.146) Loss 3.4336 (2.6616) Acc@1 29.443 (40.674) Acc@5 55.518 (69.225) Mem 11221MB [2024-08-19 23:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.122) Loss 3.6855 (3.0324) Acc@1 23.926 (35.761) Acc@5 49.316 (62.277) Mem 11221MB [2024-08-19 23:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 36.358 Acc@5 62.806 [2024-08-19 23:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 36.4% [2024-08-19 23:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 36.36% [2024-08-19 23:52:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-19 23:52:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-19 23:52:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 7.0547 (7.0547) Acc@1 0.195 (0.195) Acc@5 1.074 (1.074) Mem 11221MB [2024-08-19 23:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 7.0156 (7.0621) Acc@1 0.244 (0.093) Acc@5 0.537 (0.555) Mem 11221MB [2024-08-19 23:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 6.9844 (7.0240) Acc@1 0.391 (0.137) Acc@5 0.928 (0.725) Mem 11221MB [2024-08-19 23:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.174 Acc@5 0.834 [2024-08-19 23:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.2% [2024-08-19 23:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.17% [2024-08-19 23:52:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-19 23:52:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-19 23:52:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][0/625] eta 0:08:27 lr 0.000601 wd 0.0500 time 0.8127 (0.8127) data time 0.4776 (0.4776) model time 0.0000 (0.0000) loss 4.9355 (4.9355) grad_norm 1.6215 (1.6215) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:52:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][10/625] eta 0:04:14 lr 0.000603 wd 0.0500 time 0.3739 (0.4142) data time 0.0008 (0.0445) model time 0.0000 (0.0000) loss 6.2040 (5.5200) grad_norm 1.6243 (2.0632) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:52:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][20/625] eta 0:03:59 lr 0.000605 wd 0.0500 time 0.3751 (0.3958) data time 0.0008 (0.0238) model time 0.0000 (0.0000) loss 5.6369 (5.4960) grad_norm 2.8257 (2.0074) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:52:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][30/625] eta 0:03:51 lr 0.000606 wd 0.0500 time 0.3767 (0.3891) data time 0.0010 (0.0165) model time 0.0000 (0.0000) loss 4.9641 (5.3695) grad_norm 1.4232 (1.9904) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:52:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][40/625] eta 0:03:45 lr 0.000608 wd 0.0500 time 0.3751 (0.3857) data time 0.0008 (0.0128) model time 0.0000 (0.0000) loss 4.7894 (5.4219) grad_norm 1.8575 (1.9756) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:52:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][50/625] eta 0:03:41 lr 0.000609 wd 0.0500 time 0.3766 (0.3861) data time 0.0010 (0.0105) model time 0.0000 (0.0000) loss 5.5648 (5.4402) grad_norm 2.6524 (2.0497) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][60/625] eta 0:03:36 lr 0.000611 wd 0.0500 time 0.3705 (0.3841) data time 0.0009 (0.0091) model time 0.3697 (0.3723) loss 5.5748 (5.4295) grad_norm 2.7374 (2.1297) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][70/625] eta 0:03:32 lr 0.000613 wd 0.0500 time 0.3715 (0.3831) data time 0.0012 (0.0079) model time 0.3703 (0.3741) loss 5.7895 (5.4114) grad_norm 1.5565 (2.1115) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][80/625] eta 0:03:29 lr 0.000614 wd 0.0500 time 0.5659 (0.3846) data time 0.0007 (0.0071) model time 0.5652 (0.3809) loss 4.5790 (5.4290) grad_norm 2.0694 (2.0804) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][90/625] eta 0:03:25 lr 0.000616 wd 0.0500 time 0.3771 (0.3836) data time 0.0009 (0.0064) model time 0.3762 (0.3793) loss 5.0925 (5.3960) grad_norm 2.0293 (2.0735) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][100/625] eta 0:03:20 lr 0.000617 wd 0.0500 time 0.3708 (0.3824) data time 0.0008 (0.0059) model time 0.3701 (0.3774) loss 6.0721 (5.4132) grad_norm 1.6874 (2.0832) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][110/625] eta 0:03:16 lr 0.000619 wd 0.0500 time 0.3609 (0.3815) data time 0.0008 (0.0055) model time 0.3601 (0.3764) loss 4.5910 (5.3879) grad_norm 2.2301 (2.0666) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][120/625] eta 0:03:12 lr 0.000621 wd 0.0500 time 0.3762 (0.3809) data time 0.0011 (0.0051) model time 0.3751 (0.3758) loss 5.9608 (5.3925) grad_norm 2.4988 (2.0744) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][130/625] eta 0:03:08 lr 0.000622 wd 0.0500 time 0.3750 (0.3803) data time 0.0008 (0.0048) model time 0.3741 (0.3754) loss 5.8664 (5.3777) grad_norm 2.1739 (2.1001) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][140/625] eta 0:03:04 lr 0.000624 wd 0.0500 time 0.3884 (0.3802) data time 0.0009 (0.0046) model time 0.3875 (0.3757) loss 5.7006 (5.3714) grad_norm 1.8298 (2.1074) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][150/625] eta 0:03:00 lr 0.000625 wd 0.0500 time 0.3777 (0.3799) data time 0.0008 (0.0043) model time 0.3768 (0.3755) loss 5.6061 (5.3594) grad_norm 2.1841 (2.0965) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][160/625] eta 0:02:56 lr 0.000627 wd 0.0500 time 0.3711 (0.3796) data time 0.0008 (0.0041) model time 0.3703 (0.3755) loss 5.5878 (5.3591) grad_norm 1.4825 (2.0807) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][170/625] eta 0:02:52 lr 0.000629 wd 0.0500 time 0.3653 (0.3801) data time 0.0008 (0.0041) model time 0.3645 (0.3762) loss 5.6979 (5.3597) grad_norm 1.6385 (2.0636) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][180/625] eta 0:02:48 lr 0.000630 wd 0.0500 time 0.3740 (0.3797) data time 0.0012 (0.0039) model time 0.3728 (0.3760) loss 5.0154 (5.3450) grad_norm 2.0511 (2.0538) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][190/625] eta 0:02:45 lr 0.000632 wd 0.0500 time 0.3673 (0.3796) data time 0.0008 (0.0037) model time 0.3664 (0.3759) loss 5.6649 (5.3436) grad_norm 2.2135 (2.0595) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][200/625] eta 0:02:41 lr 0.000633 wd 0.0500 time 0.3698 (0.3794) data time 0.0009 (0.0036) model time 0.3690 (0.3758) loss 5.4583 (5.3341) grad_norm 1.7573 (2.0551) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][210/625] eta 0:02:37 lr 0.000635 wd 0.0500 time 0.3801 (0.3791) data time 0.0010 (0.0035) model time 0.3790 (0.3756) loss 5.1540 (5.3356) grad_norm 1.9942 (2.0450) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][220/625] eta 0:02:33 lr 0.000637 wd 0.0500 time 0.3742 (0.3789) data time 0.0008 (0.0034) model time 0.3733 (0.3755) loss 5.8648 (5.3504) grad_norm 1.6895 (2.0426) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][230/625] eta 0:02:29 lr 0.000638 wd 0.0500 time 0.3705 (0.3787) data time 0.0009 (0.0033) model time 0.3696 (0.3754) loss 4.5743 (5.3462) grad_norm 1.7824 (2.0502) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][240/625] eta 0:02:25 lr 0.000640 wd 0.0500 time 0.3883 (0.3785) data time 0.0010 (0.0032) model time 0.3872 (0.3752) loss 5.6612 (5.3421) grad_norm 2.5383 (2.0587) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][250/625] eta 0:02:21 lr 0.000641 wd 0.0500 time 0.3818 (0.3784) data time 0.0007 (0.0031) model time 0.3810 (0.3753) loss 6.2322 (5.3411) grad_norm 1.9424 (2.0526) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][260/625] eta 0:02:18 lr 0.000643 wd 0.0500 time 0.3679 (0.3783) data time 0.0008 (0.0030) model time 0.3671 (0.3751) loss 5.5648 (5.3344) grad_norm 2.0537 (2.0501) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][270/625] eta 0:02:14 lr 0.000645 wd 0.0500 time 0.3684 (0.3780) data time 0.0010 (0.0030) model time 0.3674 (0.3749) loss 5.1830 (5.3346) grad_norm 1.6535 (2.0477) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][280/625] eta 0:02:10 lr 0.000646 wd 0.0500 time 0.3679 (0.3779) data time 0.0009 (0.0029) model time 0.3671 (0.3749) loss 5.4019 (5.3339) grad_norm 1.7506 (2.0437) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][290/625] eta 0:02:06 lr 0.000648 wd 0.0500 time 0.6166 (0.3786) data time 0.0010 (0.0028) model time 0.6156 (0.3758) loss 5.4891 (5.3423) grad_norm 2.1829 (2.0563) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][300/625] eta 0:02:02 lr 0.000649 wd 0.0500 time 0.3715 (0.3784) data time 0.0010 (0.0028) model time 0.3704 (0.3756) loss 5.2619 (5.3529) grad_norm 1.8892 (2.0559) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][310/625] eta 0:01:59 lr 0.000651 wd 0.0500 time 0.3663 (0.3782) data time 0.0010 (0.0027) model time 0.3653 (0.3754) loss 5.5482 (5.3557) grad_norm 1.7584 (2.0449) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][320/625] eta 0:01:55 lr 0.000653 wd 0.0500 time 0.3841 (0.3781) data time 0.0010 (0.0027) model time 0.3831 (0.3754) loss 5.1057 (5.3450) grad_norm 1.9262 (2.0339) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][330/625] eta 0:01:51 lr 0.000654 wd 0.0500 time 0.3709 (0.3779) data time 0.0011 (0.0026) model time 0.3698 (0.3752) loss 5.9838 (5.3421) grad_norm 1.6320 (2.0307) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][340/625] eta 0:01:47 lr 0.000656 wd 0.0500 time 0.3708 (0.3778) data time 0.0008 (0.0026) model time 0.3700 (0.3751) loss 5.7221 (5.3293) grad_norm 1.7569 (2.0277) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][350/625] eta 0:01:43 lr 0.000657 wd 0.0500 time 0.3791 (0.3777) data time 0.0008 (0.0026) model time 0.3783 (0.3751) loss 5.5195 (5.3254) grad_norm 1.8799 (2.0259) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][360/625] eta 0:01:40 lr 0.000659 wd 0.0500 time 0.3726 (0.3776) data time 0.0009 (0.0025) model time 0.3717 (0.3751) loss 5.6677 (5.3211) grad_norm 1.9754 (2.0265) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][370/625] eta 0:01:36 lr 0.000661 wd 0.0500 time 0.3788 (0.3775) data time 0.0008 (0.0025) model time 0.3780 (0.3749) loss 4.7173 (5.3146) grad_norm 2.1403 (2.0218) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][380/625] eta 0:01:32 lr 0.000662 wd 0.0500 time 0.3768 (0.3779) data time 0.0008 (0.0024) model time 0.3760 (0.3754) loss 5.2941 (5.3181) grad_norm 1.6455 (2.0215) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][390/625] eta 0:01:28 lr 0.000664 wd 0.0500 time 0.3694 (0.3778) data time 0.0011 (0.0024) model time 0.3683 (0.3754) loss 4.7631 (5.3172) grad_norm 2.5864 (2.0208) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][400/625] eta 0:01:24 lr 0.000665 wd 0.0500 time 0.3722 (0.3777) data time 0.0008 (0.0024) model time 0.3714 (0.3753) loss 5.5913 (5.3129) grad_norm 1.8256 (2.0185) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][410/625] eta 0:01:21 lr 0.000667 wd 0.0500 time 0.3783 (0.3776) data time 0.0011 (0.0023) model time 0.3772 (0.3752) loss 4.2154 (5.3087) grad_norm 1.5875 (2.0167) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][420/625] eta 0:01:17 lr 0.000669 wd 0.0500 time 0.3702 (0.3775) data time 0.0008 (0.0023) model time 0.3693 (0.3751) loss 5.8800 (5.3117) grad_norm 2.1440 (2.0146) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][430/625] eta 0:01:13 lr 0.000670 wd 0.0500 time 0.3791 (0.3774) data time 0.0010 (0.0023) model time 0.3781 (0.3751) loss 5.7248 (5.3141) grad_norm 1.5833 (2.0124) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][440/625] eta 0:01:09 lr 0.000672 wd 0.0500 time 0.3717 (0.3773) data time 0.0008 (0.0023) model time 0.3709 (0.3750) loss 5.5788 (5.3124) grad_norm 3.1983 (2.0225) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][450/625] eta 0:01:06 lr 0.000673 wd 0.0500 time 0.3778 (0.3773) data time 0.0008 (0.0022) model time 0.3770 (0.3749) loss 5.2128 (5.3075) grad_norm 1.5239 (2.0221) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][460/625] eta 0:01:02 lr 0.000675 wd 0.0500 time 0.3782 (0.3772) data time 0.0010 (0.0022) model time 0.3772 (0.3749) loss 5.7443 (5.3101) grad_norm 2.0692 (2.0148) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][470/625] eta 0:00:58 lr 0.000677 wd 0.0500 time 0.3702 (0.3771) data time 0.0008 (0.0022) model time 0.3694 (0.3749) loss 6.0847 (5.3120) grad_norm 1.5974 (2.0095) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][480/625] eta 0:00:54 lr 0.000678 wd 0.0500 time 0.3833 (0.3776) data time 0.0009 (0.0022) model time 0.3824 (0.3754) loss 4.9722 (5.3056) grad_norm 1.9883 (2.0081) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][490/625] eta 0:00:50 lr 0.000680 wd 0.0500 time 0.3687 (0.3775) data time 0.0010 (0.0021) model time 0.3677 (0.3754) loss 5.7224 (5.3059) grad_norm 1.7436 (2.0083) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][500/625] eta 0:00:47 lr 0.000681 wd 0.0500 time 0.3885 (0.3775) data time 0.0008 (0.0021) model time 0.3877 (0.3754) loss 5.7697 (5.3052) grad_norm 1.5871 (2.0070) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][510/625] eta 0:00:43 lr 0.000683 wd 0.0500 time 0.3751 (0.3775) data time 0.0010 (0.0021) model time 0.3741 (0.3754) loss 4.4227 (5.3037) grad_norm 2.0135 (2.0032) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][520/625] eta 0:00:39 lr 0.000685 wd 0.0500 time 0.3743 (0.3775) data time 0.0008 (0.0021) model time 0.3735 (0.3754) loss 4.8745 (5.2966) grad_norm 1.8725 (1.9976) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][530/625] eta 0:00:35 lr 0.000686 wd 0.0500 time 0.3653 (0.3774) data time 0.0011 (0.0021) model time 0.3642 (0.3753) loss 5.2301 (5.2967) grad_norm 2.6270 (1.9952) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][540/625] eta 0:00:32 lr 0.000688 wd 0.0500 time 0.3823 (0.3774) data time 0.0010 (0.0021) model time 0.3813 (0.3753) loss 5.7642 (5.2965) grad_norm 1.8201 (1.9971) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][550/625] eta 0:00:28 lr 0.000689 wd 0.0500 time 0.3666 (0.3773) data time 0.0008 (0.0020) model time 0.3658 (0.3752) loss 4.6276 (5.2929) grad_norm 2.0067 (1.9986) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][560/625] eta 0:00:24 lr 0.000691 wd 0.0500 time 0.3348 (0.3776) data time 0.0011 (0.0020) model time 0.3338 (0.3755) loss 5.6355 (5.2929) grad_norm 2.2026 (2.0013) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][570/625] eta 0:00:20 lr 0.000693 wd 0.0500 time 0.3732 (0.3775) data time 0.0009 (0.0020) model time 0.3723 (0.3755) loss 5.8050 (5.2948) grad_norm 2.0138 (1.9983) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][580/625] eta 0:00:16 lr 0.000694 wd 0.0500 time 0.3718 (0.3774) data time 0.0008 (0.0020) model time 0.3710 (0.3754) loss 5.4626 (5.2921) grad_norm 1.5747 (1.9985) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][590/625] eta 0:00:13 lr 0.000696 wd 0.0500 time 0.3746 (0.3774) data time 0.0011 (0.0020) model time 0.3736 (0.3754) loss 5.6263 (5.2900) grad_norm 1.9996 (1.9942) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][600/625] eta 0:00:09 lr 0.000697 wd 0.0500 time 0.3719 (0.3777) data time 0.0011 (0.0020) model time 0.3709 (0.3758) loss 5.5720 (5.2863) grad_norm 1.7471 (1.9884) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][610/625] eta 0:00:05 lr 0.000699 wd 0.0500 time 0.3557 (0.3776) data time 0.0005 (0.0020) model time 0.3551 (0.3757) loss 4.1859 (5.2876) grad_norm 1.7189 (1.9892) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [6/300][620/625] eta 0:00:01 lr 0.000701 wd 0.0500 time 0.3544 (0.3773) data time 0.0008 (0.0019) model time 0.3536 (0.3753) loss 5.7121 (5.2882) grad_norm 1.3131 (1.9856) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 6 training takes 0:03:55 [2024-08-19 23:56:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-19 23:56:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-19 23:56:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 2.0488 (2.0488) Acc@1 54.639 (54.639) Acc@5 79.248 (79.248) Mem 11221MB [2024-08-19 23:56:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 3.1895 (2.3512) Acc@1 33.447 (46.254) Acc@5 60.107 (74.529) Mem 11221MB [2024-08-19 23:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 3.3594 (2.7155) Acc@1 27.979 (40.846) Acc@5 55.713 (67.808) Mem 11221MB [2024-08-19 23:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 41.163 Acc@5 68.064 [2024-08-19 23:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 41.2% [2024-08-19 23:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 41.16% [2024-08-19 23:56:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-19 23:56:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-19 23:56:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.530 (0.530) Loss 7.1133 (7.1133) Acc@1 0.195 (0.195) Acc@5 1.025 (1.025) Mem 11221MB [2024-08-19 23:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 7.0273 (7.0788) Acc@1 0.146 (0.120) Acc@5 0.586 (0.635) Mem 11221MB [2024-08-19 23:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 6.9453 (7.0227) Acc@1 0.488 (0.200) Acc@5 1.758 (0.893) Mem 11221MB [2024-08-19 23:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.252 Acc@5 1.032 [2024-08-19 23:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.3% [2024-08-19 23:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.25% [2024-08-19 23:56:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-19 23:56:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-19 23:56:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][0/625] eta 0:08:09 lr 0.000701 wd 0.0500 time 0.7835 (0.7835) data time 0.4422 (0.4422) model time 0.0000 (0.0000) loss 5.2750 (5.2750) grad_norm 1.6631 (1.6631) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][10/625] eta 0:04:23 lr 0.000703 wd 0.0500 time 0.3363 (0.4286) data time 0.0008 (0.0412) model time 0.0000 (0.0000) loss 4.6568 (5.0498) grad_norm 1.2956 (1.6243) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][20/625] eta 0:04:03 lr 0.000704 wd 0.0500 time 0.3766 (0.4032) data time 0.0011 (0.0221) model time 0.0000 (0.0000) loss 5.6157 (5.1817) grad_norm 1.7463 (1.6881) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][30/625] eta 0:03:54 lr 0.000706 wd 0.0500 time 0.3726 (0.3945) data time 0.0008 (0.0154) model time 0.0000 (0.0000) loss 4.7119 (5.2554) grad_norm 2.3780 (1.8134) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:56:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][40/625] eta 0:03:48 lr 0.000708 wd 0.0500 time 0.3751 (0.3907) data time 0.0011 (0.0119) model time 0.0000 (0.0000) loss 4.6895 (5.2203) grad_norm 1.6852 (1.8516) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:57:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][50/625] eta 0:03:42 lr 0.000709 wd 0.0500 time 0.3736 (0.3875) data time 0.0008 (0.0098) model time 0.0000 (0.0000) loss 4.8767 (5.1942) grad_norm 1.7296 (1.8508) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:57:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][60/625] eta 0:03:37 lr 0.000711 wd 0.0500 time 0.3712 (0.3849) data time 0.0009 (0.0084) model time 0.3703 (0.3706) loss 4.5462 (5.1462) grad_norm 1.7118 (1.8601) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:57:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][70/625] eta 0:03:32 lr 0.000712 wd 0.0500 time 0.3672 (0.3830) data time 0.0011 (0.0074) model time 0.3661 (0.3704) loss 5.4938 (5.1825) grad_norm 1.7802 (1.8784) loss_scale 32768.0000 (32768.0000) mem 11221MB [2024-08-19 23:57:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][80/625] eta 0:03:27 lr 0.000714 wd 0.0500 time 0.3759 (0.3815) data time 0.0008 (0.0066) model time 0.3751 (0.3701) loss 4.4261 (5.1563) grad_norm 1.7210 (inf) loss_scale 16384.0000 (30947.5556) mem 11221MB [2024-08-19 23:57:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][90/625] eta 0:03:23 lr 0.000716 wd 0.0500 time 0.3689 (0.3803) data time 0.0009 (0.0060) model time 0.3680 (0.3700) loss 5.6314 (5.1739) grad_norm 2.5884 (inf) loss_scale 16384.0000 (29347.1648) mem 11221MB [2024-08-19 23:57:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][100/625] eta 0:03:20 lr 0.000717 wd 0.0500 time 0.3675 (0.3817) data time 0.0011 (0.0055) model time 0.3663 (0.3746) loss 4.3731 (5.1907) grad_norm 1.7709 (inf) loss_scale 16384.0000 (28063.6832) mem 11221MB [2024-08-19 23:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][110/625] eta 0:03:16 lr 0.000719 wd 0.0500 time 0.3833 (0.3812) data time 0.0008 (0.0051) model time 0.3825 (0.3747) loss 4.1014 (5.2052) grad_norm 2.3470 (inf) loss_scale 16384.0000 (27011.4595) mem 11221MB [2024-08-19 23:57:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][120/625] eta 0:03:12 lr 0.000720 wd 0.0500 time 0.3698 (0.3804) data time 0.0012 (0.0048) model time 0.3686 (0.3741) loss 5.5912 (5.1955) grad_norm 1.9951 (inf) loss_scale 16384.0000 (26133.1570) mem 11221MB [2024-08-19 23:57:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][130/625] eta 0:03:08 lr 0.000722 wd 0.0500 time 0.3819 (0.3800) data time 0.0010 (0.0045) model time 0.3809 (0.3741) loss 5.4287 (5.1980) grad_norm 1.7029 (inf) loss_scale 16384.0000 (25388.9466) mem 11221MB [2024-08-19 23:57:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][140/625] eta 0:03:04 lr 0.000724 wd 0.0500 time 0.3767 (0.3797) data time 0.0009 (0.0043) model time 0.3758 (0.3741) loss 4.8360 (5.1944) grad_norm 1.6514 (inf) loss_scale 16384.0000 (24750.2979) mem 11221MB [2024-08-19 23:57:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][150/625] eta 0:03:00 lr 0.000725 wd 0.0500 time 0.3643 (0.3791) data time 0.0009 (0.0041) model time 0.3635 (0.3736) loss 5.6068 (5.2028) grad_norm 1.3523 (inf) loss_scale 16384.0000 (24196.2384) mem 11221MB [2024-08-19 23:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][160/625] eta 0:02:56 lr 0.000727 wd 0.0500 time 0.3820 (0.3789) data time 0.0008 (0.0039) model time 0.3812 (0.3737) loss 4.2618 (5.2020) grad_norm 1.6323 (inf) loss_scale 16384.0000 (23711.0062) mem 11221MB [2024-08-19 23:57:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][170/625] eta 0:02:52 lr 0.000728 wd 0.0500 time 0.3906 (0.3787) data time 0.0010 (0.0037) model time 0.3896 (0.3738) loss 5.4821 (5.1989) grad_norm 1.5008 (inf) loss_scale 16384.0000 (23282.5263) mem 11221MB [2024-08-19 23:57:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][180/625] eta 0:02:48 lr 0.000730 wd 0.0500 time 0.3702 (0.3785) data time 0.0011 (0.0036) model time 0.3690 (0.3738) loss 5.0246 (5.2072) grad_norm 1.7209 (inf) loss_scale 16384.0000 (22901.3923) mem 11221MB [2024-08-19 23:57:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][190/625] eta 0:02:44 lr 0.000732 wd 0.0500 time 0.3713 (0.3784) data time 0.0011 (0.0035) model time 0.3702 (0.3739) loss 5.4672 (5.2058) grad_norm 1.7125 (inf) loss_scale 16384.0000 (22560.1675) mem 11221MB [2024-08-19 23:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][200/625] eta 0:02:40 lr 0.000733 wd 0.0500 time 0.3775 (0.3781) data time 0.0010 (0.0034) model time 0.3765 (0.3737) loss 5.2478 (5.1959) grad_norm 1.3634 (inf) loss_scale 16384.0000 (22252.8955) mem 11221MB [2024-08-19 23:58:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][210/625] eta 0:02:36 lr 0.000735 wd 0.0500 time 0.3711 (0.3778) data time 0.0010 (0.0033) model time 0.3701 (0.3736) loss 5.5163 (5.2054) grad_norm 1.9073 (inf) loss_scale 16384.0000 (21974.7488) mem 11221MB [2024-08-19 23:58:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][220/625] eta 0:02:33 lr 0.000736 wd 0.0500 time 0.3710 (0.3785) data time 0.0011 (0.0032) model time 0.3699 (0.3746) loss 5.2283 (5.2044) grad_norm 1.9504 (inf) loss_scale 16384.0000 (21721.7738) mem 11221MB [2024-08-19 23:58:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][230/625] eta 0:02:29 lr 0.000738 wd 0.0500 time 0.3703 (0.3783) data time 0.0009 (0.0031) model time 0.3694 (0.3745) loss 4.0916 (5.1976) grad_norm 2.7802 (inf) loss_scale 16384.0000 (21490.7013) mem 11221MB [2024-08-19 23:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][240/625] eta 0:02:25 lr 0.000740 wd 0.0500 time 0.3722 (0.3780) data time 0.0009 (0.0030) model time 0.3713 (0.3743) loss 4.6135 (5.1863) grad_norm 2.1102 (inf) loss_scale 16384.0000 (21278.8050) mem 11221MB [2024-08-19 23:58:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][250/625] eta 0:02:21 lr 0.000741 wd 0.0500 time 0.3711 (0.3779) data time 0.0011 (0.0029) model time 0.3700 (0.3743) loss 5.0534 (5.1871) grad_norm 1.4275 (inf) loss_scale 16384.0000 (21083.7928) mem 11221MB [2024-08-19 23:58:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][260/625] eta 0:02:17 lr 0.000743 wd 0.0500 time 0.3733 (0.3777) data time 0.0009 (0.0029) model time 0.3724 (0.3742) loss 5.3307 (5.1875) grad_norm 1.6757 (inf) loss_scale 16384.0000 (20903.7241) mem 11221MB [2024-08-19 23:58:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][270/625] eta 0:02:14 lr 0.000744 wd 0.0500 time 0.3655 (0.3776) data time 0.0009 (0.0028) model time 0.3646 (0.3742) loss 5.6745 (5.1908) grad_norm 1.6841 (inf) loss_scale 16384.0000 (20736.9446) mem 11221MB [2024-08-19 23:58:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][280/625] eta 0:02:10 lr 0.000746 wd 0.0500 time 0.3758 (0.3774) data time 0.0011 (0.0027) model time 0.3747 (0.3740) loss 5.1837 (5.1891) grad_norm 1.3151 (inf) loss_scale 16384.0000 (20582.0356) mem 11221MB [2024-08-19 23:58:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][290/625] eta 0:02:06 lr 0.000748 wd 0.0500 time 0.3737 (0.3773) data time 0.0008 (0.0027) model time 0.3729 (0.3740) loss 5.9599 (5.1921) grad_norm 1.9526 (inf) loss_scale 16384.0000 (20437.7732) mem 11221MB [2024-08-19 23:58:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][300/625] eta 0:02:02 lr 0.000749 wd 0.0500 time 0.3719 (0.3773) data time 0.0008 (0.0026) model time 0.3710 (0.3740) loss 4.0499 (5.1912) grad_norm 1.7311 (inf) loss_scale 16384.0000 (20303.0963) mem 11221MB [2024-08-19 23:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][310/625] eta 0:01:59 lr 0.000751 wd 0.0500 time 0.3824 (0.3780) data time 0.0010 (0.0026) model time 0.3814 (0.3750) loss 5.0077 (5.1917) grad_norm 2.0191 (inf) loss_scale 16384.0000 (20177.0804) mem 11221MB [2024-08-19 23:58:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][320/625] eta 0:01:55 lr 0.000752 wd 0.0500 time 0.3699 (0.3778) data time 0.0011 (0.0025) model time 0.3688 (0.3748) loss 5.3158 (5.1980) grad_norm 2.0943 (inf) loss_scale 16384.0000 (20058.9159) mem 11221MB [2024-08-19 23:58:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][330/625] eta 0:01:51 lr 0.000754 wd 0.0500 time 0.3741 (0.3777) data time 0.0010 (0.0025) model time 0.3732 (0.3748) loss 5.0212 (5.2074) grad_norm 1.3619 (inf) loss_scale 16384.0000 (19947.8912) mem 11221MB [2024-08-19 23:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][340/625] eta 0:01:47 lr 0.000756 wd 0.0500 time 0.3750 (0.3777) data time 0.0011 (0.0025) model time 0.3740 (0.3748) loss 5.7492 (5.2071) grad_norm 1.8959 (inf) loss_scale 16384.0000 (19843.3783) mem 11221MB [2024-08-19 23:58:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][350/625] eta 0:01:43 lr 0.000757 wd 0.0500 time 0.3712 (0.3776) data time 0.0008 (0.0024) model time 0.3704 (0.3747) loss 5.8909 (5.2092) grad_norm 1.5146 (inf) loss_scale 16384.0000 (19744.8205) mem 11221MB [2024-08-19 23:58:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][360/625] eta 0:01:40 lr 0.000759 wd 0.0500 time 0.3898 (0.3777) data time 0.0011 (0.0024) model time 0.3887 (0.3749) loss 5.4841 (5.2042) grad_norm 1.7178 (inf) loss_scale 16384.0000 (19651.7230) mem 11221MB [2024-08-19 23:59:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][370/625] eta 0:01:36 lr 0.000760 wd 0.0500 time 0.3859 (0.3777) data time 0.0011 (0.0024) model time 0.3848 (0.3749) loss 5.4903 (5.1988) grad_norm 1.7292 (inf) loss_scale 16384.0000 (19563.6442) mem 11221MB [2024-08-19 23:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][380/625] eta 0:01:32 lr 0.000762 wd 0.0500 time 0.3938 (0.3777) data time 0.0010 (0.0023) model time 0.3928 (0.3750) loss 4.1792 (5.1971) grad_norm 1.7862 (inf) loss_scale 16384.0000 (19480.1890) mem 11221MB [2024-08-19 23:59:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][390/625] eta 0:01:28 lr 0.000764 wd 0.0500 time 0.3659 (0.3777) data time 0.0012 (0.0023) model time 0.3647 (0.3750) loss 4.9731 (5.1973) grad_norm 2.4125 (inf) loss_scale 16384.0000 (19401.0026) mem 11221MB [2024-08-19 23:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][400/625] eta 0:01:24 lr 0.000765 wd 0.0500 time 0.3731 (0.3777) data time 0.0008 (0.0023) model time 0.3723 (0.3750) loss 5.8942 (5.1890) grad_norm 1.5122 (inf) loss_scale 16384.0000 (19325.7656) mem 11221MB [2024-08-19 23:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][410/625] eta 0:01:21 lr 0.000767 wd 0.0500 time 0.3723 (0.3781) data time 0.0014 (0.0023) model time 0.3709 (0.3755) loss 5.2654 (5.1879) grad_norm 1.4976 (inf) loss_scale 16384.0000 (19254.1898) mem 11221MB [2024-08-19 23:59:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][420/625] eta 0:01:17 lr 0.000768 wd 0.0500 time 0.3885 (0.3781) data time 0.0009 (0.0022) model time 0.3876 (0.3756) loss 3.9599 (5.1877) grad_norm 1.7782 (inf) loss_scale 16384.0000 (19186.0143) mem 11221MB [2024-08-19 23:59:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][430/625] eta 0:01:13 lr 0.000770 wd 0.0500 time 0.3795 (0.3781) data time 0.0008 (0.0022) model time 0.3787 (0.3756) loss 4.9146 (5.1915) grad_norm 2.0959 (inf) loss_scale 16384.0000 (19121.0023) mem 11221MB [2024-08-19 23:59:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][440/625] eta 0:01:09 lr 0.000772 wd 0.0500 time 0.3616 (0.3780) data time 0.0010 (0.0022) model time 0.3606 (0.3756) loss 5.7448 (5.1906) grad_norm 2.0889 (inf) loss_scale 16384.0000 (19058.9388) mem 11221MB [2024-08-19 23:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][450/625] eta 0:01:06 lr 0.000773 wd 0.0500 time 0.3774 (0.3780) data time 0.0010 (0.0022) model time 0.3764 (0.3755) loss 5.3759 (5.1816) grad_norm 1.7201 (inf) loss_scale 16384.0000 (18999.6275) mem 11221MB [2024-08-19 23:59:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][460/625] eta 0:01:02 lr 0.000775 wd 0.0500 time 0.3912 (0.3780) data time 0.0011 (0.0021) model time 0.3901 (0.3756) loss 5.3689 (5.1817) grad_norm 1.2238 (inf) loss_scale 16384.0000 (18942.8894) mem 11221MB [2024-08-19 23:59:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][470/625] eta 0:00:58 lr 0.000776 wd 0.0500 time 0.3667 (0.3779) data time 0.0008 (0.0021) model time 0.3659 (0.3756) loss 3.7711 (5.1815) grad_norm 2.6445 (inf) loss_scale 16384.0000 (18888.5605) mem 11221MB [2024-08-19 23:59:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][480/625] eta 0:00:54 lr 0.000778 wd 0.0500 time 0.3747 (0.3779) data time 0.0008 (0.0021) model time 0.3739 (0.3755) loss 4.3951 (5.1765) grad_norm 1.5428 (inf) loss_scale 16384.0000 (18836.4906) mem 11221MB [2024-08-19 23:59:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][490/625] eta 0:00:51 lr 0.000780 wd 0.0500 time 0.3896 (0.3779) data time 0.0010 (0.0021) model time 0.3886 (0.3756) loss 4.6422 (5.1662) grad_norm 1.2942 (inf) loss_scale 16384.0000 (18786.5418) mem 11221MB [2024-08-19 23:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][500/625] eta 0:00:47 lr 0.000781 wd 0.0500 time 0.3704 (0.3783) data time 0.0012 (0.0021) model time 0.3693 (0.3760) loss 4.9540 (5.1588) grad_norm 1.5058 (inf) loss_scale 16384.0000 (18738.5868) mem 11221MB [2024-08-19 23:59:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][510/625] eta 0:00:43 lr 0.000783 wd 0.0500 time 0.3739 (0.3782) data time 0.0011 (0.0021) model time 0.3728 (0.3759) loss 5.1820 (5.1571) grad_norm 2.1263 (inf) loss_scale 16384.0000 (18692.5088) mem 11221MB [2024-08-19 23:59:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][520/625] eta 0:00:39 lr 0.000784 wd 0.0500 time 0.6057 (0.3785) data time 0.0009 (0.0021) model time 0.6048 (0.3763) loss 4.3102 (5.1522) grad_norm 2.0835 (inf) loss_scale 16384.0000 (18648.1996) mem 11221MB [2024-08-20 00:00:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][530/625] eta 0:00:35 lr 0.000786 wd 0.0500 time 0.3636 (0.3784) data time 0.0008 (0.0020) model time 0.3628 (0.3762) loss 4.9541 (5.1501) grad_norm 2.2380 (inf) loss_scale 16384.0000 (18605.5593) mem 11221MB [2024-08-20 00:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][540/625] eta 0:00:32 lr 0.000788 wd 0.0500 time 0.3727 (0.3784) data time 0.0010 (0.0020) model time 0.3716 (0.3762) loss 5.4807 (5.1476) grad_norm 2.4630 (inf) loss_scale 16384.0000 (18564.4954) mem 11221MB [2024-08-20 00:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][550/625] eta 0:00:28 lr 0.000789 wd 0.0500 time 0.3781 (0.3783) data time 0.0008 (0.0020) model time 0.3774 (0.3762) loss 4.3864 (5.1421) grad_norm 1.4315 (inf) loss_scale 16384.0000 (18524.9220) mem 11221MB [2024-08-20 00:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][560/625] eta 0:00:24 lr 0.000791 wd 0.0500 time 0.3718 (0.3783) data time 0.0010 (0.0020) model time 0.3708 (0.3762) loss 5.8833 (5.1453) grad_norm 2.2742 (inf) loss_scale 16384.0000 (18486.7594) mem 11221MB [2024-08-20 00:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][570/625] eta 0:00:20 lr 0.000792 wd 0.0500 time 0.3670 (0.3783) data time 0.0008 (0.0020) model time 0.3662 (0.3761) loss 5.8199 (5.1473) grad_norm 1.6156 (inf) loss_scale 16384.0000 (18449.9335) mem 11221MB [2024-08-20 00:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][580/625] eta 0:00:17 lr 0.000794 wd 0.0500 time 0.3787 (0.3782) data time 0.0010 (0.0020) model time 0.3777 (0.3761) loss 5.0197 (5.1438) grad_norm 1.9468 (inf) loss_scale 16384.0000 (18414.3752) mem 11221MB [2024-08-20 00:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][590/625] eta 0:00:13 lr 0.000796 wd 0.0500 time 0.3840 (0.3782) data time 0.0010 (0.0020) model time 0.3830 (0.3761) loss 5.2125 (5.1416) grad_norm 1.5791 (inf) loss_scale 16384.0000 (18380.0203) mem 11221MB [2024-08-20 00:00:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][600/625] eta 0:00:09 lr 0.000797 wd 0.0500 time 0.3828 (0.3783) data time 0.0011 (0.0019) model time 0.3817 (0.3762) loss 5.1294 (5.1406) grad_norm 2.0174 (inf) loss_scale 16384.0000 (18346.8087) mem 11221MB [2024-08-20 00:00:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][610/625] eta 0:00:05 lr 0.000799 wd 0.0500 time 0.5948 (0.3785) data time 0.0005 (0.0019) model time 0.5942 (0.3765) loss 4.3996 (5.1442) grad_norm 1.7396 (inf) loss_scale 16384.0000 (18314.6841) mem 11221MB [2024-08-20 00:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [7/300][620/625] eta 0:00:01 lr 0.000800 wd 0.0500 time 0.3564 (0.3782) data time 0.0006 (0.0019) model time 0.3558 (0.3762) loss 5.9266 (5.1438) grad_norm 1.9386 (inf) loss_scale 16384.0000 (18283.5942) mem 11221MB [2024-08-20 00:00:39 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 7 training takes 0:03:56 [2024-08-20 00:00:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:00:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:00:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.623 (0.623) Loss 1.8486 (1.8486) Acc@1 58.887 (58.887) Acc@5 81.934 (81.934) Mem 11221MB [2024-08-20 00:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.153) Loss 2.8965 (2.0926) Acc@1 39.014 (50.950) Acc@5 65.723 (78.675) Mem 11221MB [2024-08-20 00:00:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.126) Loss 2.9766 (2.4705) Acc@1 35.840 (45.185) Acc@5 62.549 (72.040) Mem 11221MB [2024-08-20 00:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 45.429 Acc@5 72.139 [2024-08-20 00:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 45.4% [2024-08-20 00:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 45.43% [2024-08-20 00:00:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-20 00:00:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-20 00:00:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 7.0000 (7.0000) Acc@1 0.293 (0.293) Acc@5 1.855 (1.855) Mem 11221MB [2024-08-20 00:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 6.9766 (7.0163) Acc@1 0.195 (0.222) Acc@5 1.025 (1.065) Mem 11221MB [2024-08-20 00:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 6.8789 (6.9613) Acc@1 0.781 (0.300) Acc@5 2.100 (1.290) Mem 11221MB [2024-08-20 00:00:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.392 Acc@5 1.478 [2024-08-20 00:00:47 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.4% [2024-08-20 00:00:47 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.39% [2024-08-20 00:00:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-20 00:00:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-20 00:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][0/625] eta 0:08:01 lr 0.000801 wd 0.0500 time 0.7709 (0.7709) data time 0.4247 (0.4247) model time 0.0000 (0.0000) loss 5.1349 (5.1349) grad_norm 1.9644 (1.9644) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][10/625] eta 0:04:13 lr 0.000803 wd 0.0500 time 0.3861 (0.4124) data time 0.0010 (0.0396) model time 0.0000 (0.0000) loss 5.1004 (4.9795) grad_norm 1.3131 (1.6840) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][20/625] eta 0:04:03 lr 0.000804 wd 0.0500 time 0.3723 (0.4029) data time 0.0008 (0.0213) model time 0.0000 (0.0000) loss 3.9995 (5.0913) grad_norm 1.5198 (1.6620) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][30/625] eta 0:03:54 lr 0.000806 wd 0.0500 time 0.3824 (0.3947) data time 0.0008 (0.0148) model time 0.0000 (0.0000) loss 4.6895 (5.0204) grad_norm 1.5680 (1.6757) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][40/625] eta 0:03:48 lr 0.000808 wd 0.0500 time 0.3701 (0.3902) data time 0.0010 (0.0114) model time 0.0000 (0.0000) loss 5.3100 (5.0177) grad_norm 1.6954 (1.6804) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][50/625] eta 0:03:43 lr 0.000809 wd 0.0500 time 0.3822 (0.3886) data time 0.0008 (0.0094) model time 0.0000 (0.0000) loss 4.0288 (5.0262) grad_norm 2.3437 (1.7366) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][60/625] eta 0:03:38 lr 0.000811 wd 0.0500 time 0.3722 (0.3861) data time 0.0011 (0.0080) model time 0.3711 (0.3719) loss 5.5054 (5.0731) grad_norm 1.5556 (1.7386) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][70/625] eta 0:03:33 lr 0.000812 wd 0.0500 time 0.3816 (0.3847) data time 0.0008 (0.0071) model time 0.3807 (0.3735) loss 4.2043 (5.0228) grad_norm 1.4655 (1.7342) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][80/625] eta 0:03:29 lr 0.000814 wd 0.0500 time 0.3708 (0.3839) data time 0.0011 (0.0064) model time 0.3698 (0.3747) loss 5.4632 (5.0276) grad_norm 1.3788 (1.7296) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][90/625] eta 0:03:37 lr 0.000816 wd 0.0500 time 0.3833 (0.4061) data time 0.0011 (0.0058) model time 0.3822 (0.4272) loss 5.5456 (5.0234) grad_norm 2.4619 (1.7790) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][100/625] eta 0:03:31 lr 0.000817 wd 0.0500 time 0.3882 (0.4034) data time 0.0010 (0.0054) model time 0.3872 (0.4171) loss 4.7920 (5.0005) grad_norm 2.1590 (1.8074) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][110/625] eta 0:03:26 lr 0.000819 wd 0.0500 time 0.3708 (0.4008) data time 0.0008 (0.0051) model time 0.3700 (0.4098) loss 4.1138 (5.0075) grad_norm 1.5584 (1.8032) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][120/625] eta 0:03:21 lr 0.000820 wd 0.0500 time 0.3737 (0.3989) data time 0.0007 (0.0047) model time 0.3730 (0.4051) loss 4.9324 (4.9948) grad_norm 1.6859 (1.8016) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][130/625] eta 0:03:16 lr 0.000822 wd 0.0500 time 0.3825 (0.3972) data time 0.0010 (0.0045) model time 0.3814 (0.4014) loss 5.4399 (5.0096) grad_norm 1.5188 (1.7834) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][140/625] eta 0:03:11 lr 0.000824 wd 0.0500 time 0.3655 (0.3955) data time 0.0011 (0.0042) model time 0.3644 (0.3982) loss 5.5728 (5.0197) grad_norm 1.4072 (1.7772) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][150/625] eta 0:03:07 lr 0.000825 wd 0.0500 time 0.3817 (0.3943) data time 0.0011 (0.0040) model time 0.3807 (0.3959) loss 4.9069 (5.0200) grad_norm 1.7579 (1.7699) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][160/625] eta 0:03:02 lr 0.000827 wd 0.0500 time 0.3819 (0.3935) data time 0.0010 (0.0038) model time 0.3808 (0.3946) loss 5.2655 (5.0276) grad_norm 2.5267 (1.7771) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][170/625] eta 0:02:59 lr 0.000828 wd 0.0500 time 0.3655 (0.3937) data time 0.0009 (0.0037) model time 0.3646 (0.3946) loss 4.3510 (5.0255) grad_norm 3.3450 (1.7915) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:01:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][180/625] eta 0:02:54 lr 0.000830 wd 0.0500 time 0.3724 (0.3927) data time 0.0008 (0.0035) model time 0.3716 (0.3931) loss 5.5705 (5.0359) grad_norm 2.0654 (1.8000) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][190/625] eta 0:02:50 lr 0.000832 wd 0.0500 time 0.3817 (0.3920) data time 0.0011 (0.0034) model time 0.3806 (0.3920) loss 5.3722 (5.0217) grad_norm 1.4874 (1.7942) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][200/625] eta 0:02:46 lr 0.000833 wd 0.0500 time 0.3760 (0.3918) data time 0.0008 (0.0033) model time 0.3752 (0.3916) loss 5.6724 (5.0238) grad_norm 1.8154 (1.7882) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][210/625] eta 0:02:42 lr 0.000835 wd 0.0500 time 0.3862 (0.3912) data time 0.0010 (0.0032) model time 0.3852 (0.3908) loss 5.2747 (5.0370) grad_norm 1.9186 (1.8052) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][220/625] eta 0:02:38 lr 0.000836 wd 0.0500 time 0.3686 (0.3905) data time 0.0009 (0.0031) model time 0.3678 (0.3899) loss 4.9505 (5.0405) grad_norm 2.4515 (1.8301) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][230/625] eta 0:02:34 lr 0.000838 wd 0.0500 time 0.3778 (0.3900) data time 0.0011 (0.0030) model time 0.3766 (0.3892) loss 5.6165 (5.0317) grad_norm 2.0842 (1.8445) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][240/625] eta 0:02:30 lr 0.000840 wd 0.0500 time 0.3879 (0.3896) data time 0.0010 (0.0029) model time 0.3869 (0.3887) loss 5.5936 (5.0476) grad_norm 1.9173 (1.8433) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][250/625] eta 0:02:25 lr 0.000841 wd 0.0500 time 0.3764 (0.3891) data time 0.0008 (0.0029) model time 0.3755 (0.3880) loss 5.4264 (5.0490) grad_norm 2.2787 (1.8391) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][260/625] eta 0:02:21 lr 0.000843 wd 0.0500 time 0.3755 (0.3887) data time 0.0009 (0.0028) model time 0.3747 (0.3875) loss 5.6431 (5.0512) grad_norm 1.3512 (1.8423) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][270/625] eta 0:02:17 lr 0.000844 wd 0.0500 time 0.3875 (0.3883) data time 0.0010 (0.0027) model time 0.3865 (0.3870) loss 5.0219 (5.0490) grad_norm 1.8910 (1.8308) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][280/625] eta 0:02:13 lr 0.000846 wd 0.0500 time 0.3817 (0.3879) data time 0.0010 (0.0027) model time 0.3808 (0.3865) loss 5.5768 (5.0488) grad_norm 1.3186 (1.8254) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][290/625] eta 0:02:10 lr 0.000848 wd 0.0500 time 0.3856 (0.3888) data time 0.0012 (0.0026) model time 0.3844 (0.3876) loss 4.9237 (5.0506) grad_norm 1.7222 (1.8255) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][300/625] eta 0:02:06 lr 0.000849 wd 0.0500 time 0.3677 (0.3883) data time 0.0011 (0.0026) model time 0.3666 (0.3871) loss 5.4750 (5.0545) grad_norm 2.4204 (1.8331) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][310/625] eta 0:02:02 lr 0.000851 wd 0.0500 time 0.3809 (0.3880) data time 0.0011 (0.0025) model time 0.3798 (0.3867) loss 4.3898 (5.0524) grad_norm 1.4006 (1.8340) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][320/625] eta 0:01:58 lr 0.000852 wd 0.0500 time 0.3787 (0.3882) data time 0.0011 (0.0025) model time 0.3776 (0.3869) loss 4.3719 (5.0379) grad_norm 1.7665 (1.8302) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:02:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][330/625] eta 0:01:54 lr 0.000854 wd 0.0500 time 0.3770 (0.3878) data time 0.0008 (0.0025) model time 0.3762 (0.3865) loss 4.8003 (5.0402) grad_norm 2.1541 (1.8274) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][340/625] eta 0:01:50 lr 0.000856 wd 0.0500 time 0.3848 (0.3876) data time 0.0011 (0.0024) model time 0.3837 (0.3862) loss 5.5886 (5.0399) grad_norm 1.7122 (1.8216) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][350/625] eta 0:01:46 lr 0.000857 wd 0.0500 time 0.3664 (0.3871) data time 0.0010 (0.0024) model time 0.3654 (0.3857) loss 5.4228 (5.0438) grad_norm 1.4940 (1.8207) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][360/625] eta 0:01:42 lr 0.000859 wd 0.0500 time 0.3815 (0.3869) data time 0.0009 (0.0024) model time 0.3806 (0.3854) loss 4.2884 (5.0343) grad_norm 2.0080 (1.8160) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][370/625] eta 0:01:38 lr 0.000860 wd 0.0500 time 0.3818 (0.3866) data time 0.0011 (0.0023) model time 0.3807 (0.3851) loss 5.3105 (5.0288) grad_norm 1.5179 (1.8129) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][380/625] eta 0:01:34 lr 0.000862 wd 0.0500 time 0.3786 (0.3863) data time 0.0012 (0.0023) model time 0.3773 (0.3848) loss 4.5105 (5.0304) grad_norm 1.9204 (1.8135) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][390/625] eta 0:01:30 lr 0.000864 wd 0.0500 time 0.3728 (0.3862) data time 0.0011 (0.0023) model time 0.3718 (0.3846) loss 4.2582 (5.0252) grad_norm 1.4273 (1.8261) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][400/625] eta 0:01:26 lr 0.000865 wd 0.0500 time 0.3783 (0.3860) data time 0.0008 (0.0023) model time 0.3775 (0.3844) loss 5.3922 (5.0330) grad_norm 2.3044 (1.8251) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][410/625] eta 0:01:22 lr 0.000867 wd 0.0500 time 0.3778 (0.3858) data time 0.0012 (0.0022) model time 0.3766 (0.3842) loss 4.7265 (5.0303) grad_norm 1.4020 (1.8233) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][420/625] eta 0:01:19 lr 0.000868 wd 0.0500 time 0.3660 (0.3855) data time 0.0009 (0.0022) model time 0.3651 (0.3839) loss 4.1662 (5.0283) grad_norm 1.6233 (1.8219) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][430/625] eta 0:01:15 lr 0.000870 wd 0.0500 time 0.3844 (0.3854) data time 0.0008 (0.0022) model time 0.3836 (0.3838) loss 5.4272 (5.0223) grad_norm 1.5146 (1.8171) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][440/625] eta 0:01:11 lr 0.000872 wd 0.0500 time 0.3703 (0.3853) data time 0.0011 (0.0022) model time 0.3693 (0.3837) loss 5.5957 (5.0208) grad_norm 1.4022 (1.8165) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][450/625] eta 0:01:07 lr 0.000873 wd 0.0500 time 0.3808 (0.3852) data time 0.0008 (0.0022) model time 0.3800 (0.3836) loss 4.7049 (5.0253) grad_norm 2.0093 (1.8142) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][460/625] eta 0:01:03 lr 0.000875 wd 0.0500 time 0.3714 (0.3851) data time 0.0008 (0.0021) model time 0.3706 (0.3834) loss 5.4863 (5.0271) grad_norm 1.5807 (1.8109) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][470/625] eta 0:00:59 lr 0.000876 wd 0.0500 time 0.3652 (0.3850) data time 0.0011 (0.0021) model time 0.3641 (0.3833) loss 4.6498 (5.0295) grad_norm 1.5738 (1.8136) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][480/625] eta 0:00:55 lr 0.000878 wd 0.0500 time 0.3611 (0.3848) data time 0.0008 (0.0021) model time 0.3603 (0.3832) loss 5.0603 (5.0266) grad_norm 2.1236 (1.8117) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:03:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][490/625] eta 0:00:51 lr 0.000880 wd 0.0500 time 0.3887 (0.3847) data time 0.0010 (0.0021) model time 0.3877 (0.3830) loss 4.9490 (5.0215) grad_norm 1.5884 (1.8104) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][500/625] eta 0:00:48 lr 0.000881 wd 0.0500 time 0.3746 (0.3855) data time 0.0009 (0.0021) model time 0.3738 (0.3839) loss 4.8098 (5.0190) grad_norm 1.5310 (1.8052) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][510/625] eta 0:00:44 lr 0.000883 wd 0.0500 time 0.3791 (0.3854) data time 0.0008 (0.0020) model time 0.3783 (0.3838) loss 4.7518 (5.0193) grad_norm 1.9326 (1.8046) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][520/625] eta 0:00:40 lr 0.000884 wd 0.0500 time 0.3688 (0.3853) data time 0.0011 (0.0020) model time 0.3678 (0.3837) loss 5.2730 (5.0185) grad_norm 2.9294 (1.8090) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][530/625] eta 0:00:36 lr 0.000886 wd 0.0500 time 0.3736 (0.3855) data time 0.0011 (0.0020) model time 0.3725 (0.3839) loss 4.7808 (5.0176) grad_norm 2.4551 (1.8267) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][540/625] eta 0:00:32 lr 0.000888 wd 0.0500 time 0.3836 (0.3853) data time 0.0008 (0.0020) model time 0.3828 (0.3838) loss 5.0455 (5.0125) grad_norm 1.5708 (1.8230) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][550/625] eta 0:00:28 lr 0.000889 wd 0.0500 time 0.3865 (0.3852) data time 0.0011 (0.0020) model time 0.3854 (0.3837) loss 5.2460 (5.0077) grad_norm 1.4069 (1.8175) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][560/625] eta 0:00:25 lr 0.000891 wd 0.0500 time 0.3733 (0.3852) data time 0.0008 (0.0020) model time 0.3725 (0.3836) loss 4.2609 (5.0051) grad_norm 1.4056 (1.8138) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][570/625] eta 0:00:21 lr 0.000892 wd 0.0500 time 0.3678 (0.3851) data time 0.0008 (0.0020) model time 0.3670 (0.3835) loss 4.6695 (5.0091) grad_norm 1.3779 (1.8086) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][580/625] eta 0:00:17 lr 0.000894 wd 0.0500 time 0.3766 (0.3849) data time 0.0007 (0.0019) model time 0.3759 (0.3833) loss 5.7774 (5.0062) grad_norm 1.5223 (1.8038) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][590/625] eta 0:00:13 lr 0.000896 wd 0.0500 time 0.3772 (0.3848) data time 0.0008 (0.0019) model time 0.3764 (0.3832) loss 5.2926 (5.0066) grad_norm 2.0007 (1.8073) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][600/625] eta 0:00:09 lr 0.000897 wd 0.0500 time 0.3765 (0.3846) data time 0.0009 (0.0019) model time 0.3756 (0.3831) loss 3.8535 (5.0049) grad_norm 1.8730 (1.8091) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][610/625] eta 0:00:05 lr 0.000899 wd 0.0500 time 0.3684 (0.3845) data time 0.0007 (0.0019) model time 0.3677 (0.3829) loss 4.0684 (5.0017) grad_norm 1.4991 (1.8048) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [8/300][620/625] eta 0:00:01 lr 0.000900 wd 0.0500 time 0.3554 (0.3842) data time 0.0005 (0.0019) model time 0.3549 (0.3825) loss 4.8140 (4.9993) grad_norm 1.5943 (1.8040) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:04:48 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 8 training takes 0:04:00 [2024-08-20 00:04:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:04:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.530 (0.530) Loss 1.6416 (1.6416) Acc@1 62.988 (62.988) Acc@5 84.473 (84.473) Mem 11221MB [2024-08-20 00:04:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 2.6035 (1.8753) Acc@1 43.701 (55.611) Acc@5 70.459 (81.969) Mem 11221MB [2024-08-20 00:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.121) Loss 2.8887 (2.2508) Acc@1 39.014 (49.644) Acc@5 64.209 (75.451) Mem 11221MB [2024-08-20 00:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 49.832 Acc@5 75.590 [2024-08-20 00:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 49.8% [2024-08-20 00:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 49.83% [2024-08-20 00:04:52 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-20 00:04:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-20 00:04:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.522 (0.522) Loss 6.7578 (6.7578) Acc@1 0.391 (0.391) Acc@5 2.979 (2.979) Mem 11221MB [2024-08-20 00:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 6.8750 (6.8430) Acc@1 0.293 (0.462) Acc@5 1.270 (1.949) Mem 11221MB [2024-08-20 00:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.121) Loss 6.7812 (6.8265) Acc@1 0.781 (0.463) Acc@5 2.930 (2.081) Mem 11221MB [2024-08-20 00:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.562 Acc@5 2.281 [2024-08-20 00:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 0.6% [2024-08-20 00:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.56% [2024-08-20 00:04:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-20 00:04:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-20 00:04:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][0/625] eta 0:07:56 lr 0.000901 wd 0.0500 time 0.7628 (0.7628) data time 0.4129 (0.4129) model time 0.0000 (0.0000) loss 5.2437 (5.2437) grad_norm 2.9297 (2.9297) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][10/625] eta 0:04:23 lr 0.000903 wd 0.0500 time 0.3399 (0.4280) data time 0.0008 (0.0385) model time 0.0000 (0.0000) loss 5.4912 (5.3388) grad_norm 2.2610 (2.0982) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][20/625] eta 0:04:04 lr 0.000904 wd 0.0500 time 0.3787 (0.4039) data time 0.0010 (0.0207) model time 0.0000 (0.0000) loss 5.0062 (5.0784) grad_norm 1.5057 (1.8520) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][30/625] eta 0:03:54 lr 0.000906 wd 0.0500 time 0.3678 (0.3941) data time 0.0008 (0.0145) model time 0.0000 (0.0000) loss 4.6723 (4.9719) grad_norm 2.0408 (1.9251) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][40/625] eta 0:03:48 lr 0.000907 wd 0.0500 time 0.3935 (0.3902) data time 0.0009 (0.0112) model time 0.0000 (0.0000) loss 5.6620 (4.9481) grad_norm 1.5030 (1.8605) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][50/625] eta 0:03:42 lr 0.000909 wd 0.0500 time 0.3683 (0.3873) data time 0.0009 (0.0092) model time 0.0000 (0.0000) loss 5.0166 (4.9401) grad_norm 1.5590 (1.8590) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][60/625] eta 0:03:37 lr 0.000911 wd 0.0500 time 0.3732 (0.3855) data time 0.0010 (0.0079) model time 0.3721 (0.3751) loss 5.1162 (4.9492) grad_norm 1.4014 (1.8308) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][70/625] eta 0:03:35 lr 0.000912 wd 0.0500 time 0.3915 (0.3891) data time 0.0008 (0.0070) model time 0.3907 (0.3923) loss 5.2439 (4.9515) grad_norm 1.3361 (1.7897) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][80/625] eta 0:03:31 lr 0.000914 wd 0.0500 time 0.3641 (0.3873) data time 0.0011 (0.0063) model time 0.3630 (0.3860) loss 5.4093 (4.9679) grad_norm 1.7860 (1.7888) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][90/625] eta 0:03:26 lr 0.000915 wd 0.0500 time 0.3700 (0.3860) data time 0.0011 (0.0057) model time 0.3690 (0.3832) loss 5.0805 (4.9435) grad_norm 1.5784 (1.7973) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][100/625] eta 0:03:22 lr 0.000917 wd 0.0500 time 0.3834 (0.3853) data time 0.0008 (0.0053) model time 0.3826 (0.3819) loss 5.1558 (4.9192) grad_norm 1.7441 (1.8046) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][110/625] eta 0:03:18 lr 0.000919 wd 0.0500 time 0.3790 (0.3846) data time 0.0008 (0.0049) model time 0.3782 (0.3812) loss 4.8724 (4.8940) grad_norm 2.4642 (1.7945) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][120/625] eta 0:03:13 lr 0.000920 wd 0.0500 time 0.3781 (0.3841) data time 0.0009 (0.0046) model time 0.3771 (0.3805) loss 5.4694 (4.8912) grad_norm 1.8919 (1.7846) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][130/625] eta 0:03:10 lr 0.000922 wd 0.0500 time 0.3720 (0.3850) data time 0.0010 (0.0044) model time 0.3709 (0.3822) loss 4.7543 (4.9038) grad_norm 2.5637 (1.7916) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][140/625] eta 0:03:06 lr 0.000923 wd 0.0500 time 0.3650 (0.3842) data time 0.0011 (0.0042) model time 0.3638 (0.3812) loss 4.5189 (4.9123) grad_norm 1.8532 (1.7815) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][150/625] eta 0:03:02 lr 0.000925 wd 0.0500 time 0.3623 (0.3837) data time 0.0011 (0.0040) model time 0.3612 (0.3806) loss 5.3965 (4.9027) grad_norm 1.7874 (1.7840) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:05:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][160/625] eta 0:02:58 lr 0.000927 wd 0.0500 time 0.3914 (0.3834) data time 0.0011 (0.0038) model time 0.3903 (0.3803) loss 5.3919 (4.8984) grad_norm 2.0164 (1.7870) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][170/625] eta 0:02:54 lr 0.000928 wd 0.0500 time 0.3613 (0.3828) data time 0.0011 (0.0037) model time 0.3602 (0.3796) loss 4.7332 (4.8887) grad_norm 1.6105 (1.7752) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][180/625] eta 0:02:50 lr 0.000930 wd 0.0500 time 0.3797 (0.3825) data time 0.0011 (0.0035) model time 0.3787 (0.3793) loss 5.2256 (4.8933) grad_norm 1.4628 (1.7744) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][190/625] eta 0:02:46 lr 0.000931 wd 0.0500 time 0.3744 (0.3821) data time 0.0013 (0.0034) model time 0.3732 (0.3789) loss 4.0255 (4.8884) grad_norm 2.2143 (1.7802) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][200/625] eta 0:02:42 lr 0.000933 wd 0.0500 time 0.3704 (0.3817) data time 0.0011 (0.0033) model time 0.3693 (0.3785) loss 4.9583 (4.9038) grad_norm 1.8278 (1.7733) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][210/625] eta 0:02:38 lr 0.000935 wd 0.0500 time 0.3617 (0.3816) data time 0.0008 (0.0032) model time 0.3610 (0.3785) loss 5.5111 (4.9091) grad_norm 1.5770 (1.7730) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][220/625] eta 0:02:34 lr 0.000936 wd 0.0500 time 0.3746 (0.3815) data time 0.0010 (0.0031) model time 0.3736 (0.3785) loss 5.2877 (4.9088) grad_norm 2.1233 (1.7793) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][230/625] eta 0:02:30 lr 0.000938 wd 0.0500 time 0.3759 (0.3813) data time 0.0011 (0.0030) model time 0.3748 (0.3783) loss 5.3882 (4.9031) grad_norm 1.4529 (1.7752) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][240/625] eta 0:02:26 lr 0.000939 wd 0.0500 time 0.3750 (0.3810) data time 0.0009 (0.0030) model time 0.3741 (0.3780) loss 5.1931 (4.9183) grad_norm 3.3603 (1.7855) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][250/625] eta 0:02:23 lr 0.000941 wd 0.0500 time 0.3946 (0.3817) data time 0.0011 (0.0029) model time 0.3936 (0.3790) loss 5.1566 (4.9312) grad_norm 2.1502 (1.7849) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][260/625] eta 0:02:19 lr 0.000943 wd 0.0500 time 0.3758 (0.3814) data time 0.0011 (0.0028) model time 0.3748 (0.3787) loss 5.1055 (4.9336) grad_norm 1.8584 (1.7870) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][270/625] eta 0:02:15 lr 0.000944 wd 0.0500 time 0.3664 (0.3812) data time 0.0008 (0.0028) model time 0.3656 (0.3786) loss 3.8977 (4.9262) grad_norm 2.2438 (1.7929) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][280/625] eta 0:02:11 lr 0.000946 wd 0.0500 time 0.3821 (0.3811) data time 0.0009 (0.0027) model time 0.3813 (0.3784) loss 4.4844 (4.9182) grad_norm 1.5350 (1.7926) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][290/625] eta 0:02:07 lr 0.000947 wd 0.0500 time 0.3678 (0.3809) data time 0.0010 (0.0027) model time 0.3668 (0.3783) loss 5.1719 (4.9237) grad_norm 2.2010 (1.7879) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][300/625] eta 0:02:03 lr 0.000949 wd 0.0500 time 0.3738 (0.3807) data time 0.0008 (0.0026) model time 0.3730 (0.3781) loss 5.3942 (4.9252) grad_norm 1.1379 (1.7825) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][310/625] eta 0:01:59 lr 0.000951 wd 0.0500 time 0.3998 (0.3807) data time 0.0008 (0.0026) model time 0.3990 (0.3781) loss 5.8486 (4.9290) grad_norm 1.4563 (1.7737) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][320/625] eta 0:01:56 lr 0.000952 wd 0.0500 time 0.3639 (0.3805) data time 0.0010 (0.0025) model time 0.3629 (0.3779) loss 4.0432 (4.9297) grad_norm 1.3906 (1.7721) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][330/625] eta 0:01:52 lr 0.000954 wd 0.0500 time 0.3819 (0.3804) data time 0.0011 (0.0025) model time 0.3808 (0.3779) loss 5.1973 (4.9329) grad_norm 1.3294 (1.7688) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][340/625] eta 0:01:48 lr 0.000955 wd 0.0500 time 0.3832 (0.3809) data time 0.0008 (0.0025) model time 0.3825 (0.3786) loss 5.3709 (4.9312) grad_norm 2.9118 (1.7749) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][350/625] eta 0:01:44 lr 0.000957 wd 0.0500 time 0.3790 (0.3809) data time 0.0008 (0.0024) model time 0.3782 (0.3785) loss 4.3109 (4.9313) grad_norm 1.9648 (1.7772) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][360/625] eta 0:01:40 lr 0.000959 wd 0.0500 time 0.3741 (0.3807) data time 0.0008 (0.0024) model time 0.3732 (0.3784) loss 4.7118 (4.9352) grad_norm 1.8429 (1.7844) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][370/625] eta 0:01:37 lr 0.000960 wd 0.0500 time 0.3912 (0.3812) data time 0.0010 (0.0024) model time 0.3902 (0.3790) loss 4.7825 (4.9353) grad_norm 3.1596 (1.7878) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][380/625] eta 0:01:33 lr 0.000962 wd 0.0500 time 0.3610 (0.3811) data time 0.0011 (0.0023) model time 0.3599 (0.3789) loss 4.2354 (4.9356) grad_norm 2.0632 (1.7925) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][390/625] eta 0:01:29 lr 0.000963 wd 0.0500 time 0.3809 (0.3810) data time 0.0010 (0.0023) model time 0.3798 (0.3788) loss 4.2501 (4.9382) grad_norm 1.1679 (1.7884) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][400/625] eta 0:01:25 lr 0.000965 wd 0.0500 time 0.3888 (0.3809) data time 0.0009 (0.0023) model time 0.3878 (0.3787) loss 5.4309 (4.9340) grad_norm 1.5852 (1.7879) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][410/625] eta 0:01:21 lr 0.000967 wd 0.0500 time 0.3652 (0.3808) data time 0.0012 (0.0023) model time 0.3640 (0.3786) loss 4.8165 (4.9230) grad_norm 1.4915 (1.7867) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][420/625] eta 0:01:18 lr 0.000968 wd 0.0500 time 0.3941 (0.3808) data time 0.0008 (0.0022) model time 0.3932 (0.3786) loss 5.2694 (4.9219) grad_norm 1.6417 (1.7880) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][430/625] eta 0:01:14 lr 0.000970 wd 0.0500 time 0.3908 (0.3807) data time 0.0010 (0.0022) model time 0.3897 (0.3785) loss 5.0748 (4.9188) grad_norm 1.2749 (1.7835) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][440/625] eta 0:01:10 lr 0.000971 wd 0.0500 time 0.3633 (0.3805) data time 0.0008 (0.0022) model time 0.3625 (0.3783) loss 5.3752 (4.9189) grad_norm 1.6304 (1.7792) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][450/625] eta 0:01:06 lr 0.000973 wd 0.0500 time 0.3769 (0.3804) data time 0.0010 (0.0022) model time 0.3760 (0.3782) loss 5.0538 (4.9201) grad_norm 1.9322 (1.7769) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][460/625] eta 0:01:02 lr 0.000975 wd 0.0500 time 0.3781 (0.3803) data time 0.0011 (0.0021) model time 0.3770 (0.3782) loss 5.2854 (4.9214) grad_norm 3.8661 (1.7792) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:07:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][470/625] eta 0:00:58 lr 0.000976 wd 0.0500 time 0.3709 (0.3802) data time 0.0010 (0.0021) model time 0.3699 (0.3781) loss 4.9203 (4.9212) grad_norm 2.1740 (1.7776) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][480/625] eta 0:00:55 lr 0.000978 wd 0.0500 time 0.3749 (0.3801) data time 0.0010 (0.0021) model time 0.3739 (0.3780) loss 4.0689 (4.9162) grad_norm 1.8792 (1.7750) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][490/625] eta 0:00:51 lr 0.000979 wd 0.0500 time 0.3939 (0.3801) data time 0.0010 (0.0021) model time 0.3928 (0.3779) loss 5.2492 (4.9111) grad_norm 2.0789 (1.7753) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][500/625] eta 0:00:47 lr 0.000981 wd 0.0500 time 0.3727 (0.3800) data time 0.0008 (0.0021) model time 0.3719 (0.3779) loss 4.7305 (4.9104) grad_norm 1.5917 (1.7723) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][510/625] eta 0:00:43 lr 0.000983 wd 0.0500 time 0.3677 (0.3798) data time 0.0008 (0.0021) model time 0.3669 (0.3777) loss 4.1622 (4.9107) grad_norm 1.9868 (1.7763) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][520/625] eta 0:00:39 lr 0.000984 wd 0.0500 time 0.6055 (0.3802) data time 0.0010 (0.0021) model time 0.6044 (0.3781) loss 5.0304 (4.9050) grad_norm 1.2602 (1.7720) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][530/625] eta 0:00:36 lr 0.000986 wd 0.0500 time 0.3644 (0.3801) data time 0.0011 (0.0021) model time 0.3633 (0.3780) loss 5.2068 (4.9036) grad_norm 1.4287 (1.7686) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][540/625] eta 0:00:32 lr 0.000987 wd 0.0500 time 0.3735 (0.3799) data time 0.0008 (0.0020) model time 0.3726 (0.3779) loss 5.8308 (4.9046) grad_norm 1.3857 (1.7667) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][550/625] eta 0:00:28 lr 0.000989 wd 0.0500 time 0.3928 (0.3800) data time 0.0012 (0.0020) model time 0.3916 (0.3779) loss 4.5013 (4.9043) grad_norm 1.5413 (1.7661) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][560/625] eta 0:00:24 lr 0.000991 wd 0.0500 time 0.3756 (0.3800) data time 0.0010 (0.0020) model time 0.3747 (0.3779) loss 4.9644 (4.9040) grad_norm 2.0069 (1.7639) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][570/625] eta 0:00:20 lr 0.000992 wd 0.0500 time 0.3711 (0.3799) data time 0.0007 (0.0020) model time 0.3704 (0.3779) loss 5.4227 (4.9058) grad_norm 2.2473 (1.7651) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][580/625] eta 0:00:17 lr 0.000994 wd 0.0500 time 0.3807 (0.3806) data time 0.0011 (0.0020) model time 0.3796 (0.3786) loss 4.5627 (4.9047) grad_norm 2.2721 (1.7705) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][590/625] eta 0:00:13 lr 0.000995 wd 0.0500 time 0.3733 (0.3805) data time 0.0010 (0.0020) model time 0.3723 (0.3786) loss 4.4898 (4.8928) grad_norm 2.2057 (1.7694) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][600/625] eta 0:00:09 lr 0.000997 wd 0.0500 time 0.3825 (0.3804) data time 0.0008 (0.0020) model time 0.3817 (0.3785) loss 4.0158 (4.8923) grad_norm 1.6568 (1.7677) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][610/625] eta 0:00:05 lr 0.000999 wd 0.0500 time 0.3683 (0.3804) data time 0.0005 (0.0019) model time 0.3678 (0.3784) loss 5.7880 (4.8917) grad_norm 2.0454 (1.7694) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [9/300][620/625] eta 0:00:01 lr 0.001000 wd 0.0500 time 0.3554 (0.3801) data time 0.0005 (0.0019) model time 0.3548 (0.3782) loss 5.3479 (4.8923) grad_norm 2.3572 (1.7710) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:08:55 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 9 training takes 0:03:57 [2024-08-20 00:08:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:08:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:08:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.532 (0.532) Loss 1.4111 (1.4111) Acc@1 66.748 (66.748) Acc@5 88.525 (88.525) Mem 11221MB [2024-08-20 00:08:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 2.4492 (1.7272) Acc@1 46.484 (59.175) Acc@5 71.729 (84.415) Mem 11221MB [2024-08-20 00:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.121) Loss 2.6367 (2.0877) Acc@1 41.602 (52.881) Acc@5 68.994 (77.960) Mem 11221MB [2024-08-20 00:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 52.769 Acc@5 77.937 [2024-08-20 00:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 52.8% [2024-08-20 00:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 52.77% [2024-08-20 00:08:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-20 00:09:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-20 00:09:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.518 (0.518) Loss 6.4844 (6.4844) Acc@1 0.537 (0.537) Acc@5 4.688 (4.688) Mem 11221MB [2024-08-20 00:09:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 6.7305 (6.6278) Acc@1 0.781 (0.879) Acc@5 2.197 (3.431) Mem 11221MB [2024-08-20 00:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 6.6523 (6.6434) Acc@1 1.172 (0.839) Acc@5 3.711 (3.409) Mem 11221MB [2024-08-20 00:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 0.970 Acc@5 3.701 [2024-08-20 00:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 1.0% [2024-08-20 00:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 0.97% [2024-08-20 00:09:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-20 00:09:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-20 00:09:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][0/625] eta 0:08:33 lr 0.001001 wd 0.0500 time 0.8209 (0.8209) data time 0.4829 (0.4829) model time 0.0000 (0.0000) loss 5.7011 (5.7011) grad_norm 2.0619 (2.0619) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][10/625] eta 0:04:18 lr 0.001003 wd 0.0500 time 0.3849 (0.4205) data time 0.0009 (0.0450) model time 0.0000 (0.0000) loss 5.5514 (4.9890) grad_norm 1.2414 (1.6374) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][20/625] eta 0:04:01 lr 0.001004 wd 0.0500 time 0.3692 (0.3991) data time 0.0008 (0.0241) model time 0.0000 (0.0000) loss 5.1228 (5.0752) grad_norm 1.7674 (1.6282) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][30/625] eta 0:03:53 lr 0.001006 wd 0.0500 time 0.3778 (0.3931) data time 0.0008 (0.0167) model time 0.0000 (0.0000) loss 5.2514 (4.9635) grad_norm 1.7152 (1.6556) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][40/625] eta 0:03:47 lr 0.001007 wd 0.0500 time 0.3882 (0.3887) data time 0.0007 (0.0133) model time 0.0000 (0.0000) loss 5.9631 (4.9941) grad_norm 2.1547 (1.6985) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][50/625] eta 0:03:42 lr 0.001009 wd 0.0500 time 0.3713 (0.3871) data time 0.0011 (0.0109) model time 0.0000 (0.0000) loss 5.3559 (5.0083) grad_norm 1.3545 (1.6810) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][60/625] eta 0:03:37 lr 0.001011 wd 0.0500 time 0.3772 (0.3857) data time 0.0008 (0.0093) model time 0.3764 (0.3775) loss 5.4558 (5.0498) grad_norm 1.5284 (1.6856) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][70/625] eta 0:03:33 lr 0.001012 wd 0.0500 time 0.3975 (0.3852) data time 0.0009 (0.0081) model time 0.3966 (0.3794) loss 4.0120 (4.9842) grad_norm 1.6280 (1.6633) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][80/625] eta 0:03:29 lr 0.001014 wd 0.0500 time 0.3786 (0.3850) data time 0.0011 (0.0073) model time 0.3775 (0.3804) loss 5.1288 (4.9792) grad_norm 1.6267 (1.6569) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][90/625] eta 0:03:25 lr 0.001015 wd 0.0500 time 0.3783 (0.3841) data time 0.0008 (0.0066) model time 0.3775 (0.3792) loss 4.0146 (4.9647) grad_norm 2.1192 (1.6763) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][100/625] eta 0:03:21 lr 0.001017 wd 0.0500 time 0.4013 (0.3837) data time 0.0009 (0.0061) model time 0.4005 (0.3793) loss 3.6486 (4.9498) grad_norm 1.2011 (1.6589) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][110/625] eta 0:03:17 lr 0.001019 wd 0.0500 time 0.3700 (0.3834) data time 0.0009 (0.0056) model time 0.3690 (0.3791) loss 6.0525 (4.9340) grad_norm 1.5133 (1.6704) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][120/625] eta 0:03:13 lr 0.001020 wd 0.0500 time 0.3732 (0.3833) data time 0.0010 (0.0052) model time 0.3722 (0.3794) loss 5.1109 (4.9339) grad_norm 1.6904 (1.6715) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][130/625] eta 0:03:09 lr 0.001022 wd 0.0500 time 0.3912 (0.3833) data time 0.0008 (0.0050) model time 0.3904 (0.3796) loss 5.5710 (4.9465) grad_norm 1.7290 (1.6573) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][140/625] eta 0:03:05 lr 0.001023 wd 0.0500 time 0.3711 (0.3828) data time 0.0009 (0.0048) model time 0.3701 (0.3791) loss 5.5892 (4.9310) grad_norm 1.2843 (1.6491) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:10:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][150/625] eta 0:03:01 lr 0.001025 wd 0.0500 time 0.3779 (0.3824) data time 0.0011 (0.0045) model time 0.3769 (0.3787) loss 3.5934 (4.9172) grad_norm 1.7031 (1.6684) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:10:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][160/625] eta 0:02:58 lr 0.001027 wd 0.0500 time 0.3880 (0.3847) data time 0.0010 (0.0043) model time 0.3870 (0.3824) loss 3.8380 (4.9050) grad_norm 1.3463 (1.6787) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:10:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][170/625] eta 0:02:54 lr 0.001028 wd 0.0500 time 0.3712 (0.3842) data time 0.0008 (0.0041) model time 0.3704 (0.3817) loss 5.5943 (4.9159) grad_norm 1.6434 (1.6891) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:10:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][180/625] eta 0:02:50 lr 0.001030 wd 0.0500 time 0.3735 (0.3839) data time 0.0011 (0.0040) model time 0.3723 (0.3813) loss 4.5760 (4.8951) grad_norm 1.6139 (1.6857) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][190/625] eta 0:02:46 lr 0.001031 wd 0.0500 time 0.3907 (0.3839) data time 0.0010 (0.0039) model time 0.3897 (0.3814) loss 5.0965 (4.8910) grad_norm 1.7126 (1.6794) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][200/625] eta 0:02:43 lr 0.001033 wd 0.0500 time 0.3857 (0.3836) data time 0.0009 (0.0037) model time 0.3848 (0.3811) loss 5.6605 (4.8737) grad_norm 2.1925 (1.7089) loss_scale 32768.0000 (16710.0498) mem 11221MB [2024-08-20 00:10:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][210/625] eta 0:02:39 lr 0.001035 wd 0.0500 time 0.3708 (0.3833) data time 0.0008 (0.0036) model time 0.3700 (0.3807) loss 3.8685 (4.8565) grad_norm 2.2991 (1.7238) loss_scale 32768.0000 (17471.0900) mem 11221MB [2024-08-20 00:10:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][220/625] eta 0:02:35 lr 0.001036 wd 0.0500 time 0.3841 (0.3832) data time 0.0008 (0.0035) model time 0.3833 (0.3807) loss 4.1197 (4.8491) grad_norm 2.5240 (1.7269) loss_scale 32768.0000 (18163.2579) mem 11221MB [2024-08-20 00:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][230/625] eta 0:02:31 lr 0.001038 wd 0.0500 time 0.3796 (0.3830) data time 0.0008 (0.0034) model time 0.3788 (0.3806) loss 4.8878 (4.8343) grad_norm 2.5069 (1.7416) loss_scale 32768.0000 (18795.4978) mem 11221MB [2024-08-20 00:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][240/625] eta 0:02:27 lr 0.001039 wd 0.0500 time 0.3701 (0.3828) data time 0.0009 (0.0033) model time 0.3692 (0.3803) loss 5.7826 (4.8318) grad_norm 1.5927 (1.7442) loss_scale 32768.0000 (19375.2697) mem 11221MB [2024-08-20 00:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][250/625] eta 0:02:23 lr 0.001041 wd 0.0500 time 0.3846 (0.3827) data time 0.0011 (0.0033) model time 0.3836 (0.3802) loss 5.0135 (4.8207) grad_norm 1.8493 (1.7446) loss_scale 32768.0000 (19908.8446) mem 11221MB [2024-08-20 00:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][260/625] eta 0:02:19 lr 0.001043 wd 0.0500 time 0.3760 (0.3824) data time 0.0011 (0.0032) model time 0.3750 (0.3799) loss 5.3196 (4.8266) grad_norm 1.6366 (1.7449) loss_scale 32768.0000 (20401.5326) mem 11221MB [2024-08-20 00:10:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][270/625] eta 0:02:15 lr 0.001044 wd 0.0500 time 0.3737 (0.3824) data time 0.0010 (0.0031) model time 0.3727 (0.3799) loss 3.7072 (4.8226) grad_norm 1.5424 (1.7405) loss_scale 32768.0000 (20857.8598) mem 11221MB [2024-08-20 00:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][280/625] eta 0:02:11 lr 0.001046 wd 0.0500 time 0.3870 (0.3822) data time 0.0013 (0.0031) model time 0.3858 (0.3798) loss 4.0324 (4.8215) grad_norm 1.9066 (1.7356) loss_scale 32768.0000 (21281.7082) mem 11221MB [2024-08-20 00:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][290/625] eta 0:02:07 lr 0.001047 wd 0.0500 time 0.3754 (0.3820) data time 0.0008 (0.0030) model time 0.3746 (0.3796) loss 5.3700 (4.8276) grad_norm 1.8519 (1.7286) loss_scale 32768.0000 (21676.4261) mem 11221MB [2024-08-20 00:10:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][300/625] eta 0:02:04 lr 0.001049 wd 0.0500 time 0.3750 (0.3818) data time 0.0008 (0.0030) model time 0.3743 (0.3794) loss 5.4263 (4.8279) grad_norm 1.4335 (1.7266) loss_scale 32768.0000 (22044.9169) mem 11221MB [2024-08-20 00:11:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][310/625] eta 0:02:00 lr 0.001051 wd 0.0500 time 0.3847 (0.3817) data time 0.0008 (0.0029) model time 0.3838 (0.3793) loss 5.6775 (4.8227) grad_norm 2.2556 (inf) loss_scale 16384.0000 (22073.6206) mem 11221MB [2024-08-20 00:11:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][320/625] eta 0:01:56 lr 0.001052 wd 0.0500 time 0.3599 (0.3815) data time 0.0012 (0.0028) model time 0.3587 (0.3792) loss 4.7638 (4.8285) grad_norm 1.5018 (inf) loss_scale 16384.0000 (21896.3738) mem 11221MB [2024-08-20 00:11:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][330/625] eta 0:01:52 lr 0.001054 wd 0.0500 time 0.3716 (0.3813) data time 0.0008 (0.0028) model time 0.3707 (0.3790) loss 3.8587 (4.8194) grad_norm 2.0660 (inf) loss_scale 16384.0000 (21729.8369) mem 11221MB [2024-08-20 00:11:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][340/625] eta 0:01:48 lr 0.001055 wd 0.0500 time 0.3868 (0.3824) data time 0.0008 (0.0027) model time 0.3860 (0.3803) loss 4.7247 (4.8258) grad_norm 1.6697 (inf) loss_scale 16384.0000 (21573.0674) mem 11221MB [2024-08-20 00:11:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][350/625] eta 0:01:45 lr 0.001057 wd 0.0500 time 0.3735 (0.3822) data time 0.0009 (0.0027) model time 0.3726 (0.3801) loss 4.1454 (4.8263) grad_norm 1.6634 (inf) loss_scale 16384.0000 (21425.2308) mem 11221MB [2024-08-20 00:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][360/625] eta 0:01:41 lr 0.001059 wd 0.0500 time 0.3639 (0.3821) data time 0.0010 (0.0027) model time 0.3629 (0.3800) loss 5.2988 (4.8162) grad_norm 1.6473 (inf) loss_scale 16384.0000 (21285.5845) mem 11221MB [2024-08-20 00:11:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][370/625] eta 0:01:37 lr 0.001060 wd 0.0500 time 0.3801 (0.3820) data time 0.0008 (0.0026) model time 0.3793 (0.3798) loss 5.7524 (4.8242) grad_norm 1.6198 (inf) loss_scale 16384.0000 (21153.4663) mem 11221MB [2024-08-20 00:11:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][380/625] eta 0:01:33 lr 0.001062 wd 0.0500 time 0.3711 (0.3818) data time 0.0009 (0.0026) model time 0.3702 (0.3796) loss 3.8048 (4.8180) grad_norm 2.2724 (inf) loss_scale 16384.0000 (21028.2835) mem 11221MB [2024-08-20 00:11:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][390/625] eta 0:01:29 lr 0.001063 wd 0.0500 time 0.3771 (0.3817) data time 0.0008 (0.0026) model time 0.3763 (0.3796) loss 3.5800 (4.8166) grad_norm 1.5018 (inf) loss_scale 16384.0000 (20909.5038) mem 11221MB [2024-08-20 00:11:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][400/625] eta 0:01:25 lr 0.001065 wd 0.0500 time 0.3762 (0.3816) data time 0.0009 (0.0025) model time 0.3753 (0.3795) loss 3.5237 (4.8123) grad_norm 1.5783 (inf) loss_scale 16384.0000 (20796.6484) mem 11221MB [2024-08-20 00:11:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][410/625] eta 0:01:22 lr 0.001067 wd 0.0500 time 0.3684 (0.3814) data time 0.0009 (0.0025) model time 0.3675 (0.3793) loss 5.0248 (4.8109) grad_norm 1.2225 (inf) loss_scale 16384.0000 (20689.2847) mem 11221MB [2024-08-20 00:11:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][420/625] eta 0:01:18 lr 0.001068 wd 0.0500 time 0.3736 (0.3813) data time 0.0009 (0.0025) model time 0.3728 (0.3793) loss 3.5861 (4.8038) grad_norm 1.7184 (inf) loss_scale 16384.0000 (20587.0214) mem 11221MB [2024-08-20 00:11:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][430/625] eta 0:01:14 lr 0.001070 wd 0.0500 time 0.3854 (0.3813) data time 0.0011 (0.0024) model time 0.3843 (0.3792) loss 5.1896 (4.7959) grad_norm 1.3736 (inf) loss_scale 16384.0000 (20489.5035) mem 11221MB [2024-08-20 00:11:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][440/625] eta 0:01:10 lr 0.001071 wd 0.0500 time 0.3765 (0.3811) data time 0.0008 (0.0024) model time 0.3757 (0.3791) loss 3.8497 (4.7925) grad_norm 1.4724 (inf) loss_scale 16384.0000 (20396.4082) mem 11221MB [2024-08-20 00:11:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][450/625] eta 0:01:06 lr 0.001073 wd 0.0500 time 0.3681 (0.3810) data time 0.0010 (0.0024) model time 0.3670 (0.3790) loss 3.9362 (4.7963) grad_norm 1.3698 (inf) loss_scale 16384.0000 (20307.4412) mem 11221MB [2024-08-20 00:12:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][460/625] eta 0:01:03 lr 0.001075 wd 0.0500 time 0.3830 (0.3819) data time 0.0008 (0.0024) model time 0.3822 (0.3800) loss 5.0705 (4.7999) grad_norm 1.4782 (inf) loss_scale 16384.0000 (20222.3341) mem 11221MB [2024-08-20 00:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][470/625] eta 0:00:59 lr 0.001076 wd 0.0500 time 0.3657 (0.3817) data time 0.0011 (0.0023) model time 0.3645 (0.3798) loss 5.1474 (4.8029) grad_norm 2.2883 (inf) loss_scale 16384.0000 (20140.8408) mem 11221MB [2024-08-20 00:12:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][480/625] eta 0:00:55 lr 0.001078 wd 0.0500 time 0.3744 (0.3816) data time 0.0011 (0.0023) model time 0.3733 (0.3796) loss 5.2432 (4.8052) grad_norm 1.9363 (inf) loss_scale 16384.0000 (20062.7360) mem 11221MB [2024-08-20 00:12:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][490/625] eta 0:00:51 lr 0.001079 wd 0.0500 time 0.3867 (0.3816) data time 0.0009 (0.0023) model time 0.3859 (0.3796) loss 5.3104 (4.8021) grad_norm 1.6548 (inf) loss_scale 16384.0000 (19987.8126) mem 11221MB [2024-08-20 00:12:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][500/625] eta 0:00:47 lr 0.001081 wd 0.0500 time 0.3744 (0.3815) data time 0.0008 (0.0023) model time 0.3735 (0.3796) loss 5.4013 (4.7925) grad_norm 1.6419 (inf) loss_scale 16384.0000 (19915.8802) mem 11221MB [2024-08-20 00:12:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][510/625] eta 0:00:43 lr 0.001083 wd 0.0500 time 0.3755 (0.3814) data time 0.0011 (0.0023) model time 0.3744 (0.3795) loss 4.9654 (4.7903) grad_norm 1.5818 (inf) loss_scale 16384.0000 (19846.7632) mem 11221MB [2024-08-20 00:12:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][520/625] eta 0:00:40 lr 0.001084 wd 0.0500 time 0.3847 (0.3814) data time 0.0011 (0.0023) model time 0.3836 (0.3795) loss 4.1354 (4.7842) grad_norm 1.5067 (inf) loss_scale 16384.0000 (19780.2994) mem 11221MB [2024-08-20 00:12:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][530/625] eta 0:00:36 lr 0.001086 wd 0.0500 time 0.3782 (0.3813) data time 0.0010 (0.0022) model time 0.3772 (0.3794) loss 4.9236 (4.7874) grad_norm 1.4257 (inf) loss_scale 16384.0000 (19716.3390) mem 11221MB [2024-08-20 00:12:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][540/625] eta 0:00:32 lr 0.001087 wd 0.0500 time 0.3731 (0.3812) data time 0.0008 (0.0022) model time 0.3722 (0.3793) loss 5.2436 (4.7884) grad_norm 1.4244 (inf) loss_scale 16384.0000 (19654.7431) mem 11221MB [2024-08-20 00:12:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][550/625] eta 0:00:28 lr 0.001089 wd 0.0500 time 0.3816 (0.3812) data time 0.0009 (0.0022) model time 0.3807 (0.3793) loss 5.0564 (4.7872) grad_norm 1.2857 (inf) loss_scale 16384.0000 (19595.3829) mem 11221MB [2024-08-20 00:12:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][560/625] eta 0:00:24 lr 0.001091 wd 0.0500 time 0.3623 (0.3811) data time 0.0008 (0.0022) model time 0.3615 (0.3792) loss 4.9021 (4.7866) grad_norm 1.6410 (inf) loss_scale 16384.0000 (19538.1390) mem 11221MB [2024-08-20 00:12:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][570/625] eta 0:00:20 lr 0.001092 wd 0.0500 time 0.3748 (0.3810) data time 0.0008 (0.0022) model time 0.3740 (0.3791) loss 4.4338 (4.7841) grad_norm 1.6012 (inf) loss_scale 16384.0000 (19482.9002) mem 11221MB [2024-08-20 00:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][580/625] eta 0:00:17 lr 0.001094 wd 0.0500 time 0.3814 (0.3810) data time 0.0013 (0.0021) model time 0.3801 (0.3791) loss 5.0355 (4.7849) grad_norm 1.1112 (inf) loss_scale 16384.0000 (19429.5628) mem 11221MB [2024-08-20 00:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][590/625] eta 0:00:13 lr 0.001095 wd 0.0500 time 0.3775 (0.3809) data time 0.0009 (0.0021) model time 0.3766 (0.3790) loss 4.2542 (4.7797) grad_norm 1.7771 (inf) loss_scale 16384.0000 (19378.0305) mem 11221MB [2024-08-20 00:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][600/625] eta 0:00:09 lr 0.001097 wd 0.0500 time 0.3824 (0.3809) data time 0.0009 (0.0021) model time 0.3815 (0.3791) loss 3.8060 (4.7765) grad_norm 1.4827 (inf) loss_scale 16384.0000 (19328.2130) mem 11221MB [2024-08-20 00:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][610/625] eta 0:00:05 lr 0.001099 wd 0.0500 time 0.3776 (0.3808) data time 0.0005 (0.0021) model time 0.3770 (0.3790) loss 5.4385 (4.7734) grad_norm 1.3845 (inf) loss_scale 16384.0000 (19280.0262) mem 11221MB [2024-08-20 00:13:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [10/300][620/625] eta 0:00:01 lr 0.001100 wd 0.0500 time 0.3600 (0.3806) data time 0.0006 (0.0021) model time 0.3595 (0.3787) loss 4.2730 (4.7678) grad_norm 1.4090 (inf) loss_scale 16384.0000 (19233.3913) mem 11221MB [2024-08-20 00:13:02 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 10 training takes 0:03:57 [2024-08-20 00:13:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:13:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.521 (0.521) Loss 1.3418 (1.3418) Acc@1 67.920 (67.920) Acc@5 88.623 (88.623) Mem 11221MB [2024-08-20 00:13:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 2.2832 (1.6140) Acc@1 49.072 (60.795) Acc@5 75.684 (85.875) Mem 11221MB [2024-08-20 00:13:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.122) Loss 2.4668 (1.9802) Acc@1 46.289 (54.787) Acc@5 71.143 (79.799) Mem 11221MB [2024-08-20 00:13:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 54.846 Acc@5 79.924 [2024-08-20 00:13:06 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 54.8% [2024-08-20 00:13:06 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 54.85% [2024-08-20 00:13:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-20 00:13:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-20 00:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 6.2188 (6.2188) Acc@1 1.416 (1.416) Acc@5 7.031 (7.031) Mem 11221MB [2024-08-20 00:13:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 6.5195 (6.3949) Acc@1 0.928 (1.425) Acc@5 4.297 (5.731) Mem 11221MB [2024-08-20 00:13:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 6.4453 (6.4182) Acc@1 1.660 (1.451) Acc@5 5.713 (5.629) Mem 11221MB [2024-08-20 00:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 1.723 Acc@5 6.128 [2024-08-20 00:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 1.7% [2024-08-20 00:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 1.72% [2024-08-20 00:13:11 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-20 00:13:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-20 00:13:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][0/625] eta 0:08:14 lr 0.001101 wd 0.0500 time 0.7908 (0.7908) data time 0.4535 (0.4535) model time 0.0000 (0.0000) loss 4.2958 (4.2958) grad_norm 1.6499 (1.6499) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][10/625] eta 0:04:13 lr 0.001102 wd 0.0500 time 0.3912 (0.4115) data time 0.0008 (0.0422) model time 0.0000 (0.0000) loss 5.5064 (4.5276) grad_norm 1.9263 (1.6682) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][20/625] eta 0:03:58 lr 0.001104 wd 0.0500 time 0.3695 (0.3950) data time 0.0011 (0.0227) model time 0.0000 (0.0000) loss 5.0174 (4.6308) grad_norm 1.4069 (1.7559) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][30/625] eta 0:03:55 lr 0.001106 wd 0.0500 time 0.3698 (0.3955) data time 0.0009 (0.0158) model time 0.0000 (0.0000) loss 4.0632 (4.5468) grad_norm 1.2944 (1.6658) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][40/625] eta 0:03:48 lr 0.001107 wd 0.0500 time 0.3819 (0.3909) data time 0.0010 (0.0122) model time 0.0000 (0.0000) loss 4.4492 (4.6016) grad_norm 1.7074 (1.6441) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][50/625] eta 0:03:43 lr 0.001109 wd 0.0500 time 0.3682 (0.3880) data time 0.0010 (0.0101) model time 0.0000 (0.0000) loss 3.9976 (4.5940) grad_norm 1.5703 (1.6448) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][60/625] eta 0:03:38 lr 0.001110 wd 0.0500 time 0.3709 (0.3865) data time 0.0009 (0.0086) model time 0.3701 (0.3777) loss 5.4100 (4.6521) grad_norm 1.3560 (1.6212) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][70/625] eta 0:03:33 lr 0.001112 wd 0.0500 time 0.3779 (0.3853) data time 0.0011 (0.0075) model time 0.3768 (0.3774) loss 4.8043 (4.6285) grad_norm 1.9766 (1.6238) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][80/625] eta 0:03:29 lr 0.001114 wd 0.0500 time 0.3687 (0.3845) data time 0.0008 (0.0067) model time 0.3678 (0.3775) loss 4.8305 (4.5978) grad_norm 1.5234 (1.6219) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][90/625] eta 0:03:25 lr 0.001115 wd 0.0500 time 0.3783 (0.3837) data time 0.0011 (0.0061) model time 0.3772 (0.3770) loss 4.1919 (4.6070) grad_norm 1.5491 (1.6369) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][100/625] eta 0:03:21 lr 0.001117 wd 0.0500 time 0.4015 (0.3833) data time 0.0011 (0.0057) model time 0.4003 (0.3774) loss 5.2985 (4.6069) grad_norm 1.8345 (1.6378) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][110/625] eta 0:03:17 lr 0.001118 wd 0.0500 time 0.3679 (0.3829) data time 0.0011 (0.0053) model time 0.3669 (0.3773) loss 4.3912 (4.6093) grad_norm 1.8046 (1.6580) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:13:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][120/625] eta 0:03:13 lr 0.001120 wd 0.0500 time 0.3677 (0.3822) data time 0.0008 (0.0049) model time 0.3669 (0.3768) loss 4.0951 (4.5983) grad_norm 1.7203 (1.6545) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][130/625] eta 0:03:09 lr 0.001122 wd 0.0500 time 0.3862 (0.3821) data time 0.0012 (0.0046) model time 0.3850 (0.3771) loss 3.7069 (4.5597) grad_norm 1.8222 (1.6441) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][140/625] eta 0:03:05 lr 0.001123 wd 0.0500 time 0.3657 (0.3818) data time 0.0012 (0.0044) model time 0.3645 (0.3770) loss 5.3430 (4.5774) grad_norm 2.4943 (1.6487) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][150/625] eta 0:03:01 lr 0.001125 wd 0.0500 time 0.3645 (0.3813) data time 0.0011 (0.0042) model time 0.3634 (0.3767) loss 4.9750 (4.5756) grad_norm 1.5150 (1.6667) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][160/625] eta 0:02:57 lr 0.001126 wd 0.0500 time 0.3892 (0.3809) data time 0.0008 (0.0040) model time 0.3884 (0.3764) loss 5.5175 (4.5813) grad_norm 1.1693 (1.6594) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][170/625] eta 0:02:53 lr 0.001128 wd 0.0500 time 0.3690 (0.3818) data time 0.0010 (0.0038) model time 0.3680 (0.3780) loss 4.8680 (4.5779) grad_norm 1.6827 (1.6739) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][180/625] eta 0:02:49 lr 0.001130 wd 0.0500 time 0.3824 (0.3815) data time 0.0010 (0.0037) model time 0.3814 (0.3778) loss 4.8721 (4.5714) grad_norm 1.5635 (1.6796) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][190/625] eta 0:02:45 lr 0.001131 wd 0.0500 time 0.3749 (0.3814) data time 0.0009 (0.0036) model time 0.3740 (0.3777) loss 4.7898 (4.5911) grad_norm 1.6241 (1.6890) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][200/625] eta 0:02:42 lr 0.001133 wd 0.0500 time 0.3713 (0.3812) data time 0.0008 (0.0035) model time 0.3705 (0.3777) loss 4.0237 (4.5839) grad_norm 1.3855 (1.6905) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][210/625] eta 0:02:38 lr 0.001134 wd 0.0500 time 0.3802 (0.3811) data time 0.0010 (0.0034) model time 0.3792 (0.3776) loss 4.2117 (4.5754) grad_norm 1.2101 (1.6806) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][220/625] eta 0:02:34 lr 0.001136 wd 0.0500 time 0.4097 (0.3810) data time 0.0008 (0.0033) model time 0.4089 (0.3777) loss 4.2434 (4.5827) grad_norm 1.9550 (1.6785) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][230/625] eta 0:02:30 lr 0.001138 wd 0.0500 time 0.3692 (0.3807) data time 0.0008 (0.0032) model time 0.3684 (0.3774) loss 4.2716 (4.5885) grad_norm 1.2967 (1.6810) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][240/625] eta 0:02:26 lr 0.001139 wd 0.0500 time 0.3842 (0.3817) data time 0.0014 (0.0031) model time 0.3828 (0.3787) loss 4.4074 (4.5781) grad_norm 2.4442 (1.6895) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][250/625] eta 0:02:23 lr 0.001141 wd 0.0500 time 0.3672 (0.3815) data time 0.0009 (0.0030) model time 0.3662 (0.3786) loss 5.5074 (4.5814) grad_norm 1.3942 (1.6861) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][260/625] eta 0:02:19 lr 0.001142 wd 0.0500 time 0.3735 (0.3815) data time 0.0011 (0.0030) model time 0.3724 (0.3786) loss 4.5753 (4.5904) grad_norm 1.7559 (1.6886) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][270/625] eta 0:02:15 lr 0.001144 wd 0.0500 time 0.3898 (0.3815) data time 0.0009 (0.0030) model time 0.3889 (0.3787) loss 5.7465 (4.5928) grad_norm 2.0541 (1.6860) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:14:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][280/625] eta 0:02:11 lr 0.001146 wd 0.0500 time 0.3694 (0.3815) data time 0.0008 (0.0029) model time 0.3686 (0.3787) loss 5.4435 (4.6121) grad_norm 1.1141 (1.6735) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][290/625] eta 0:02:07 lr 0.001147 wd 0.0500 time 0.3802 (0.3814) data time 0.0010 (0.0028) model time 0.3792 (0.3787) loss 4.1597 (4.6185) grad_norm 1.3991 (1.6699) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][300/625] eta 0:02:03 lr 0.001149 wd 0.0500 time 0.3759 (0.3814) data time 0.0017 (0.0028) model time 0.3741 (0.3787) loss 5.3776 (4.6262) grad_norm 1.5047 (1.6816) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][310/625] eta 0:02:00 lr 0.001150 wd 0.0500 time 0.3769 (0.3812) data time 0.0008 (0.0027) model time 0.3761 (0.3785) loss 4.7750 (4.6162) grad_norm 1.6975 (1.6985) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][320/625] eta 0:01:56 lr 0.001152 wd 0.0500 time 0.3618 (0.3812) data time 0.0011 (0.0027) model time 0.3608 (0.3785) loss 5.3835 (4.6177) grad_norm 1.4760 (1.6976) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][330/625] eta 0:01:52 lr 0.001154 wd 0.0500 time 0.3836 (0.3811) data time 0.0007 (0.0027) model time 0.3829 (0.3785) loss 3.8606 (4.6187) grad_norm 1.4582 (1.7006) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][340/625] eta 0:01:48 lr 0.001155 wd 0.0500 time 0.3660 (0.3810) data time 0.0009 (0.0026) model time 0.3652 (0.3784) loss 5.3348 (4.6190) grad_norm 2.1997 (1.6995) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][350/625] eta 0:01:44 lr 0.001157 wd 0.0500 time 0.3850 (0.3815) data time 0.0009 (0.0026) model time 0.3840 (0.3791) loss 4.9125 (4.6227) grad_norm 1.5547 (1.7041) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][360/625] eta 0:01:41 lr 0.001158 wd 0.0500 time 0.3784 (0.3814) data time 0.0008 (0.0025) model time 0.3776 (0.3790) loss 3.9555 (4.6196) grad_norm 1.1947 (1.7046) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][370/625] eta 0:01:37 lr 0.001160 wd 0.0500 time 0.3703 (0.3813) data time 0.0009 (0.0025) model time 0.3695 (0.3789) loss 4.0116 (4.6228) grad_norm 1.5433 (1.7025) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][380/625] eta 0:01:33 lr 0.001162 wd 0.0500 time 0.3691 (0.3810) data time 0.0012 (0.0025) model time 0.3679 (0.3786) loss 5.0481 (4.6208) grad_norm 1.3332 (1.6950) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][390/625] eta 0:01:29 lr 0.001163 wd 0.0500 time 0.3806 (0.3809) data time 0.0011 (0.0024) model time 0.3795 (0.3785) loss 5.1190 (4.6306) grad_norm 1.8927 (1.6892) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][400/625] eta 0:01:25 lr 0.001165 wd 0.0500 time 0.3663 (0.3807) data time 0.0011 (0.0024) model time 0.3652 (0.3784) loss 3.7220 (4.6326) grad_norm 1.2912 (1.6833) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][410/625] eta 0:01:21 lr 0.001166 wd 0.0500 time 0.3732 (0.3806) data time 0.0015 (0.0024) model time 0.3717 (0.3782) loss 4.8895 (4.6278) grad_norm 1.7341 (1.6901) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][420/625] eta 0:01:18 lr 0.001168 wd 0.0500 time 0.3885 (0.3810) data time 0.0009 (0.0023) model time 0.3877 (0.3788) loss 4.6067 (4.6281) grad_norm 1.4720 (1.6917) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][430/625] eta 0:01:14 lr 0.001170 wd 0.0500 time 0.3767 (0.3810) data time 0.0011 (0.0023) model time 0.3756 (0.3788) loss 4.4418 (4.6340) grad_norm 1.6411 (1.6991) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][440/625] eta 0:01:10 lr 0.001171 wd 0.0500 time 0.3716 (0.3810) data time 0.0011 (0.0023) model time 0.3705 (0.3788) loss 4.2628 (4.6351) grad_norm 1.1362 (1.7006) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][450/625] eta 0:01:06 lr 0.001173 wd 0.0500 time 0.3974 (0.3809) data time 0.0009 (0.0023) model time 0.3966 (0.3788) loss 3.2190 (4.6302) grad_norm 1.3481 (1.6999) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][460/625] eta 0:01:02 lr 0.001174 wd 0.0500 time 0.3713 (0.3809) data time 0.0011 (0.0022) model time 0.3703 (0.3787) loss 5.1834 (4.6296) grad_norm 2.0773 (1.7029) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][470/625] eta 0:00:59 lr 0.001176 wd 0.0500 time 0.3679 (0.3811) data time 0.0008 (0.0022) model time 0.3671 (0.3790) loss 5.4618 (4.6353) grad_norm 1.6309 (1.6977) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][480/625] eta 0:00:55 lr 0.001178 wd 0.0500 time 0.3872 (0.3811) data time 0.0011 (0.0022) model time 0.3861 (0.3790) loss 5.0112 (4.6364) grad_norm 1.3798 (1.6927) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][490/625] eta 0:00:51 lr 0.001179 wd 0.0500 time 0.3738 (0.3810) data time 0.0009 (0.0022) model time 0.3729 (0.3789) loss 4.7458 (4.6381) grad_norm 1.7165 (1.6867) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][500/625] eta 0:00:47 lr 0.001181 wd 0.0500 time 0.3724 (0.3810) data time 0.0009 (0.0022) model time 0.3715 (0.3789) loss 5.3472 (4.6363) grad_norm 1.4492 (1.6886) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][510/625] eta 0:00:43 lr 0.001182 wd 0.0500 time 0.3695 (0.3809) data time 0.0009 (0.0021) model time 0.3686 (0.3789) loss 5.1838 (4.6382) grad_norm 1.7649 (1.6846) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][520/625] eta 0:00:39 lr 0.001184 wd 0.0500 time 0.3549 (0.3809) data time 0.0009 (0.0021) model time 0.3540 (0.3789) loss 3.2434 (4.6382) grad_norm 1.5311 (1.6811) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][530/625] eta 0:00:36 lr 0.001186 wd 0.0500 time 0.3679 (0.3808) data time 0.0011 (0.0021) model time 0.3668 (0.3788) loss 4.6913 (4.6326) grad_norm 1.8082 (1.6839) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][540/625] eta 0:00:32 lr 0.001187 wd 0.0500 time 0.3948 (0.3812) data time 0.0011 (0.0021) model time 0.3936 (0.3793) loss 4.4331 (4.6317) grad_norm 1.3459 (1.6798) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][550/625] eta 0:00:28 lr 0.001189 wd 0.0500 time 0.3724 (0.3811) data time 0.0011 (0.0021) model time 0.3712 (0.3792) loss 4.7776 (4.6350) grad_norm 1.6165 (1.6794) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][560/625] eta 0:00:24 lr 0.001190 wd 0.0500 time 0.3747 (0.3810) data time 0.0008 (0.0021) model time 0.3738 (0.3791) loss 4.1502 (4.6332) grad_norm 1.2141 (1.6770) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][570/625] eta 0:00:20 lr 0.001192 wd 0.0500 time 0.3892 (0.3810) data time 0.0009 (0.0020) model time 0.3883 (0.3790) loss 4.6041 (4.6292) grad_norm 1.6834 (1.6734) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][580/625] eta 0:00:17 lr 0.001194 wd 0.0500 time 0.3788 (0.3809) data time 0.0010 (0.0020) model time 0.3778 (0.3790) loss 5.1780 (4.6320) grad_norm 1.7710 (1.6680) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:16:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][590/625] eta 0:00:13 lr 0.001195 wd 0.0500 time 0.3692 (0.3808) data time 0.0011 (0.0020) model time 0.3680 (0.3789) loss 4.4935 (4.6338) grad_norm 1.7924 (1.6683) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][600/625] eta 0:00:09 lr 0.001197 wd 0.0500 time 0.3735 (0.3807) data time 0.0009 (0.0020) model time 0.3726 (0.3788) loss 4.2694 (4.6366) grad_norm 1.6267 (1.6692) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][610/625] eta 0:00:05 lr 0.001198 wd 0.0500 time 0.3603 (0.3806) data time 0.0005 (0.0020) model time 0.3597 (0.3787) loss 4.3237 (4.6364) grad_norm 1.4301 (1.6694) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [11/300][620/625] eta 0:00:01 lr 0.001200 wd 0.0500 time 0.3599 (0.3803) data time 0.0005 (0.0020) model time 0.3593 (0.3784) loss 3.6376 (4.6294) grad_norm 2.4532 (1.6667) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:10 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 11 training takes 0:03:57 [2024-08-20 00:17:10 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:17:11 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:17:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.521 (0.521) Loss 1.3281 (1.3281) Acc@1 70.020 (70.020) Acc@5 89.990 (89.990) Mem 11221MB [2024-08-20 00:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 2.2324 (1.5640) Acc@1 49.023 (63.414) Acc@5 77.539 (87.598) Mem 11221MB [2024-08-20 00:17:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 2.3867 (1.9052) Acc@1 47.510 (57.078) Acc@5 73.193 (81.692) Mem 11221MB [2024-08-20 00:17:14 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 57.074 Acc@5 81.710 [2024-08-20 00:17:14 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 57.1% [2024-08-20 00:17:14 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 57.07% [2024-08-20 00:17:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-20 00:17:15 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-20 00:17:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.493 (0.493) Loss 5.9375 (5.9375) Acc@1 2.979 (2.979) Acc@5 11.230 (11.230) Mem 11221MB [2024-08-20 00:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 6.2461 (6.1246) Acc@1 2.148 (2.761) Acc@5 8.008 (9.264) Mem 11221MB [2024-08-20 00:17:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 6.1719 (6.1406) Acc@1 3.174 (2.837) Acc@5 8.350 (9.261) Mem 11221MB [2024-08-20 00:17:18 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 3.281 Acc@5 10.151 [2024-08-20 00:17:18 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 3.3% [2024-08-20 00:17:18 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 3.28% [2024-08-20 00:17:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-20 00:17:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-20 00:17:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][0/625] eta 0:08:08 lr 0.001201 wd 0.0500 time 0.7817 (0.7817) data time 0.4390 (0.4390) model time 0.0000 (0.0000) loss 4.6512 (4.6512) grad_norm 2.2324 (2.2324) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][10/625] eta 0:04:16 lr 0.001202 wd 0.0500 time 0.3907 (0.4174) data time 0.0013 (0.0410) model time 0.0000 (0.0000) loss 5.1907 (4.5014) grad_norm 2.4273 (2.0565) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][20/625] eta 0:04:00 lr 0.001204 wd 0.0500 time 0.3746 (0.3977) data time 0.0011 (0.0220) model time 0.0000 (0.0000) loss 4.0460 (4.4915) grad_norm 1.7267 (1.7688) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][30/625] eta 0:03:52 lr 0.001206 wd 0.0500 time 0.3713 (0.3911) data time 0.0008 (0.0153) model time 0.0000 (0.0000) loss 5.1469 (4.5340) grad_norm 1.2702 (1.6509) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][40/625] eta 0:03:46 lr 0.001207 wd 0.0500 time 0.3966 (0.3880) data time 0.0009 (0.0118) model time 0.0000 (0.0000) loss 5.2817 (4.4791) grad_norm 1.8153 (1.7261) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][50/625] eta 0:03:41 lr 0.001209 wd 0.0500 time 0.3679 (0.3856) data time 0.0010 (0.0097) model time 0.0000 (0.0000) loss 4.9905 (4.5125) grad_norm 2.2745 (1.7355) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][60/625] eta 0:03:37 lr 0.001210 wd 0.0500 time 0.3704 (0.3846) data time 0.0011 (0.0083) model time 0.3693 (0.3782) loss 4.4230 (4.4671) grad_norm 1.4636 (1.7692) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][70/625] eta 0:03:34 lr 0.001212 wd 0.0500 time 0.3795 (0.3867) data time 0.0009 (0.0073) model time 0.3787 (0.3884) loss 3.6632 (4.4364) grad_norm 1.8254 (1.7424) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][80/625] eta 0:03:30 lr 0.001214 wd 0.0500 time 0.3627 (0.3856) data time 0.0009 (0.0066) model time 0.3618 (0.3843) loss 4.2902 (4.4834) grad_norm 1.5275 (1.7162) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][90/625] eta 0:03:25 lr 0.001215 wd 0.0500 time 0.3801 (0.3847) data time 0.0012 (0.0060) model time 0.3789 (0.3824) loss 5.0928 (4.5131) grad_norm 2.1733 (1.7005) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:17:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][100/625] eta 0:03:21 lr 0.001217 wd 0.0500 time 0.4010 (0.3841) data time 0.0012 (0.0055) model time 0.3999 (0.3815) loss 4.9471 (4.5340) grad_norm 1.4946 (1.6889) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][110/625] eta 0:03:17 lr 0.001218 wd 0.0500 time 0.3651 (0.3833) data time 0.0010 (0.0051) model time 0.3641 (0.3801) loss 4.5012 (4.5396) grad_norm 1.3143 (1.6938) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][120/625] eta 0:03:13 lr 0.001220 wd 0.0500 time 0.3696 (0.3829) data time 0.0011 (0.0048) model time 0.3685 (0.3797) loss 4.8240 (4.5212) grad_norm 1.8584 (1.6842) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][130/625] eta 0:03:09 lr 0.001222 wd 0.0500 time 0.3910 (0.3827) data time 0.0009 (0.0045) model time 0.3902 (0.3797) loss 4.3203 (4.5345) grad_norm 1.3034 (1.6696) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][140/625] eta 0:03:05 lr 0.001223 wd 0.0500 time 0.3728 (0.3825) data time 0.0013 (0.0043) model time 0.3716 (0.3796) loss 4.6547 (4.5411) grad_norm 1.6728 (1.6752) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][150/625] eta 0:03:01 lr 0.001225 wd 0.0500 time 0.3691 (0.3821) data time 0.0009 (0.0041) model time 0.3681 (0.3791) loss 4.5403 (4.5502) grad_norm 3.1900 (1.6753) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][160/625] eta 0:02:57 lr 0.001226 wd 0.0500 time 0.3848 (0.3821) data time 0.0011 (0.0039) model time 0.3836 (0.3793) loss 4.3698 (4.5527) grad_norm 1.5616 (1.6646) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][170/625] eta 0:02:53 lr 0.001228 wd 0.0500 time 0.3673 (0.3824) data time 0.0010 (0.0037) model time 0.3663 (0.3798) loss 3.6068 (4.5655) grad_norm 1.4487 (1.6612) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][180/625] eta 0:02:50 lr 0.001230 wd 0.0500 time 0.3652 (0.3822) data time 0.0010 (0.0036) model time 0.3642 (0.3797) loss 4.4955 (4.5447) grad_norm 2.1928 (1.6627) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][190/625] eta 0:02:46 lr 0.001231 wd 0.0500 time 0.3863 (0.3820) data time 0.0010 (0.0035) model time 0.3853 (0.3795) loss 4.7456 (4.5498) grad_norm 1.5550 (1.6606) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][200/625] eta 0:02:42 lr 0.001233 wd 0.0500 time 0.3635 (0.3818) data time 0.0011 (0.0034) model time 0.3625 (0.3793) loss 4.8235 (4.5651) grad_norm 1.4583 (1.6574) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][210/625] eta 0:02:38 lr 0.001234 wd 0.0500 time 0.3715 (0.3816) data time 0.0010 (0.0033) model time 0.3706 (0.3791) loss 3.9989 (4.5748) grad_norm 1.5289 (1.6543) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][220/625] eta 0:02:34 lr 0.001236 wd 0.0500 time 0.4060 (0.3814) data time 0.0011 (0.0032) model time 0.4049 (0.3789) loss 3.1795 (4.5455) grad_norm 1.6424 (1.6519) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][230/625] eta 0:02:30 lr 0.001238 wd 0.0500 time 0.3717 (0.3812) data time 0.0010 (0.0031) model time 0.3707 (0.3787) loss 4.9822 (4.5474) grad_norm 1.2175 (1.6464) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][240/625] eta 0:02:26 lr 0.001239 wd 0.0500 time 0.3748 (0.3811) data time 0.0012 (0.0030) model time 0.3736 (0.3787) loss 4.1988 (4.5387) grad_norm 1.5184 (1.6440) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][250/625] eta 0:02:22 lr 0.001241 wd 0.0500 time 0.3808 (0.3809) data time 0.0011 (0.0029) model time 0.3798 (0.3785) loss 3.9175 (4.5404) grad_norm 1.6014 (1.6529) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:18:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][260/625] eta 0:02:18 lr 0.001242 wd 0.0500 time 0.3693 (0.3808) data time 0.0008 (0.0029) model time 0.3685 (0.3784) loss 4.5314 (4.5478) grad_norm 1.2157 (1.6433) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][270/625] eta 0:02:15 lr 0.001244 wd 0.0500 time 0.3807 (0.3807) data time 0.0008 (0.0028) model time 0.3799 (0.3784) loss 4.8406 (4.5412) grad_norm 2.8025 (1.6401) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][280/625] eta 0:02:11 lr 0.001246 wd 0.0500 time 0.3684 (0.3816) data time 0.0010 (0.0027) model time 0.3674 (0.3795) loss 4.2347 (4.5414) grad_norm 2.2412 (1.6608) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][290/625] eta 0:02:07 lr 0.001247 wd 0.0500 time 0.3748 (0.3819) data time 0.0008 (0.0027) model time 0.3740 (0.3799) loss 3.8267 (4.5463) grad_norm 1.3969 (1.6608) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][300/625] eta 0:02:04 lr 0.001249 wd 0.0500 time 0.3844 (0.3817) data time 0.0008 (0.0027) model time 0.3836 (0.3797) loss 5.6365 (4.5430) grad_norm 1.9976 (1.6607) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][310/625] eta 0:02:00 lr 0.001250 wd 0.0500 time 0.3675 (0.3814) data time 0.0010 (0.0026) model time 0.3665 (0.3794) loss 5.1136 (4.5491) grad_norm 1.1640 (1.6598) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][320/625] eta 0:01:56 lr 0.001252 wd 0.0500 time 0.3692 (0.3814) data time 0.0010 (0.0026) model time 0.3682 (0.3793) loss 5.0821 (4.5587) grad_norm 1.6158 (1.6618) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][330/625] eta 0:01:52 lr 0.001254 wd 0.0500 time 0.3937 (0.3813) data time 0.0008 (0.0025) model time 0.3929 (0.3793) loss 3.8415 (4.5513) grad_norm 1.4314 (1.6612) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][340/625] eta 0:01:48 lr 0.001255 wd 0.0500 time 0.3655 (0.3813) data time 0.0011 (0.0025) model time 0.3644 (0.3793) loss 5.2918 (4.5511) grad_norm 1.2403 (1.6630) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][350/625] eta 0:01:44 lr 0.001257 wd 0.0500 time 0.3682 (0.3812) data time 0.0010 (0.0025) model time 0.3672 (0.3792) loss 3.6375 (4.5557) grad_norm 1.8310 (1.6684) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][360/625] eta 0:01:41 lr 0.001258 wd 0.0500 time 0.4008 (0.3812) data time 0.0008 (0.0024) model time 0.3999 (0.3792) loss 4.2016 (4.5516) grad_norm 4.1185 (1.6811) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][370/625] eta 0:01:37 lr 0.001260 wd 0.0500 time 0.3704 (0.3810) data time 0.0011 (0.0024) model time 0.3693 (0.3791) loss 4.6826 (4.5583) grad_norm 1.7197 (1.6819) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][380/625] eta 0:01:33 lr 0.001262 wd 0.0500 time 0.3814 (0.3810) data time 0.0009 (0.0024) model time 0.3805 (0.3790) loss 4.4081 (4.5583) grad_norm 1.5780 (1.6761) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][390/625] eta 0:01:29 lr 0.001263 wd 0.0500 time 0.3817 (0.3810) data time 0.0010 (0.0024) model time 0.3807 (0.3790) loss 4.7072 (4.5572) grad_norm 1.4264 (1.6805) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][400/625] eta 0:01:25 lr 0.001265 wd 0.0500 time 0.3713 (0.3810) data time 0.0011 (0.0024) model time 0.3702 (0.3790) loss 5.0097 (4.5596) grad_norm 1.3171 (1.6807) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:19:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][410/625] eta 0:01:21 lr 0.001266 wd 0.0500 time 0.3756 (0.3810) data time 0.0011 (0.0023) model time 0.3744 (0.3791) loss 4.5750 (4.5662) grad_norm 1.5039 (1.6806) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][420/625] eta 0:01:18 lr 0.001268 wd 0.0500 time 0.3929 (0.3811) data time 0.0010 (0.0023) model time 0.3919 (0.3792) loss 4.2178 (4.5592) grad_norm 1.4105 (1.6830) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][430/625] eta 0:01:14 lr 0.001270 wd 0.0500 time 0.3572 (0.3811) data time 0.0011 (0.0023) model time 0.3562 (0.3792) loss 4.3960 (4.5583) grad_norm 1.4368 (1.6774) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][440/625] eta 0:01:10 lr 0.001271 wd 0.0500 time 0.3730 (0.3810) data time 0.0010 (0.0023) model time 0.3720 (0.3791) loss 5.4292 (4.5620) grad_norm 2.3148 (1.6794) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][450/625] eta 0:01:06 lr 0.001273 wd 0.0500 time 0.3829 (0.3809) data time 0.0011 (0.0022) model time 0.3818 (0.3791) loss 3.8906 (4.5649) grad_norm 1.7933 (1.6788) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][460/625] eta 0:01:02 lr 0.001274 wd 0.0500 time 0.3692 (0.3813) data time 0.0012 (0.0023) model time 0.3680 (0.3795) loss 4.4253 (4.5595) grad_norm 1.8224 (1.6815) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][470/625] eta 0:00:59 lr 0.001276 wd 0.0500 time 0.3741 (0.3812) data time 0.0009 (0.0022) model time 0.3732 (0.3794) loss 3.7370 (4.5515) grad_norm 1.8879 (1.6844) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][480/625] eta 0:00:55 lr 0.001278 wd 0.0500 time 0.3924 (0.3812) data time 0.0010 (0.0022) model time 0.3914 (0.3793) loss 5.0686 (4.5521) grad_norm 1.4066 (1.6864) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][490/625] eta 0:00:51 lr 0.001279 wd 0.0500 time 0.3664 (0.3812) data time 0.0008 (0.0022) model time 0.3656 (0.3794) loss 3.4947 (4.5520) grad_norm 1.4392 (1.6867) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][500/625] eta 0:00:47 lr 0.001281 wd 0.0500 time 0.3698 (0.3816) data time 0.0008 (0.0022) model time 0.3690 (0.3798) loss 4.4022 (4.5524) grad_norm 1.5609 (1.6815) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][510/625] eta 0:00:43 lr 0.001282 wd 0.0500 time 0.3802 (0.3816) data time 0.0013 (0.0022) model time 0.3790 (0.3798) loss 4.3637 (4.5498) grad_norm 1.4174 (1.6767) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][520/625] eta 0:00:40 lr 0.001284 wd 0.0500 time 0.3717 (0.3815) data time 0.0008 (0.0022) model time 0.3709 (0.3797) loss 3.7903 (4.5533) grad_norm 1.9499 (1.6732) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][530/625] eta 0:00:36 lr 0.001286 wd 0.0500 time 0.3792 (0.3815) data time 0.0010 (0.0021) model time 0.3782 (0.3798) loss 3.9614 (4.5509) grad_norm 1.4798 (1.6749) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][540/625] eta 0:00:32 lr 0.001287 wd 0.0500 time 0.3867 (0.3815) data time 0.0010 (0.0021) model time 0.3856 (0.3797) loss 4.1920 (4.5504) grad_norm 1.7924 (1.6747) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][550/625] eta 0:00:28 lr 0.001289 wd 0.0500 time 0.3665 (0.3814) data time 0.0008 (0.0021) model time 0.3657 (0.3797) loss 4.4220 (4.5444) grad_norm 1.4270 (1.6710) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][560/625] eta 0:00:24 lr 0.001290 wd 0.0500 time 0.3721 (0.3814) data time 0.0008 (0.0021) model time 0.3713 (0.3797) loss 4.6571 (4.5427) grad_norm 1.1871 (1.6709) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:20:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][570/625] eta 0:00:20 lr 0.001292 wd 0.0500 time 0.3900 (0.3815) data time 0.0011 (0.0021) model time 0.3889 (0.3797) loss 5.3028 (4.5476) grad_norm 1.2033 (1.6694) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][580/625] eta 0:00:17 lr 0.001294 wd 0.0500 time 0.3626 (0.3818) data time 0.0011 (0.0021) model time 0.3615 (0.3800) loss 4.3982 (4.5473) grad_norm 1.2826 (1.6648) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][590/625] eta 0:00:13 lr 0.001295 wd 0.0500 time 0.3722 (0.3817) data time 0.0010 (0.0021) model time 0.3712 (0.3800) loss 5.5522 (4.5502) grad_norm 2.3127 (1.6644) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][600/625] eta 0:00:09 lr 0.001297 wd 0.0500 time 0.3915 (0.3817) data time 0.0011 (0.0020) model time 0.3904 (0.3800) loss 4.8149 (4.5501) grad_norm 1.7458 (1.6628) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][610/625] eta 0:00:05 lr 0.001298 wd 0.0500 time 0.3536 (0.3815) data time 0.0005 (0.0020) model time 0.3530 (0.3798) loss 4.0100 (4.5476) grad_norm 1.5645 (1.6634) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [12/300][620/625] eta 0:00:01 lr 0.001300 wd 0.0500 time 0.3562 (0.3812) data time 0.0008 (0.0020) model time 0.3555 (0.3795) loss 4.7092 (4.5471) grad_norm 1.4874 (1.6615) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 12 training takes 0:03:58 [2024-08-20 00:21:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:21:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.520 (0.520) Loss 1.2109 (1.2109) Acc@1 72.168 (72.168) Acc@5 90.820 (90.820) Mem 11221MB [2024-08-20 00:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.148) Loss 2.0449 (1.4583) Acc@1 54.248 (64.582) Acc@5 80.273 (88.703) Mem 11221MB [2024-08-20 00:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.122) Loss 2.2539 (1.8062) Acc@1 50.977 (58.419) Acc@5 74.951 (82.938) Mem 11221MB [2024-08-20 00:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 58.289 Acc@5 82.837 [2024-08-20 00:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 58.3% [2024-08-20 00:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 58.29% [2024-08-20 00:21:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-20 00:21:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-20 00:21:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.507 (0.507) Loss 5.6016 (5.6016) Acc@1 4.639 (4.639) Acc@5 17.383 (17.383) Mem 11221MB [2024-08-20 00:21:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.145) Loss 5.8828 (5.7777) Acc@1 4.492 (4.976) Acc@5 14.307 (15.057) Mem 11221MB [2024-08-20 00:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 5.8477 (5.7933) Acc@1 5.029 (5.160) Acc@5 13.379 (14.946) Mem 11221MB [2024-08-20 00:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 5.852 Acc@5 16.363 [2024-08-20 00:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 5.9% [2024-08-20 00:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 5.85% [2024-08-20 00:21:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-20 00:21:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-20 00:21:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][0/625] eta 0:07:31 lr 0.001301 wd 0.0500 time 0.7231 (0.7231) data time 0.3711 (0.3711) model time 0.0000 (0.0000) loss 4.9168 (4.9168) grad_norm 1.3104 (1.3104) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][10/625] eta 0:04:11 lr 0.001302 wd 0.0500 time 0.3989 (0.4097) data time 0.0008 (0.0350) model time 0.0000 (0.0000) loss 4.0178 (4.5991) grad_norm 1.5224 (1.4615) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][20/625] eta 0:03:58 lr 0.001304 wd 0.0500 time 0.3676 (0.3935) data time 0.0013 (0.0190) model time 0.0000 (0.0000) loss 4.9113 (4.6767) grad_norm 1.4366 (1.4315) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][30/625] eta 0:03:50 lr 0.001305 wd 0.0500 time 0.3680 (0.3880) data time 0.0009 (0.0133) model time 0.0000 (0.0000) loss 5.4872 (4.6051) grad_norm 2.0247 (1.5481) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][40/625] eta 0:03:45 lr 0.001307 wd 0.0500 time 0.3886 (0.3852) data time 0.0010 (0.0103) model time 0.0000 (0.0000) loss 4.4937 (4.5588) grad_norm 0.9820 (1.5137) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][50/625] eta 0:03:40 lr 0.001309 wd 0.0500 time 0.3689 (0.3840) data time 0.0008 (0.0085) model time 0.0000 (0.0000) loss 4.4484 (4.5680) grad_norm 3.8912 (1.5755) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][60/625] eta 0:03:36 lr 0.001310 wd 0.0500 time 0.3817 (0.3834) data time 0.0011 (0.0074) model time 0.3807 (0.3783) loss 4.8061 (4.5828) grad_norm 1.2639 (1.6379) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][70/625] eta 0:03:32 lr 0.001312 wd 0.0500 time 0.3745 (0.3829) data time 0.0008 (0.0066) model time 0.3737 (0.3784) loss 4.6514 (4.5534) grad_norm 1.7063 (1.6353) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:21:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][80/625] eta 0:03:28 lr 0.001313 wd 0.0500 time 0.3728 (0.3820) data time 0.0008 (0.0059) model time 0.3720 (0.3771) loss 3.9204 (4.5555) grad_norm 1.4155 (1.6265) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][90/625] eta 0:03:23 lr 0.001315 wd 0.0500 time 0.3794 (0.3811) data time 0.0008 (0.0054) model time 0.3785 (0.3761) loss 4.7834 (4.5374) grad_norm 1.4067 (1.6150) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][100/625] eta 0:03:21 lr 0.001317 wd 0.0500 time 0.3992 (0.3831) data time 0.0009 (0.0049) model time 0.3983 (0.3808) loss 3.7023 (4.5318) grad_norm 1.5067 (1.6346) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][110/625] eta 0:03:17 lr 0.001318 wd 0.0500 time 0.3679 (0.3825) data time 0.0011 (0.0046) model time 0.3669 (0.3800) loss 4.0737 (4.5121) grad_norm 1.4350 (1.6149) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][120/625] eta 0:03:12 lr 0.001320 wd 0.0500 time 0.3654 (0.3818) data time 0.0011 (0.0043) model time 0.3643 (0.3789) loss 4.7647 (4.5162) grad_norm 1.9403 (1.6021) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][130/625] eta 0:03:08 lr 0.001321 wd 0.0500 time 0.3870 (0.3815) data time 0.0008 (0.0041) model time 0.3862 (0.3787) loss 5.0951 (4.5211) grad_norm 2.1771 (1.6062) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][140/625] eta 0:03:04 lr 0.001323 wd 0.0500 time 0.3717 (0.3810) data time 0.0012 (0.0039) model time 0.3704 (0.3780) loss 5.0044 (4.5446) grad_norm 1.5298 (1.6138) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][150/625] eta 0:03:00 lr 0.001325 wd 0.0500 time 0.3726 (0.3806) data time 0.0011 (0.0037) model time 0.3715 (0.3776) loss 4.1333 (4.5484) grad_norm 1.5643 (1.6028) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][160/625] eta 0:02:56 lr 0.001326 wd 0.0500 time 0.3828 (0.3804) data time 0.0009 (0.0035) model time 0.3819 (0.3774) loss 4.4832 (4.5536) grad_norm 1.2881 (1.5993) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][170/625] eta 0:02:53 lr 0.001328 wd 0.0500 time 0.3643 (0.3809) data time 0.0008 (0.0034) model time 0.3634 (0.3783) loss 4.4932 (4.5573) grad_norm 1.9373 (1.5942) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][180/625] eta 0:02:49 lr 0.001329 wd 0.0500 time 0.3839 (0.3806) data time 0.0011 (0.0033) model time 0.3828 (0.3781) loss 3.8296 (4.5540) grad_norm 1.4910 (1.6020) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][190/625] eta 0:02:45 lr 0.001331 wd 0.0500 time 0.3822 (0.3805) data time 0.0013 (0.0032) model time 0.3809 (0.3780) loss 3.3376 (4.5398) grad_norm 1.5231 (1.6105) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][200/625] eta 0:02:41 lr 0.001333 wd 0.0500 time 0.3741 (0.3804) data time 0.0008 (0.0031) model time 0.3733 (0.3779) loss 5.1023 (4.5465) grad_norm 1.2483 (1.6205) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][210/625] eta 0:02:37 lr 0.001334 wd 0.0500 time 0.3750 (0.3802) data time 0.0008 (0.0030) model time 0.3742 (0.3778) loss 4.8338 (4.5447) grad_norm 1.7123 (1.6224) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][220/625] eta 0:02:34 lr 0.001336 wd 0.0500 time 0.3903 (0.3803) data time 0.0008 (0.0030) model time 0.3895 (0.3778) loss 3.7738 (4.5289) grad_norm 1.5595 (1.6266) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][230/625] eta 0:02:30 lr 0.001337 wd 0.0500 time 0.3680 (0.3801) data time 0.0010 (0.0029) model time 0.3670 (0.3777) loss 4.3393 (4.5325) grad_norm 1.9872 (1.6259) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:22:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][240/625] eta 0:02:26 lr 0.001339 wd 0.0500 time 0.3876 (0.3801) data time 0.0011 (0.0028) model time 0.3865 (0.3778) loss 5.0507 (4.5414) grad_norm 1.2078 (1.6202) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:23:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][250/625] eta 0:02:22 lr 0.001341 wd 0.0500 time 0.3823 (0.3800) data time 0.0008 (0.0027) model time 0.3815 (0.3777) loss 4.2349 (4.5403) grad_norm 2.2417 (1.6201) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:23:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][260/625] eta 0:02:18 lr 0.001342 wd 0.0500 time 0.3751 (0.3799) data time 0.0008 (0.0027) model time 0.3743 (0.3776) loss 4.7697 (4.5396) grad_norm 2.0955 (1.6193) loss_scale 16384.0000 (16384.0000) mem 11221MB [2024-08-20 00:23:09 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-20 00:23:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:23:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:25:13 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-20 00:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-20 00:25:16 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-20 00:25:35 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-20 00:25:35 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-20 00:25:38 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-20 00:25:40 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-20 00:25:40 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 13) [2024-08-20 00:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-20 00:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][270/625] eta 0:17:45 lr 0.001344 wd 0.0500 time 0.3439 (3.0015) data time 0.0006 (0.1860) model time 0.3433 (2.8154) loss 5.2671 (5.0725) grad_norm 1.5899 (1.6224) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][280/625] eta 0:07:07 lr 0.001345 wd 0.0500 time 0.3545 (1.2395) data time 0.0008 (0.0636) model time 0.3537 (1.1759) loss 4.9875 (4.8729) grad_norm 2.0620 (1.8113) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][290/625] eta 0:04:59 lr 0.001347 wd 0.0500 time 0.3413 (0.8928) data time 0.0009 (0.0385) model time 0.3403 (0.8543) loss 5.0183 (4.8197) grad_norm 1.2303 (1.7007) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][300/625] eta 0:03:59 lr 0.001349 wd 0.0500 time 0.3464 (0.7378) data time 0.0008 (0.0278) model time 0.3456 (0.7101) loss 4.9390 (4.8069) grad_norm 1.4843 (1.7493) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][310/625] eta 0:03:27 lr 0.001350 wd 0.0500 time 0.3867 (0.6585) data time 0.0009 (0.0218) model time 0.3858 (0.6367) loss 5.0401 (4.7501) grad_norm 1.6038 (1.7243) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][320/625] eta 0:03:03 lr 0.001352 wd 0.0500 time 0.3419 (0.6025) data time 0.0007 (0.0180) model time 0.3413 (0.5846) loss 3.4965 (4.7135) grad_norm 1.4162 (1.6533) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][330/625] eta 0:02:46 lr 0.001353 wd 0.0500 time 0.3518 (0.5638) data time 0.0008 (0.0153) model time 0.3510 (0.5484) loss 4.7925 (4.6924) grad_norm 1.5349 (1.6329) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][340/625] eta 0:02:32 lr 0.001355 wd 0.0500 time 0.3621 (0.5356) data time 0.0008 (0.0134) model time 0.3613 (0.5222) loss 3.9044 (4.6409) grad_norm 1.4589 (1.6133) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][350/625] eta 0:02:21 lr 0.001357 wd 0.0500 time 0.3354 (0.5139) data time 0.0006 (0.0121) model time 0.3348 (0.5018) loss 4.4531 (4.6162) grad_norm 1.3387 (1.5961) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][360/625] eta 0:02:11 lr 0.001358 wd 0.0500 time 0.3550 (0.4969) data time 0.0008 (0.0109) model time 0.3541 (0.4859) loss 4.4831 (4.6162) grad_norm 1.6310 (1.6073) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][370/625] eta 0:02:03 lr 0.001360 wd 0.0500 time 0.3652 (0.4831) data time 0.0008 (0.0100) model time 0.3644 (0.4732) loss 3.9514 (4.6388) grad_norm 1.2083 (1.6197) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][380/625] eta 0:01:55 lr 0.001361 wd 0.0500 time 0.3386 (0.4715) data time 0.0007 (0.0092) model time 0.3379 (0.4623) loss 4.0995 (4.6364) grad_norm 1.5799 (1.6157) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][390/625] eta 0:01:48 lr 0.001363 wd 0.0500 time 0.3507 (0.4616) data time 0.0007 (0.0085) model time 0.3500 (0.4531) loss 3.8607 (4.6294) grad_norm 1.5868 (1.6458) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][400/625] eta 0:01:42 lr 0.001365 wd 0.0500 time 0.3609 (0.4536) data time 0.0006 (0.0080) model time 0.3603 (0.4456) loss 4.4370 (4.6215) grad_norm 2.0495 (1.6364) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][410/625] eta 0:01:35 lr 0.001366 wd 0.0500 time 0.3399 (0.4464) data time 0.0009 (0.0075) model time 0.3390 (0.4388) loss 4.7123 (4.6066) grad_norm 1.6816 (1.6309) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][420/625] eta 0:01:30 lr 0.001368 wd 0.0500 time 0.3417 (0.4400) data time 0.0008 (0.0071) model time 0.3408 (0.4329) loss 4.9361 (4.5989) grad_norm 1.4245 (1.6248) loss_scale 16384.0000 (16384.0000) mem 11219MB [2024-08-20 00:26:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][430/625] eta 0:01:24 lr 0.001369 wd 0.0500 time 0.3524 (0.4346) data time 0.0008 (0.0067) model time 0.3515 (0.4278) loss 5.0202 (4.5996) grad_norm 1.3062 (1.6143) loss_scale 32768.0000 (16483.2970) mem 11219MB [2024-08-20 00:26:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][440/625] eta 0:01:19 lr 0.001371 wd 0.0500 time 0.3378 (0.4296) data time 0.0007 (0.0064) model time 0.3371 (0.4232) loss 4.6604 (4.5869) grad_norm 1.7371 (1.6095) loss_scale 32768.0000 (17413.8514) mem 11219MB [2024-08-20 00:27:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][450/625] eta 0:01:14 lr 0.001373 wd 0.0500 time 0.3525 (0.4254) data time 0.0009 (0.0061) model time 0.3516 (0.4193) loss 4.9244 (4.5746) grad_norm 1.7182 (1.6020) loss_scale 32768.0000 (18243.8054) mem 11219MB [2024-08-20 00:27:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][460/625] eta 0:01:09 lr 0.001374 wd 0.0500 time 0.3762 (0.4219) data time 0.0008 (0.0058) model time 0.3754 (0.4161) loss 3.2402 (4.5592) grad_norm 1.3461 (inf) loss_scale 16384.0000 (18232.4513) mem 11219MB [2024-08-20 00:27:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][470/625] eta 0:01:04 lr 0.001376 wd 0.0500 time 0.3413 (0.4185) data time 0.0009 (0.0056) model time 0.3405 (0.4128) loss 3.6021 (4.5372) grad_norm 1.5505 (inf) loss_scale 16384.0000 (18142.2829) mem 11219MB [2024-08-20 00:27:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][480/625] eta 0:01:00 lr 0.001377 wd 0.0500 time 0.3480 (0.4154) data time 0.0008 (0.0054) model time 0.3472 (0.4099) loss 4.4016 (4.5308) grad_norm 1.4402 (inf) loss_scale 16384.0000 (18060.5023) mem 11219MB [2024-08-20 00:27:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][490/625] eta 0:00:55 lr 0.001379 wd 0.0500 time 0.5675 (0.4144) data time 0.0006 (0.0052) model time 0.5669 (0.4092) loss 4.6775 (4.5331) grad_norm 1.3772 (inf) loss_scale 16384.0000 (17985.9911) mem 11219MB [2024-08-20 00:27:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][500/625] eta 0:00:51 lr 0.001381 wd 0.0500 time 0.3377 (0.4123) data time 0.0008 (0.0050) model time 0.3369 (0.4073) loss 4.7606 (4.5231) grad_norm 1.6333 (inf) loss_scale 16384.0000 (17917.8213) mem 11219MB [2024-08-20 00:27:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][510/625] eta 0:00:47 lr 0.001382 wd 0.0500 time 0.3476 (0.4097) data time 0.0010 (0.0049) model time 0.3466 (0.4049) loss 4.3998 (4.5266) grad_norm 1.8848 (inf) loss_scale 16384.0000 (17855.2163) mem 11219MB [2024-08-20 00:27:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][520/625] eta 0:00:42 lr 0.001384 wd 0.0500 time 0.3606 (0.4074) data time 0.0006 (0.0047) model time 0.3600 (0.4027) loss 4.4649 (4.5197) grad_norm 1.3068 (inf) loss_scale 16384.0000 (17797.5216) mem 11219MB [2024-08-20 00:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][530/625] eta 0:00:38 lr 0.001385 wd 0.0500 time 0.3389 (0.4052) data time 0.0007 (0.0045) model time 0.3382 (0.4007) loss 3.3241 (4.5091) grad_norm 1.4390 (inf) loss_scale 16384.0000 (17744.1811) mem 11219MB [2024-08-20 00:27:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][540/625] eta 0:00:34 lr 0.001387 wd 0.0500 time 0.3449 (0.4032) data time 0.0008 (0.0044) model time 0.3440 (0.3988) loss 4.8099 (4.5080) grad_norm 1.9744 (inf) loss_scale 16384.0000 (17694.7200) mem 11219MB [2024-08-20 00:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][550/625] eta 0:00:30 lr 0.001389 wd 0.0500 time 0.3545 (0.4014) data time 0.0008 (0.0043) model time 0.3537 (0.3971) loss 4.5932 (4.5044) grad_norm 1.3899 (inf) loss_scale 16384.0000 (17648.7298) mem 11219MB [2024-08-20 00:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][560/625] eta 0:00:25 lr 0.001390 wd 0.0500 time 0.3378 (0.3995) data time 0.0008 (0.0042) model time 0.3370 (0.3954) loss 4.4909 (4.4995) grad_norm 1.6493 (inf) loss_scale 16384.0000 (17605.8576) mem 11219MB [2024-08-20 00:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][570/625] eta 0:00:21 lr 0.001392 wd 0.0500 time 0.3516 (0.3981) data time 0.0008 (0.0041) model time 0.3508 (0.3940) loss 3.8884 (4.4908) grad_norm 1.7970 (inf) loss_scale 16384.0000 (17565.7967) mem 11219MB [2024-08-20 00:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][580/625] eta 0:00:17 lr 0.001393 wd 0.0500 time 0.3602 (0.3966) data time 0.0008 (0.0040) model time 0.3593 (0.3927) loss 4.7031 (4.4924) grad_norm 1.2953 (inf) loss_scale 16384.0000 (17528.2794) mem 11219MB [2024-08-20 00:27:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][590/625] eta 0:00:13 lr 0.001395 wd 0.0500 time 0.3430 (0.3953) data time 0.0007 (0.0040) model time 0.3423 (0.3913) loss 5.4716 (4.5044) grad_norm 2.4399 (inf) loss_scale 16384.0000 (17493.0708) mem 11219MB [2024-08-20 00:27:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][600/625] eta 0:00:09 lr 0.001397 wd 0.0500 time 0.3478 (0.3939) data time 0.0006 (0.0039) model time 0.3472 (0.3901) loss 4.9211 (4.5026) grad_norm 1.0679 (inf) loss_scale 16384.0000 (17459.9642) mem 11219MB [2024-08-20 00:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][610/625] eta 0:00:05 lr 0.001398 wd 0.0500 time 0.3347 (0.3939) data time 0.0004 (0.0038) model time 0.3343 (0.3901) loss 3.9121 (4.4989) grad_norm 1.5120 (inf) loss_scale 16384.0000 (17428.7768) mem 11219MB [2024-08-20 00:28:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [13/300][620/625] eta 0:00:01 lr 0.001400 wd 0.0500 time 0.3383 (0.3924) data time 0.0007 (0.0037) model time 0.3376 (0.3887) loss 4.2815 (4.5056) grad_norm 1.5366 (inf) loss_scale 16384.0000 (17399.3465) mem 11219MB [2024-08-20 00:28:05 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 13 training takes 0:02:20 [2024-08-20 00:28:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:28:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.442 (0.442) Loss 1.2773 (1.2773) Acc@1 70.703 (70.703) Acc@5 91.748 (91.748) Mem 11219MB [2024-08-20 00:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 2.0410 (1.4789) Acc@1 55.420 (65.678) Acc@5 79.346 (88.943) Mem 11219MB [2024-08-20 00:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 2.1602 (1.7968) Acc@1 51.709 (59.561) Acc@5 78.857 (83.694) Mem 11219MB [2024-08-20 00:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 59.441 Acc@5 83.599 [2024-08-20 00:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 59.4% [2024-08-20 00:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 59.44% [2024-08-20 00:28:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-20 00:28:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-20 00:28:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.422 (0.422) Loss 4.1406 (4.1406) Acc@1 26.611 (26.611) Acc@5 50.293 (50.293) Mem 11219MB [2024-08-20 00:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.128) Loss 4.4531 (4.0939) Acc@1 20.459 (24.738) Acc@5 43.018 (49.743) Mem 11219MB [2024-08-20 00:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.110) Loss 4.4258 (4.1989) Acc@1 20.166 (24.140) Acc@5 42.188 (48.117) Mem 11219MB [2024-08-20 00:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 25.282 Acc@5 49.630 [2024-08-20 00:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 25.3% [2024-08-20 00:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 25.28% [2024-08-20 00:28:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-20 00:28:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-20 00:28:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][0/625] eta 0:07:58 lr 0.001401 wd 0.0500 time 0.7650 (0.7650) data time 0.3971 (0.3971) model time 0.0000 (0.0000) loss 5.4168 (5.4168) grad_norm 2.1982 (2.1982) loss_scale 16384.0000 (16384.0000) mem 11227MB [2024-08-20 00:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][10/625] eta 0:04:00 lr 0.001402 wd 0.0500 time 0.3649 (0.3909) data time 0.0009 (0.0370) model time 0.0000 (0.0000) loss 5.0496 (4.5115) grad_norm 1.8001 (1.6002) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:28:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][20/625] eta 0:03:45 lr 0.001404 wd 0.0500 time 0.3376 (0.3723) data time 0.0007 (0.0198) model time 0.0000 (0.0000) loss 4.6531 (4.4410) grad_norm 2.2571 (1.6215) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:28:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][30/625] eta 0:03:37 lr 0.001405 wd 0.0500 time 0.3453 (0.3649) data time 0.0008 (0.0137) model time 0.0000 (0.0000) loss 3.7945 (4.3195) grad_norm 1.2204 (1.6615) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:28:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][40/625] eta 0:03:31 lr 0.001407 wd 0.0500 time 0.3609 (0.3614) data time 0.0008 (0.0106) model time 0.0000 (0.0000) loss 4.0423 (4.4142) grad_norm 1.4468 (1.6555) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:28:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][50/625] eta 0:03:26 lr 0.001409 wd 0.0500 time 0.3371 (0.3591) data time 0.0007 (0.0087) model time 0.0000 (0.0000) loss 4.5045 (4.4525) grad_norm 1.1388 (1.6297) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:28:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][60/625] eta 0:03:22 lr 0.001410 wd 0.0500 time 0.3431 (0.3579) data time 0.0007 (0.0074) model time 0.3424 (0.3513) loss 4.9997 (4.4362) grad_norm 1.2445 (1.6032) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:28:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][70/625] eta 0:03:18 lr 0.001412 wd 0.0500 time 0.3566 (0.3573) data time 0.0008 (0.0065) model time 0.3557 (0.3518) loss 4.4851 (4.4917) grad_norm 1.5028 (1.5801) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][80/625] eta 0:03:14 lr 0.001413 wd 0.0500 time 0.3386 (0.3563) data time 0.0008 (0.0058) model time 0.3378 (0.3508) loss 4.1716 (4.5109) grad_norm 2.1363 (1.5688) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][90/625] eta 0:03:10 lr 0.001415 wd 0.0500 time 0.3519 (0.3559) data time 0.0007 (0.0053) model time 0.3512 (0.3509) loss 4.8143 (4.5052) grad_norm 1.2303 (1.5602) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][100/625] eta 0:03:06 lr 0.001417 wd 0.0500 time 0.3616 (0.3553) data time 0.0007 (0.0048) model time 0.3609 (0.3505) loss 4.8352 (4.4820) grad_norm 1.9192 (1.5899) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][110/625] eta 0:03:02 lr 0.001418 wd 0.0500 time 0.3359 (0.3547) data time 0.0009 (0.0045) model time 0.3350 (0.3501) loss 4.0557 (4.4469) grad_norm 1.6059 (1.5762) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][120/625] eta 0:02:58 lr 0.001420 wd 0.0500 time 0.3389 (0.3543) data time 0.0011 (0.0042) model time 0.3378 (0.3499) loss 4.0828 (4.4410) grad_norm 1.4019 (1.5737) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][130/625] eta 0:02:55 lr 0.001421 wd 0.0500 time 0.3722 (0.3542) data time 0.0006 (0.0040) model time 0.3716 (0.3502) loss 4.5380 (4.4646) grad_norm 1.6945 (1.5696) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][140/625] eta 0:02:51 lr 0.001423 wd 0.0500 time 0.3373 (0.3538) data time 0.0008 (0.0038) model time 0.3366 (0.3499) loss 3.2725 (4.4651) grad_norm 1.9336 (1.5606) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][150/625] eta 0:02:48 lr 0.001425 wd 0.0500 time 0.3538 (0.3538) data time 0.0008 (0.0036) model time 0.3529 (0.3502) loss 4.8678 (4.4835) grad_norm 1.2915 (1.5475) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][160/625] eta 0:02:44 lr 0.001426 wd 0.0500 time 0.3671 (0.3535) data time 0.0006 (0.0034) model time 0.3665 (0.3500) loss 4.4750 (4.4878) grad_norm 2.1070 (1.5393) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][170/625] eta 0:02:41 lr 0.001428 wd 0.0500 time 0.3422 (0.3557) data time 0.0008 (0.0033) model time 0.3414 (0.3533) loss 4.3774 (4.4712) grad_norm 1.2074 (1.5785) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][180/625] eta 0:02:38 lr 0.001429 wd 0.0500 time 0.3610 (0.3565) data time 0.0007 (0.0031) model time 0.3603 (0.3545) loss 4.4901 (4.4695) grad_norm 1.5787 (1.5690) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][190/625] eta 0:02:34 lr 0.001431 wd 0.0500 time 0.3431 (0.3562) data time 0.0008 (0.0030) model time 0.3423 (0.3542) loss 3.4349 (4.4714) grad_norm 1.7045 (1.5719) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][200/625] eta 0:02:31 lr 0.001433 wd 0.0500 time 0.3457 (0.3559) data time 0.0009 (0.0029) model time 0.3448 (0.3539) loss 4.4589 (4.4810) grad_norm 2.4579 (1.5847) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][210/625] eta 0:02:27 lr 0.001434 wd 0.0500 time 0.3556 (0.3557) data time 0.0008 (0.0028) model time 0.3548 (0.3536) loss 4.6718 (4.4916) grad_norm 1.6515 (1.5797) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][220/625] eta 0:02:23 lr 0.001436 wd 0.0500 time 0.3466 (0.3553) data time 0.0007 (0.0027) model time 0.3459 (0.3532) loss 3.9531 (4.4921) grad_norm 2.0024 (1.5860) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][230/625] eta 0:02:20 lr 0.001437 wd 0.0500 time 0.3451 (0.3550) data time 0.0006 (0.0027) model time 0.3445 (0.3529) loss 5.2549 (4.5004) grad_norm 1.9122 (1.6057) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:29:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][240/625] eta 0:02:16 lr 0.001439 wd 0.0500 time 0.3526 (0.3549) data time 0.0008 (0.0026) model time 0.3518 (0.3528) loss 4.6784 (4.4996) grad_norm 1.2389 (1.5970) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][250/625] eta 0:02:13 lr 0.001441 wd 0.0500 time 0.3496 (0.3548) data time 0.0006 (0.0025) model time 0.3490 (0.3528) loss 4.4247 (4.4904) grad_norm 1.5097 (1.5879) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][260/625] eta 0:02:09 lr 0.001442 wd 0.0500 time 0.3482 (0.3546) data time 0.0008 (0.0025) model time 0.3474 (0.3526) loss 4.6102 (4.4910) grad_norm 1.6191 (1.5853) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][270/625] eta 0:02:05 lr 0.001444 wd 0.0500 time 0.3541 (0.3545) data time 0.0007 (0.0024) model time 0.3534 (0.3525) loss 5.5475 (4.4928) grad_norm 1.2141 (1.5771) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][280/625] eta 0:02:02 lr 0.001445 wd 0.0500 time 0.3507 (0.3543) data time 0.0006 (0.0024) model time 0.3501 (0.3523) loss 2.8603 (4.4863) grad_norm 1.3600 (1.5773) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][290/625] eta 0:01:59 lr 0.001447 wd 0.0500 time 0.3497 (0.3554) data time 0.0008 (0.0023) model time 0.3489 (0.3536) loss 5.1481 (4.4728) grad_norm 1.7265 (1.5684) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][300/625] eta 0:01:55 lr 0.001449 wd 0.0500 time 0.3718 (0.3559) data time 0.0007 (0.0023) model time 0.3711 (0.3543) loss 4.1853 (4.4656) grad_norm 1.4193 (1.5660) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][310/625] eta 0:01:52 lr 0.001450 wd 0.0500 time 0.3450 (0.3557) data time 0.0007 (0.0022) model time 0.3444 (0.3541) loss 4.8200 (4.4658) grad_norm 2.5646 (1.5671) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][320/625] eta 0:01:48 lr 0.001452 wd 0.0500 time 0.3516 (0.3556) data time 0.0008 (0.0022) model time 0.3508 (0.3540) loss 3.6871 (4.4705) grad_norm 1.6798 (1.5706) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][330/625] eta 0:01:44 lr 0.001453 wd 0.0500 time 0.3545 (0.3555) data time 0.0006 (0.0022) model time 0.3539 (0.3538) loss 3.3186 (4.4672) grad_norm 1.5478 (1.5774) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][340/625] eta 0:01:41 lr 0.001455 wd 0.0500 time 0.3488 (0.3553) data time 0.0006 (0.0021) model time 0.3481 (0.3536) loss 3.7328 (4.4580) grad_norm 1.2659 (1.5769) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][350/625] eta 0:01:37 lr 0.001457 wd 0.0500 time 0.3495 (0.3553) data time 0.0008 (0.0021) model time 0.3486 (0.3537) loss 4.6212 (4.4594) grad_norm 1.6663 (1.5746) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][360/625] eta 0:01:34 lr 0.001458 wd 0.0500 time 0.3809 (0.3553) data time 0.0008 (0.0021) model time 0.3802 (0.3537) loss 3.4733 (4.4495) grad_norm 1.1341 (1.5717) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][370/625] eta 0:01:30 lr 0.001460 wd 0.0500 time 0.3544 (0.3553) data time 0.0008 (0.0020) model time 0.3535 (0.3537) loss 4.7746 (4.4543) grad_norm 2.5790 (1.5718) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][380/625] eta 0:01:27 lr 0.001461 wd 0.0500 time 0.3428 (0.3552) data time 0.0009 (0.0020) model time 0.3419 (0.3536) loss 5.0962 (4.4648) grad_norm 2.9022 (1.5811) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][390/625] eta 0:01:23 lr 0.001463 wd 0.0500 time 0.3510 (0.3552) data time 0.0009 (0.0020) model time 0.3502 (0.3536) loss 4.1468 (4.4618) grad_norm 1.3409 (1.5784) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][400/625] eta 0:01:19 lr 0.001465 wd 0.0500 time 0.3470 (0.3550) data time 0.0009 (0.0020) model time 0.3461 (0.3535) loss 4.7158 (4.4638) grad_norm 1.1500 (1.5717) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:30:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][410/625] eta 0:01:16 lr 0.001466 wd 0.0500 time 0.3435 (0.3549) data time 0.0009 (0.0019) model time 0.3426 (0.3533) loss 4.9542 (4.4631) grad_norm 1.3539 (1.5707) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][420/625] eta 0:01:12 lr 0.001468 wd 0.0500 time 0.3545 (0.3548) data time 0.0008 (0.0019) model time 0.3537 (0.3533) loss 4.8214 (4.4680) grad_norm 1.5068 (1.5794) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][430/625] eta 0:01:09 lr 0.001469 wd 0.0500 time 0.3435 (0.3547) data time 0.0009 (0.0019) model time 0.3426 (0.3532) loss 4.5337 (4.4670) grad_norm 1.1135 (1.5792) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][440/625] eta 0:01:05 lr 0.001471 wd 0.0500 time 0.3438 (0.3546) data time 0.0008 (0.0019) model time 0.3430 (0.3530) loss 4.1023 (4.4652) grad_norm 1.8515 (1.5929) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][450/625] eta 0:01:02 lr 0.001473 wd 0.0500 time 0.3588 (0.3544) data time 0.0006 (0.0018) model time 0.3582 (0.3529) loss 3.9779 (4.4544) grad_norm 1.6567 (1.5922) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][460/625] eta 0:00:58 lr 0.001474 wd 0.0500 time 0.3506 (0.3544) data time 0.0006 (0.0018) model time 0.3500 (0.3528) loss 3.1424 (4.4533) grad_norm 1.8089 (1.5935) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][470/625] eta 0:00:54 lr 0.001476 wd 0.0500 time 0.3420 (0.3543) data time 0.0009 (0.0018) model time 0.3412 (0.3528) loss 4.6511 (4.4459) grad_norm 1.0938 (1.5890) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][480/625] eta 0:00:51 lr 0.001477 wd 0.0500 time 0.3484 (0.3542) data time 0.0006 (0.0018) model time 0.3478 (0.3526) loss 3.7490 (4.4409) grad_norm 1.2270 (1.5849) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][490/625] eta 0:00:47 lr 0.001479 wd 0.0500 time 0.3447 (0.3541) data time 0.0007 (0.0018) model time 0.3441 (0.3526) loss 5.3272 (4.4382) grad_norm 1.9277 (1.5843) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][500/625] eta 0:00:44 lr 0.001481 wd 0.0500 time 0.5795 (0.3553) data time 0.0007 (0.0018) model time 0.5788 (0.3539) loss 3.6738 (4.4373) grad_norm 1.3535 (1.5809) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][510/625] eta 0:00:40 lr 0.001482 wd 0.0500 time 0.3658 (0.3553) data time 0.0006 (0.0018) model time 0.3651 (0.3538) loss 4.8680 (4.4381) grad_norm 1.5820 (1.5805) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][520/625] eta 0:00:37 lr 0.001484 wd 0.0500 time 0.3393 (0.3552) data time 0.0008 (0.0018) model time 0.3385 (0.3537) loss 3.2466 (4.4318) grad_norm 1.3128 (1.5758) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][530/625] eta 0:00:33 lr 0.001485 wd 0.0500 time 0.3414 (0.3550) data time 0.0008 (0.0017) model time 0.3406 (0.3536) loss 3.0300 (4.4316) grad_norm 2.0906 (1.5756) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][540/625] eta 0:00:30 lr 0.001487 wd 0.0500 time 0.3638 (0.3550) data time 0.0007 (0.0017) model time 0.3631 (0.3535) loss 5.3897 (4.4278) grad_norm 1.6901 (1.5843) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][550/625] eta 0:00:26 lr 0.001489 wd 0.0500 time 0.3413 (0.3549) data time 0.0006 (0.0017) model time 0.3407 (0.3534) loss 4.8584 (4.4295) grad_norm 1.4855 (1.5851) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][560/625] eta 0:00:23 lr 0.001490 wd 0.0500 time 0.3497 (0.3548) data time 0.0008 (0.0017) model time 0.3489 (0.3534) loss 4.4747 (4.4285) grad_norm 1.7254 (1.5825) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:31:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][570/625] eta 0:00:19 lr 0.001492 wd 0.0500 time 0.3651 (0.3547) data time 0.0009 (0.0017) model time 0.3642 (0.3533) loss 3.9995 (4.4326) grad_norm 1.0478 (1.5793) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][580/625] eta 0:00:15 lr 0.001493 wd 0.0500 time 0.3500 (0.3547) data time 0.0009 (0.0017) model time 0.3491 (0.3533) loss 4.4351 (4.4301) grad_norm 1.5515 (1.5765) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][590/625] eta 0:00:12 lr 0.001495 wd 0.0500 time 0.3473 (0.3546) data time 0.0006 (0.0017) model time 0.3467 (0.3532) loss 3.3857 (4.4234) grad_norm 1.4576 (1.5801) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][600/625] eta 0:00:08 lr 0.001497 wd 0.0500 time 0.3545 (0.3545) data time 0.0007 (0.0016) model time 0.3537 (0.3531) loss 5.5447 (4.4228) grad_norm 1.3134 (1.5750) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][610/625] eta 0:00:05 lr 0.001498 wd 0.0500 time 0.3325 (0.3544) data time 0.0007 (0.0016) model time 0.3318 (0.3530) loss 4.5344 (4.4244) grad_norm 1.4134 (1.5757) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [14/300][620/625] eta 0:00:01 lr 0.001500 wd 0.0500 time 0.3388 (0.3542) data time 0.0004 (0.0016) model time 0.3384 (0.3528) loss 4.9688 (4.4239) grad_norm 1.4667 (1.5730) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:15 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 14 training takes 0:03:41 [2024-08-20 00:32:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:32:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-20 00:32:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.437 (0.437) Loss 1.1572 (1.1572) Acc@1 73.926 (73.926) Acc@5 91.943 (91.943) Mem 11224MB [2024-08-20 00:32:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 2.0488 (1.3978) Acc@1 53.809 (66.970) Acc@5 80.225 (89.897) Mem 11224MB [2024-08-20 00:32:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 2.1191 (1.6787) Acc@1 51.514 (61.686) Acc@5 77.979 (85.110) Mem 11224MB [2024-08-20 00:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 61.708 Acc@5 85.089 [2024-08-20 00:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 61.7% [2024-08-20 00:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 61.71% [2024-08-20 00:32:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-20 00:32:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-20 00:32:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.435 (0.435) Loss 3.5859 (3.5859) Acc@1 34.180 (34.180) Acc@5 59.375 (59.375) Mem 11224MB [2024-08-20 00:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 4.0117 (3.6083) Acc@1 26.367 (30.833) Acc@5 50.537 (57.986) Mem 11224MB [2024-08-20 00:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 4.0469 (3.7492) Acc@1 24.707 (29.808) Acc@5 48.193 (55.680) Mem 11224MB [2024-08-20 00:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 30.912 Acc@5 56.872 [2024-08-20 00:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 30.9% [2024-08-20 00:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 30.91% [2024-08-20 00:32:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-20 00:32:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-20 00:32:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][0/625] eta 0:07:00 lr 0.001501 wd 0.0500 time 0.6731 (0.6731) data time 0.3649 (0.3649) model time 0.0000 (0.0000) loss 4.7125 (4.7125) grad_norm 1.6309 (1.6309) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][10/625] eta 0:03:54 lr 0.001502 wd 0.0500 time 0.3703 (0.3819) data time 0.0010 (0.0342) model time 0.0000 (0.0000) loss 4.8320 (4.4109) grad_norm 1.6592 (1.6628) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][20/625] eta 0:03:42 lr 0.001504 wd 0.0500 time 0.3372 (0.3674) data time 0.0006 (0.0183) model time 0.0000 (0.0000) loss 3.7929 (4.3759) grad_norm 1.2335 (1.6919) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][30/625] eta 0:03:35 lr 0.001505 wd 0.0500 time 0.3492 (0.3622) data time 0.0007 (0.0127) model time 0.0000 (0.0000) loss 5.0526 (4.4956) grad_norm 1.4498 (1.6533) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][40/625] eta 0:03:30 lr 0.001507 wd 0.0500 time 0.3710 (0.3600) data time 0.0006 (0.0098) model time 0.0000 (0.0000) loss 3.4973 (4.3651) grad_norm 1.4190 (1.6269) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][50/625] eta 0:03:26 lr 0.001508 wd 0.0500 time 0.3415 (0.3583) data time 0.0006 (0.0080) model time 0.0000 (0.0000) loss 4.9295 (4.3637) grad_norm 1.0326 (1.6202) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][60/625] eta 0:03:21 lr 0.001510 wd 0.0500 time 0.3504 (0.3572) data time 0.0008 (0.0069) model time 0.3497 (0.3506) loss 3.9400 (4.3971) grad_norm 1.8257 (1.6142) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][70/625] eta 0:03:21 lr 0.001512 wd 0.0500 time 0.3597 (0.3630) data time 0.0006 (0.0060) model time 0.3591 (0.3743) loss 5.0216 (4.4058) grad_norm 1.3517 (1.6289) loss_scale 16384.0000 (16384.0000) mem 11224MB [2024-08-20 00:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-20 00:32:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-20 00:32:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-21 09:15:04 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-21 09:15:05 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-21 09:15:19 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-26 03:52:06 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-26 03:52:08 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-26 03:52:22 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-26 03:52:33 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-26 03:52:33 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-26 03:52:35 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-26 03:52:37 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-26 03:52:37 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 15) [2024-08-26 03:52:38 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-26 03:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][80/625] eta 0:15:29 lr 0.001513 wd 0.0500 time 0.3749 (1.7048) data time 0.0011 (0.0885) model time 0.3738 (1.6163) loss 5.1748 (5.0133) grad_norm 1.8559 (1.6769) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][90/625] eta 0:09:16 lr 0.001515 wd 0.0500 time 0.3756 (1.0402) data time 0.0009 (0.0448) model time 0.3747 (0.9954) loss 4.6818 (4.7492) grad_norm 2.0984 (1.5763) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][100/625] eta 0:07:14 lr 0.001516 wd 0.0500 time 0.3780 (0.8273) data time 0.0012 (0.0303) model time 0.3768 (0.7970) loss 5.1249 (4.8052) grad_norm 1.2128 (1.5442) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][110/625] eta 0:06:08 lr 0.001518 wd 0.0500 time 0.3794 (0.7151) data time 0.0008 (0.0230) model time 0.3786 (0.6921) loss 3.7236 (4.6524) grad_norm 1.6527 (1.4988) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][120/625] eta 0:05:29 lr 0.001520 wd 0.0500 time 0.3754 (0.6534) data time 0.0011 (0.0187) model time 0.3743 (0.6347) loss 3.8608 (4.6074) grad_norm 1.3400 (1.5093) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][130/625] eta 0:05:00 lr 0.001521 wd 0.0500 time 0.3796 (0.6074) data time 0.0009 (0.0158) model time 0.3788 (0.5916) loss 4.6217 (4.5874) grad_norm 1.2733 (1.5408) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][140/625] eta 0:04:38 lr 0.001523 wd 0.0500 time 0.3761 (0.5745) data time 0.0008 (0.0137) model time 0.3752 (0.5607) loss 3.4071 (4.5606) grad_norm 2.4965 (1.5644) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][150/625] eta 0:04:21 lr 0.001524 wd 0.0500 time 0.3809 (0.5497) data time 0.0011 (0.0122) model time 0.3798 (0.5376) loss 4.8200 (4.5220) grad_norm 1.4871 (1.5597) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][160/625] eta 0:04:06 lr 0.001526 wd 0.0500 time 0.3751 (0.5306) data time 0.0008 (0.0109) model time 0.3742 (0.5197) loss 5.0258 (4.4949) grad_norm 1.1040 (1.5323) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][170/625] eta 0:03:54 lr 0.001528 wd 0.0500 time 0.3889 (0.5154) data time 0.0011 (0.0100) model time 0.3878 (0.5054) loss 5.1132 (4.5078) grad_norm 1.4842 (1.5203) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][180/625] eta 0:03:43 lr 0.001529 wd 0.0500 time 0.3824 (0.5029) data time 0.0012 (0.0092) model time 0.3812 (0.4937) loss 4.3394 (4.5133) grad_norm 2.4225 (1.5313) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][190/625] eta 0:03:34 lr 0.001531 wd 0.0500 time 0.3748 (0.4925) data time 0.0011 (0.0085) model time 0.3737 (0.4839) loss 5.2148 (4.5209) grad_norm 1.4111 (1.5400) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][200/625] eta 0:03:25 lr 0.001532 wd 0.0500 time 0.3788 (0.4834) data time 0.0008 (0.0080) model time 0.3779 (0.4755) loss 4.5627 (4.4990) grad_norm 1.2057 (1.5204) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][210/625] eta 0:03:17 lr 0.001534 wd 0.0500 time 0.3723 (0.4759) data time 0.0011 (0.0075) model time 0.3712 (0.4684) loss 3.2138 (4.4848) grad_norm 1.4402 (1.5179) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][220/625] eta 0:03:10 lr 0.001536 wd 0.0500 time 0.3785 (0.4693) data time 0.0011 (0.0071) model time 0.3774 (0.4623) loss 4.6240 (4.4790) grad_norm 1.6779 (1.5231) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:53:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][230/625] eta 0:03:03 lr 0.001537 wd 0.0500 time 0.3713 (0.4636) data time 0.0011 (0.0067) model time 0.3702 (0.4569) loss 4.5648 (4.4693) grad_norm 1.6440 (1.5404) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][240/625] eta 0:02:56 lr 0.001539 wd 0.0500 time 0.3720 (0.4586) data time 0.0008 (0.0064) model time 0.3712 (0.4522) loss 3.4099 (4.4706) grad_norm 1.5944 (1.5458) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][250/625] eta 0:02:50 lr 0.001540 wd 0.0500 time 0.3786 (0.4541) data time 0.0008 (0.0061) model time 0.3778 (0.4480) loss 3.7468 (4.4493) grad_norm 2.6354 (1.5615) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][260/625] eta 0:02:44 lr 0.001542 wd 0.0500 time 0.3886 (0.4501) data time 0.0009 (0.0058) model time 0.3878 (0.4443) loss 4.1724 (4.4492) grad_norm 1.5555 (1.5628) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][270/625] eta 0:02:38 lr 0.001544 wd 0.0500 time 0.3742 (0.4465) data time 0.0013 (0.0056) model time 0.3729 (0.4409) loss 4.2407 (4.4304) grad_norm 1.3750 (1.5624) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][280/625] eta 0:02:32 lr 0.001545 wd 0.0500 time 0.4014 (0.4433) data time 0.0008 (0.0054) model time 0.4006 (0.4379) loss 4.5792 (4.4219) grad_norm 1.8964 (1.5608) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][290/625] eta 0:02:27 lr 0.001547 wd 0.0500 time 0.3802 (0.4403) data time 0.0010 (0.0052) model time 0.3791 (0.4351) loss 4.1844 (4.4148) grad_norm 1.8698 (1.5620) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][300/625] eta 0:02:22 lr 0.001548 wd 0.0500 time 0.3868 (0.4395) data time 0.0011 (0.0050) model time 0.3858 (0.4345) loss 5.1400 (4.4198) grad_norm 2.3945 (1.5759) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][310/625] eta 0:02:17 lr 0.001550 wd 0.0500 time 0.3952 (0.4378) data time 0.0010 (0.0049) model time 0.3942 (0.4329) loss 4.4781 (4.4131) grad_norm 2.0382 (1.5742) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][320/625] eta 0:02:12 lr 0.001552 wd 0.0500 time 0.3743 (0.4354) data time 0.0008 (0.0047) model time 0.3735 (0.4307) loss 3.5632 (4.4053) grad_norm 1.7739 (1.5868) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][330/625] eta 0:02:07 lr 0.001553 wd 0.0500 time 0.3832 (0.4332) data time 0.0010 (0.0046) model time 0.3822 (0.4287) loss 3.6887 (4.4009) grad_norm 1.3913 (1.5739) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][340/625] eta 0:02:02 lr 0.001555 wd 0.0500 time 0.3927 (0.4313) data time 0.0009 (0.0045) model time 0.3918 (0.4268) loss 5.4155 (4.3893) grad_norm 1.6217 (1.5679) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][350/625] eta 0:01:58 lr 0.001556 wd 0.0500 time 0.3753 (0.4293) data time 0.0011 (0.0043) model time 0.3742 (0.4250) loss 5.0754 (4.3988) grad_norm 1.7250 (1.5657) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][360/625] eta 0:01:53 lr 0.001558 wd 0.0500 time 0.3804 (0.4276) data time 0.0011 (0.0042) model time 0.3793 (0.4234) loss 3.8947 (4.3925) grad_norm 1.1496 (1.5576) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][370/625] eta 0:01:48 lr 0.001560 wd 0.0500 time 0.3889 (0.4260) data time 0.0010 (0.0041) model time 0.3879 (0.4219) loss 3.7905 (4.3734) grad_norm 1.4394 (1.5550) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][380/625] eta 0:01:44 lr 0.001561 wd 0.0500 time 0.3729 (0.4246) data time 0.0011 (0.0040) model time 0.3718 (0.4205) loss 4.4382 (4.3696) grad_norm 2.4585 (1.5588) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][390/625] eta 0:01:39 lr 0.001563 wd 0.0500 time 0.3728 (0.4232) data time 0.0011 (0.0040) model time 0.3718 (0.4192) loss 3.9318 (4.3826) grad_norm 3.0008 (1.5630) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][400/625] eta 0:01:34 lr 0.001564 wd 0.0500 time 0.3956 (0.4219) data time 0.0009 (0.0039) model time 0.3947 (0.4180) loss 5.0330 (4.3887) grad_norm 1.2461 (1.5598) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][410/625] eta 0:01:30 lr 0.001566 wd 0.0500 time 0.3747 (0.4206) data time 0.0010 (0.0038) model time 0.3736 (0.4168) loss 4.6611 (4.3879) grad_norm 1.2355 (1.5571) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][420/625] eta 0:01:26 lr 0.001568 wd 0.0500 time 0.3818 (0.4206) data time 0.0011 (0.0037) model time 0.3807 (0.4169) loss 3.7665 (4.3904) grad_norm 1.3788 (1.5547) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][430/625] eta 0:01:21 lr 0.001569 wd 0.0500 time 0.3818 (0.4195) data time 0.0009 (0.0036) model time 0.3809 (0.4158) loss 5.2347 (4.3936) grad_norm 1.2043 (1.5475) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][440/625] eta 0:01:17 lr 0.001571 wd 0.0500 time 0.3844 (0.4184) data time 0.0011 (0.0036) model time 0.3833 (0.4148) loss 4.7957 (4.3920) grad_norm 1.3458 (1.5501) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][450/625] eta 0:01:13 lr 0.001572 wd 0.0500 time 0.3870 (0.4173) data time 0.0008 (0.0035) model time 0.3862 (0.4138) loss 4.0900 (4.3899) grad_norm 1.8043 (1.5462) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][460/625] eta 0:01:08 lr 0.001574 wd 0.0500 time 0.3673 (0.4163) data time 0.0011 (0.0035) model time 0.3662 (0.4128) loss 3.5376 (4.3790) grad_norm 2.5378 (1.5478) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][470/625] eta 0:01:04 lr 0.001576 wd 0.0500 time 0.3824 (0.4153) data time 0.0010 (0.0034) model time 0.3813 (0.4119) loss 4.1125 (4.3852) grad_norm 1.5964 (1.5466) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][480/625] eta 0:01:00 lr 0.001577 wd 0.0500 time 0.3763 (0.4145) data time 0.0009 (0.0034) model time 0.3754 (0.4111) loss 4.2779 (4.3907) grad_norm 1.7253 (1.5446) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][490/625] eta 0:00:55 lr 0.001579 wd 0.0500 time 0.3744 (0.4141) data time 0.0008 (0.0033) model time 0.3735 (0.4108) loss 4.3274 (4.3887) grad_norm 1.3300 (1.5520) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][500/625] eta 0:00:51 lr 0.001580 wd 0.0500 time 0.3739 (0.4133) data time 0.0012 (0.0033) model time 0.3727 (0.4101) loss 4.7016 (4.3992) grad_norm 1.5395 (1.5476) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][510/625] eta 0:00:47 lr 0.001582 wd 0.0500 time 0.3757 (0.4125) data time 0.0012 (0.0032) model time 0.3745 (0.4093) loss 3.6666 (4.4022) grad_norm 1.6224 (1.5485) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][520/625] eta 0:00:43 lr 0.001584 wd 0.0500 time 0.3758 (0.4118) data time 0.0009 (0.0032) model time 0.3749 (0.4086) loss 4.8452 (4.4036) grad_norm 1.2099 (1.5539) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][530/625] eta 0:00:39 lr 0.001585 wd 0.0500 time 0.3752 (0.4111) data time 0.0009 (0.0031) model time 0.3743 (0.4080) loss 4.4912 (4.3935) grad_norm 1.3297 (1.5519) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][540/625] eta 0:00:34 lr 0.001587 wd 0.0500 time 0.3833 (0.4105) data time 0.0014 (0.0031) model time 0.3820 (0.4074) loss 4.4923 (4.3852) grad_norm 1.0714 (1.5512) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:55:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][550/625] eta 0:00:30 lr 0.001588 wd 0.0500 time 0.3763 (0.4099) data time 0.0012 (0.0030) model time 0.3750 (0.4068) loss 3.8366 (4.3819) grad_norm 1.5725 (1.5542) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][560/625] eta 0:00:26 lr 0.001590 wd 0.0500 time 0.3737 (0.4092) data time 0.0009 (0.0030) model time 0.3728 (0.4062) loss 3.8427 (4.3866) grad_norm 1.1820 (1.5549) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][570/625] eta 0:00:22 lr 0.001592 wd 0.0500 time 0.3784 (0.4085) data time 0.0008 (0.0030) model time 0.3776 (0.4056) loss 3.5431 (4.3843) grad_norm 1.1313 (1.5531) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][580/625] eta 0:00:18 lr 0.001593 wd 0.0500 time 0.3926 (0.4080) data time 0.0011 (0.0029) model time 0.3915 (0.4051) loss 4.9525 (4.3900) grad_norm 1.0842 (1.5516) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:56:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][590/625] eta 0:00:14 lr 0.001595 wd 0.0500 time 0.3830 (0.4075) data time 0.0008 (0.0029) model time 0.3822 (0.4046) loss 4.6642 (4.3923) grad_norm 2.5127 (1.5565) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][600/625] eta 0:00:10 lr 0.001596 wd 0.0500 time 0.3702 (0.4069) data time 0.0011 (0.0029) model time 0.3691 (0.4040) loss 4.3486 (4.3849) grad_norm 1.5707 (1.5577) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][610/625] eta 0:00:06 lr 0.001598 wd 0.0500 time 0.3643 (0.4067) data time 0.0006 (0.0028) model time 0.3637 (0.4038) loss 4.9156 (4.3844) grad_norm 2.2214 (1.5581) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [15/300][620/625] eta 0:00:02 lr 0.001600 wd 0.0500 time 0.3662 (0.4060) data time 0.0008 (0.0028) model time 0.3654 (0.4032) loss 3.3811 (4.3834) grad_norm 1.7622 (1.5565) loss_scale 16384.0000 (16384.0000) mem 11218MB [2024-08-26 03:56:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 15 training takes 0:03:44 [2024-08-26 03:56:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 03:56:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 03:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.487 (0.487) Loss 1.1035 (1.1035) Acc@1 74.805 (74.805) Acc@5 92.334 (92.334) Mem 11218MB [2024-08-26 03:56:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.8818 (1.3121) Acc@1 56.201 (68.053) Acc@5 82.129 (90.470) Mem 11218MB [2024-08-26 03:56:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.9922 (1.6109) Acc@1 55.273 (62.598) Acc@5 79.736 (85.789) Mem 11218MB [2024-08-26 03:56:38 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 62.336 Acc@5 85.653 [2024-08-26 03:56:38 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 62.3% [2024-08-26 03:56:38 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 62.34% [2024-08-26 03:56:38 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 03:56:39 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 03:56:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.503 (0.503) Loss 3.0273 (3.0273) Acc@1 42.188 (42.188) Acc@5 67.139 (67.139) Mem 11218MB [2024-08-26 03:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.145) Loss 3.5742 (3.1310) Acc@1 31.201 (37.438) Acc@5 58.447 (65.186) Mem 11218MB [2024-08-26 03:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 3.6699 (3.3116) Acc@1 28.809 (35.584) Acc@5 54.443 (62.128) Mem 11218MB [2024-08-26 03:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 36.508 Acc@5 63.022 [2024-08-26 03:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 36.5% [2024-08-26 03:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 36.51% [2024-08-26 03:56:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 03:56:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 03:56:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][0/625] eta 0:09:58 lr 0.001600 wd 0.0500 time 0.9575 (0.9575) data time 0.5165 (0.5165) model time 0.0000 (0.0000) loss 4.3966 (4.3966) grad_norm 1.7362 (1.7362) loss_scale 16384.0000 (16384.0000) mem 11228MB [2024-08-26 03:56:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][10/625] eta 0:04:35 lr 0.001602 wd 0.0500 time 0.3816 (0.4478) data time 0.0013 (0.0481) model time 0.0000 (0.0000) loss 4.2211 (4.4862) grad_norm 1.5221 (1.5718) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:56:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][20/625] eta 0:04:10 lr 0.001604 wd 0.0500 time 0.3767 (0.4146) data time 0.0011 (0.0258) model time 0.0000 (0.0000) loss 4.2210 (4.4890) grad_norm 1.1801 (1.4702) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:56:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][30/625] eta 0:03:59 lr 0.001605 wd 0.0500 time 0.3758 (0.4024) data time 0.0010 (0.0178) model time 0.0000 (0.0000) loss 3.8480 (4.4586) grad_norm 1.7337 (1.4639) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][40/625] eta 0:03:52 lr 0.001607 wd 0.0500 time 0.3861 (0.3967) data time 0.0008 (0.0138) model time 0.0000 (0.0000) loss 5.2487 (4.4673) grad_norm 1.0619 (1.4758) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][50/625] eta 0:03:46 lr 0.001608 wd 0.0500 time 0.3787 (0.3933) data time 0.0008 (0.0113) model time 0.0000 (0.0000) loss 5.0224 (4.4216) grad_norm 1.1866 (1.4526) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][60/625] eta 0:03:42 lr 0.001610 wd 0.0500 time 0.3799 (0.3944) data time 0.0009 (0.0096) model time 0.3790 (0.3987) loss 4.9246 (4.4115) grad_norm 2.4547 (1.4978) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][70/625] eta 0:03:37 lr 0.001612 wd 0.0500 time 0.3725 (0.3917) data time 0.0011 (0.0084) model time 0.3714 (0.3864) loss 4.4388 (4.4365) grad_norm 1.0935 (1.4539) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][80/625] eta 0:03:32 lr 0.001613 wd 0.0500 time 0.3734 (0.3902) data time 0.0011 (0.0076) model time 0.3723 (0.3837) loss 4.1087 (4.4311) grad_norm 2.7425 (1.4650) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][90/625] eta 0:03:28 lr 0.001615 wd 0.0500 time 0.3769 (0.3889) data time 0.0008 (0.0068) model time 0.3761 (0.3820) loss 3.4593 (4.3861) grad_norm 1.3116 (1.4784) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][100/625] eta 0:03:23 lr 0.001616 wd 0.0500 time 0.3730 (0.3876) data time 0.0009 (0.0063) model time 0.3721 (0.3805) loss 4.0631 (4.3935) grad_norm 1.8074 (1.4897) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][110/625] eta 0:03:19 lr 0.001618 wd 0.0500 time 0.3786 (0.3867) data time 0.0008 (0.0058) model time 0.3778 (0.3799) loss 3.9675 (4.3551) grad_norm 1.4310 (1.4759) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][120/625] eta 0:03:14 lr 0.001620 wd 0.0500 time 0.3746 (0.3860) data time 0.0009 (0.0054) model time 0.3738 (0.3795) loss 4.5262 (4.3870) grad_norm 1.3048 (1.4632) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][130/625] eta 0:03:10 lr 0.001621 wd 0.0500 time 0.3699 (0.3854) data time 0.0011 (0.0051) model time 0.3687 (0.3792) loss 4.8402 (4.3869) grad_norm 1.1060 (1.4659) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][140/625] eta 0:03:06 lr 0.001623 wd 0.0500 time 0.3814 (0.3848) data time 0.0012 (0.0048) model time 0.3803 (0.3788) loss 4.0214 (4.3668) grad_norm 2.2868 (1.4934) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][150/625] eta 0:03:02 lr 0.001624 wd 0.0500 time 0.3803 (0.3843) data time 0.0009 (0.0046) model time 0.3794 (0.3786) loss 5.1932 (4.3618) grad_norm 1.7957 (1.4929) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][160/625] eta 0:02:58 lr 0.001626 wd 0.0500 time 0.3751 (0.3841) data time 0.0011 (0.0044) model time 0.3740 (0.3787) loss 4.7976 (4.3410) grad_norm 1.5855 (1.5018) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][170/625] eta 0:02:54 lr 0.001628 wd 0.0500 time 0.3797 (0.3837) data time 0.0008 (0.0042) model time 0.3789 (0.3785) loss 4.1662 (4.3378) grad_norm 1.2837 (1.4954) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][180/625] eta 0:02:51 lr 0.001629 wd 0.0500 time 0.5433 (0.3843) data time 0.0008 (0.0040) model time 0.5425 (0.3797) loss 4.5356 (4.3451) grad_norm 2.0426 (1.4917) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:57:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][190/625] eta 0:02:47 lr 0.001631 wd 0.0500 time 0.3804 (0.3840) data time 0.0011 (0.0039) model time 0.3792 (0.3795) loss 4.1869 (4.3480) grad_norm 1.4174 (1.4894) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][200/625] eta 0:02:43 lr 0.001632 wd 0.0500 time 0.3764 (0.3838) data time 0.0011 (0.0037) model time 0.3753 (0.3794) loss 3.6905 (4.3444) grad_norm 1.4582 (1.4895) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][210/625] eta 0:02:39 lr 0.001634 wd 0.0500 time 0.5837 (0.3844) data time 0.0008 (0.0036) model time 0.5829 (0.3804) loss 5.0879 (4.3583) grad_norm 1.2501 (1.4855) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][220/625] eta 0:02:35 lr 0.001636 wd 0.0500 time 0.3769 (0.3850) data time 0.0010 (0.0035) model time 0.3759 (0.3814) loss 3.9519 (4.3641) grad_norm 1.4276 (1.4819) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][230/625] eta 0:02:31 lr 0.001637 wd 0.0500 time 0.3754 (0.3847) data time 0.0012 (0.0034) model time 0.3742 (0.3810) loss 3.4043 (4.3609) grad_norm 2.7131 (1.4902) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][240/625] eta 0:02:28 lr 0.001639 wd 0.0500 time 0.3829 (0.3845) data time 0.0009 (0.0033) model time 0.3820 (0.3809) loss 4.4185 (4.3646) grad_norm 1.3188 (1.4986) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][250/625] eta 0:02:24 lr 0.001640 wd 0.0500 time 0.3730 (0.3842) data time 0.0008 (0.0032) model time 0.3721 (0.3807) loss 5.1894 (4.3605) grad_norm 1.5898 (1.5005) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][260/625] eta 0:02:20 lr 0.001642 wd 0.0500 time 0.3731 (0.3839) data time 0.0008 (0.0032) model time 0.3723 (0.3805) loss 5.0351 (4.3471) grad_norm 2.7481 (1.5103) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][270/625] eta 0:02:16 lr 0.001644 wd 0.0500 time 0.3785 (0.3844) data time 0.0008 (0.0031) model time 0.3777 (0.3811) loss 3.4462 (4.3419) grad_norm 1.3701 (1.5194) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][280/625] eta 0:02:12 lr 0.001645 wd 0.0500 time 0.3788 (0.3841) data time 0.0011 (0.0030) model time 0.3778 (0.3809) loss 3.5668 (4.3302) grad_norm 1.6244 (1.5148) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][290/625] eta 0:02:08 lr 0.001647 wd 0.0500 time 0.3761 (0.3839) data time 0.0008 (0.0029) model time 0.3752 (0.3807) loss 3.2384 (4.3231) grad_norm 1.9456 (1.5143) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][300/625] eta 0:02:04 lr 0.001648 wd 0.0500 time 0.5777 (0.3843) data time 0.0008 (0.0029) model time 0.5769 (0.3813) loss 4.6614 (4.3287) grad_norm 1.4202 (1.5060) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][310/625] eta 0:02:00 lr 0.001650 wd 0.0500 time 0.3786 (0.3840) data time 0.0010 (0.0028) model time 0.3776 (0.3810) loss 4.2048 (4.3262) grad_norm 1.5893 (1.5061) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][320/625] eta 0:01:57 lr 0.001652 wd 0.0500 time 0.3711 (0.3838) data time 0.0009 (0.0029) model time 0.3702 (0.3807) loss 4.3881 (4.3308) grad_norm 1.5300 (1.5007) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][330/625] eta 0:01:53 lr 0.001653 wd 0.0500 time 0.3741 (0.3836) data time 0.0010 (0.0028) model time 0.3732 (0.3805) loss 3.8825 (4.3229) grad_norm 1.5791 (1.4964) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][340/625] eta 0:01:49 lr 0.001655 wd 0.0500 time 0.3784 (0.3835) data time 0.0010 (0.0028) model time 0.3774 (0.3805) loss 4.2200 (4.3215) grad_norm 1.4904 (1.4934) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:58:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][350/625] eta 0:01:45 lr 0.001656 wd 0.0500 time 0.3776 (0.3832) data time 0.0008 (0.0027) model time 0.3768 (0.3803) loss 4.7738 (4.3220) grad_norm 1.1812 (1.4889) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][360/625] eta 0:01:41 lr 0.001658 wd 0.0500 time 0.3776 (0.3831) data time 0.0008 (0.0027) model time 0.3768 (0.3801) loss 5.1488 (4.3179) grad_norm 1.7564 (1.4986) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][370/625] eta 0:01:37 lr 0.001660 wd 0.0500 time 0.3724 (0.3829) data time 0.0008 (0.0026) model time 0.3716 (0.3800) loss 5.0892 (4.3282) grad_norm 1.7900 (1.4997) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][380/625] eta 0:01:33 lr 0.001661 wd 0.0500 time 0.3763 (0.3827) data time 0.0008 (0.0026) model time 0.3755 (0.3798) loss 3.0817 (4.3307) grad_norm 1.0931 (1.4972) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][390/625] eta 0:01:29 lr 0.001663 wd 0.0500 time 0.3692 (0.3826) data time 0.0010 (0.0026) model time 0.3682 (0.3797) loss 4.6290 (4.3221) grad_norm 1.2808 (1.4926) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][400/625] eta 0:01:26 lr 0.001664 wd 0.0500 time 0.3718 (0.3835) data time 0.0010 (0.0025) model time 0.3708 (0.3808) loss 4.6179 (4.3132) grad_norm 1.6897 (1.5026) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][410/625] eta 0:01:22 lr 0.001666 wd 0.0500 time 0.3762 (0.3833) data time 0.0011 (0.0025) model time 0.3751 (0.3807) loss 4.3825 (4.3152) grad_norm 1.4832 (1.5027) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][420/625] eta 0:01:18 lr 0.001668 wd 0.0500 time 0.3795 (0.3833) data time 0.0009 (0.0025) model time 0.3786 (0.3807) loss 4.8054 (4.3236) grad_norm 1.2601 (1.4980) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][430/625] eta 0:01:14 lr 0.001669 wd 0.0500 time 0.3792 (0.3832) data time 0.0011 (0.0024) model time 0.3781 (0.3806) loss 4.6171 (4.3213) grad_norm 1.3075 (1.4964) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][440/625] eta 0:01:10 lr 0.001671 wd 0.0500 time 0.3755 (0.3831) data time 0.0010 (0.0024) model time 0.3745 (0.3805) loss 4.8941 (4.3195) grad_norm 1.6415 (1.4947) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][450/625] eta 0:01:07 lr 0.001672 wd 0.0500 time 0.3781 (0.3830) data time 0.0008 (0.0024) model time 0.3772 (0.3804) loss 3.2946 (4.3185) grad_norm 2.1160 (1.4962) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][460/625] eta 0:01:03 lr 0.001674 wd 0.0500 time 0.3767 (0.3831) data time 0.0008 (0.0023) model time 0.3758 (0.3806) loss 5.0693 (4.3288) grad_norm 1.3616 (1.4951) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][470/625] eta 0:00:59 lr 0.001676 wd 0.0500 time 0.3810 (0.3830) data time 0.0008 (0.0023) model time 0.3802 (0.3806) loss 3.3081 (4.3189) grad_norm 1.4576 (1.4932) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][480/625] eta 0:00:55 lr 0.001677 wd 0.0500 time 0.3821 (0.3829) data time 0.0008 (0.0023) model time 0.3812 (0.3805) loss 5.2164 (4.3189) grad_norm 1.8547 (1.4984) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][490/625] eta 0:00:51 lr 0.001679 wd 0.0500 time 0.3794 (0.3829) data time 0.0010 (0.0023) model time 0.3783 (0.3805) loss 4.4140 (4.3238) grad_norm 1.3757 (1.4949) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][500/625] eta 0:00:47 lr 0.001680 wd 0.0500 time 0.3761 (0.3828) data time 0.0008 (0.0022) model time 0.3753 (0.3804) loss 4.4590 (4.3236) grad_norm 1.6603 (1.4957) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 03:59:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][510/625] eta 0:00:44 lr 0.001682 wd 0.0500 time 0.3789 (0.3832) data time 0.0009 (0.0022) model time 0.3780 (0.3808) loss 5.0649 (4.3274) grad_norm 1.9811 (1.4941) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:00:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][520/625] eta 0:00:40 lr 0.001684 wd 0.0500 time 0.3743 (0.3839) data time 0.0011 (0.0022) model time 0.3732 (0.3816) loss 4.1062 (4.3205) grad_norm 1.3372 (1.4940) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:00:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][530/625] eta 0:00:36 lr 0.001685 wd 0.0500 time 0.3732 (0.3837) data time 0.0009 (0.0022) model time 0.3723 (0.3815) loss 3.1926 (4.3202) grad_norm 1.0410 (1.4901) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:00:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][540/625] eta 0:00:32 lr 0.001687 wd 0.0500 time 0.3765 (0.3836) data time 0.0008 (0.0022) model time 0.3757 (0.3814) loss 3.7559 (4.3176) grad_norm 1.4347 (1.4899) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][550/625] eta 0:00:28 lr 0.001688 wd 0.0500 time 0.3754 (0.3835) data time 0.0008 (0.0021) model time 0.3745 (0.3813) loss 4.2903 (4.3190) grad_norm 0.9941 (1.4877) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][560/625] eta 0:00:24 lr 0.001690 wd 0.0500 time 0.3803 (0.3834) data time 0.0008 (0.0021) model time 0.3795 (0.3812) loss 5.0963 (4.3229) grad_norm 1.9500 (1.4887) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:00:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][570/625] eta 0:00:21 lr 0.001692 wd 0.0500 time 0.3886 (0.3834) data time 0.0008 (0.0021) model time 0.3878 (0.3812) loss 5.1018 (4.3241) grad_norm 1.0082 (1.4882) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:00:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][580/625] eta 0:00:17 lr 0.001693 wd 0.0500 time 0.3897 (0.3837) data time 0.0011 (0.0021) model time 0.3887 (0.3816) loss 3.9311 (4.3253) grad_norm 1.3060 (1.4875) loss_scale 32768.0000 (16496.7986) mem 11225MB [2024-08-26 04:00:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][590/625] eta 0:00:13 lr 0.001695 wd 0.0500 time 0.3802 (0.3837) data time 0.0011 (0.0021) model time 0.3791 (0.3816) loss 4.0597 (4.3246) grad_norm 1.2846 (1.4858) loss_scale 32768.0000 (16772.1151) mem 11225MB [2024-08-26 04:00:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][600/625] eta 0:00:09 lr 0.001696 wd 0.0500 time 0.3953 (0.3836) data time 0.0010 (0.0021) model time 0.3943 (0.3815) loss 4.6081 (4.3234) grad_norm 1.1741 (1.4849) loss_scale 32768.0000 (17038.2696) mem 11225MB [2024-08-26 04:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][610/625] eta 0:00:05 lr 0.001698 wd 0.0500 time 0.3647 (0.3836) data time 0.0008 (0.0020) model time 0.3639 (0.3815) loss 3.9874 (4.3189) grad_norm 1.3374 (1.4834) loss_scale 32768.0000 (17295.7119) mem 11225MB [2024-08-26 04:00:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [16/300][620/625] eta 0:00:01 lr 0.001700 wd 0.0500 time 0.3670 (0.3833) data time 0.0005 (0.0020) model time 0.3665 (0.3812) loss 4.9481 (4.3235) grad_norm 1.0059 (1.4839) loss_scale 32768.0000 (17544.8631) mem 11225MB [2024-08-26 04:00:42 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 16 training takes 0:03:59 [2024-08-26 04:00:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:00:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:00:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.498 (0.498) Loss 1.1123 (1.1123) Acc@1 73.193 (73.193) Acc@5 91.602 (91.602) Mem 11225MB [2024-08-26 04:00:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.142) Loss 1.7852 (1.2680) Acc@1 59.326 (69.078) Acc@5 83.936 (91.109) Mem 11225MB [2024-08-26 04:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 1.9795 (1.5710) Acc@1 56.055 (63.646) Acc@5 79.199 (86.435) Mem 11225MB [2024-08-26 04:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 63.562 Acc@5 86.320 [2024-08-26 04:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 63.6% [2024-08-26 04:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 63.56% [2024-08-26 04:00:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:00:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:00:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 2.5391 (2.5391) Acc@1 49.121 (49.121) Acc@5 74.707 (74.707) Mem 11225MB [2024-08-26 04:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 3.1914 (2.6989) Acc@1 36.523 (43.488) Acc@5 64.209 (71.524) Mem 11225MB [2024-08-26 04:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 3.3203 (2.9162) Acc@1 33.057 (40.858) Acc@5 60.596 (67.852) Mem 11225MB [2024-08-26 04:00:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 41.591 Acc@5 68.410 [2024-08-26 04:00:50 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 41.6% [2024-08-26 04:00:50 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 41.59% [2024-08-26 04:00:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:00:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:00:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][0/625] eta 0:08:14 lr 0.001700 wd 0.0500 time 0.7910 (0.7910) data time 0.4607 (0.4607) model time 0.0000 (0.0000) loss 4.4655 (4.4655) grad_norm 1.2811 (1.2811) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][10/625] eta 0:04:16 lr 0.001702 wd 0.0500 time 0.3818 (0.4176) data time 0.0009 (0.0430) model time 0.0000 (0.0000) loss 4.4860 (4.5886) grad_norm 1.0222 (1.6393) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:00:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][20/625] eta 0:04:01 lr 0.001703 wd 0.0500 time 0.3789 (0.3994) data time 0.0009 (0.0231) model time 0.0000 (0.0000) loss 5.2686 (4.4636) grad_norm 1.4377 (1.5990) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][30/625] eta 0:03:53 lr 0.001705 wd 0.0500 time 0.3792 (0.3926) data time 0.0012 (0.0160) model time 0.0000 (0.0000) loss 3.8742 (4.4163) grad_norm 1.6334 (1.6971) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][40/625] eta 0:03:47 lr 0.001707 wd 0.0500 time 0.4044 (0.3896) data time 0.0011 (0.0123) model time 0.0000 (0.0000) loss 4.4808 (4.3714) grad_norm 1.6184 (1.6615) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][50/625] eta 0:03:42 lr 0.001708 wd 0.0500 time 0.3845 (0.3870) data time 0.0011 (0.0101) model time 0.0000 (0.0000) loss 4.4833 (4.3222) grad_norm 1.8394 (1.6283) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][60/625] eta 0:03:41 lr 0.001710 wd 0.0500 time 0.3833 (0.3925) data time 0.0008 (0.0087) model time 0.3825 (0.4193) loss 4.2832 (4.3139) grad_norm 1.4615 (1.5996) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][70/625] eta 0:03:36 lr 0.001711 wd 0.0500 time 0.3769 (0.3910) data time 0.0011 (0.0076) model time 0.3759 (0.3999) loss 3.0439 (4.2888) grad_norm 1.2972 (1.5591) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][80/625] eta 0:03:33 lr 0.001713 wd 0.0500 time 0.3825 (0.3922) data time 0.0011 (0.0068) model time 0.3814 (0.3999) loss 4.6852 (4.3060) grad_norm 1.2543 (1.5334) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][90/625] eta 0:03:29 lr 0.001715 wd 0.0500 time 0.3789 (0.3907) data time 0.0011 (0.0062) model time 0.3777 (0.3942) loss 4.0234 (4.3172) grad_norm 1.6775 (1.5237) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][100/625] eta 0:03:24 lr 0.001716 wd 0.0500 time 0.3820 (0.3896) data time 0.0009 (0.0057) model time 0.3811 (0.3910) loss 5.0820 (4.3323) grad_norm 1.4536 (1.5082) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][110/625] eta 0:03:20 lr 0.001718 wd 0.0500 time 0.3751 (0.3885) data time 0.0010 (0.0053) model time 0.3740 (0.3887) loss 4.2509 (4.3300) grad_norm 2.1711 (1.5264) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][120/625] eta 0:03:15 lr 0.001719 wd 0.0500 time 0.3819 (0.3877) data time 0.0010 (0.0049) model time 0.3809 (0.3870) loss 4.5944 (4.3369) grad_norm 1.0593 (inf) loss_scale 16384.0000 (31684.7603) mem 11225MB [2024-08-26 04:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][130/625] eta 0:03:11 lr 0.001721 wd 0.0500 time 0.3771 (0.3869) data time 0.0009 (0.0047) model time 0.3762 (0.3857) loss 5.3452 (4.3331) grad_norm 2.5643 (inf) loss_scale 16384.0000 (30516.7634) mem 11225MB [2024-08-26 04:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][140/625] eta 0:03:07 lr 0.001723 wd 0.0500 time 0.3776 (0.3863) data time 0.0009 (0.0044) model time 0.3768 (0.3847) loss 4.6254 (4.3021) grad_norm 1.9320 (inf) loss_scale 16384.0000 (29514.4397) mem 11225MB [2024-08-26 04:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][150/625] eta 0:03:03 lr 0.001724 wd 0.0500 time 0.3793 (0.3858) data time 0.0011 (0.0042) model time 0.3782 (0.3841) loss 4.4265 (4.2978) grad_norm 1.1764 (inf) loss_scale 16384.0000 (28644.8742) mem 11225MB [2024-08-26 04:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][160/625] eta 0:02:59 lr 0.001726 wd 0.0500 time 0.3813 (0.3855) data time 0.0011 (0.0040) model time 0.3802 (0.3837) loss 4.8098 (4.2982) grad_norm 1.0730 (inf) loss_scale 16384.0000 (27883.3292) mem 11225MB [2024-08-26 04:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][170/625] eta 0:02:55 lr 0.001727 wd 0.0500 time 0.3782 (0.3851) data time 0.0008 (0.0038) model time 0.3775 (0.3831) loss 5.0886 (4.3027) grad_norm 1.4172 (inf) loss_scale 16384.0000 (27210.8538) mem 11225MB [2024-08-26 04:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][180/625] eta 0:02:51 lr 0.001729 wd 0.0500 time 0.3815 (0.3847) data time 0.0011 (0.0037) model time 0.3804 (0.3825) loss 4.5192 (4.3133) grad_norm 1.3672 (inf) loss_scale 16384.0000 (26612.6851) mem 11225MB [2024-08-26 04:02:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][190/625] eta 0:02:47 lr 0.001731 wd 0.0500 time 0.3826 (0.3844) data time 0.0011 (0.0036) model time 0.3815 (0.3822) loss 4.2122 (4.3223) grad_norm 1.6053 (inf) loss_scale 16384.0000 (26077.1518) mem 11225MB [2024-08-26 04:02:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][200/625] eta 0:02:43 lr 0.001732 wd 0.0500 time 0.3744 (0.3841) data time 0.0011 (0.0035) model time 0.3734 (0.3819) loss 4.2023 (4.3264) grad_norm 2.0552 (inf) loss_scale 16384.0000 (25594.9055) mem 11225MB [2024-08-26 04:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][210/625] eta 0:02:39 lr 0.001734 wd 0.0500 time 0.3783 (0.3839) data time 0.0008 (0.0034) model time 0.3775 (0.3817) loss 4.6586 (4.3331) grad_norm 0.8763 (inf) loss_scale 16384.0000 (25158.3697) mem 11225MB [2024-08-26 04:02:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][220/625] eta 0:02:35 lr 0.001735 wd 0.0500 time 0.3797 (0.3837) data time 0.0011 (0.0033) model time 0.3786 (0.3814) loss 4.3873 (4.3324) grad_norm 0.8786 (inf) loss_scale 16384.0000 (24761.3394) mem 11225MB [2024-08-26 04:02:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][230/625] eta 0:02:31 lr 0.001737 wd 0.0500 time 0.3833 (0.3835) data time 0.0011 (0.0032) model time 0.3823 (0.3813) loss 4.9270 (4.3459) grad_norm 1.2755 (inf) loss_scale 16384.0000 (24398.6840) mem 11225MB [2024-08-26 04:02:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][240/625] eta 0:02:27 lr 0.001739 wd 0.0500 time 0.3799 (0.3842) data time 0.0010 (0.0031) model time 0.3789 (0.3822) loss 4.2349 (4.3371) grad_norm 1.2823 (inf) loss_scale 16384.0000 (24066.1245) mem 11225MB [2024-08-26 04:02:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][250/625] eta 0:02:23 lr 0.001740 wd 0.0500 time 0.3771 (0.3839) data time 0.0009 (0.0030) model time 0.3762 (0.3819) loss 4.5116 (4.3313) grad_norm 1.3047 (inf) loss_scale 16384.0000 (23760.0637) mem 11225MB [2024-08-26 04:02:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][260/625] eta 0:02:20 lr 0.001742 wd 0.0500 time 0.3811 (0.3837) data time 0.0011 (0.0029) model time 0.3800 (0.3817) loss 5.0273 (4.3348) grad_norm 1.8068 (inf) loss_scale 16384.0000 (23477.4559) mem 11225MB [2024-08-26 04:02:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][270/625] eta 0:02:16 lr 0.001743 wd 0.0500 time 0.3784 (0.3856) data time 0.0008 (0.0029) model time 0.3776 (0.3841) loss 3.5611 (4.3330) grad_norm 1.9121 (inf) loss_scale 16384.0000 (23215.7048) mem 11225MB [2024-08-26 04:02:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][280/625] eta 0:02:12 lr 0.001745 wd 0.0500 time 0.3773 (0.3853) data time 0.0008 (0.0028) model time 0.3765 (0.3838) loss 5.2030 (4.3447) grad_norm 1.4825 (inf) loss_scale 16384.0000 (22972.5836) mem 11225MB [2024-08-26 04:02:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][290/625] eta 0:02:09 lr 0.001747 wd 0.0500 time 0.3805 (0.3852) data time 0.0011 (0.0028) model time 0.3794 (0.3836) loss 4.5831 (4.3454) grad_norm 1.1742 (inf) loss_scale 16384.0000 (22746.1718) mem 11225MB [2024-08-26 04:02:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][300/625] eta 0:02:05 lr 0.001748 wd 0.0500 time 0.3782 (0.3849) data time 0.0008 (0.0027) model time 0.3774 (0.3833) loss 3.8896 (4.3267) grad_norm 1.5219 (inf) loss_scale 16384.0000 (22534.8040) mem 11225MB [2024-08-26 04:02:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][310/625] eta 0:02:01 lr 0.001750 wd 0.0500 time 0.3838 (0.3847) data time 0.0011 (0.0027) model time 0.3827 (0.3831) loss 3.3497 (4.3096) grad_norm 1.6023 (inf) loss_scale 16384.0000 (22337.0289) mem 11225MB [2024-08-26 04:02:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][320/625] eta 0:01:57 lr 0.001751 wd 0.0500 time 0.3736 (0.3845) data time 0.0011 (0.0026) model time 0.3725 (0.3829) loss 4.6382 (4.3068) grad_norm 2.6447 (inf) loss_scale 16384.0000 (22151.5763) mem 11225MB [2024-08-26 04:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][330/625] eta 0:01:53 lr 0.001753 wd 0.0500 time 0.3756 (0.3844) data time 0.0010 (0.0026) model time 0.3746 (0.3827) loss 3.4008 (4.3106) grad_norm 1.5921 (inf) loss_scale 16384.0000 (21977.3293) mem 11225MB [2024-08-26 04:03:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][340/625] eta 0:01:49 lr 0.001755 wd 0.0500 time 0.3806 (0.3843) data time 0.0008 (0.0025) model time 0.3798 (0.3827) loss 3.8787 (4.3115) grad_norm 1.0136 (inf) loss_scale 16384.0000 (21813.3021) mem 11225MB [2024-08-26 04:03:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][350/625] eta 0:01:45 lr 0.001756 wd 0.0500 time 0.3754 (0.3842) data time 0.0011 (0.0025) model time 0.3742 (0.3825) loss 4.1245 (4.3121) grad_norm 1.0445 (inf) loss_scale 16384.0000 (21658.6211) mem 11225MB [2024-08-26 04:03:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][360/625] eta 0:01:41 lr 0.001758 wd 0.0500 time 0.3796 (0.3846) data time 0.0008 (0.0024) model time 0.3788 (0.3830) loss 5.2645 (4.3145) grad_norm 1.1752 (inf) loss_scale 16384.0000 (21512.5097) mem 11225MB [2024-08-26 04:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][370/625] eta 0:01:38 lr 0.001759 wd 0.0500 time 0.3924 (0.3844) data time 0.0011 (0.0024) model time 0.3913 (0.3829) loss 4.2902 (4.3035) grad_norm 1.7618 (inf) loss_scale 16384.0000 (21374.2749) mem 11225MB [2024-08-26 04:03:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][380/625] eta 0:01:34 lr 0.001761 wd 0.0500 time 0.3811 (0.3843) data time 0.0011 (0.0024) model time 0.3800 (0.3827) loss 3.5106 (4.3018) grad_norm 1.1336 (inf) loss_scale 16384.0000 (21243.2966) mem 11225MB [2024-08-26 04:03:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][390/625] eta 0:01:30 lr 0.001763 wd 0.0500 time 0.3804 (0.3847) data time 0.0009 (0.0023) model time 0.3795 (0.3832) loss 4.0667 (4.2914) grad_norm 1.0912 (inf) loss_scale 16384.0000 (21119.0179) mem 11225MB [2024-08-26 04:03:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][400/625] eta 0:01:26 lr 0.001764 wd 0.0500 time 0.3773 (0.3846) data time 0.0011 (0.0023) model time 0.3762 (0.3830) loss 3.7323 (4.2917) grad_norm 1.6002 (inf) loss_scale 16384.0000 (21000.9377) mem 11225MB [2024-08-26 04:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][410/625] eta 0:01:22 lr 0.001766 wd 0.0500 time 0.3806 (0.3844) data time 0.0009 (0.0023) model time 0.3797 (0.3829) loss 4.9475 (4.2931) grad_norm 1.8515 (inf) loss_scale 16384.0000 (20888.6034) mem 11225MB [2024-08-26 04:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][420/625] eta 0:01:18 lr 0.001767 wd 0.0500 time 0.3734 (0.3843) data time 0.0009 (0.0023) model time 0.3725 (0.3827) loss 5.2718 (4.3005) grad_norm 1.3090 (inf) loss_scale 16384.0000 (20781.6057) mem 11225MB [2024-08-26 04:03:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][430/625] eta 0:01:14 lr 0.001769 wd 0.0500 time 0.4142 (0.3843) data time 0.0008 (0.0022) model time 0.4133 (0.3827) loss 4.2017 (4.3062) grad_norm 1.1823 (inf) loss_scale 16384.0000 (20679.5731) mem 11225MB [2024-08-26 04:03:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][440/625] eta 0:01:11 lr 0.001771 wd 0.0500 time 0.3771 (0.3841) data time 0.0011 (0.0022) model time 0.3760 (0.3825) loss 4.1456 (4.3036) grad_norm 1.7431 (inf) loss_scale 16384.0000 (20582.1678) mem 11225MB [2024-08-26 04:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][450/625] eta 0:01:07 lr 0.001772 wd 0.0500 time 0.5896 (0.3850) data time 0.0009 (0.0022) model time 0.5888 (0.3835) loss 4.3767 (4.3072) grad_norm 1.5909 (inf) loss_scale 16384.0000 (20489.0820) mem 11225MB [2024-08-26 04:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][460/625] eta 0:01:03 lr 0.001774 wd 0.0500 time 0.3946 (0.3849) data time 0.0012 (0.0022) model time 0.3934 (0.3834) loss 3.6420 (4.3093) grad_norm 1.0839 (inf) loss_scale 16384.0000 (20400.0347) mem 11225MB [2024-08-26 04:03:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][470/625] eta 0:00:59 lr 0.001775 wd 0.0500 time 0.3725 (0.3848) data time 0.0011 (0.0021) model time 0.3715 (0.3833) loss 4.2928 (4.2994) grad_norm 1.7884 (inf) loss_scale 16384.0000 (20314.7686) mem 11225MB [2024-08-26 04:03:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][480/625] eta 0:00:55 lr 0.001777 wd 0.0500 time 0.3941 (0.3847) data time 0.0011 (0.0021) model time 0.3930 (0.3832) loss 4.7773 (4.2903) grad_norm 1.2137 (inf) loss_scale 16384.0000 (20233.0478) mem 11225MB [2024-08-26 04:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][490/625] eta 0:00:51 lr 0.001779 wd 0.0500 time 0.3754 (0.3846) data time 0.0009 (0.0021) model time 0.3746 (0.3831) loss 4.1412 (4.2894) grad_norm 1.0000 (inf) loss_scale 16384.0000 (20154.6558) mem 11225MB [2024-08-26 04:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][500/625] eta 0:00:48 lr 0.001780 wd 0.0500 time 0.3826 (0.3845) data time 0.0009 (0.0021) model time 0.3818 (0.3830) loss 4.6745 (4.2895) grad_norm 1.0107 (inf) loss_scale 16384.0000 (20079.3932) mem 11225MB [2024-08-26 04:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][510/625] eta 0:00:44 lr 0.001782 wd 0.0500 time 0.3774 (0.3844) data time 0.0008 (0.0021) model time 0.3766 (0.3829) loss 3.5782 (4.2898) grad_norm 1.1701 (inf) loss_scale 16384.0000 (20007.0763) mem 11225MB [2024-08-26 04:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][520/625] eta 0:00:40 lr 0.001783 wd 0.0500 time 0.3853 (0.3843) data time 0.0011 (0.0020) model time 0.3842 (0.3828) loss 4.7585 (4.2944) grad_norm 1.1957 (inf) loss_scale 16384.0000 (19937.5355) mem 11225MB [2024-08-26 04:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][530/625] eta 0:00:36 lr 0.001785 wd 0.0500 time 0.3743 (0.3842) data time 0.0011 (0.0020) model time 0.3732 (0.3827) loss 3.7826 (4.2945) grad_norm 1.4276 (inf) loss_scale 16384.0000 (19870.6139) mem 11225MB [2024-08-26 04:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][540/625] eta 0:00:32 lr 0.001787 wd 0.0500 time 0.3835 (0.3841) data time 0.0011 (0.0020) model time 0.3824 (0.3826) loss 4.1102 (4.2918) grad_norm 1.1343 (inf) loss_scale 16384.0000 (19806.1664) mem 11225MB [2024-08-26 04:04:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][550/625] eta 0:00:28 lr 0.001788 wd 0.0500 time 0.3743 (0.3841) data time 0.0007 (0.0020) model time 0.3736 (0.3826) loss 4.6967 (4.2923) grad_norm 1.2966 (inf) loss_scale 16384.0000 (19744.0581) mem 11225MB [2024-08-26 04:04:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][560/625] eta 0:00:24 lr 0.001790 wd 0.0500 time 0.3853 (0.3840) data time 0.0010 (0.0020) model time 0.3842 (0.3825) loss 4.3413 (4.2910) grad_norm 1.2518 (inf) loss_scale 16384.0000 (19684.1640) mem 11225MB [2024-08-26 04:04:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][570/625] eta 0:00:21 lr 0.001791 wd 0.0500 time 0.3775 (0.3850) data time 0.0011 (0.0020) model time 0.3764 (0.3836) loss 3.5990 (4.2857) grad_norm 1.4678 (inf) loss_scale 16384.0000 (19626.3678) mem 11225MB [2024-08-26 04:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][580/625] eta 0:00:17 lr 0.001793 wd 0.0500 time 0.3725 (0.3849) data time 0.0010 (0.0020) model time 0.3715 (0.3835) loss 5.3183 (4.2810) grad_norm 1.7574 (inf) loss_scale 16384.0000 (19570.5611) mem 11225MB [2024-08-26 04:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][590/625] eta 0:00:13 lr 0.001795 wd 0.0500 time 0.3788 (0.3848) data time 0.0008 (0.0019) model time 0.3780 (0.3834) loss 4.2394 (4.2838) grad_norm 1.9158 (inf) loss_scale 16384.0000 (19516.6430) mem 11225MB [2024-08-26 04:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][600/625] eta 0:00:09 lr 0.001796 wd 0.0500 time 0.3879 (0.3851) data time 0.0011 (0.0019) model time 0.3868 (0.3837) loss 4.6096 (4.2832) grad_norm 1.4698 (inf) loss_scale 16384.0000 (19464.5191) mem 11225MB [2024-08-26 04:04:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][610/625] eta 0:00:05 lr 0.001798 wd 0.0500 time 0.3644 (0.3849) data time 0.0008 (0.0019) model time 0.3636 (0.3835) loss 4.7053 (4.2799) grad_norm 1.3175 (inf) loss_scale 16384.0000 (19414.1015) mem 11225MB [2024-08-26 04:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [17/300][620/625] eta 0:00:01 lr 0.001799 wd 0.0500 time 0.3697 (0.3846) data time 0.0005 (0.0019) model time 0.3692 (0.3833) loss 5.0300 (4.2840) grad_norm 1.5638 (inf) loss_scale 16384.0000 (19365.3076) mem 11225MB [2024-08-26 04:04:51 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 17 training takes 0:04:00 [2024-08-26 04:04:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:04:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 1.1543 (1.1543) Acc@1 75.000 (75.000) Acc@5 92.529 (92.529) Mem 11225MB [2024-08-26 04:04:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 1.9580 (1.3399) Acc@1 55.322 (69.269) Acc@5 82.031 (90.976) Mem 11225MB [2024-08-26 04:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 2.0098 (1.6119) Acc@1 55.518 (63.918) Acc@5 80.664 (86.565) Mem 11225MB [2024-08-26 04:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 63.848 Acc@5 86.444 [2024-08-26 04:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 63.8% [2024-08-26 04:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 63.85% [2024-08-26 04:04:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:04:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 2.1367 (2.1367) Acc@1 54.492 (54.492) Acc@5 79.004 (79.004) Mem 11225MB [2024-08-26 04:04:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.143) Loss 2.8633 (2.3338) Acc@1 40.674 (48.815) Acc@5 68.604 (76.349) Mem 11225MB [2024-08-26 04:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 3.0156 (2.5804) Acc@1 37.061 (45.561) Acc@5 64.795 (72.194) Mem 11225MB [2024-08-26 04:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 46.139 Acc@5 72.645 [2024-08-26 04:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 46.1% [2024-08-26 04:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 46.14% [2024-08-26 04:04:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:05:01 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][0/625] eta 0:07:53 lr 0.001800 wd 0.0500 time 0.7582 (0.7582) data time 0.4087 (0.4087) model time 0.0000 (0.0000) loss 3.6250 (3.6250) grad_norm 2.4316 (2.4316) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][10/625] eta 0:04:15 lr 0.001802 wd 0.0500 time 0.3789 (0.4153) data time 0.0012 (0.0383) model time 0.0000 (0.0000) loss 4.2800 (4.2168) grad_norm 3.4781 (1.6423) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][20/625] eta 0:04:02 lr 0.001803 wd 0.0500 time 0.3852 (0.4004) data time 0.0012 (0.0206) model time 0.0000 (0.0000) loss 4.7063 (4.2769) grad_norm 1.3915 (1.8383) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][30/625] eta 0:03:54 lr 0.001805 wd 0.0500 time 0.3819 (0.3938) data time 0.0011 (0.0143) model time 0.0000 (0.0000) loss 4.3449 (4.3297) grad_norm 1.6335 (1.6636) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][40/625] eta 0:03:48 lr 0.001807 wd 0.0500 time 0.3788 (0.3906) data time 0.0009 (0.0112) model time 0.0000 (0.0000) loss 3.5412 (4.2404) grad_norm 1.1816 (1.5480) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][50/625] eta 0:03:44 lr 0.001808 wd 0.0500 time 0.3781 (0.3906) data time 0.0009 (0.0093) model time 0.0000 (0.0000) loss 3.3952 (4.1774) grad_norm 1.5770 (1.5070) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][60/625] eta 0:03:39 lr 0.001810 wd 0.0500 time 0.3841 (0.3890) data time 0.0009 (0.0080) model time 0.3832 (0.3797) loss 5.0973 (4.1904) grad_norm 1.2367 (1.5243) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][70/625] eta 0:03:35 lr 0.001811 wd 0.0500 time 0.3791 (0.3876) data time 0.0010 (0.0070) model time 0.3781 (0.3789) loss 4.1675 (4.1847) grad_norm 1.1319 (1.5270) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][80/625] eta 0:03:30 lr 0.001813 wd 0.0500 time 0.3749 (0.3864) data time 0.0008 (0.0063) model time 0.3741 (0.3781) loss 5.1475 (4.2223) grad_norm 1.3812 (1.5168) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][90/625] eta 0:03:28 lr 0.001815 wd 0.0500 time 0.3788 (0.3901) data time 0.0011 (0.0057) model time 0.3777 (0.3882) loss 4.2288 (4.1984) grad_norm 1.6341 (1.4903) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][100/625] eta 0:03:24 lr 0.001816 wd 0.0500 time 0.4055 (0.3892) data time 0.0010 (0.0053) model time 0.4045 (0.3866) loss 4.4022 (4.2033) grad_norm 1.5499 (1.4912) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][110/625] eta 0:03:19 lr 0.001818 wd 0.0500 time 0.3806 (0.3881) data time 0.0011 (0.0049) model time 0.3794 (0.3848) loss 2.9646 (4.1772) grad_norm 1.5001 (1.4767) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][120/625] eta 0:03:15 lr 0.001819 wd 0.0500 time 0.3787 (0.3873) data time 0.0009 (0.0046) model time 0.3779 (0.3837) loss 4.3585 (4.1597) grad_norm 1.5502 (1.5030) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][130/625] eta 0:03:11 lr 0.001821 wd 0.0500 time 0.3802 (0.3865) data time 0.0011 (0.0043) model time 0.3791 (0.3828) loss 4.4442 (4.1506) grad_norm 1.5337 (1.5127) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][140/625] eta 0:03:07 lr 0.001823 wd 0.0500 time 0.3780 (0.3861) data time 0.0009 (0.0041) model time 0.3771 (0.3825) loss 4.3876 (4.1554) grad_norm 1.6759 (1.5003) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:05:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][150/625] eta 0:03:03 lr 0.001824 wd 0.0500 time 0.3776 (0.3857) data time 0.0008 (0.0039) model time 0.3768 (0.3821) loss 4.1250 (4.1278) grad_norm 1.6448 (1.5033) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][160/625] eta 0:02:59 lr 0.001826 wd 0.0500 time 0.3883 (0.3853) data time 0.0009 (0.0037) model time 0.3874 (0.3817) loss 4.5173 (4.1422) grad_norm 2.0806 (1.4957) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][170/625] eta 0:02:56 lr 0.001827 wd 0.0500 time 0.5900 (0.3874) data time 0.0011 (0.0036) model time 0.5889 (0.3848) loss 4.3017 (4.1384) grad_norm 1.6513 (1.4948) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][180/625] eta 0:02:52 lr 0.001829 wd 0.0500 time 0.3766 (0.3870) data time 0.0008 (0.0035) model time 0.3758 (0.3844) loss 4.9751 (4.1291) grad_norm 1.4650 (1.4877) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][190/625] eta 0:02:48 lr 0.001831 wd 0.0500 time 0.3794 (0.3866) data time 0.0008 (0.0034) model time 0.3786 (0.3839) loss 3.1704 (4.1279) grad_norm 1.1742 (1.4819) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][200/625] eta 0:02:44 lr 0.001832 wd 0.0500 time 0.3800 (0.3862) data time 0.0009 (0.0033) model time 0.3791 (0.3834) loss 5.1244 (4.1443) grad_norm 1.5176 (1.4896) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][210/625] eta 0:02:40 lr 0.001834 wd 0.0500 time 0.3783 (0.3860) data time 0.0009 (0.0032) model time 0.3775 (0.3833) loss 4.1402 (4.1591) grad_norm 1.1441 (1.4852) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][220/625] eta 0:02:36 lr 0.001835 wd 0.0500 time 0.3849 (0.3858) data time 0.0010 (0.0031) model time 0.3838 (0.3831) loss 4.8951 (4.1651) grad_norm 1.6509 (1.4875) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][230/625] eta 0:02:32 lr 0.001837 wd 0.0500 time 0.3650 (0.3855) data time 0.0008 (0.0030) model time 0.3642 (0.3828) loss 3.5214 (4.1727) grad_norm 1.8517 (1.4972) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][240/625] eta 0:02:28 lr 0.001839 wd 0.0500 time 0.3778 (0.3852) data time 0.0011 (0.0029) model time 0.3767 (0.3826) loss 4.5954 (4.1794) grad_norm 0.9292 (1.5008) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][250/625] eta 0:02:24 lr 0.001840 wd 0.0500 time 0.3938 (0.3852) data time 0.0009 (0.0029) model time 0.3929 (0.3826) loss 3.7170 (4.1839) grad_norm 1.4758 (1.4935) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][260/625] eta 0:02:20 lr 0.001842 wd 0.0500 time 0.3821 (0.3850) data time 0.0008 (0.0028) model time 0.3813 (0.3824) loss 4.6531 (4.1730) grad_norm 2.6943 (1.4994) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][270/625] eta 0:02:16 lr 0.001843 wd 0.0500 time 0.3701 (0.3847) data time 0.0008 (0.0027) model time 0.3693 (0.3822) loss 4.2956 (4.1794) grad_norm 1.0737 (1.4977) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][280/625] eta 0:02:12 lr 0.001845 wd 0.0500 time 0.3853 (0.3845) data time 0.0008 (0.0027) model time 0.3845 (0.3820) loss 4.8279 (4.1760) grad_norm 1.4008 (1.4904) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][290/625] eta 0:02:08 lr 0.001847 wd 0.0500 time 0.3784 (0.3843) data time 0.0008 (0.0026) model time 0.3776 (0.3817) loss 5.0353 (4.1809) grad_norm 1.5398 (1.5017) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][300/625] eta 0:02:05 lr 0.001848 wd 0.0500 time 0.3855 (0.3855) data time 0.0011 (0.0026) model time 0.3843 (0.3833) loss 4.7148 (4.1829) grad_norm 1.0024 (1.4985) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][310/625] eta 0:02:01 lr 0.001850 wd 0.0500 time 0.3836 (0.3853) data time 0.0011 (0.0025) model time 0.3825 (0.3831) loss 4.6204 (4.1876) grad_norm 2.4088 (1.4943) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][320/625] eta 0:01:57 lr 0.001851 wd 0.0500 time 0.3745 (0.3852) data time 0.0008 (0.0025) model time 0.3737 (0.3830) loss 4.8490 (4.1920) grad_norm 1.9763 (1.4946) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][330/625] eta 0:01:53 lr 0.001853 wd 0.0500 time 0.3719 (0.3850) data time 0.0011 (0.0024) model time 0.3708 (0.3828) loss 4.5229 (4.1990) grad_norm 1.2209 (1.4917) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][340/625] eta 0:01:49 lr 0.001855 wd 0.0500 time 0.3976 (0.3848) data time 0.0008 (0.0024) model time 0.3969 (0.3826) loss 4.6290 (4.2014) grad_norm 1.1716 (1.4906) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][350/625] eta 0:01:45 lr 0.001856 wd 0.0500 time 0.3807 (0.3847) data time 0.0013 (0.0024) model time 0.3795 (0.3825) loss 4.5079 (4.2042) grad_norm 1.6806 (1.4896) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][360/625] eta 0:01:42 lr 0.001858 wd 0.0500 time 0.3836 (0.3851) data time 0.0009 (0.0023) model time 0.3827 (0.3830) loss 4.5784 (4.2111) grad_norm 1.0343 (1.4863) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][370/625] eta 0:01:38 lr 0.001859 wd 0.0500 time 0.3821 (0.3850) data time 0.0009 (0.0023) model time 0.3811 (0.3828) loss 3.3983 (4.2098) grad_norm 1.2834 (1.4851) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][380/625] eta 0:01:34 lr 0.001861 wd 0.0500 time 0.3724 (0.3853) data time 0.0012 (0.0023) model time 0.3712 (0.3832) loss 4.3877 (4.2090) grad_norm 1.2081 (1.4853) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][390/625] eta 0:01:30 lr 0.001863 wd 0.0500 time 0.3758 (0.3851) data time 0.0011 (0.0023) model time 0.3747 (0.3831) loss 3.9364 (4.2181) grad_norm 1.3927 (1.4817) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][400/625] eta 0:01:26 lr 0.001864 wd 0.0500 time 0.3824 (0.3850) data time 0.0013 (0.0022) model time 0.3811 (0.3830) loss 3.8770 (4.2128) grad_norm 2.1230 (1.4813) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][410/625] eta 0:01:22 lr 0.001866 wd 0.0500 time 0.3742 (0.3848) data time 0.0011 (0.0022) model time 0.3731 (0.3828) loss 4.4397 (4.2173) grad_norm 1.1077 (1.4838) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][420/625] eta 0:01:18 lr 0.001867 wd 0.0500 time 0.3816 (0.3847) data time 0.0008 (0.0022) model time 0.3808 (0.3827) loss 2.9163 (4.2068) grad_norm 0.9832 (1.4837) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][430/625] eta 0:01:14 lr 0.001869 wd 0.0500 time 0.3778 (0.3846) data time 0.0011 (0.0022) model time 0.3767 (0.3826) loss 4.6151 (4.2056) grad_norm 1.4428 (1.4802) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][440/625] eta 0:01:11 lr 0.001871 wd 0.0500 time 0.3721 (0.3844) data time 0.0011 (0.0021) model time 0.3709 (0.3824) loss 3.8379 (4.2043) grad_norm 1.1916 (1.4773) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][450/625] eta 0:01:07 lr 0.001872 wd 0.0500 time 0.3740 (0.3842) data time 0.0008 (0.0021) model time 0.3732 (0.3822) loss 3.5363 (4.2007) grad_norm 1.7217 (1.4775) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:07:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][460/625] eta 0:01:03 lr 0.001874 wd 0.0500 time 0.3906 (0.3841) data time 0.0009 (0.0021) model time 0.3898 (0.3821) loss 4.9882 (4.2054) grad_norm 1.8941 (1.4844) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][470/625] eta 0:00:59 lr 0.001875 wd 0.0500 time 0.3759 (0.3839) data time 0.0010 (0.0021) model time 0.3749 (0.3819) loss 4.7909 (4.2091) grad_norm 1.4460 (1.4845) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][480/625] eta 0:00:55 lr 0.001877 wd 0.0500 time 0.3746 (0.3846) data time 0.0009 (0.0021) model time 0.3738 (0.3827) loss 3.7785 (4.2045) grad_norm 1.3138 (1.4810) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][490/625] eta 0:00:51 lr 0.001879 wd 0.0500 time 0.3911 (0.3849) data time 0.0008 (0.0020) model time 0.3903 (0.3831) loss 3.4171 (4.2048) grad_norm 1.2142 (1.4765) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][500/625] eta 0:00:48 lr 0.001880 wd 0.0500 time 0.3721 (0.3848) data time 0.0008 (0.0020) model time 0.3713 (0.3829) loss 3.1277 (4.2036) grad_norm 0.9082 (1.4741) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][510/625] eta 0:00:44 lr 0.001882 wd 0.0500 time 0.3694 (0.3846) data time 0.0011 (0.0020) model time 0.3683 (0.3827) loss 4.3824 (4.2062) grad_norm 1.1467 (1.4749) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][520/625] eta 0:00:40 lr 0.001883 wd 0.0500 time 0.3801 (0.3844) data time 0.0011 (0.0020) model time 0.3790 (0.3825) loss 4.5686 (4.2102) grad_norm 1.8324 (1.4721) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][530/625] eta 0:00:36 lr 0.001885 wd 0.0500 time 0.3838 (0.3843) data time 0.0011 (0.0020) model time 0.3828 (0.3824) loss 4.5147 (4.2139) grad_norm 0.9950 (1.4672) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][540/625] eta 0:00:32 lr 0.001887 wd 0.0500 time 0.3736 (0.3841) data time 0.0011 (0.0020) model time 0.3726 (0.3823) loss 4.2085 (4.2119) grad_norm 1.8109 (1.4671) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][550/625] eta 0:00:28 lr 0.001888 wd 0.0500 time 0.3751 (0.3840) data time 0.0010 (0.0019) model time 0.3741 (0.3821) loss 4.3360 (4.2124) grad_norm 1.3409 (1.4695) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][560/625] eta 0:00:24 lr 0.001890 wd 0.0500 time 0.3301 (0.3842) data time 0.0012 (0.0019) model time 0.3289 (0.3824) loss 4.1956 (4.2130) grad_norm 1.2196 (1.4672) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][570/625] eta 0:00:21 lr 0.001891 wd 0.0500 time 0.3713 (0.3841) data time 0.0009 (0.0019) model time 0.3704 (0.3823) loss 2.8290 (4.2113) grad_norm 1.0696 (1.4642) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][580/625] eta 0:00:17 lr 0.001893 wd 0.0500 time 0.3738 (0.3839) data time 0.0010 (0.0019) model time 0.3728 (0.3821) loss 5.1547 (4.2144) grad_norm 1.3377 (1.4628) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][590/625] eta 0:00:13 lr 0.001895 wd 0.0500 time 0.3721 (0.3838) data time 0.0011 (0.0019) model time 0.3711 (0.3820) loss 3.9317 (4.2110) grad_norm 0.9703 (1.4632) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][600/625] eta 0:00:09 lr 0.001896 wd 0.0500 time 0.5988 (0.3844) data time 0.0010 (0.0019) model time 0.5977 (0.3827) loss 4.5535 (4.2139) grad_norm 1.5199 (1.4623) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][610/625] eta 0:00:05 lr 0.001898 wd 0.0500 time 0.3762 (0.3843) data time 0.0008 (0.0019) model time 0.3754 (0.3825) loss 4.5011 (4.2141) grad_norm 1.1991 (1.4619) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [18/300][620/625] eta 0:00:01 lr 0.001899 wd 0.0500 time 0.3660 (0.3840) data time 0.0006 (0.0019) model time 0.3655 (0.3822) loss 4.0469 (4.2117) grad_norm 1.0165 (1.4635) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:00 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 18 training takes 0:03:59 [2024-08-26 04:09:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:09:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:09:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 1.0195 (1.0195) Acc@1 75.732 (75.732) Acc@5 93.164 (93.164) Mem 11225MB [2024-08-26 04:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.7930 (1.2509) Acc@1 58.398 (70.264) Acc@5 84.521 (91.499) Mem 11225MB [2024-08-26 04:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.9648 (1.5317) Acc@1 55.908 (64.981) Acc@5 80.322 (87.267) Mem 11225MB [2024-08-26 04:09:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 64.863 Acc@5 87.174 [2024-08-26 04:09:05 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 64.9% [2024-08-26 04:09:05 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 64.86% [2024-08-26 04:09:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:09:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 1.8223 (1.8223) Acc@1 59.424 (59.424) Acc@5 82.275 (82.275) Mem 11225MB [2024-08-26 04:09:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.142) Loss 2.5918 (2.0370) Acc@1 45.312 (53.680) Acc@5 72.217 (80.096) Mem 11225MB [2024-08-26 04:09:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 2.7441 (2.2992) Acc@1 41.113 (49.793) Acc@5 68.555 (75.781) Mem 11225MB [2024-08-26 04:09:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 50.252 Acc@5 76.112 [2024-08-26 04:09:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 50.3% [2024-08-26 04:09:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 50.25% [2024-08-26 04:09:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:09:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:09:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][0/625] eta 0:07:52 lr 0.001900 wd 0.0500 time 0.7560 (0.7560) data time 0.4042 (0.4042) model time 0.0000 (0.0000) loss 4.0600 (4.0600) grad_norm 1.1995 (1.1995) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][10/625] eta 0:04:24 lr 0.001902 wd 0.0500 time 0.3327 (0.4302) data time 0.0009 (0.0379) model time 0.0000 (0.0000) loss 4.3376 (4.1955) grad_norm 1.1410 (1.4319) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][20/625] eta 0:04:05 lr 0.001903 wd 0.0500 time 0.3717 (0.4050) data time 0.0008 (0.0204) model time 0.0000 (0.0000) loss 5.1388 (4.3107) grad_norm 1.4932 (1.4094) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][30/625] eta 0:03:55 lr 0.001905 wd 0.0500 time 0.3699 (0.3962) data time 0.0011 (0.0142) model time 0.0000 (0.0000) loss 3.8543 (4.3120) grad_norm 1.3999 (1.4004) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][40/625] eta 0:03:52 lr 0.001906 wd 0.0500 time 0.3816 (0.3967) data time 0.0008 (0.0110) model time 0.0000 (0.0000) loss 4.8129 (4.2491) grad_norm 1.2237 (1.3935) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][50/625] eta 0:03:45 lr 0.001908 wd 0.0500 time 0.3712 (0.3925) data time 0.0011 (0.0091) model time 0.0000 (0.0000) loss 3.4670 (4.1950) grad_norm 1.2265 (1.4103) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][60/625] eta 0:03:40 lr 0.001910 wd 0.0500 time 0.3764 (0.3899) data time 0.0008 (0.0078) model time 0.3756 (0.3755) loss 5.3936 (4.2567) grad_norm 1.7190 (1.4135) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][70/625] eta 0:03:35 lr 0.001911 wd 0.0500 time 0.3880 (0.3881) data time 0.0011 (0.0069) model time 0.3869 (0.3759) loss 4.6617 (4.2381) grad_norm 1.4164 (1.4534) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][80/625] eta 0:03:30 lr 0.001913 wd 0.0500 time 0.3719 (0.3870) data time 0.0011 (0.0061) model time 0.3708 (0.3765) loss 3.9156 (4.2257) grad_norm 1.4505 (1.4996) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][90/625] eta 0:03:26 lr 0.001914 wd 0.0500 time 0.3800 (0.3859) data time 0.0010 (0.0056) model time 0.3789 (0.3764) loss 4.6551 (4.2474) grad_norm 1.3171 (1.4809) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][100/625] eta 0:03:22 lr 0.001916 wd 0.0500 time 0.3816 (0.3852) data time 0.0011 (0.0052) model time 0.3806 (0.3767) loss 3.6070 (4.2501) grad_norm 1.0074 (1.4636) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][110/625] eta 0:03:18 lr 0.001918 wd 0.0500 time 0.3754 (0.3863) data time 0.0008 (0.0048) model time 0.3746 (0.3800) loss 4.9880 (4.2538) grad_norm 1.3029 (1.4558) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][120/625] eta 0:03:14 lr 0.001919 wd 0.0500 time 0.3754 (0.3856) data time 0.0011 (0.0045) model time 0.3743 (0.3794) loss 4.2540 (4.2531) grad_norm 1.0105 (1.4603) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][130/625] eta 0:03:10 lr 0.001921 wd 0.0500 time 0.3748 (0.3849) data time 0.0008 (0.0043) model time 0.3739 (0.3789) loss 4.7388 (4.2813) grad_norm 2.0302 (1.4637) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][140/625] eta 0:03:06 lr 0.001922 wd 0.0500 time 0.3821 (0.3845) data time 0.0013 (0.0040) model time 0.3808 (0.3788) loss 4.4401 (4.2869) grad_norm 1.4841 (1.4771) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][150/625] eta 0:03:02 lr 0.001924 wd 0.0500 time 0.3721 (0.3841) data time 0.0011 (0.0038) model time 0.3710 (0.3786) loss 4.3737 (4.2815) grad_norm 1.2564 (1.4653) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][160/625] eta 0:02:58 lr 0.001926 wd 0.0500 time 0.3850 (0.3838) data time 0.0008 (0.0037) model time 0.3842 (0.3786) loss 3.6465 (4.2661) grad_norm 1.4479 (1.4676) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][170/625] eta 0:02:54 lr 0.001927 wd 0.0500 time 0.3760 (0.3845) data time 0.0009 (0.0035) model time 0.3751 (0.3799) loss 4.9440 (4.2738) grad_norm 1.0424 (1.4701) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][180/625] eta 0:02:50 lr 0.001929 wd 0.0500 time 0.3765 (0.3841) data time 0.0009 (0.0034) model time 0.3757 (0.3796) loss 3.3844 (4.2507) grad_norm 1.3893 (1.4570) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][190/625] eta 0:02:46 lr 0.001930 wd 0.0500 time 0.3732 (0.3837) data time 0.0008 (0.0033) model time 0.3724 (0.3793) loss 5.0558 (4.2539) grad_norm 1.2953 (1.4539) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][200/625] eta 0:02:42 lr 0.001932 wd 0.0500 time 0.3764 (0.3833) data time 0.0011 (0.0032) model time 0.3753 (0.3791) loss 3.8166 (4.2494) grad_norm 1.3054 (1.4490) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][210/625] eta 0:02:38 lr 0.001934 wd 0.0500 time 0.3744 (0.3830) data time 0.0011 (0.0031) model time 0.3733 (0.3788) loss 4.5000 (4.2544) grad_norm 0.9757 (1.4430) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][220/625] eta 0:02:35 lr 0.001935 wd 0.0500 time 0.3759 (0.3837) data time 0.0010 (0.0030) model time 0.3749 (0.3799) loss 2.9055 (4.2630) grad_norm 2.1824 (1.4395) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][230/625] eta 0:02:31 lr 0.001937 wd 0.0500 time 0.3735 (0.3833) data time 0.0012 (0.0029) model time 0.3723 (0.3795) loss 4.5401 (4.2557) grad_norm 1.3516 (1.4462) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][240/625] eta 0:02:27 lr 0.001938 wd 0.0500 time 0.5854 (0.3839) data time 0.0012 (0.0029) model time 0.5842 (0.3805) loss 4.7042 (4.2549) grad_norm 1.3676 (1.4447) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][250/625] eta 0:02:23 lr 0.001940 wd 0.0500 time 0.3800 (0.3837) data time 0.0011 (0.0028) model time 0.3790 (0.3803) loss 4.7512 (4.2622) grad_norm 1.5109 (1.4494) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][260/625] eta 0:02:19 lr 0.001942 wd 0.0500 time 0.3781 (0.3835) data time 0.0010 (0.0027) model time 0.3772 (0.3802) loss 4.4991 (4.2614) grad_norm 1.2242 (1.4411) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][270/625] eta 0:02:16 lr 0.001943 wd 0.0500 time 0.3772 (0.3833) data time 0.0011 (0.0027) model time 0.3761 (0.3800) loss 4.1777 (4.2675) grad_norm 1.1743 (1.4372) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:10:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][280/625] eta 0:02:12 lr 0.001945 wd 0.0500 time 0.3790 (0.3831) data time 0.0011 (0.0026) model time 0.3779 (0.3798) loss 4.2791 (4.2758) grad_norm 1.6207 (1.4395) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][290/625] eta 0:02:08 lr 0.001946 wd 0.0500 time 0.3804 (0.3837) data time 0.0008 (0.0026) model time 0.3795 (0.3807) loss 3.2031 (4.2760) grad_norm 1.1761 (1.4375) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][300/625] eta 0:02:04 lr 0.001948 wd 0.0500 time 0.3761 (0.3835) data time 0.0008 (0.0025) model time 0.3753 (0.3805) loss 4.6338 (4.2871) grad_norm 1.9877 (1.4375) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][310/625] eta 0:02:00 lr 0.001950 wd 0.0500 time 0.3847 (0.3835) data time 0.0008 (0.0025) model time 0.3839 (0.3805) loss 3.6231 (4.2695) grad_norm 1.2995 (1.4323) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][320/625] eta 0:01:57 lr 0.001951 wd 0.0500 time 0.3806 (0.3840) data time 0.0011 (0.0024) model time 0.3795 (0.3812) loss 3.9697 (4.2622) grad_norm 1.6625 (1.4303) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][330/625] eta 0:01:53 lr 0.001953 wd 0.0500 time 0.3714 (0.3838) data time 0.0011 (0.0024) model time 0.3704 (0.3810) loss 4.5142 (4.2554) grad_norm 1.7753 (1.4337) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][340/625] eta 0:01:49 lr 0.001954 wd 0.0500 time 0.3748 (0.3836) data time 0.0009 (0.0024) model time 0.3739 (0.3808) loss 5.1496 (4.2619) grad_norm 1.1742 (1.4343) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][350/625] eta 0:01:45 lr 0.001956 wd 0.0500 time 0.3686 (0.3834) data time 0.0009 (0.0023) model time 0.3678 (0.3806) loss 2.9407 (4.2583) grad_norm 1.1482 (1.4355) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][360/625] eta 0:01:41 lr 0.001958 wd 0.0500 time 0.3760 (0.3832) data time 0.0011 (0.0023) model time 0.3749 (0.3805) loss 4.5343 (4.2566) grad_norm 1.3420 (1.4296) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][370/625] eta 0:01:37 lr 0.001959 wd 0.0500 time 0.3728 (0.3831) data time 0.0012 (0.0023) model time 0.3716 (0.3804) loss 4.2516 (4.2525) grad_norm 1.5019 (1.4287) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][380/625] eta 0:01:33 lr 0.001961 wd 0.0500 time 0.3771 (0.3830) data time 0.0008 (0.0022) model time 0.3762 (0.3804) loss 4.7849 (4.2534) grad_norm 1.5781 (1.4273) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][390/625] eta 0:01:29 lr 0.001962 wd 0.0500 time 0.3728 (0.3829) data time 0.0009 (0.0022) model time 0.3719 (0.3803) loss 4.8739 (4.2546) grad_norm 1.2385 (1.4265) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][400/625] eta 0:01:26 lr 0.001964 wd 0.0500 time 0.3831 (0.3828) data time 0.0010 (0.0022) model time 0.3820 (0.3802) loss 4.3287 (4.2521) grad_norm 1.3354 (1.4314) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][410/625] eta 0:01:22 lr 0.001966 wd 0.0500 time 0.3651 (0.3832) data time 0.0009 (0.0022) model time 0.3642 (0.3807) loss 5.4199 (4.2506) grad_norm 0.9271 (1.4299) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][420/625] eta 0:01:18 lr 0.001967 wd 0.0500 time 0.3809 (0.3831) data time 0.0011 (0.0021) model time 0.3798 (0.3806) loss 4.3064 (4.2405) grad_norm 1.2442 (1.4299) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][430/625] eta 0:01:14 lr 0.001969 wd 0.0500 time 0.3821 (0.3835) data time 0.0011 (0.0021) model time 0.3811 (0.3811) loss 3.8887 (4.2271) grad_norm 1.2582 (1.4289) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:11:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][440/625] eta 0:01:10 lr 0.001970 wd 0.0500 time 0.3847 (0.3834) data time 0.0011 (0.0021) model time 0.3835 (0.3810) loss 4.6200 (4.2227) grad_norm 1.2808 (1.4282) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][450/625] eta 0:01:07 lr 0.001972 wd 0.0500 time 0.3782 (0.3833) data time 0.0012 (0.0021) model time 0.3770 (0.3809) loss 4.4816 (4.2184) grad_norm 1.0811 (1.4271) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][460/625] eta 0:01:03 lr 0.001974 wd 0.0500 time 0.3800 (0.3832) data time 0.0008 (0.0021) model time 0.3792 (0.3809) loss 4.3903 (4.2103) grad_norm 1.7841 (1.4259) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][470/625] eta 0:00:59 lr 0.001975 wd 0.0500 time 0.3784 (0.3831) data time 0.0009 (0.0020) model time 0.3776 (0.3808) loss 4.2348 (4.2166) grad_norm 0.7822 (1.4242) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][480/625] eta 0:00:55 lr 0.001977 wd 0.0500 time 0.3836 (0.3830) data time 0.0008 (0.0020) model time 0.3828 (0.3807) loss 3.4514 (4.2098) grad_norm 1.2090 (1.4165) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][490/625] eta 0:00:51 lr 0.001978 wd 0.0500 time 0.3834 (0.3829) data time 0.0010 (0.0020) model time 0.3824 (0.3806) loss 3.2439 (4.2141) grad_norm 2.1840 (1.4189) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][500/625] eta 0:00:47 lr 0.001980 wd 0.0500 time 0.3801 (0.3836) data time 0.0008 (0.0020) model time 0.3793 (0.3814) loss 4.6338 (4.2177) grad_norm 1.8653 (1.4204) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][510/625] eta 0:00:44 lr 0.001982 wd 0.0500 time 0.3732 (0.3835) data time 0.0011 (0.0020) model time 0.3721 (0.3813) loss 3.8420 (4.2217) grad_norm 2.2339 (1.4239) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][520/625] eta 0:00:40 lr 0.001983 wd 0.0500 time 0.6076 (0.3838) data time 0.0011 (0.0019) model time 0.6066 (0.3817) loss 4.1252 (4.2170) grad_norm 1.3629 (1.4229) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][530/625] eta 0:00:36 lr 0.001985 wd 0.0500 time 0.3760 (0.3836) data time 0.0011 (0.0019) model time 0.3749 (0.3815) loss 4.6654 (4.2162) grad_norm 1.1456 (1.4210) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][540/625] eta 0:00:32 lr 0.001986 wd 0.0500 time 0.3776 (0.3835) data time 0.0010 (0.0019) model time 0.3766 (0.3814) loss 3.7642 (4.2132) grad_norm 1.2878 (1.4208) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][550/625] eta 0:00:28 lr 0.001988 wd 0.0500 time 0.3943 (0.3838) data time 0.0010 (0.0019) model time 0.3933 (0.3817) loss 3.7200 (4.2152) grad_norm 1.5680 (1.4211) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][560/625] eta 0:00:24 lr 0.001990 wd 0.0500 time 0.3790 (0.3836) data time 0.0008 (0.0019) model time 0.3782 (0.3816) loss 4.7122 (4.2117) grad_norm 1.3310 (1.4199) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][570/625] eta 0:00:21 lr 0.001991 wd 0.0500 time 0.3705 (0.3835) data time 0.0009 (0.0019) model time 0.3696 (0.3814) loss 4.6856 (4.2129) grad_norm 1.3077 (1.4174) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][580/625] eta 0:00:17 lr 0.001993 wd 0.0500 time 0.3815 (0.3834) data time 0.0011 (0.0019) model time 0.3804 (0.3814) loss 4.4565 (4.2102) grad_norm 1.5396 (1.4155) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:12:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][590/625] eta 0:00:13 lr 0.001994 wd 0.0500 time 0.3826 (0.3833) data time 0.0010 (0.0019) model time 0.3816 (0.3813) loss 4.1415 (4.2132) grad_norm 1.2224 (1.4157) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][600/625] eta 0:00:09 lr 0.001996 wd 0.0500 time 0.3778 (0.3832) data time 0.0008 (0.0018) model time 0.3769 (0.3812) loss 3.4921 (4.2092) grad_norm 2.3981 (1.4185) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][610/625] eta 0:00:05 lr 0.001998 wd 0.0500 time 0.3728 (0.3832) data time 0.0006 (0.0018) model time 0.3723 (0.3812) loss 3.8339 (4.2133) grad_norm 1.1194 (1.4170) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [19/300][620/625] eta 0:00:01 lr 0.001999 wd 0.0500 time 0.3657 (0.3833) data time 0.0008 (0.0018) model time 0.3649 (0.3813) loss 4.0969 (4.2119) grad_norm 1.7629 (1.4194) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:09 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 19 training takes 0:03:59 [2024-08-26 04:13:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:13:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 1.0020 (1.0020) Acc@1 76.514 (76.514) Acc@5 93.652 (93.652) Mem 11225MB [2024-08-26 04:13:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 1.7373 (1.1968) Acc@1 60.303 (71.245) Acc@5 85.205 (92.032) Mem 11225MB [2024-08-26 04:13:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.9248 (1.4752) Acc@1 57.275 (65.904) Acc@5 81.543 (88.049) Mem 11225MB [2024-08-26 04:13:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 65.773 Acc@5 87.986 [2024-08-26 04:13:13 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 65.8% [2024-08-26 04:13:13 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 65.77% [2024-08-26 04:13:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:13:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:13:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 1.5811 (1.5811) Acc@1 62.744 (62.744) Acc@5 85.840 (85.840) Mem 11225MB [2024-08-26 04:13:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 2.3613 (1.8028) Acc@1 48.486 (57.657) Acc@5 74.658 (83.017) Mem 11225MB [2024-08-26 04:13:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 2.5215 (2.0748) Acc@1 44.629 (53.344) Acc@5 70.703 (78.604) Mem 11225MB [2024-08-26 04:13:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 53.625 Acc@5 78.837 [2024-08-26 04:13:17 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 53.6% [2024-08-26 04:13:17 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 53.62% [2024-08-26 04:13:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:13:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][0/625] eta 0:08:12 lr 0.002000 wd 0.0500 time 0.7885 (0.7885) data time 0.4490 (0.4490) model time 0.0000 (0.0000) loss 4.2819 (4.2819) grad_norm 1.6116 (1.6116) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][10/625] eta 0:04:15 lr 0.002000 wd 0.0500 time 0.3764 (0.4150) data time 0.0011 (0.0419) model time 0.0000 (0.0000) loss 3.7706 (3.9162) grad_norm 1.2551 (1.5032) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][20/625] eta 0:04:05 lr 0.002000 wd 0.0500 time 0.3801 (0.4062) data time 0.0009 (0.0225) model time 0.0000 (0.0000) loss 2.6125 (4.0355) grad_norm 1.5812 (1.4618) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][30/625] eta 0:03:56 lr 0.002000 wd 0.0500 time 0.3753 (0.3968) data time 0.0009 (0.0156) model time 0.0000 (0.0000) loss 3.1346 (3.9903) grad_norm 1.4491 (1.4244) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][40/625] eta 0:03:49 lr 0.002000 wd 0.0500 time 0.3805 (0.3921) data time 0.0012 (0.0121) model time 0.0000 (0.0000) loss 4.2141 (4.0532) grad_norm 1.9148 (1.5115) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][50/625] eta 0:03:43 lr 0.002000 wd 0.0500 time 0.3731 (0.3891) data time 0.0011 (0.0099) model time 0.0000 (0.0000) loss 4.2488 (4.0464) grad_norm 1.2656 (1.4697) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][60/625] eta 0:03:40 lr 0.002000 wd 0.0500 time 0.3748 (0.3911) data time 0.0011 (0.0085) model time 0.3737 (0.4001) loss 4.0274 (4.0735) grad_norm 1.2735 (1.4529) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][70/625] eta 0:03:36 lr 0.002000 wd 0.0500 time 0.3773 (0.3893) data time 0.0008 (0.0075) model time 0.3765 (0.3887) loss 5.3208 (4.1225) grad_norm 0.9410 (1.4445) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][80/625] eta 0:03:32 lr 0.002000 wd 0.0500 time 0.3705 (0.3896) data time 0.0010 (0.0067) model time 0.3694 (0.3893) loss 4.0190 (4.1149) grad_norm 1.4234 (1.4263) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][90/625] eta 0:03:27 lr 0.002000 wd 0.0500 time 0.3807 (0.3880) data time 0.0008 (0.0061) model time 0.3799 (0.3856) loss 4.5732 (4.0800) grad_norm 0.9651 (1.4234) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][100/625] eta 0:03:23 lr 0.002000 wd 0.0500 time 0.3867 (0.3869) data time 0.0008 (0.0056) model time 0.3859 (0.3836) loss 3.7838 (4.0903) grad_norm 1.8869 (1.4182) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][110/625] eta 0:03:18 lr 0.002000 wd 0.0500 time 0.3743 (0.3861) data time 0.0011 (0.0052) model time 0.3732 (0.3825) loss 4.3073 (4.0987) grad_norm 0.9653 (1.4139) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][120/625] eta 0:03:14 lr 0.002000 wd 0.0500 time 0.3675 (0.3852) data time 0.0008 (0.0049) model time 0.3667 (0.3811) loss 2.8748 (4.0849) grad_norm 1.5395 (1.4150) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][130/625] eta 0:03:10 lr 0.002000 wd 0.0500 time 0.3821 (0.3845) data time 0.0008 (0.0046) model time 0.3813 (0.3804) loss 2.9080 (4.0831) grad_norm 1.2256 (1.4182) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][140/625] eta 0:03:06 lr 0.002000 wd 0.0500 time 0.3672 (0.3839) data time 0.0012 (0.0044) model time 0.3660 (0.3797) loss 4.6381 (4.0873) grad_norm 1.2396 (1.4139) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][150/625] eta 0:03:02 lr 0.002000 wd 0.0500 time 0.3670 (0.3835) data time 0.0011 (0.0042) model time 0.3659 (0.3794) loss 4.4383 (4.0964) grad_norm 1.4700 (1.4025) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][160/625] eta 0:02:58 lr 0.002000 wd 0.0500 time 0.3862 (0.3833) data time 0.0009 (0.0040) model time 0.3853 (0.3794) loss 3.8106 (4.1116) grad_norm 1.3794 (1.4053) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][170/625] eta 0:02:54 lr 0.002000 wd 0.0500 time 0.3754 (0.3828) data time 0.0011 (0.0038) model time 0.3743 (0.3789) loss 3.9913 (4.1066) grad_norm 1.4984 (1.4080) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][180/625] eta 0:02:50 lr 0.002000 wd 0.0500 time 0.3790 (0.3827) data time 0.0011 (0.0037) model time 0.3779 (0.3790) loss 4.6749 (4.1345) grad_norm 1.0435 (1.4196) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][190/625] eta 0:02:46 lr 0.002000 wd 0.0500 time 0.3775 (0.3825) data time 0.0011 (0.0036) model time 0.3765 (0.3789) loss 3.7915 (4.1541) grad_norm 1.0402 (1.4090) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][200/625] eta 0:02:43 lr 0.002000 wd 0.0500 time 0.3748 (0.3842) data time 0.0011 (0.0035) model time 0.3737 (0.3814) loss 3.9807 (4.1563) grad_norm 1.3609 (1.4057) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][210/625] eta 0:02:39 lr 0.002000 wd 0.0500 time 0.3815 (0.3840) data time 0.0011 (0.0033) model time 0.3804 (0.3811) loss 4.3867 (4.1604) grad_norm 1.2926 (1.4289) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][220/625] eta 0:02:35 lr 0.002000 wd 0.0500 time 0.3800 (0.3838) data time 0.0009 (0.0032) model time 0.3792 (0.3810) loss 4.8820 (4.1550) grad_norm 1.3176 (1.4179) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][230/625] eta 0:02:31 lr 0.002000 wd 0.0500 time 0.3807 (0.3837) data time 0.0009 (0.0032) model time 0.3798 (0.3809) loss 3.3214 (4.1452) grad_norm 1.2899 (1.4193) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:14:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][240/625] eta 0:02:27 lr 0.002000 wd 0.0500 time 0.5583 (0.3841) data time 0.0011 (0.0031) model time 0.5572 (0.3816) loss 4.5576 (4.1569) grad_norm 0.8908 (1.4170) loss_scale 32768.0000 (16587.9502) mem 11225MB [2024-08-26 04:14:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][250/625] eta 0:02:23 lr 0.002000 wd 0.0500 time 0.3869 (0.3839) data time 0.0008 (0.0030) model time 0.3861 (0.3814) loss 5.0583 (4.1614) grad_norm 1.2437 (1.4110) loss_scale 32768.0000 (17232.5737) mem 11225MB [2024-08-26 04:14:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][260/625] eta 0:02:20 lr 0.002000 wd 0.0500 time 0.3794 (0.3838) data time 0.0012 (0.0029) model time 0.3782 (0.3813) loss 4.9540 (4.1714) grad_norm 1.3416 (1.4042) loss_scale 32768.0000 (17827.8008) mem 11225MB [2024-08-26 04:15:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][270/625] eta 0:02:16 lr 0.002000 wd 0.0500 time 0.3790 (0.3836) data time 0.0011 (0.0029) model time 0.3779 (0.3812) loss 4.6979 (4.1674) grad_norm 1.2145 (1.3981) loss_scale 32768.0000 (18379.0996) mem 11225MB [2024-08-26 04:15:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][280/625] eta 0:02:12 lr 0.002000 wd 0.0500 time 0.3897 (0.3836) data time 0.0008 (0.0028) model time 0.3889 (0.3812) loss 2.9741 (4.1586) grad_norm 1.1017 (1.3972) loss_scale 32768.0000 (18891.1601) mem 11225MB [2024-08-26 04:15:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][290/625] eta 0:02:08 lr 0.002000 wd 0.0500 time 0.3830 (0.3843) data time 0.0011 (0.0027) model time 0.3819 (0.3820) loss 4.0129 (4.1572) grad_norm 1.2992 (1.3992) loss_scale 32768.0000 (19368.0275) mem 11225MB [2024-08-26 04:15:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][300/625] eta 0:02:04 lr 0.002000 wd 0.0500 time 0.3808 (0.3841) data time 0.0010 (0.0027) model time 0.3797 (0.3819) loss 3.0117 (4.1515) grad_norm 2.0918 (1.3980) loss_scale 32768.0000 (19813.2093) mem 11225MB [2024-08-26 04:15:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][310/625] eta 0:02:00 lr 0.002000 wd 0.0500 time 0.3902 (0.3840) data time 0.0008 (0.0026) model time 0.3894 (0.3818) loss 3.4957 (4.1503) grad_norm 1.0963 (1.3927) loss_scale 32768.0000 (20229.7621) mem 11225MB [2024-08-26 04:15:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][320/625] eta 0:01:57 lr 0.002000 wd 0.0500 time 0.3746 (0.3844) data time 0.0007 (0.0026) model time 0.3738 (0.3823) loss 4.6324 (4.1554) grad_norm 1.0315 (1.3892) loss_scale 32768.0000 (20620.3614) mem 11225MB [2024-08-26 04:15:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][330/625] eta 0:01:53 lr 0.002000 wd 0.0500 time 0.3820 (0.3842) data time 0.0009 (0.0026) model time 0.3811 (0.3821) loss 4.6742 (4.1662) grad_norm 1.4614 (1.3911) loss_scale 32768.0000 (20987.3595) mem 11225MB [2024-08-26 04:15:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][340/625] eta 0:01:49 lr 0.002000 wd 0.0500 time 0.3814 (0.3841) data time 0.0013 (0.0026) model time 0.3802 (0.3819) loss 3.9189 (4.1726) grad_norm 1.2511 (1.4024) loss_scale 32768.0000 (21332.8328) mem 11225MB [2024-08-26 04:15:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][350/625] eta 0:01:45 lr 0.002000 wd 0.0500 time 0.3824 (0.3840) data time 0.0011 (0.0025) model time 0.3813 (0.3818) loss 4.6196 (4.1673) grad_norm 1.1552 (1.4017) loss_scale 32768.0000 (21658.6211) mem 11225MB [2024-08-26 04:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][360/625] eta 0:01:41 lr 0.002000 wd 0.0500 time 0.5915 (0.3844) data time 0.0011 (0.0025) model time 0.5904 (0.3824) loss 4.5906 (4.1677) grad_norm 1.3600 (1.3965) loss_scale 32768.0000 (21966.3601) mem 11225MB [2024-08-26 04:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][370/625] eta 0:01:37 lr 0.002000 wd 0.0500 time 0.3925 (0.3843) data time 0.0009 (0.0025) model time 0.3916 (0.3822) loss 4.2889 (4.1689) grad_norm 1.1602 (1.3959) loss_scale 32768.0000 (22257.5094) mem 11225MB [2024-08-26 04:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][380/625] eta 0:01:34 lr 0.002000 wd 0.0500 time 0.3466 (0.3840) data time 0.0011 (0.0024) model time 0.3455 (0.3819) loss 4.2897 (4.1587) grad_norm inf (inf) loss_scale 16384.0000 (22490.3727) mem 11225MB [2024-08-26 04:15:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][390/625] eta 0:01:30 lr 0.002000 wd 0.0500 time 0.3783 (0.3843) data time 0.0009 (0.0024) model time 0.3775 (0.3823) loss 4.7940 (4.1544) grad_norm 1.1385 (inf) loss_scale 16384.0000 (22334.1995) mem 11225MB [2024-08-26 04:15:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][400/625] eta 0:01:26 lr 0.002000 wd 0.0500 time 0.3734 (0.3843) data time 0.0012 (0.0024) model time 0.3722 (0.3823) loss 4.5528 (4.1512) grad_norm 1.1726 (inf) loss_scale 16384.0000 (22185.8155) mem 11225MB [2024-08-26 04:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][410/625] eta 0:01:22 lr 0.002000 wd 0.0500 time 0.3752 (0.3842) data time 0.0009 (0.0024) model time 0.3743 (0.3822) loss 3.4239 (4.1527) grad_norm 1.0929 (inf) loss_scale 16384.0000 (22044.6521) mem 11225MB [2024-08-26 04:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][420/625] eta 0:01:18 lr 0.002000 wd 0.0500 time 0.3790 (0.3841) data time 0.0010 (0.0023) model time 0.3780 (0.3821) loss 3.7880 (4.1489) grad_norm 1.1777 (inf) loss_scale 16384.0000 (21910.1948) mem 11225MB [2024-08-26 04:16:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][430/625] eta 0:01:14 lr 0.002000 wd 0.0500 time 0.3742 (0.3840) data time 0.0010 (0.0023) model time 0.3732 (0.3820) loss 3.6974 (4.1470) grad_norm 1.2339 (inf) loss_scale 16384.0000 (21781.9768) mem 11225MB [2024-08-26 04:16:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][440/625] eta 0:01:11 lr 0.002000 wd 0.0500 time 0.3747 (0.3839) data time 0.0011 (0.0023) model time 0.3736 (0.3819) loss 3.1543 (4.1503) grad_norm 1.3300 (inf) loss_scale 16384.0000 (21659.5737) mem 11225MB [2024-08-26 04:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][450/625] eta 0:01:07 lr 0.002000 wd 0.0500 time 0.3721 (0.3837) data time 0.0009 (0.0022) model time 0.3712 (0.3818) loss 3.9046 (4.1503) grad_norm 1.4487 (inf) loss_scale 16384.0000 (21542.5987) mem 11225MB [2024-08-26 04:16:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][460/625] eta 0:01:03 lr 0.002000 wd 0.0500 time 0.3768 (0.3836) data time 0.0012 (0.0022) model time 0.3756 (0.3817) loss 4.6385 (4.1496) grad_norm 1.4074 (inf) loss_scale 16384.0000 (21430.6985) mem 11225MB [2024-08-26 04:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][470/625] eta 0:00:59 lr 0.002000 wd 0.0500 time 0.3824 (0.3839) data time 0.0008 (0.0022) model time 0.3816 (0.3820) loss 3.7542 (4.1482) grad_norm 1.6261 (inf) loss_scale 16384.0000 (21323.5499) mem 11225MB [2024-08-26 04:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][480/625] eta 0:00:55 lr 0.002000 wd 0.0500 time 0.3814 (0.3838) data time 0.0010 (0.0022) model time 0.3804 (0.3819) loss 4.4827 (4.1456) grad_norm 0.9809 (inf) loss_scale 16384.0000 (21220.8565) mem 11225MB [2024-08-26 04:16:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][490/625] eta 0:00:51 lr 0.002000 wd 0.0500 time 0.3812 (0.3837) data time 0.0008 (0.0022) model time 0.3804 (0.3819) loss 4.4442 (4.1418) grad_norm 1.1786 (inf) loss_scale 16384.0000 (21122.3462) mem 11225MB [2024-08-26 04:16:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][500/625] eta 0:00:47 lr 0.002000 wd 0.0500 time 0.3852 (0.3837) data time 0.0009 (0.0022) model time 0.3843 (0.3818) loss 4.9181 (4.1422) grad_norm 1.3654 (inf) loss_scale 16384.0000 (21027.7685) mem 11225MB [2024-08-26 04:16:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][510/625] eta 0:00:44 lr 0.002000 wd 0.0500 time 0.3799 (0.3839) data time 0.0011 (0.0021) model time 0.3788 (0.3821) loss 4.6346 (4.1513) grad_norm 1.5584 (inf) loss_scale 16384.0000 (20936.8924) mem 11225MB [2024-08-26 04:16:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][520/625] eta 0:00:40 lr 0.002000 wd 0.0500 time 0.3800 (0.3838) data time 0.0008 (0.0021) model time 0.3792 (0.3820) loss 3.9285 (4.1500) grad_norm 3.4399 (inf) loss_scale 16384.0000 (20849.5048) mem 11225MB [2024-08-26 04:16:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][530/625] eta 0:00:36 lr 0.002000 wd 0.0500 time 0.3796 (0.3841) data time 0.0008 (0.0021) model time 0.3788 (0.3823) loss 5.2357 (4.1574) grad_norm 1.1311 (inf) loss_scale 16384.0000 (20765.4087) mem 11225MB [2024-08-26 04:16:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][540/625] eta 0:00:32 lr 0.002000 wd 0.0500 time 0.3728 (0.3841) data time 0.0008 (0.0021) model time 0.3720 (0.3823) loss 4.3902 (4.1590) grad_norm 1.5625 (inf) loss_scale 16384.0000 (20684.4214) mem 11225MB [2024-08-26 04:16:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][550/625] eta 0:00:28 lr 0.002000 wd 0.0500 time 0.3833 (0.3839) data time 0.0008 (0.0021) model time 0.3824 (0.3821) loss 5.0562 (4.1565) grad_norm 1.0539 (inf) loss_scale 16384.0000 (20606.3739) mem 11225MB [2024-08-26 04:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][560/625] eta 0:00:24 lr 0.002000 wd 0.0500 time 0.3767 (0.3838) data time 0.0011 (0.0020) model time 0.3756 (0.3821) loss 4.2754 (4.1604) grad_norm 1.6189 (inf) loss_scale 16384.0000 (20531.1087) mem 11225MB [2024-08-26 04:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][570/625] eta 0:00:21 lr 0.002000 wd 0.0500 time 0.3756 (0.3841) data time 0.0009 (0.0020) model time 0.3747 (0.3824) loss 4.2659 (4.1584) grad_norm 1.9430 (inf) loss_scale 16384.0000 (20458.4799) mem 11225MB [2024-08-26 04:17:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][580/625] eta 0:00:17 lr 0.002000 wd 0.0500 time 0.3835 (0.3840) data time 0.0012 (0.0020) model time 0.3823 (0.3823) loss 4.3760 (4.1636) grad_norm 0.9199 (inf) loss_scale 16384.0000 (20388.3511) mem 11225MB [2024-08-26 04:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][590/625] eta 0:00:13 lr 0.002000 wd 0.0500 time 0.3788 (0.3842) data time 0.0008 (0.0020) model time 0.3780 (0.3825) loss 5.2495 (4.1650) grad_norm 1.2549 (inf) loss_scale 16384.0000 (20320.5956) mem 11225MB [2024-08-26 04:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][600/625] eta 0:00:09 lr 0.002000 wd 0.0500 time 0.3802 (0.3842) data time 0.0009 (0.0020) model time 0.3794 (0.3825) loss 3.7799 (4.1654) grad_norm 1.3387 (inf) loss_scale 16384.0000 (20255.0948) mem 11225MB [2024-08-26 04:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][610/625] eta 0:00:05 lr 0.002000 wd 0.0500 time 0.3677 (0.3840) data time 0.0009 (0.0020) model time 0.3668 (0.3823) loss 4.2264 (4.1665) grad_norm 1.3854 (inf) loss_scale 16384.0000 (20191.7381) mem 11225MB [2024-08-26 04:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [20/300][620/625] eta 0:00:01 lr 0.002000 wd 0.0500 time 0.3704 (0.3838) data time 0.0008 (0.0020) model time 0.3697 (0.3821) loss 4.2927 (4.1684) grad_norm 1.3557 (inf) loss_scale 16384.0000 (20130.4219) mem 11225MB [2024-08-26 04:17:18 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 20 training takes 0:03:59 [2024-08-26 04:17:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:17:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:17:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.9595 (0.9595) Acc@1 77.100 (77.100) Acc@5 94.189 (94.189) Mem 11225MB [2024-08-26 04:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 1.6758 (1.1595) Acc@1 60.449 (71.902) Acc@5 85.938 (92.338) Mem 11225MB [2024-08-26 04:17:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.9062 (1.4284) Acc@1 57.422 (66.643) Acc@5 81.445 (88.281) Mem 11225MB [2024-08-26 04:17:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 66.641 Acc@5 88.276 [2024-08-26 04:17:22 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 66.6% [2024-08-26 04:17:22 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 66.64% [2024-08-26 04:17:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:17:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 1.3955 (1.3955) Acc@1 66.650 (66.650) Acc@5 88.330 (88.330) Mem 11225MB [2024-08-26 04:17:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.143) Loss 2.1797 (1.6159) Acc@1 51.025 (61.093) Acc@5 77.539 (85.392) Mem 11225MB [2024-08-26 04:17:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 2.3379 (1.8923) Acc@1 47.656 (56.436) Acc@5 73.779 (80.878) Mem 11225MB [2024-08-26 04:17:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 56.590 Acc@5 81.018 [2024-08-26 04:17:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 56.6% [2024-08-26 04:17:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 56.59% [2024-08-26 04:17:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:17:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][0/625] eta 0:08:16 lr 0.002000 wd 0.0500 time 0.7939 (0.7939) data time 0.4544 (0.4544) model time 0.0000 (0.0000) loss 4.3182 (4.3182) grad_norm 1.1025 (1.1025) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][10/625] eta 0:04:27 lr 0.002000 wd 0.0500 time 0.3321 (0.4348) data time 0.0010 (0.0424) model time 0.0000 (0.0000) loss 4.8173 (4.2781) grad_norm 1.5737 (1.3619) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:17:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][20/625] eta 0:04:07 lr 0.002000 wd 0.0500 time 0.3833 (0.4083) data time 0.0008 (0.0227) model time 0.0000 (0.0000) loss 3.8528 (4.0926) grad_norm 1.3844 (1.3968) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:17:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][30/625] eta 0:03:57 lr 0.002000 wd 0.0500 time 0.3678 (0.3985) data time 0.0009 (0.0158) model time 0.0000 (0.0000) loss 4.4398 (4.1661) grad_norm 2.1862 (1.3534) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:17:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][40/625] eta 0:03:50 lr 0.002000 wd 0.0500 time 0.3803 (0.3939) data time 0.0010 (0.0122) model time 0.0000 (0.0000) loss 3.6132 (4.1896) grad_norm 1.8712 (1.3797) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:17:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][50/625] eta 0:03:47 lr 0.002000 wd 0.0500 time 0.3755 (0.3949) data time 0.0011 (0.0100) model time 0.0000 (0.0000) loss 4.4880 (4.1969) grad_norm 1.9657 (1.4167) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][60/625] eta 0:03:41 lr 0.002000 wd 0.0500 time 0.3801 (0.3922) data time 0.0010 (0.0086) model time 0.3790 (0.3771) loss 4.8277 (4.2117) grad_norm 1.2070 (1.3866) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:17:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][70/625] eta 0:03:36 lr 0.002000 wd 0.0500 time 0.3741 (0.3904) data time 0.0014 (0.0076) model time 0.3727 (0.3777) loss 3.7441 (4.2008) grad_norm 1.5886 (1.3771) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:17:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][80/625] eta 0:03:31 lr 0.002000 wd 0.0500 time 0.3763 (0.3890) data time 0.0011 (0.0068) model time 0.3752 (0.3777) loss 4.7243 (4.1954) grad_norm 2.0335 (1.3718) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][90/625] eta 0:03:27 lr 0.002000 wd 0.0500 time 0.3809 (0.3881) data time 0.0008 (0.0061) model time 0.3801 (0.3782) loss 5.0183 (4.2172) grad_norm 1.4786 (1.3738) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][100/625] eta 0:03:24 lr 0.002000 wd 0.0500 time 0.3775 (0.3891) data time 0.0009 (0.0057) model time 0.3766 (0.3820) loss 3.1136 (4.2021) grad_norm 1.5720 (1.3816) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][110/625] eta 0:03:19 lr 0.002000 wd 0.0500 time 0.3801 (0.3883) data time 0.0011 (0.0052) model time 0.3790 (0.3814) loss 4.3785 (4.2124) grad_norm 1.3376 (1.3687) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][120/625] eta 0:03:15 lr 0.002000 wd 0.0500 time 0.3781 (0.3876) data time 0.0008 (0.0049) model time 0.3773 (0.3810) loss 4.0469 (4.2032) grad_norm 1.3807 (1.3747) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][130/625] eta 0:03:12 lr 0.002000 wd 0.0500 time 0.3863 (0.3882) data time 0.0011 (0.0046) model time 0.3852 (0.3827) loss 3.6916 (4.1902) grad_norm 1.1810 (1.3716) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][140/625] eta 0:03:07 lr 0.002000 wd 0.0500 time 0.3866 (0.3876) data time 0.0011 (0.0044) model time 0.3855 (0.3822) loss 3.7240 (4.1745) grad_norm 0.9180 (1.3692) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][150/625] eta 0:03:04 lr 0.002000 wd 0.0500 time 0.3773 (0.3884) data time 0.0008 (0.0042) model time 0.3765 (0.3839) loss 5.0440 (4.1858) grad_norm 1.2767 (1.3745) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][160/625] eta 0:03:00 lr 0.002000 wd 0.0500 time 0.3831 (0.3878) data time 0.0011 (0.0040) model time 0.3821 (0.3833) loss 4.3472 (4.1553) grad_norm 1.2693 (1.3663) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][170/625] eta 0:02:56 lr 0.002000 wd 0.0500 time 0.3761 (0.3871) data time 0.0008 (0.0038) model time 0.3753 (0.3826) loss 4.7402 (4.1537) grad_norm 3.0498 (1.3786) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][180/625] eta 0:02:52 lr 0.002000 wd 0.0500 time 0.3788 (0.3866) data time 0.0011 (0.0037) model time 0.3777 (0.3822) loss 3.5261 (4.1388) grad_norm 1.4678 (1.3909) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][190/625] eta 0:02:47 lr 0.002000 wd 0.0500 time 0.3719 (0.3862) data time 0.0010 (0.0035) model time 0.3709 (0.3818) loss 4.6608 (4.1338) grad_norm 1.0874 (1.3731) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][200/625] eta 0:02:43 lr 0.002000 wd 0.0500 time 0.3798 (0.3858) data time 0.0011 (0.0034) model time 0.3787 (0.3816) loss 3.7346 (4.1168) grad_norm 1.2272 (1.3792) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][210/625] eta 0:02:39 lr 0.002000 wd 0.0500 time 0.3802 (0.3855) data time 0.0008 (0.0033) model time 0.3794 (0.3813) loss 3.7284 (4.1178) grad_norm 1.0967 (1.3730) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][220/625] eta 0:02:35 lr 0.002000 wd 0.0500 time 0.3748 (0.3852) data time 0.0008 (0.0032) model time 0.3740 (0.3811) loss 3.0797 (4.1089) grad_norm 1.2161 (1.3661) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:18:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][230/625] eta 0:02:31 lr 0.002000 wd 0.0500 time 0.3710 (0.3848) data time 0.0008 (0.0031) model time 0.3702 (0.3808) loss 4.3652 (4.1112) grad_norm 1.4096 (1.3687) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][240/625] eta 0:02:28 lr 0.002000 wd 0.0500 time 0.3730 (0.3845) data time 0.0010 (0.0030) model time 0.3720 (0.3805) loss 3.3869 (4.1075) grad_norm 2.1687 (1.3702) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][250/625] eta 0:02:24 lr 0.002000 wd 0.0500 time 0.3841 (0.3842) data time 0.0008 (0.0030) model time 0.3833 (0.3803) loss 3.1354 (4.1043) grad_norm 1.3012 (1.3696) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][260/625] eta 0:02:20 lr 0.002000 wd 0.0500 time 0.5759 (0.3846) data time 0.0010 (0.0029) model time 0.5748 (0.3810) loss 3.4807 (4.0949) grad_norm 1.1214 (1.3623) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][270/625] eta 0:02:16 lr 0.002000 wd 0.0500 time 0.3795 (0.3844) data time 0.0008 (0.0028) model time 0.3787 (0.3808) loss 4.4799 (4.0894) grad_norm 1.0502 (1.3570) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][280/625] eta 0:02:12 lr 0.002000 wd 0.0500 time 0.3731 (0.3841) data time 0.0008 (0.0027) model time 0.3722 (0.3806) loss 3.5541 (4.0936) grad_norm 1.2387 (1.3623) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][290/625] eta 0:02:08 lr 0.002000 wd 0.0500 time 0.3741 (0.3838) data time 0.0009 (0.0027) model time 0.3733 (0.3803) loss 4.2946 (4.1020) grad_norm 1.3643 (1.3699) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][300/625] eta 0:02:04 lr 0.002000 wd 0.0500 time 0.3741 (0.3835) data time 0.0011 (0.0026) model time 0.3730 (0.3801) loss 4.4226 (4.1104) grad_norm 1.2120 (1.3711) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][310/625] eta 0:02:00 lr 0.002000 wd 0.0500 time 0.3761 (0.3841) data time 0.0011 (0.0026) model time 0.3750 (0.3808) loss 4.1929 (4.1059) grad_norm 0.9845 (1.3699) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][320/625] eta 0:01:57 lr 0.002000 wd 0.0500 time 0.3857 (0.3839) data time 0.0011 (0.0026) model time 0.3847 (0.3807) loss 3.4449 (4.0982) grad_norm 1.1669 (1.3661) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][330/625] eta 0:01:53 lr 0.002000 wd 0.0500 time 0.3771 (0.3837) data time 0.0008 (0.0025) model time 0.3763 (0.3805) loss 4.6280 (4.0935) grad_norm 1.4512 (1.3608) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][340/625] eta 0:01:49 lr 0.002000 wd 0.0500 time 0.3793 (0.3853) data time 0.0011 (0.0025) model time 0.3783 (0.3825) loss 4.4306 (4.0988) grad_norm 2.1822 (1.3586) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][350/625] eta 0:01:45 lr 0.002000 wd 0.0500 time 0.3838 (0.3851) data time 0.0009 (0.0024) model time 0.3830 (0.3823) loss 3.6517 (4.0958) grad_norm 0.9892 (1.3587) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][360/625] eta 0:01:42 lr 0.002000 wd 0.0500 time 0.3784 (0.3849) data time 0.0012 (0.0024) model time 0.3772 (0.3821) loss 4.4395 (4.1009) grad_norm 1.1035 (1.3551) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][370/625] eta 0:01:38 lr 0.002000 wd 0.0500 time 0.3755 (0.3847) data time 0.0011 (0.0024) model time 0.3744 (0.3819) loss 4.0806 (4.1069) grad_norm 1.5468 (1.3566) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][380/625] eta 0:01:34 lr 0.002000 wd 0.0500 time 0.3752 (0.3844) data time 0.0011 (0.0023) model time 0.3741 (0.3817) loss 3.5875 (4.1091) grad_norm 1.2662 (1.3567) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][390/625] eta 0:01:30 lr 0.002000 wd 0.0500 time 0.3832 (0.3843) data time 0.0011 (0.0023) model time 0.3821 (0.3816) loss 2.8806 (4.0984) grad_norm 0.9798 (1.3531) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][400/625] eta 0:01:26 lr 0.002000 wd 0.0500 time 0.3726 (0.3841) data time 0.0008 (0.0023) model time 0.3718 (0.3814) loss 3.8507 (4.0951) grad_norm 1.1664 (1.3516) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][410/625] eta 0:01:22 lr 0.002000 wd 0.0500 time 0.3755 (0.3840) data time 0.0009 (0.0022) model time 0.3746 (0.3813) loss 3.1428 (4.0877) grad_norm 1.1174 (1.3539) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][420/625] eta 0:01:18 lr 0.002000 wd 0.0500 time 0.3821 (0.3838) data time 0.0009 (0.0022) model time 0.3812 (0.3811) loss 3.2719 (4.0846) grad_norm 1.5522 (1.3506) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][430/625] eta 0:01:14 lr 0.002000 wd 0.0500 time 0.3764 (0.3836) data time 0.0008 (0.0022) model time 0.3756 (0.3810) loss 4.7307 (4.0834) grad_norm 1.6324 (1.3498) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][440/625] eta 0:01:10 lr 0.002000 wd 0.0500 time 0.3740 (0.3834) data time 0.0011 (0.0022) model time 0.3729 (0.3808) loss 2.6554 (4.0812) grad_norm 2.1956 (1.3558) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][450/625] eta 0:01:07 lr 0.002000 wd 0.0500 time 0.5841 (0.3842) data time 0.0008 (0.0021) model time 0.5833 (0.3817) loss 4.2597 (4.0840) grad_norm 1.6673 (1.3560) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][460/625] eta 0:01:03 lr 0.002000 wd 0.0500 time 0.3753 (0.3845) data time 0.0008 (0.0021) model time 0.3745 (0.3821) loss 4.9075 (4.0849) grad_norm 2.0003 (1.3537) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][470/625] eta 0:00:59 lr 0.002000 wd 0.0500 time 0.3738 (0.3844) data time 0.0008 (0.0021) model time 0.3730 (0.3820) loss 4.7735 (4.0818) grad_norm 1.1161 (1.3590) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][480/625] eta 0:00:55 lr 0.002000 wd 0.0500 time 0.3757 (0.3843) data time 0.0011 (0.0021) model time 0.3746 (0.3819) loss 4.5291 (4.0862) grad_norm 1.0938 (1.3580) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][490/625] eta 0:00:51 lr 0.002000 wd 0.0500 time 0.3791 (0.3846) data time 0.0011 (0.0021) model time 0.3780 (0.3823) loss 4.1882 (4.0870) grad_norm 1.3364 (1.3566) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][500/625] eta 0:00:48 lr 0.002000 wd 0.0500 time 0.3804 (0.3845) data time 0.0010 (0.0020) model time 0.3794 (0.3822) loss 4.0588 (4.0909) grad_norm 1.1401 (1.3554) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][510/625] eta 0:00:44 lr 0.002000 wd 0.0500 time 0.3707 (0.3844) data time 0.0010 (0.0020) model time 0.3697 (0.3820) loss 3.5094 (4.0891) grad_norm 1.1508 (1.3522) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][520/625] eta 0:00:40 lr 0.002000 wd 0.0500 time 0.6065 (0.3847) data time 0.0008 (0.0020) model time 0.6057 (0.3824) loss 3.9600 (4.0867) grad_norm 1.6676 (1.3501) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][530/625] eta 0:00:36 lr 0.002000 wd 0.0500 time 0.3811 (0.3845) data time 0.0011 (0.0020) model time 0.3800 (0.3823) loss 4.1712 (4.0900) grad_norm 1.1523 (1.3533) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][540/625] eta 0:00:32 lr 0.002000 wd 0.0500 time 0.3787 (0.3844) data time 0.0011 (0.0020) model time 0.3777 (0.3822) loss 4.3919 (4.0916) grad_norm 1.2664 (1.3521) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:20:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][550/625] eta 0:00:28 lr 0.002000 wd 0.0500 time 0.3807 (0.3843) data time 0.0008 (0.0020) model time 0.3799 (0.3821) loss 5.2636 (4.0903) grad_norm 1.6448 (1.3524) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][560/625] eta 0:00:24 lr 0.002000 wd 0.0500 time 0.3738 (0.3842) data time 0.0010 (0.0019) model time 0.3728 (0.3820) loss 4.2971 (4.0890) grad_norm 0.8587 (1.3496) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][570/625] eta 0:00:21 lr 0.002000 wd 0.0500 time 0.3807 (0.3844) data time 0.0009 (0.0019) model time 0.3798 (0.3823) loss 4.3222 (4.0896) grad_norm 1.7431 (1.3509) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][580/625] eta 0:00:17 lr 0.002000 wd 0.0500 time 0.3804 (0.3843) data time 0.0008 (0.0019) model time 0.3796 (0.3822) loss 4.6901 (4.0957) grad_norm 3.0539 (1.3527) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][590/625] eta 0:00:13 lr 0.002000 wd 0.0500 time 0.3839 (0.3842) data time 0.0008 (0.0019) model time 0.3831 (0.3821) loss 4.8234 (4.0906) grad_norm 1.0186 (1.3552) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][600/625] eta 0:00:09 lr 0.002000 wd 0.0500 time 0.3793 (0.3842) data time 0.0011 (0.0019) model time 0.3782 (0.3820) loss 4.2720 (4.0898) grad_norm 1.3065 (1.3539) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][610/625] eta 0:00:05 lr 0.002000 wd 0.0500 time 0.3677 (0.3844) data time 0.0008 (0.0019) model time 0.3669 (0.3823) loss 3.3947 (4.0898) grad_norm 1.2291 (1.3545) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [21/300][620/625] eta 0:00:01 lr 0.002000 wd 0.0500 time 0.3661 (0.3841) data time 0.0008 (0.0019) model time 0.3653 (0.3820) loss 3.5925 (4.0888) grad_norm 0.9472 (1.3576) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 21 training takes 0:03:59 [2024-08-26 04:21:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:21:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.500 (0.500) Loss 0.9487 (0.9487) Acc@1 78.125 (78.125) Acc@5 94.824 (94.824) Mem 11225MB [2024-08-26 04:21:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.102 (0.143) Loss 1.6191 (1.1376) Acc@1 62.549 (72.661) Acc@5 85.693 (92.711) Mem 11225MB [2024-08-26 04:21:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.8574 (1.4080) Acc@1 58.398 (67.243) Acc@5 82.520 (88.795) Mem 11225MB [2024-08-26 04:21:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 67.039 Acc@5 88.690 [2024-08-26 04:21:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 67.0% [2024-08-26 04:21:32 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 67.04% [2024-08-26 04:21:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:21:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 1.2461 (1.2461) Acc@1 68.945 (68.945) Acc@5 89.258 (89.258) Mem 11225MB [2024-08-26 04:21:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 2.0273 (1.4676) Acc@1 53.125 (63.663) Acc@5 79.590 (87.225) Mem 11225MB [2024-08-26 04:21:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 2.1836 (1.7449) Acc@1 50.098 (58.856) Acc@5 75.830 (82.764) Mem 11225MB [2024-08-26 04:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 58.983 Acc@5 82.859 [2024-08-26 04:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 59.0% [2024-08-26 04:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 58.98% [2024-08-26 04:21:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:21:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:21:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][0/625] eta 0:08:44 lr 0.002000 wd 0.0500 time 0.8389 (0.8389) data time 0.4888 (0.4888) model time 0.0000 (0.0000) loss 4.1509 (4.1509) grad_norm 1.5179 (1.5179) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][10/625] eta 0:04:17 lr 0.002000 wd 0.0500 time 0.3644 (0.4186) data time 0.0008 (0.0455) model time 0.0000 (0.0000) loss 4.2483 (4.2462) grad_norm 2.0426 (1.3836) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][20/625] eta 0:04:06 lr 0.002000 wd 0.0500 time 0.5768 (0.4076) data time 0.0011 (0.0244) model time 0.0000 (0.0000) loss 4.2208 (4.2572) grad_norm 1.4884 (1.5504) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][30/625] eta 0:04:00 lr 0.002000 wd 0.0500 time 0.3817 (0.4044) data time 0.0010 (0.0169) model time 0.0000 (0.0000) loss 3.8734 (4.0918) grad_norm 1.2495 (1.4217) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][40/625] eta 0:03:52 lr 0.002000 wd 0.0500 time 0.3824 (0.3979) data time 0.0009 (0.0131) model time 0.0000 (0.0000) loss 3.3841 (3.9928) grad_norm 1.0204 (1.3925) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:21:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][50/625] eta 0:03:46 lr 0.002000 wd 0.0500 time 0.3782 (0.3940) data time 0.0008 (0.0107) model time 0.0000 (0.0000) loss 4.7524 (4.0189) grad_norm 1.0860 (1.3886) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][60/625] eta 0:03:40 lr 0.002000 wd 0.0500 time 0.3798 (0.3911) data time 0.0011 (0.0091) model time 0.3787 (0.3756) loss 4.0728 (3.9969) grad_norm 1.1459 (1.3914) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][70/625] eta 0:03:36 lr 0.002000 wd 0.0500 time 0.3737 (0.3893) data time 0.0008 (0.0080) model time 0.3730 (0.3762) loss 4.3483 (4.0433) grad_norm 1.0004 (1.3614) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][80/625] eta 0:03:31 lr 0.002000 wd 0.0500 time 0.3790 (0.3877) data time 0.0008 (0.0072) model time 0.3782 (0.3760) loss 4.7172 (4.0305) grad_norm 1.6354 (1.3334) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][90/625] eta 0:03:28 lr 0.002000 wd 0.0500 time 0.3755 (0.3889) data time 0.0010 (0.0065) model time 0.3745 (0.3814) loss 4.5364 (4.0543) grad_norm 1.1193 (1.3290) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][100/625] eta 0:03:23 lr 0.002000 wd 0.0500 time 0.3789 (0.3879) data time 0.0012 (0.0060) model time 0.3777 (0.3807) loss 3.9880 (4.0830) grad_norm 1.3370 (1.3135) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][110/625] eta 0:03:19 lr 0.002000 wd 0.0500 time 0.3769 (0.3869) data time 0.0008 (0.0055) model time 0.3761 (0.3798) loss 3.5875 (4.0757) grad_norm 1.3186 (1.3116) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][120/625] eta 0:03:15 lr 0.002000 wd 0.0500 time 0.3773 (0.3862) data time 0.0008 (0.0052) model time 0.3765 (0.3794) loss 4.0706 (4.0764) grad_norm 1.4346 (1.3133) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][130/625] eta 0:03:10 lr 0.002000 wd 0.0500 time 0.3745 (0.3856) data time 0.0008 (0.0049) model time 0.3737 (0.3792) loss 2.8835 (4.0565) grad_norm 3.5771 (1.3409) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][140/625] eta 0:03:06 lr 0.002000 wd 0.0500 time 0.3767 (0.3851) data time 0.0009 (0.0046) model time 0.3759 (0.3790) loss 4.5955 (4.0766) grad_norm 1.2794 (1.3713) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][150/625] eta 0:03:02 lr 0.002000 wd 0.0500 time 0.3784 (0.3848) data time 0.0009 (0.0044) model time 0.3776 (0.3790) loss 4.4904 (4.0790) grad_norm 1.5482 (1.3797) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][160/625] eta 0:02:59 lr 0.002000 wd 0.0500 time 0.3722 (0.3851) data time 0.0008 (0.0042) model time 0.3714 (0.3799) loss 4.5059 (4.0739) grad_norm 1.5903 (1.3752) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][170/625] eta 0:02:55 lr 0.002000 wd 0.0500 time 0.3788 (0.3866) data time 0.0008 (0.0040) model time 0.3780 (0.3824) loss 3.0950 (4.0906) grad_norm 1.6402 (1.3769) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][180/625] eta 0:02:51 lr 0.002000 wd 0.0500 time 0.3767 (0.3860) data time 0.0009 (0.0038) model time 0.3758 (0.3817) loss 4.4293 (4.1035) grad_norm 1.2128 (1.3708) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][190/625] eta 0:02:47 lr 0.002000 wd 0.0500 time 0.3774 (0.3855) data time 0.0011 (0.0037) model time 0.3763 (0.3812) loss 4.3401 (4.0813) grad_norm 1.0497 (1.3584) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][200/625] eta 0:02:43 lr 0.002000 wd 0.0500 time 0.3744 (0.3850) data time 0.0010 (0.0036) model time 0.3734 (0.3808) loss 4.3590 (4.0844) grad_norm 1.1500 (1.3524) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:22:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][210/625] eta 0:02:39 lr 0.002000 wd 0.0500 time 0.3768 (0.3847) data time 0.0011 (0.0035) model time 0.3756 (0.3806) loss 3.2097 (4.0847) grad_norm 1.3115 (1.3570) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][220/625] eta 0:02:35 lr 0.002000 wd 0.0500 time 0.3731 (0.3844) data time 0.0011 (0.0034) model time 0.3719 (0.3803) loss 2.8964 (4.0826) grad_norm 1.1908 (1.3542) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][230/625] eta 0:02:32 lr 0.002000 wd 0.0500 time 0.6008 (0.3852) data time 0.0009 (0.0033) model time 0.5999 (0.3815) loss 4.9435 (4.0877) grad_norm 1.3744 (1.3469) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][240/625] eta 0:02:28 lr 0.002000 wd 0.0500 time 0.3834 (0.3849) data time 0.0011 (0.0032) model time 0.3823 (0.3813) loss 2.9662 (4.0848) grad_norm 1.5363 (1.3504) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][250/625] eta 0:02:24 lr 0.002000 wd 0.0500 time 0.3768 (0.3847) data time 0.0009 (0.0031) model time 0.3759 (0.3811) loss 5.0088 (4.0943) grad_norm 1.2088 (1.3471) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][260/625] eta 0:02:20 lr 0.002000 wd 0.0500 time 0.3739 (0.3845) data time 0.0011 (0.0030) model time 0.3728 (0.3810) loss 4.0282 (4.1024) grad_norm 1.3953 (1.3443) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][270/625] eta 0:02:16 lr 0.002000 wd 0.0500 time 0.3800 (0.3843) data time 0.0011 (0.0030) model time 0.3789 (0.3809) loss 3.5212 (4.0967) grad_norm 1.1289 (1.3395) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][280/625] eta 0:02:12 lr 0.002000 wd 0.0500 time 0.3810 (0.3841) data time 0.0008 (0.0029) model time 0.3802 (0.3807) loss 4.9433 (4.1041) grad_norm 1.5568 (1.3402) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][290/625] eta 0:02:08 lr 0.002000 wd 0.0500 time 0.3790 (0.3845) data time 0.0008 (0.0028) model time 0.3781 (0.3814) loss 3.6290 (4.1085) grad_norm 0.9186 (1.3420) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][300/625] eta 0:02:05 lr 0.002000 wd 0.0500 time 0.3785 (0.3850) data time 0.0011 (0.0028) model time 0.3775 (0.3821) loss 3.9494 (4.1085) grad_norm 1.3378 (1.3448) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][310/625] eta 0:02:01 lr 0.002000 wd 0.0500 time 0.3719 (0.3848) data time 0.0009 (0.0027) model time 0.3711 (0.3818) loss 4.6235 (4.1046) grad_norm 1.2785 (1.3439) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][320/625] eta 0:01:57 lr 0.002000 wd 0.0500 time 0.3799 (0.3845) data time 0.0008 (0.0027) model time 0.3790 (0.3816) loss 2.8111 (4.1061) grad_norm 2.0432 (1.3427) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][330/625] eta 0:01:53 lr 0.002000 wd 0.0500 time 0.3738 (0.3842) data time 0.0011 (0.0026) model time 0.3727 (0.3813) loss 3.9283 (4.1083) grad_norm 1.0132 (1.3391) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][340/625] eta 0:01:49 lr 0.002000 wd 0.0500 time 0.3717 (0.3845) data time 0.0011 (0.0026) model time 0.3706 (0.3817) loss 3.3611 (4.1093) grad_norm 1.1101 (1.3352) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][350/625] eta 0:01:45 lr 0.002000 wd 0.0500 time 0.3728 (0.3843) data time 0.0011 (0.0025) model time 0.3717 (0.3815) loss 4.2571 (4.1111) grad_norm 1.2673 (1.3303) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][360/625] eta 0:01:41 lr 0.002000 wd 0.0500 time 0.3721 (0.3841) data time 0.0010 (0.0025) model time 0.3710 (0.3813) loss 4.0999 (4.1170) grad_norm 1.1245 (1.3298) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:23:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][370/625] eta 0:01:37 lr 0.002000 wd 0.0500 time 0.3727 (0.3839) data time 0.0013 (0.0025) model time 0.3715 (0.3812) loss 4.9756 (4.1223) grad_norm 1.2725 (1.3281) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][380/625] eta 0:01:34 lr 0.002000 wd 0.0500 time 0.3773 (0.3838) data time 0.0013 (0.0024) model time 0.3760 (0.3811) loss 3.7707 (4.1209) grad_norm 1.4812 (1.3255) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][390/625] eta 0:01:30 lr 0.002000 wd 0.0500 time 0.3766 (0.3837) data time 0.0008 (0.0024) model time 0.3758 (0.3810) loss 4.0836 (4.1294) grad_norm 1.0998 (1.3285) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][400/625] eta 0:01:26 lr 0.002000 wd 0.0500 time 0.3783 (0.3835) data time 0.0011 (0.0024) model time 0.3772 (0.3809) loss 4.0538 (4.1360) grad_norm 1.0391 (1.3324) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][410/625] eta 0:01:22 lr 0.002000 wd 0.0500 time 0.3773 (0.3834) data time 0.0012 (0.0023) model time 0.3761 (0.3808) loss 3.8550 (4.1318) grad_norm 1.0245 (1.3338) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][420/625] eta 0:01:18 lr 0.002000 wd 0.0500 time 0.3787 (0.3844) data time 0.0011 (0.0023) model time 0.3777 (0.3819) loss 3.6375 (4.1326) grad_norm 1.5818 (1.3386) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][430/625] eta 0:01:14 lr 0.002000 wd 0.0500 time 0.3730 (0.3842) data time 0.0008 (0.0023) model time 0.3721 (0.3817) loss 4.7376 (4.1293) grad_norm 1.4648 (1.3410) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][440/625] eta 0:01:11 lr 0.002000 wd 0.0500 time 0.3711 (0.3840) data time 0.0010 (0.0023) model time 0.3701 (0.3815) loss 3.3032 (4.1266) grad_norm 1.1383 (1.3399) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][450/625] eta 0:01:07 lr 0.002000 wd 0.0500 time 0.3745 (0.3838) data time 0.0011 (0.0022) model time 0.3733 (0.3814) loss 4.4383 (4.1263) grad_norm 1.2027 (1.3369) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][460/625] eta 0:01:03 lr 0.002000 wd 0.0500 time 0.3809 (0.3841) data time 0.0008 (0.0022) model time 0.3800 (0.3818) loss 3.5251 (4.1244) grad_norm 1.0924 (1.3370) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][470/625] eta 0:00:59 lr 0.002000 wd 0.0500 time 0.3797 (0.3840) data time 0.0011 (0.0022) model time 0.3786 (0.3816) loss 3.2860 (4.1153) grad_norm 1.0288 (1.3377) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][480/625] eta 0:00:55 lr 0.002000 wd 0.0500 time 0.3774 (0.3843) data time 0.0010 (0.0022) model time 0.3764 (0.3820) loss 3.4626 (4.1141) grad_norm 1.2011 (1.3331) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][490/625] eta 0:00:51 lr 0.002000 wd 0.0500 time 0.3765 (0.3842) data time 0.0012 (0.0021) model time 0.3753 (0.3820) loss 4.2517 (4.1160) grad_norm 1.3059 (1.3331) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][500/625] eta 0:00:48 lr 0.002000 wd 0.0500 time 0.3886 (0.3845) data time 0.0008 (0.0021) model time 0.3877 (0.3823) loss 4.5018 (4.1162) grad_norm 1.2735 (1.3345) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][510/625] eta 0:00:44 lr 0.002000 wd 0.0500 time 0.3784 (0.3844) data time 0.0009 (0.0021) model time 0.3775 (0.3822) loss 4.3234 (4.1133) grad_norm 1.3200 (1.3314) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][520/625] eta 0:00:40 lr 0.002000 wd 0.0500 time 0.3712 (0.3843) data time 0.0009 (0.0021) model time 0.3703 (0.3821) loss 3.9876 (4.1072) grad_norm 1.2525 (1.3310) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][530/625] eta 0:00:36 lr 0.001999 wd 0.0500 time 0.3766 (0.3842) data time 0.0011 (0.0021) model time 0.3755 (0.3820) loss 3.6544 (4.1090) grad_norm 1.8704 (1.3311) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][540/625] eta 0:00:32 lr 0.001999 wd 0.0500 time 0.3717 (0.3849) data time 0.0012 (0.0021) model time 0.3705 (0.3828) loss 4.5156 (4.1112) grad_norm 1.6977 (1.3365) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][550/625] eta 0:00:28 lr 0.001999 wd 0.0500 time 0.3748 (0.3848) data time 0.0011 (0.0020) model time 0.3737 (0.3827) loss 4.3470 (4.1061) grad_norm 1.4604 (1.3339) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][560/625] eta 0:00:25 lr 0.001999 wd 0.0500 time 0.3772 (0.3847) data time 0.0014 (0.0020) model time 0.3758 (0.3826) loss 4.3335 (4.0978) grad_norm 1.4094 (1.3319) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][570/625] eta 0:00:21 lr 0.001999 wd 0.0500 time 0.3802 (0.3846) data time 0.0012 (0.0020) model time 0.3790 (0.3825) loss 4.0404 (4.0920) grad_norm 1.0481 (1.3316) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][580/625] eta 0:00:17 lr 0.001999 wd 0.0500 time 0.3747 (0.3846) data time 0.0010 (0.0020) model time 0.3737 (0.3825) loss 3.7973 (4.0879) grad_norm 1.2538 (1.3337) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][590/625] eta 0:00:13 lr 0.001999 wd 0.0500 time 0.3794 (0.3845) data time 0.0009 (0.0020) model time 0.3785 (0.3824) loss 3.5376 (4.0865) grad_norm 1.3000 (1.3342) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][600/625] eta 0:00:09 lr 0.001999 wd 0.0500 time 0.3740 (0.3847) data time 0.0012 (0.0020) model time 0.3728 (0.3827) loss 4.2335 (4.0850) grad_norm 1.4716 (1.3355) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][610/625] eta 0:00:05 lr 0.001999 wd 0.0500 time 0.3672 (0.3846) data time 0.0006 (0.0020) model time 0.3666 (0.3826) loss 4.9068 (4.0835) grad_norm 1.2938 (1.3350) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [22/300][620/625] eta 0:00:01 lr 0.001999 wd 0.0500 time 0.3664 (0.3843) data time 0.0008 (0.0019) model time 0.3656 (0.3823) loss 4.0381 (4.0835) grad_norm 1.3520 (1.3336) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 22 training takes 0:04:00 [2024-08-26 04:25:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:25:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:25:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 0.9326 (0.9326) Acc@1 77.637 (77.637) Acc@5 95.166 (95.166) Mem 11225MB [2024-08-26 04:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.6768 (1.1120) Acc@1 60.205 (73.699) Acc@5 85.938 (93.217) Mem 11225MB [2024-08-26 04:25:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.7969 (1.3855) Acc@1 60.107 (68.171) Acc@5 83.740 (89.281) Mem 11225MB [2024-08-26 04:25:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 68.072 Acc@5 89.255 [2024-08-26 04:25:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 68.1% [2024-08-26 04:25:41 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 68.07% [2024-08-26 04:25:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:25:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:25:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.503 (0.503) Loss 1.1338 (1.1338) Acc@1 71.045 (71.045) Acc@5 90.283 (90.283) Mem 11225MB [2024-08-26 04:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 1.8916 (1.3528) Acc@1 54.297 (65.732) Acc@5 81.836 (88.605) Mem 11225MB [2024-08-26 04:25:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 2.0625 (1.6276) Acc@1 52.197 (60.982) Acc@5 77.637 (84.315) Mem 11225MB [2024-08-26 04:25:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 61.072 Acc@5 84.391 [2024-08-26 04:25:45 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 61.1% [2024-08-26 04:25:45 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 61.07% [2024-08-26 04:25:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:25:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:25:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][0/625] eta 0:07:54 lr 0.001999 wd 0.0500 time 0.7593 (0.7593) data time 0.4119 (0.4119) model time 0.0000 (0.0000) loss 4.9321 (4.9321) grad_norm 0.8582 (0.8582) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][10/625] eta 0:04:13 lr 0.001999 wd 0.0500 time 0.3789 (0.4115) data time 0.0009 (0.0385) model time 0.0000 (0.0000) loss 3.2194 (4.4483) grad_norm 1.5523 (1.2488) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][20/625] eta 0:03:59 lr 0.001999 wd 0.0500 time 0.3762 (0.3958) data time 0.0008 (0.0212) model time 0.0000 (0.0000) loss 4.6672 (4.2867) grad_norm 1.3610 (1.2248) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][30/625] eta 0:03:52 lr 0.001999 wd 0.0500 time 0.3736 (0.3903) data time 0.0014 (0.0147) model time 0.0000 (0.0000) loss 3.6965 (4.2175) grad_norm 1.3876 (1.2640) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][40/625] eta 0:03:46 lr 0.001999 wd 0.0500 time 0.3964 (0.3876) data time 0.0011 (0.0114) model time 0.0000 (0.0000) loss 3.6265 (4.0568) grad_norm 0.9352 (1.2851) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][50/625] eta 0:03:41 lr 0.001999 wd 0.0500 time 0.3770 (0.3851) data time 0.0012 (0.0095) model time 0.0000 (0.0000) loss 4.0767 (4.1337) grad_norm 1.0586 (1.2777) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][60/625] eta 0:03:40 lr 0.001999 wd 0.0500 time 0.5867 (0.3898) data time 0.0008 (0.0081) model time 0.5859 (0.4125) loss 4.0905 (4.1281) grad_norm 2.8087 (1.3186) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][70/625] eta 0:03:38 lr 0.001999 wd 0.0500 time 0.3702 (0.3936) data time 0.0008 (0.0071) model time 0.3694 (0.4142) loss 4.9810 (4.1309) grad_norm 1.0052 (1.3369) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][80/625] eta 0:03:33 lr 0.001999 wd 0.0500 time 0.3800 (0.3917) data time 0.0008 (0.0064) model time 0.3791 (0.4018) loss 4.7665 (4.1136) grad_norm 1.2767 (1.3185) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][90/625] eta 0:03:28 lr 0.001999 wd 0.0500 time 0.3794 (0.3900) data time 0.0008 (0.0058) model time 0.3786 (0.3951) loss 2.9044 (4.1342) grad_norm 1.2553 (1.3157) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][100/625] eta 0:03:24 lr 0.001999 wd 0.0500 time 0.3686 (0.3886) data time 0.0009 (0.0054) model time 0.3677 (0.3910) loss 4.6699 (4.0943) grad_norm 1.3358 (1.3195) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][110/625] eta 0:03:20 lr 0.001999 wd 0.0500 time 0.3806 (0.3895) data time 0.0011 (0.0050) model time 0.3795 (0.3920) loss 4.3831 (4.0834) grad_norm 1.4066 (1.3040) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][120/625] eta 0:03:16 lr 0.001999 wd 0.0500 time 0.3766 (0.3885) data time 0.0008 (0.0047) model time 0.3757 (0.3899) loss 4.9513 (4.0987) grad_norm 1.8687 (1.3054) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][130/625] eta 0:03:11 lr 0.001999 wd 0.0500 time 0.3735 (0.3876) data time 0.0009 (0.0044) model time 0.3726 (0.3880) loss 4.6505 (4.1088) grad_norm 1.4327 (1.3054) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][140/625] eta 0:03:07 lr 0.001999 wd 0.0500 time 0.3825 (0.3871) data time 0.0011 (0.0042) model time 0.3814 (0.3870) loss 2.9099 (4.0918) grad_norm 1.9024 (1.3099) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][150/625] eta 0:03:03 lr 0.001999 wd 0.0500 time 0.3804 (0.3865) data time 0.0011 (0.0040) model time 0.3793 (0.3861) loss 4.1295 (4.0704) grad_norm 1.5573 (1.3174) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][160/625] eta 0:02:59 lr 0.001999 wd 0.0500 time 0.3776 (0.3861) data time 0.0011 (0.0038) model time 0.3766 (0.3853) loss 3.5451 (4.0726) grad_norm 1.1178 (1.3096) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][170/625] eta 0:02:55 lr 0.001999 wd 0.0500 time 0.3792 (0.3865) data time 0.0010 (0.0036) model time 0.3782 (0.3858) loss 4.4087 (4.0634) grad_norm 1.1369 (1.3001) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:26:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][180/625] eta 0:02:51 lr 0.001999 wd 0.0500 time 0.3876 (0.3860) data time 0.0011 (0.0035) model time 0.3865 (0.3852) loss 3.8513 (4.0740) grad_norm 1.9168 (1.2996) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][190/625] eta 0:02:47 lr 0.001999 wd 0.0500 time 0.3721 (0.3855) data time 0.0011 (0.0034) model time 0.3710 (0.3845) loss 4.3512 (4.0662) grad_norm 0.9370 (1.2982) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][200/625] eta 0:02:43 lr 0.001999 wd 0.0500 time 0.3820 (0.3852) data time 0.0009 (0.0033) model time 0.3811 (0.3841) loss 4.4116 (4.0720) grad_norm 0.8906 (1.2929) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][210/625] eta 0:02:39 lr 0.001999 wd 0.0500 time 0.3795 (0.3848) data time 0.0008 (0.0032) model time 0.3786 (0.3835) loss 4.9795 (4.0775) grad_norm 1.1268 (1.2952) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][220/625] eta 0:02:35 lr 0.001999 wd 0.0500 time 0.3863 (0.3845) data time 0.0008 (0.0031) model time 0.3855 (0.3831) loss 4.6169 (4.0780) grad_norm 1.6888 (1.2951) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][230/625] eta 0:02:31 lr 0.001999 wd 0.0500 time 0.3780 (0.3842) data time 0.0011 (0.0030) model time 0.3769 (0.3828) loss 3.6270 (4.0741) grad_norm 2.1491 (1.3037) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][240/625] eta 0:02:27 lr 0.001999 wd 0.0500 time 0.3926 (0.3840) data time 0.0008 (0.0029) model time 0.3918 (0.3825) loss 2.8827 (4.0760) grad_norm 1.6976 (1.3062) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][250/625] eta 0:02:24 lr 0.001999 wd 0.0500 time 0.3763 (0.3851) data time 0.0008 (0.0029) model time 0.3756 (0.3840) loss 4.9355 (4.0779) grad_norm 0.9279 (1.3166) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][260/625] eta 0:02:20 lr 0.001999 wd 0.0500 time 0.3751 (0.3849) data time 0.0011 (0.0028) model time 0.3739 (0.3837) loss 3.5107 (4.0689) grad_norm 1.5792 (1.3181) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][270/625] eta 0:02:16 lr 0.001999 wd 0.0500 time 0.3819 (0.3847) data time 0.0013 (0.0027) model time 0.3806 (0.3834) loss 4.0757 (4.0710) grad_norm 1.1722 (1.3170) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][280/625] eta 0:02:12 lr 0.001999 wd 0.0500 time 0.3758 (0.3852) data time 0.0008 (0.0027) model time 0.3750 (0.3840) loss 5.0129 (4.0852) grad_norm 1.7719 (1.3152) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][290/625] eta 0:02:09 lr 0.001999 wd 0.0500 time 0.3723 (0.3856) data time 0.0011 (0.0026) model time 0.3711 (0.3845) loss 4.3462 (4.0897) grad_norm 1.5731 (1.3206) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][300/625] eta 0:02:05 lr 0.001999 wd 0.0500 time 0.3768 (0.3853) data time 0.0008 (0.0026) model time 0.3760 (0.3841) loss 4.2840 (4.0826) grad_norm 1.0983 (1.3198) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][310/625] eta 0:02:01 lr 0.001999 wd 0.0500 time 0.3762 (0.3850) data time 0.0011 (0.0025) model time 0.3751 (0.3838) loss 4.5757 (4.0925) grad_norm 1.3591 (1.3132) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][320/625] eta 0:01:57 lr 0.001999 wd 0.0500 time 0.3834 (0.3855) data time 0.0008 (0.0025) model time 0.3826 (0.3843) loss 5.0005 (4.0904) grad_norm 1.3240 (1.3150) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][330/625] eta 0:01:53 lr 0.001999 wd 0.0500 time 0.3715 (0.3852) data time 0.0011 (0.0025) model time 0.3704 (0.3840) loss 3.9851 (4.0919) grad_norm 1.1716 (1.3164) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:27:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][340/625] eta 0:01:49 lr 0.001999 wd 0.0500 time 0.3781 (0.3849) data time 0.0008 (0.0024) model time 0.3772 (0.3837) loss 4.8138 (4.0933) grad_norm 1.2568 (1.3143) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][350/625] eta 0:01:45 lr 0.001999 wd 0.0500 time 0.3805 (0.3848) data time 0.0010 (0.0024) model time 0.3796 (0.3835) loss 4.5308 (4.0932) grad_norm 1.2044 (1.3080) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][360/625] eta 0:01:41 lr 0.001999 wd 0.0500 time 0.3771 (0.3846) data time 0.0010 (0.0024) model time 0.3761 (0.3833) loss 4.0923 (4.0958) grad_norm 1.7522 (1.3105) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][370/625] eta 0:01:38 lr 0.001999 wd 0.0500 time 0.3765 (0.3849) data time 0.0011 (0.0023) model time 0.3753 (0.3837) loss 4.2656 (4.0980) grad_norm 1.3149 (1.3201) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][380/625] eta 0:01:34 lr 0.001999 wd 0.0500 time 0.3828 (0.3847) data time 0.0008 (0.0023) model time 0.3820 (0.3835) loss 4.8174 (4.0978) grad_norm 0.9179 (1.3158) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][390/625] eta 0:01:30 lr 0.001999 wd 0.0500 time 0.3758 (0.3846) data time 0.0012 (0.0023) model time 0.3746 (0.3833) loss 2.7707 (4.0860) grad_norm 0.9489 (1.3120) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][400/625] eta 0:01:26 lr 0.001999 wd 0.0500 time 0.3762 (0.3844) data time 0.0011 (0.0022) model time 0.3751 (0.3831) loss 4.0846 (4.0872) grad_norm 1.1012 (1.3130) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][410/625] eta 0:01:22 lr 0.001999 wd 0.0500 time 0.3811 (0.3842) data time 0.0009 (0.0022) model time 0.3802 (0.3829) loss 4.6516 (4.0858) grad_norm 1.4027 (1.3134) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][420/625] eta 0:01:18 lr 0.001999 wd 0.0500 time 0.3758 (0.3840) data time 0.0011 (0.0022) model time 0.3748 (0.3827) loss 3.9630 (4.0907) grad_norm 1.5477 (1.3181) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][430/625] eta 0:01:14 lr 0.001999 wd 0.0500 time 0.3781 (0.3839) data time 0.0008 (0.0022) model time 0.3773 (0.3826) loss 4.8207 (4.0859) grad_norm 1.2519 (1.3141) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][440/625] eta 0:01:10 lr 0.001999 wd 0.0500 time 0.3753 (0.3837) data time 0.0012 (0.0021) model time 0.3740 (0.3824) loss 3.0736 (4.0799) grad_norm 1.3411 (1.3102) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][450/625] eta 0:01:07 lr 0.001999 wd 0.0500 time 0.3783 (0.3836) data time 0.0010 (0.0021) model time 0.3772 (0.3822) loss 2.9593 (4.0801) grad_norm 1.2426 (1.3201) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][460/625] eta 0:01:03 lr 0.001999 wd 0.0500 time 0.3746 (0.3839) data time 0.0012 (0.0021) model time 0.3734 (0.3826) loss 4.8178 (4.0774) grad_norm 2.0659 (1.3186) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][470/625] eta 0:00:59 lr 0.001999 wd 0.0500 time 0.3828 (0.3838) data time 0.0011 (0.0021) model time 0.3818 (0.3825) loss 4.5701 (4.0803) grad_norm 1.6601 (1.3209) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][480/625] eta 0:00:55 lr 0.001999 wd 0.0500 time 0.3760 (0.3837) data time 0.0009 (0.0021) model time 0.3751 (0.3823) loss 4.6906 (4.0790) grad_norm 1.1904 (1.3196) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][490/625] eta 0:00:51 lr 0.001999 wd 0.0500 time 0.3765 (0.3835) data time 0.0011 (0.0020) model time 0.3754 (0.3822) loss 4.0177 (4.0797) grad_norm 2.1105 (1.3221) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:28:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][500/625] eta 0:00:48 lr 0.001999 wd 0.0500 time 0.3721 (0.3842) data time 0.0008 (0.0020) model time 0.3713 (0.3829) loss 4.2404 (4.0721) grad_norm 1.1075 (1.3192) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:29:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][510/625] eta 0:00:44 lr 0.001999 wd 0.0500 time 0.3740 (0.3841) data time 0.0011 (0.0020) model time 0.3729 (0.3828) loss 3.7501 (4.0676) grad_norm 1.2838 (1.3189) loss_scale 32768.0000 (16576.3757) mem 11225MB [2024-08-26 04:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][520/625] eta 0:00:40 lr 0.001999 wd 0.0500 time 0.3714 (0.3840) data time 0.0012 (0.0020) model time 0.3702 (0.3827) loss 4.5803 (4.0700) grad_norm 2.3854 (1.3222) loss_scale 32768.0000 (16887.1555) mem 11225MB [2024-08-26 04:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][530/625] eta 0:00:36 lr 0.001999 wd 0.0500 time 0.3760 (0.3839) data time 0.0009 (0.0020) model time 0.3751 (0.3826) loss 5.0331 (4.0694) grad_norm 1.1796 (1.3281) loss_scale 32768.0000 (17186.2298) mem 11225MB [2024-08-26 04:29:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][540/625] eta 0:00:32 lr 0.001999 wd 0.0500 time 0.3794 (0.3838) data time 0.0011 (0.0020) model time 0.3784 (0.3825) loss 4.2120 (4.0734) grad_norm 1.1178 (1.3250) loss_scale 32768.0000 (17474.2477) mem 11225MB [2024-08-26 04:29:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][550/625] eta 0:00:28 lr 0.001999 wd 0.0500 time 0.3832 (0.3837) data time 0.0010 (0.0019) model time 0.3821 (0.3823) loss 4.5963 (4.0695) grad_norm 1.0233 (1.3228) loss_scale 32768.0000 (17751.8113) mem 11225MB [2024-08-26 04:29:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][560/625] eta 0:00:24 lr 0.001999 wd 0.0500 time 0.3795 (0.3836) data time 0.0011 (0.0019) model time 0.3785 (0.3822) loss 4.2462 (4.0674) grad_norm 1.5081 (1.3261) loss_scale 32768.0000 (18019.4795) mem 11225MB [2024-08-26 04:29:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][570/625] eta 0:00:21 lr 0.001999 wd 0.0500 time 0.3766 (0.3835) data time 0.0011 (0.0019) model time 0.3756 (0.3821) loss 3.4741 (4.0644) grad_norm 1.3675 (1.3269) loss_scale 32768.0000 (18277.7723) mem 11225MB [2024-08-26 04:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][580/625] eta 0:00:17 lr 0.001999 wd 0.0500 time 0.3788 (0.3841) data time 0.0011 (0.0019) model time 0.3777 (0.3828) loss 3.2693 (4.0662) grad_norm 1.7975 (1.3278) loss_scale 32768.0000 (18527.1738) mem 11225MB [2024-08-26 04:29:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][590/625] eta 0:00:13 lr 0.001999 wd 0.0500 time 0.3721 (0.3840) data time 0.0011 (0.0019) model time 0.3710 (0.3827) loss 4.0913 (4.0633) grad_norm 1.5021 (1.3271) loss_scale 32768.0000 (18768.1354) mem 11225MB [2024-08-26 04:29:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][600/625] eta 0:00:09 lr 0.001999 wd 0.0500 time 0.3782 (0.3839) data time 0.0010 (0.0019) model time 0.3772 (0.3826) loss 4.2744 (4.0626) grad_norm 1.2357 (1.3240) loss_scale 32768.0000 (19001.0782) mem 11225MB [2024-08-26 04:29:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][610/625] eta 0:00:05 lr 0.001999 wd 0.0500 time 0.3686 (0.3837) data time 0.0008 (0.0019) model time 0.3678 (0.3824) loss 4.2756 (4.0593) grad_norm 1.3868 (1.3241) loss_scale 32768.0000 (19226.3961) mem 11225MB [2024-08-26 04:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [23/300][620/625] eta 0:00:01 lr 0.001999 wd 0.0500 time 0.3667 (0.3838) data time 0.0008 (0.0018) model time 0.3659 (0.3825) loss 4.3229 (4.0551) grad_norm 1.9750 (1.3222) loss_scale 32768.0000 (19444.4573) mem 11225MB [2024-08-26 04:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 23 training takes 0:03:59 [2024-08-26 04:29:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:29:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:29:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.9214 (0.9214) Acc@1 77.881 (77.881) Acc@5 94.775 (94.775) Mem 11225MB [2024-08-26 04:29:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 1.5820 (1.1061) Acc@1 62.988 (73.997) Acc@5 87.158 (93.062) Mem 11225MB [2024-08-26 04:29:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 1.7725 (1.3661) Acc@1 59.863 (68.615) Acc@5 84.180 (89.560) Mem 11225MB [2024-08-26 04:29:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 68.568 Acc@5 89.561 [2024-08-26 04:29:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 68.6% [2024-08-26 04:29:51 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 68.57% [2024-08-26 04:29:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:29:52 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:29:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.491 (0.491) Loss 1.0469 (1.0469) Acc@1 73.047 (73.047) Acc@5 91.406 (91.406) Mem 11225MB [2024-08-26 04:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.142) Loss 1.7773 (1.2593) Acc@1 56.201 (67.707) Acc@5 82.861 (89.626) Mem 11225MB [2024-08-26 04:29:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.9619 (1.5308) Acc@1 54.053 (62.805) Acc@5 79.150 (85.493) Mem 11225MB [2024-08-26 04:29:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 62.826 Acc@5 85.565 [2024-08-26 04:29:55 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 62.8% [2024-08-26 04:29:55 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 62.83% [2024-08-26 04:29:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:29:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][0/625] eta 0:07:49 lr 0.001999 wd 0.0500 time 0.7508 (0.7508) data time 0.4129 (0.4129) model time 0.0000 (0.0000) loss 4.2379 (4.2379) grad_norm 0.9060 (0.9060) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:30:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][10/625] eta 0:04:12 lr 0.001999 wd 0.0500 time 0.3693 (0.4104) data time 0.0011 (0.0386) model time 0.0000 (0.0000) loss 3.3744 (4.0982) grad_norm 1.0667 (1.2481) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:30:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][20/625] eta 0:03:58 lr 0.001999 wd 0.0500 time 0.3717 (0.3936) data time 0.0008 (0.0208) model time 0.0000 (0.0000) loss 3.1715 (3.9536) grad_norm 1.0337 (1.2906) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:30:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][30/625] eta 0:03:51 lr 0.001999 wd 0.0500 time 0.3776 (0.3887) data time 0.0010 (0.0144) model time 0.0000 (0.0000) loss 4.3711 (4.0047) grad_norm 1.0511 (1.3145) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:30:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][40/625] eta 0:03:45 lr 0.001999 wd 0.0500 time 0.3760 (0.3860) data time 0.0009 (0.0112) model time 0.0000 (0.0000) loss 4.8723 (4.0629) grad_norm 1.1543 (1.2794) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][50/625] eta 0:03:40 lr 0.001999 wd 0.0500 time 0.3672 (0.3838) data time 0.0010 (0.0092) model time 0.0000 (0.0000) loss 3.6947 (4.0085) grad_norm 1.1045 (inf) loss_scale 16384.0000 (31482.9804) mem 11225MB [2024-08-26 04:30:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][60/625] eta 0:03:36 lr 0.001999 wd 0.0500 time 0.3737 (0.3827) data time 0.0008 (0.0079) model time 0.3729 (0.3762) loss 4.5837 (4.0141) grad_norm 0.9819 (inf) loss_scale 16384.0000 (29007.7377) mem 11225MB [2024-08-26 04:30:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][70/625] eta 0:03:32 lr 0.001999 wd 0.0500 time 0.3740 (0.3821) data time 0.0009 (0.0070) model time 0.3731 (0.3764) loss 4.1504 (4.0094) grad_norm 1.2531 (inf) loss_scale 16384.0000 (27229.7465) mem 11225MB [2024-08-26 04:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][80/625] eta 0:03:27 lr 0.001999 wd 0.0500 time 0.3766 (0.3813) data time 0.0010 (0.0062) model time 0.3756 (0.3758) loss 4.3149 (4.0580) grad_norm 1.2649 (inf) loss_scale 16384.0000 (25890.7654) mem 11225MB [2024-08-26 04:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][90/625] eta 0:03:23 lr 0.001999 wd 0.0500 time 0.3794 (0.3807) data time 0.0008 (0.0057) model time 0.3787 (0.3757) loss 2.7085 (4.0506) grad_norm 1.5440 (inf) loss_scale 16384.0000 (24846.0659) mem 11225MB [2024-08-26 04:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][100/625] eta 0:03:20 lr 0.001999 wd 0.0500 time 0.3794 (0.3821) data time 0.0011 (0.0052) model time 0.3783 (0.3792) loss 3.8544 (4.0267) grad_norm 1.0612 (inf) loss_scale 16384.0000 (24008.2376) mem 11225MB [2024-08-26 04:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][110/625] eta 0:03:16 lr 0.001999 wd 0.0500 time 0.3723 (0.3815) data time 0.0008 (0.0049) model time 0.3715 (0.3783) loss 3.7760 (4.0246) grad_norm 1.3224 (inf) loss_scale 16384.0000 (23321.3694) mem 11225MB [2024-08-26 04:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][120/625] eta 0:03:12 lr 0.001999 wd 0.0500 time 0.3790 (0.3810) data time 0.0008 (0.0045) model time 0.3782 (0.3779) loss 5.0129 (4.0250) grad_norm 1.1663 (inf) loss_scale 16384.0000 (22748.0331) mem 11225MB [2024-08-26 04:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][130/625] eta 0:03:08 lr 0.001999 wd 0.0500 time 0.3907 (0.3807) data time 0.0010 (0.0043) model time 0.3897 (0.3776) loss 3.4366 (4.0314) grad_norm 1.1228 (inf) loss_scale 16384.0000 (22262.2290) mem 11225MB [2024-08-26 04:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][140/625] eta 0:03:04 lr 0.001999 wd 0.0500 time 0.3758 (0.3802) data time 0.0009 (0.0041) model time 0.3749 (0.3771) loss 2.7358 (4.0378) grad_norm 1.0850 (inf) loss_scale 16384.0000 (21845.3333) mem 11225MB [2024-08-26 04:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][150/625] eta 0:03:01 lr 0.001999 wd 0.0500 time 0.5815 (0.3814) data time 0.0011 (0.0039) model time 0.5805 (0.3790) loss 3.5444 (4.0358) grad_norm 1.7125 (inf) loss_scale 16384.0000 (21483.6556) mem 11225MB [2024-08-26 04:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][160/625] eta 0:02:57 lr 0.001999 wd 0.0500 time 0.3878 (0.3811) data time 0.0009 (0.0037) model time 0.3869 (0.3788) loss 4.6233 (4.0230) grad_norm 1.4491 (inf) loss_scale 16384.0000 (21166.9068) mem 11225MB [2024-08-26 04:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][170/625] eta 0:02:53 lr 0.001999 wd 0.0500 time 0.3817 (0.3816) data time 0.0011 (0.0035) model time 0.3806 (0.3796) loss 3.3192 (4.0012) grad_norm 1.4045 (inf) loss_scale 16384.0000 (20887.2047) mem 11225MB [2024-08-26 04:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][180/625] eta 0:02:49 lr 0.001999 wd 0.0500 time 0.3737 (0.3814) data time 0.0010 (0.0034) model time 0.3727 (0.3794) loss 4.6058 (3.9975) grad_norm 1.3571 (inf) loss_scale 16384.0000 (20638.4088) mem 11225MB [2024-08-26 04:31:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][190/625] eta 0:02:45 lr 0.001999 wd 0.0500 time 0.3807 (0.3813) data time 0.0010 (0.0033) model time 0.3797 (0.3792) loss 3.9956 (3.9944) grad_norm 1.2051 (inf) loss_scale 16384.0000 (20415.6649) mem 11225MB [2024-08-26 04:31:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][200/625] eta 0:02:41 lr 0.001999 wd 0.0500 time 0.3745 (0.3810) data time 0.0011 (0.0032) model time 0.3734 (0.3790) loss 3.8234 (4.0020) grad_norm 1.1366 (inf) loss_scale 16384.0000 (20215.0846) mem 11225MB [2024-08-26 04:31:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][210/625] eta 0:02:38 lr 0.001999 wd 0.0500 time 0.3711 (0.3816) data time 0.0011 (0.0031) model time 0.3700 (0.3798) loss 3.6199 (4.0145) grad_norm 1.5753 (inf) loss_scale 16384.0000 (20033.5166) mem 11225MB [2024-08-26 04:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][220/625] eta 0:02:34 lr 0.001999 wd 0.0500 time 0.3792 (0.3815) data time 0.0011 (0.0030) model time 0.3781 (0.3796) loss 3.2900 (4.0101) grad_norm 0.9825 (inf) loss_scale 16384.0000 (19868.3801) mem 11225MB [2024-08-26 04:31:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][230/625] eta 0:02:30 lr 0.001999 wd 0.0500 time 0.3713 (0.3814) data time 0.0010 (0.0029) model time 0.3703 (0.3795) loss 3.9103 (4.0147) grad_norm 1.4510 (inf) loss_scale 16384.0000 (19717.5411) mem 11225MB [2024-08-26 04:31:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][240/625] eta 0:02:26 lr 0.001999 wd 0.0500 time 0.3780 (0.3812) data time 0.0008 (0.0028) model time 0.3772 (0.3794) loss 4.8545 (4.0074) grad_norm 1.4291 (inf) loss_scale 16384.0000 (19579.2199) mem 11225MB [2024-08-26 04:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][250/625] eta 0:02:22 lr 0.001999 wd 0.0500 time 0.3828 (0.3810) data time 0.0008 (0.0028) model time 0.3819 (0.3792) loss 4.7996 (4.0070) grad_norm 0.9393 (inf) loss_scale 16384.0000 (19451.9203) mem 11225MB [2024-08-26 04:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][260/625] eta 0:02:19 lr 0.001999 wd 0.0500 time 0.3737 (0.3810) data time 0.0009 (0.0027) model time 0.3728 (0.3792) loss 3.6227 (4.0069) grad_norm 0.9918 (inf) loss_scale 16384.0000 (19334.3755) mem 11225MB [2024-08-26 04:31:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][270/625] eta 0:02:15 lr 0.001999 wd 0.0500 time 0.3832 (0.3809) data time 0.0013 (0.0026) model time 0.3819 (0.3791) loss 3.7225 (4.0123) grad_norm 3.0121 (inf) loss_scale 16384.0000 (19225.5055) mem 11225MB [2024-08-26 04:31:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][280/625] eta 0:02:11 lr 0.001999 wd 0.0500 time 0.3903 (0.3809) data time 0.0011 (0.0026) model time 0.3892 (0.3791) loss 4.4454 (4.0167) grad_norm 2.2081 (inf) loss_scale 16384.0000 (19124.3843) mem 11225MB [2024-08-26 04:31:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][290/625] eta 0:02:07 lr 0.001999 wd 0.0500 time 0.3710 (0.3814) data time 0.0008 (0.0025) model time 0.3701 (0.3798) loss 5.0417 (4.0159) grad_norm 1.2118 (inf) loss_scale 16384.0000 (19030.2131) mem 11225MB [2024-08-26 04:31:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][300/625] eta 0:02:03 lr 0.001999 wd 0.0500 time 0.3755 (0.3813) data time 0.0010 (0.0025) model time 0.3745 (0.3796) loss 3.9709 (4.0125) grad_norm 1.0389 (inf) loss_scale 16384.0000 (18942.2990) mem 11225MB [2024-08-26 04:31:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][310/625] eta 0:02:00 lr 0.001999 wd 0.0500 time 0.3776 (0.3819) data time 0.0008 (0.0025) model time 0.3768 (0.3804) loss 4.7032 (4.0242) grad_norm 1.2296 (inf) loss_scale 16384.0000 (18860.0386) mem 11225MB [2024-08-26 04:31:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][320/625] eta 0:01:56 lr 0.001999 wd 0.0500 time 0.3578 (0.3818) data time 0.0008 (0.0024) model time 0.3570 (0.3803) loss 4.8421 (4.0404) grad_norm 1.0738 (inf) loss_scale 16384.0000 (18782.9034) mem 11225MB [2024-08-26 04:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][330/625] eta 0:01:52 lr 0.001999 wd 0.0500 time 0.3762 (0.3817) data time 0.0009 (0.0024) model time 0.3753 (0.3802) loss 4.5741 (4.0414) grad_norm 1.3119 (inf) loss_scale 16384.0000 (18710.4290) mem 11225MB [2024-08-26 04:32:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][340/625] eta 0:01:49 lr 0.001999 wd 0.0500 time 0.3757 (0.3826) data time 0.0011 (0.0023) model time 0.3746 (0.3813) loss 3.7676 (4.0395) grad_norm 1.0764 (inf) loss_scale 16384.0000 (18642.2053) mem 11225MB [2024-08-26 04:32:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][350/625] eta 0:01:45 lr 0.001999 wd 0.0500 time 0.3795 (0.3824) data time 0.0011 (0.0023) model time 0.3784 (0.3811) loss 4.6304 (4.0298) grad_norm 1.2230 (inf) loss_scale 16384.0000 (18577.8689) mem 11225MB [2024-08-26 04:32:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][360/625] eta 0:01:41 lr 0.001999 wd 0.0500 time 0.3725 (0.3823) data time 0.0013 (0.0023) model time 0.3712 (0.3809) loss 3.2040 (4.0307) grad_norm 1.0451 (inf) loss_scale 16384.0000 (18517.0970) mem 11225MB [2024-08-26 04:32:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][370/625] eta 0:01:37 lr 0.001999 wd 0.0500 time 0.3791 (0.3822) data time 0.0010 (0.0022) model time 0.3781 (0.3809) loss 4.1395 (4.0291) grad_norm 1.4722 (inf) loss_scale 16384.0000 (18459.6011) mem 11225MB [2024-08-26 04:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][380/625] eta 0:01:33 lr 0.001999 wd 0.0500 time 0.3814 (0.3822) data time 0.0011 (0.0022) model time 0.3803 (0.3808) loss 4.3984 (4.0308) grad_norm 1.2537 (inf) loss_scale 16384.0000 (18405.1234) mem 11225MB [2024-08-26 04:32:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][390/625] eta 0:01:29 lr 0.001999 wd 0.0500 time 0.3817 (0.3826) data time 0.0011 (0.0022) model time 0.3806 (0.3813) loss 3.5661 (4.0384) grad_norm 0.9296 (inf) loss_scale 16384.0000 (18353.4322) mem 11225MB [2024-08-26 04:32:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][400/625] eta 0:01:26 lr 0.001999 wd 0.0500 time 0.3699 (0.3824) data time 0.0009 (0.0022) model time 0.3691 (0.3811) loss 3.4143 (4.0347) grad_norm 1.5898 (inf) loss_scale 16384.0000 (18304.3192) mem 11225MB [2024-08-26 04:32:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][410/625] eta 0:01:22 lr 0.001999 wd 0.0500 time 0.3636 (0.3823) data time 0.0009 (0.0021) model time 0.3627 (0.3810) loss 5.0377 (4.0327) grad_norm 1.1890 (inf) loss_scale 16384.0000 (18257.5961) mem 11225MB [2024-08-26 04:32:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][420/625] eta 0:01:18 lr 0.001999 wd 0.0500 time 0.3762 (0.3822) data time 0.0008 (0.0021) model time 0.3754 (0.3809) loss 4.3015 (4.0334) grad_norm 1.2059 (inf) loss_scale 16384.0000 (18213.0926) mem 11225MB [2024-08-26 04:32:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][430/625] eta 0:01:14 lr 0.001999 wd 0.0500 time 0.3944 (0.3822) data time 0.0011 (0.0021) model time 0.3933 (0.3808) loss 3.5221 (4.0374) grad_norm 0.8550 (inf) loss_scale 16384.0000 (18170.6543) mem 11225MB [2024-08-26 04:32:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][440/625] eta 0:01:10 lr 0.001999 wd 0.0500 time 0.3877 (0.3821) data time 0.0008 (0.0021) model time 0.3869 (0.3807) loss 4.9143 (4.0403) grad_norm 1.3026 (inf) loss_scale 16384.0000 (18130.1406) mem 11225MB [2024-08-26 04:32:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][450/625] eta 0:01:06 lr 0.001999 wd 0.0500 time 0.3710 (0.3819) data time 0.0012 (0.0021) model time 0.3698 (0.3806) loss 4.5365 (4.0402) grad_norm 1.7797 (inf) loss_scale 16384.0000 (18091.4235) mem 11225MB [2024-08-26 04:32:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][460/625] eta 0:01:03 lr 0.001999 wd 0.0500 time 0.3870 (0.3823) data time 0.0008 (0.0020) model time 0.3862 (0.3810) loss 4.2797 (4.0436) grad_norm 1.6259 (inf) loss_scale 16384.0000 (18054.3861) mem 11225MB [2024-08-26 04:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][470/625] eta 0:00:59 lr 0.001999 wd 0.0500 time 0.3750 (0.3823) data time 0.0011 (0.0020) model time 0.3739 (0.3809) loss 4.0273 (4.0382) grad_norm 1.2376 (inf) loss_scale 16384.0000 (18018.9214) mem 11225MB [2024-08-26 04:33:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][480/625] eta 0:00:55 lr 0.001999 wd 0.0500 time 0.3839 (0.3822) data time 0.0009 (0.0020) model time 0.3830 (0.3808) loss 4.0743 (4.0307) grad_norm 1.2347 (inf) loss_scale 16384.0000 (17984.9314) mem 11225MB [2024-08-26 04:33:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][490/625] eta 0:00:51 lr 0.001999 wd 0.0500 time 0.3756 (0.3830) data time 0.0008 (0.0020) model time 0.3748 (0.3818) loss 4.8322 (4.0299) grad_norm 1.4259 (inf) loss_scale 16384.0000 (17952.3259) mem 11225MB [2024-08-26 04:33:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][500/625] eta 0:00:47 lr 0.001999 wd 0.0500 time 0.3767 (0.3833) data time 0.0008 (0.0020) model time 0.3759 (0.3821) loss 4.0493 (4.0287) grad_norm 1.4225 (inf) loss_scale 16384.0000 (17921.0220) mem 11225MB [2024-08-26 04:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][510/625] eta 0:00:44 lr 0.001999 wd 0.0500 time 0.3782 (0.3836) data time 0.0009 (0.0019) model time 0.3773 (0.3824) loss 4.5784 (4.0309) grad_norm 1.0755 (inf) loss_scale 16384.0000 (17890.9432) mem 11225MB [2024-08-26 04:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][520/625] eta 0:00:40 lr 0.001999 wd 0.0500 time 0.3826 (0.3836) data time 0.0008 (0.0019) model time 0.3818 (0.3824) loss 4.8426 (4.0323) grad_norm 1.1849 (inf) loss_scale 16384.0000 (17862.0192) mem 11225MB [2024-08-26 04:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][530/625] eta 0:00:36 lr 0.001999 wd 0.0500 time 0.3753 (0.3834) data time 0.0008 (0.0019) model time 0.3745 (0.3823) loss 3.1203 (4.0322) grad_norm 1.7090 (inf) loss_scale 16384.0000 (17834.1846) mem 11225MB [2024-08-26 04:33:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][540/625] eta 0:00:32 lr 0.001999 wd 0.0500 time 0.3911 (0.3834) data time 0.0010 (0.0019) model time 0.3900 (0.3822) loss 3.5109 (4.0331) grad_norm 1.4293 (inf) loss_scale 16384.0000 (17807.3789) mem 11225MB [2024-08-26 04:33:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][550/625] eta 0:00:28 lr 0.001999 wd 0.0500 time 0.3827 (0.3833) data time 0.0010 (0.0019) model time 0.3816 (0.3821) loss 2.9705 (4.0321) grad_norm 1.3391 (inf) loss_scale 16384.0000 (17781.5463) mem 11225MB [2024-08-26 04:33:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][560/625] eta 0:00:24 lr 0.001999 wd 0.0500 time 0.3833 (0.3833) data time 0.0011 (0.0019) model time 0.3822 (0.3821) loss 4.5014 (4.0269) grad_norm 1.6316 (inf) loss_scale 16384.0000 (17756.6346) mem 11225MB [2024-08-26 04:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][570/625] eta 0:00:21 lr 0.001998 wd 0.0500 time 0.3752 (0.3832) data time 0.0011 (0.0019) model time 0.3741 (0.3820) loss 4.1361 (4.0260) grad_norm 1.2042 (inf) loss_scale 16384.0000 (17732.5954) mem 11225MB [2024-08-26 04:33:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][580/625] eta 0:00:17 lr 0.001998 wd 0.0500 time 0.3819 (0.3832) data time 0.0011 (0.0018) model time 0.3808 (0.3820) loss 3.1659 (4.0237) grad_norm 1.3349 (inf) loss_scale 16384.0000 (17709.3838) mem 11225MB [2024-08-26 04:33:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][590/625] eta 0:00:13 lr 0.001998 wd 0.0500 time 0.3834 (0.3831) data time 0.0008 (0.0018) model time 0.3826 (0.3819) loss 3.6747 (4.0178) grad_norm 1.1850 (inf) loss_scale 16384.0000 (17686.9577) mem 11225MB [2024-08-26 04:33:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][600/625] eta 0:00:09 lr 0.001998 wd 0.0500 time 0.3778 (0.3831) data time 0.0010 (0.0019) model time 0.3767 (0.3818) loss 4.2872 (4.0195) grad_norm 1.7434 (inf) loss_scale 16384.0000 (17665.2779) mem 11225MB [2024-08-26 04:33:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][610/625] eta 0:00:05 lr 0.001998 wd 0.0500 time 0.3659 (0.3837) data time 0.0008 (0.0019) model time 0.3651 (0.3825) loss 4.2760 (4.0189) grad_norm 1.0460 (inf) loss_scale 16384.0000 (17644.3077) mem 11225MB [2024-08-26 04:33:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [24/300][620/625] eta 0:00:01 lr 0.001998 wd 0.0500 time 0.3650 (0.3834) data time 0.0006 (0.0018) model time 0.3645 (0.3822) loss 4.6140 (4.0227) grad_norm 1.4840 (inf) loss_scale 16384.0000 (17624.0129) mem 11225MB [2024-08-26 04:33:56 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 24 training takes 0:03:59 [2024-08-26 04:33:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:33:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:33:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.513 (0.513) Loss 0.8618 (0.8618) Acc@1 79.980 (79.980) Acc@5 95.312 (95.312) Mem 11225MB [2024-08-26 04:33:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.5176 (1.0987) Acc@1 65.332 (74.476) Acc@5 88.525 (93.417) Mem 11225MB [2024-08-26 04:33:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.6924 (1.3501) Acc@1 63.379 (69.389) Acc@5 85.547 (89.909) Mem 11225MB [2024-08-26 04:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 69.118 Acc@5 89.813 [2024-08-26 04:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 69.1% [2024-08-26 04:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 69.12% [2024-08-26 04:34:00 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:34:01 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:34:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.501 (0.501) Loss 0.9692 (0.9692) Acc@1 74.756 (74.756) Acc@5 92.383 (92.383) Mem 11225MB [2024-08-26 04:34:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.6777 (1.1804) Acc@1 57.812 (69.438) Acc@5 84.326 (90.656) Mem 11225MB [2024-08-26 04:34:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.8672 (1.4482) Acc@1 55.908 (64.558) Acc@5 80.664 (86.658) Mem 11225MB [2024-08-26 04:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 64.571 Acc@5 86.676 [2024-08-26 04:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 64.6% [2024-08-26 04:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 64.57% [2024-08-26 04:34:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:34:05 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][0/625] eta 0:08:21 lr 0.001998 wd 0.0500 time 0.8019 (0.8019) data time 0.4686 (0.4686) model time 0.0000 (0.0000) loss 4.8046 (4.8046) grad_norm 1.1847 (1.1847) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][10/625] eta 0:04:16 lr 0.001998 wd 0.0500 time 0.3911 (0.4175) data time 0.0010 (0.0436) model time 0.0000 (0.0000) loss 4.0915 (4.0244) grad_norm 1.7275 (1.2981) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][20/625] eta 0:04:01 lr 0.001998 wd 0.0500 time 0.3747 (0.3987) data time 0.0008 (0.0233) model time 0.0000 (0.0000) loss 3.5986 (4.0382) grad_norm 1.5720 (1.3146) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][30/625] eta 0:04:00 lr 0.001998 wd 0.0500 time 0.3772 (0.4034) data time 0.0010 (0.0164) model time 0.0000 (0.0000) loss 3.4150 (3.9353) grad_norm 0.9850 (1.2589) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][40/625] eta 0:03:52 lr 0.001998 wd 0.0500 time 0.3921 (0.3981) data time 0.0008 (0.0127) model time 0.0000 (0.0000) loss 4.4367 (3.9087) grad_norm 1.6227 (1.2759) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][50/625] eta 0:03:49 lr 0.001998 wd 0.0500 time 0.6016 (0.3985) data time 0.0011 (0.0104) model time 0.0000 (0.0000) loss 4.1481 (3.8962) grad_norm 1.4460 (1.2453) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][60/625] eta 0:03:43 lr 0.001998 wd 0.0500 time 0.3763 (0.3950) data time 0.0008 (0.0089) model time 0.3755 (0.3760) loss 4.1264 (3.9521) grad_norm 1.1055 (1.2510) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][70/625] eta 0:03:38 lr 0.001998 wd 0.0500 time 0.3828 (0.3933) data time 0.0010 (0.0078) model time 0.3818 (0.3789) loss 3.9929 (3.9921) grad_norm 1.3180 (1.2601) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][80/625] eta 0:03:33 lr 0.001998 wd 0.0500 time 0.3703 (0.3912) data time 0.0012 (0.0070) model time 0.3691 (0.3777) loss 3.3879 (3.9984) grad_norm 1.3386 (1.2665) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][90/625] eta 0:03:28 lr 0.001998 wd 0.0500 time 0.3787 (0.3898) data time 0.0009 (0.0063) model time 0.3778 (0.3776) loss 4.6797 (4.0476) grad_norm 1.8075 (1.2576) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][100/625] eta 0:03:24 lr 0.001998 wd 0.0500 time 0.3894 (0.3889) data time 0.0008 (0.0058) model time 0.3886 (0.3779) loss 4.2599 (4.0321) grad_norm 1.6138 (1.2803) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][110/625] eta 0:03:19 lr 0.001998 wd 0.0500 time 0.3712 (0.3881) data time 0.0011 (0.0054) model time 0.3701 (0.3782) loss 4.2447 (4.0361) grad_norm 1.1849 (1.2899) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][120/625] eta 0:03:15 lr 0.001998 wd 0.0500 time 0.3810 (0.3873) data time 0.0011 (0.0050) model time 0.3799 (0.3780) loss 3.8094 (4.0728) grad_norm 1.5651 (1.2952) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:34:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][130/625] eta 0:03:11 lr 0.001998 wd 0.0500 time 0.3865 (0.3866) data time 0.0010 (0.0047) model time 0.3855 (0.3779) loss 3.6415 (4.0936) grad_norm 1.2906 (1.2832) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][140/625] eta 0:03:07 lr 0.001998 wd 0.0500 time 0.3768 (0.3859) data time 0.0008 (0.0045) model time 0.3760 (0.3777) loss 3.4373 (4.0797) grad_norm 1.9981 (1.2919) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][150/625] eta 0:03:03 lr 0.001998 wd 0.0500 time 0.3795 (0.3855) data time 0.0009 (0.0043) model time 0.3786 (0.3778) loss 2.4818 (4.0754) grad_norm 1.5279 (1.2921) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][160/625] eta 0:02:59 lr 0.001998 wd 0.0500 time 0.3833 (0.3850) data time 0.0012 (0.0041) model time 0.3821 (0.3777) loss 3.5383 (4.0752) grad_norm 1.1147 (1.2851) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][170/625] eta 0:02:56 lr 0.001998 wd 0.0500 time 0.3724 (0.3870) data time 0.0008 (0.0039) model time 0.3715 (0.3810) loss 2.9752 (4.0740) grad_norm 1.0481 (1.2883) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][180/625] eta 0:02:52 lr 0.001998 wd 0.0500 time 0.3781 (0.3872) data time 0.0008 (0.0037) model time 0.3773 (0.3817) loss 4.7916 (4.0615) grad_norm 1.4446 (1.2856) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][190/625] eta 0:02:48 lr 0.001998 wd 0.0500 time 0.4112 (0.3870) data time 0.0011 (0.0036) model time 0.4101 (0.3817) loss 4.1036 (4.0600) grad_norm 1.3147 (1.2783) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][200/625] eta 0:02:44 lr 0.001998 wd 0.0500 time 0.3753 (0.3866) data time 0.0013 (0.0035) model time 0.3740 (0.3814) loss 3.6587 (4.0651) grad_norm 1.3092 (1.2811) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][210/625] eta 0:02:40 lr 0.001998 wd 0.0500 time 0.3716 (0.3861) data time 0.0009 (0.0034) model time 0.3707 (0.3811) loss 4.5019 (4.0523) grad_norm 1.0751 (1.2774) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][220/625] eta 0:02:36 lr 0.001998 wd 0.0500 time 0.3792 (0.3857) data time 0.0009 (0.0033) model time 0.3783 (0.3808) loss 4.5870 (4.0575) grad_norm 1.4395 (1.2871) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][230/625] eta 0:02:32 lr 0.001998 wd 0.0500 time 0.3793 (0.3855) data time 0.0018 (0.0032) model time 0.3774 (0.3807) loss 3.6508 (4.0540) grad_norm 1.2564 (1.2836) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][240/625] eta 0:02:29 lr 0.001998 wd 0.0500 time 0.3867 (0.3874) data time 0.0011 (0.0031) model time 0.3856 (0.3832) loss 4.5259 (4.0652) grad_norm 1.2177 (1.2830) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][250/625] eta 0:02:25 lr 0.001998 wd 0.0500 time 0.3804 (0.3870) data time 0.0010 (0.0030) model time 0.3794 (0.3829) loss 4.1538 (4.0533) grad_norm 1.1780 (1.2926) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][260/625] eta 0:02:21 lr 0.001998 wd 0.0500 time 0.3772 (0.3867) data time 0.0009 (0.0029) model time 0.3763 (0.3827) loss 4.8585 (4.0518) grad_norm 1.2509 (1.2894) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][270/625] eta 0:02:17 lr 0.001998 wd 0.0500 time 0.3840 (0.3871) data time 0.0010 (0.0029) model time 0.3830 (0.3833) loss 3.4301 (4.0366) grad_norm 0.9920 (1.2881) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][280/625] eta 0:02:13 lr 0.001998 wd 0.0500 time 0.3808 (0.3868) data time 0.0008 (0.0028) model time 0.3801 (0.3831) loss 4.2238 (4.0314) grad_norm 1.7093 (1.2880) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:35:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][290/625] eta 0:02:09 lr 0.001998 wd 0.0500 time 0.3778 (0.3880) data time 0.0008 (0.0028) model time 0.3770 (0.3846) loss 3.6785 (4.0220) grad_norm 1.1602 (1.2924) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][300/625] eta 0:02:06 lr 0.001998 wd 0.0500 time 0.3821 (0.3884) data time 0.0010 (0.0027) model time 0.3811 (0.3853) loss 3.4019 (4.0113) grad_norm 1.2414 (1.2975) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][310/625] eta 0:02:02 lr 0.001998 wd 0.0500 time 0.3836 (0.3881) data time 0.0008 (0.0027) model time 0.3828 (0.3850) loss 2.7989 (4.0074) grad_norm 1.1408 (1.2955) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][320/625] eta 0:01:58 lr 0.001998 wd 0.0500 time 0.3811 (0.3879) data time 0.0008 (0.0026) model time 0.3804 (0.3848) loss 2.7243 (4.0056) grad_norm 1.2499 (1.2922) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][330/625] eta 0:01:54 lr 0.001998 wd 0.0500 time 0.3744 (0.3876) data time 0.0008 (0.0026) model time 0.3736 (0.3845) loss 4.5915 (4.0092) grad_norm 0.9215 (1.2920) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][340/625] eta 0:01:50 lr 0.001998 wd 0.0500 time 0.3750 (0.3873) data time 0.0008 (0.0025) model time 0.3742 (0.3842) loss 4.4508 (4.0115) grad_norm 1.3190 (1.2943) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][350/625] eta 0:01:46 lr 0.001998 wd 0.0500 time 0.3735 (0.3870) data time 0.0009 (0.0025) model time 0.3726 (0.3839) loss 4.5854 (4.0086) grad_norm 1.3515 (1.2909) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][360/625] eta 0:01:42 lr 0.001998 wd 0.0500 time 0.3739 (0.3868) data time 0.0009 (0.0025) model time 0.3730 (0.3837) loss 4.9084 (4.0065) grad_norm 1.1915 (1.2898) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][370/625] eta 0:01:38 lr 0.001998 wd 0.0500 time 0.3730 (0.3865) data time 0.0010 (0.0024) model time 0.3719 (0.3835) loss 4.2926 (4.0084) grad_norm 1.0860 (1.2870) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][380/625] eta 0:01:34 lr 0.001998 wd 0.0500 time 0.3706 (0.3863) data time 0.0008 (0.0024) model time 0.3698 (0.3833) loss 3.0177 (4.0035) grad_norm 1.2324 (1.2854) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][390/625] eta 0:01:30 lr 0.001998 wd 0.0500 time 0.3754 (0.3861) data time 0.0009 (0.0024) model time 0.3745 (0.3831) loss 3.0078 (4.0076) grad_norm 2.0656 (1.2875) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][400/625] eta 0:01:26 lr 0.001998 wd 0.0500 time 0.3752 (0.3860) data time 0.0012 (0.0023) model time 0.3740 (0.3830) loss 3.6548 (4.0069) grad_norm 1.5979 (1.2888) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][410/625] eta 0:01:22 lr 0.001998 wd 0.0500 time 0.3745 (0.3857) data time 0.0008 (0.0023) model time 0.3736 (0.3828) loss 3.5563 (4.0093) grad_norm 1.2669 (1.2886) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][420/625] eta 0:01:19 lr 0.001998 wd 0.0500 time 0.3782 (0.3869) data time 0.0010 (0.0023) model time 0.3771 (0.3841) loss 4.6890 (4.0123) grad_norm 2.0185 (1.2890) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][430/625] eta 0:01:15 lr 0.001998 wd 0.0500 time 0.3691 (0.3867) data time 0.0011 (0.0023) model time 0.3680 (0.3839) loss 4.2947 (4.0021) grad_norm 2.5237 (1.2908) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:36:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][440/625] eta 0:01:11 lr 0.001998 wd 0.0500 time 0.3791 (0.3865) data time 0.0010 (0.0022) model time 0.3781 (0.3838) loss 4.5330 (4.0051) grad_norm 1.4034 (1.3019) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][450/625] eta 0:01:07 lr 0.001998 wd 0.0500 time 0.3920 (0.3868) data time 0.0008 (0.0022) model time 0.3912 (0.3841) loss 2.4516 (4.0066) grad_norm 1.2536 (1.3010) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][460/625] eta 0:01:03 lr 0.001998 wd 0.0500 time 0.3718 (0.3866) data time 0.0011 (0.0022) model time 0.3707 (0.3840) loss 3.8071 (4.0073) grad_norm 1.1681 (1.2989) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][470/625] eta 0:00:59 lr 0.001998 wd 0.0500 time 0.3750 (0.3864) data time 0.0011 (0.0022) model time 0.3739 (0.3838) loss 4.2058 (4.0081) grad_norm 1.0593 (1.2962) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][480/625] eta 0:00:56 lr 0.001998 wd 0.0500 time 0.3761 (0.3862) data time 0.0008 (0.0021) model time 0.3753 (0.3836) loss 3.3365 (4.0058) grad_norm 1.4029 (1.2985) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][490/625] eta 0:00:52 lr 0.001998 wd 0.0500 time 0.3737 (0.3861) data time 0.0009 (0.0021) model time 0.3728 (0.3835) loss 4.5618 (4.0020) grad_norm 1.1838 (1.2969) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][500/625] eta 0:00:48 lr 0.001998 wd 0.0500 time 0.3764 (0.3868) data time 0.0008 (0.0021) model time 0.3756 (0.3843) loss 3.1034 (3.9941) grad_norm 3.1622 (1.3037) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][510/625] eta 0:00:44 lr 0.001998 wd 0.0500 time 0.3843 (0.3870) data time 0.0008 (0.0021) model time 0.3835 (0.3846) loss 3.2447 (3.9907) grad_norm 1.1026 (1.3044) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][520/625] eta 0:00:40 lr 0.001998 wd 0.0500 time 0.3820 (0.3868) data time 0.0011 (0.0021) model time 0.3809 (0.3844) loss 4.0537 (3.9953) grad_norm 1.1334 (1.3035) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][530/625] eta 0:00:36 lr 0.001998 wd 0.0500 time 0.3850 (0.3867) data time 0.0012 (0.0020) model time 0.3839 (0.3843) loss 4.1559 (3.9969) grad_norm 1.0126 (1.3018) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][540/625] eta 0:00:32 lr 0.001998 wd 0.0500 time 0.3723 (0.3872) data time 0.0011 (0.0021) model time 0.3712 (0.3849) loss 4.4914 (4.0021) grad_norm 1.8772 (1.3014) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][550/625] eta 0:00:29 lr 0.001998 wd 0.0500 time 0.3765 (0.3871) data time 0.0011 (0.0021) model time 0.3753 (0.3847) loss 4.3304 (4.0022) grad_norm 1.0038 (1.3009) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][560/625] eta 0:00:25 lr 0.001998 wd 0.0500 time 0.3775 (0.3869) data time 0.0009 (0.0020) model time 0.3766 (0.3846) loss 3.4062 (4.0006) grad_norm 1.3955 (1.2999) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][570/625] eta 0:00:21 lr 0.001998 wd 0.0500 time 0.3783 (0.3872) data time 0.0011 (0.0020) model time 0.3772 (0.3849) loss 4.5459 (3.9913) grad_norm 1.3930 (1.2988) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][580/625] eta 0:00:17 lr 0.001998 wd 0.0500 time 0.3808 (0.3870) data time 0.0011 (0.0020) model time 0.3797 (0.3847) loss 4.0377 (3.9893) grad_norm 1.2087 (1.2966) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][590/625] eta 0:00:13 lr 0.001998 wd 0.0500 time 0.3709 (0.3869) data time 0.0008 (0.0020) model time 0.3700 (0.3846) loss 4.8648 (3.9880) grad_norm 1.2784 (1.2962) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:37:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][600/625] eta 0:00:09 lr 0.001998 wd 0.0500 time 0.3747 (0.3867) data time 0.0010 (0.0020) model time 0.3736 (0.3845) loss 4.4674 (3.9888) grad_norm 1.0831 (1.2939) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][610/625] eta 0:00:05 lr 0.001998 wd 0.0500 time 0.3646 (0.3866) data time 0.0008 (0.0020) model time 0.3638 (0.3843) loss 4.0631 (3.9897) grad_norm 1.3569 (1.2916) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [25/300][620/625] eta 0:00:01 lr 0.001998 wd 0.0500 time 0.3740 (0.3862) data time 0.0008 (0.0020) model time 0.3733 (0.3839) loss 3.4237 (3.9891) grad_norm 1.6372 (1.2938) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:06 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 25 training takes 0:04:01 [2024-08-26 04:38:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:38:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:38:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 0.8945 (0.8945) Acc@1 80.176 (80.176) Acc@5 95.215 (95.215) Mem 11225MB [2024-08-26 04:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 1.5547 (1.1157) Acc@1 64.307 (74.197) Acc@5 86.865 (93.377) Mem 11225MB [2024-08-26 04:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 1.7451 (1.3401) Acc@1 60.596 (69.575) Acc@5 84.717 (90.030) Mem 11225MB [2024-08-26 04:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 69.440 Acc@5 89.983 [2024-08-26 04:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 69.4% [2024-08-26 04:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 69.44% [2024-08-26 04:38:10 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:38:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:38:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.492 (0.492) Loss 0.9087 (0.9087) Acc@1 76.416 (76.416) Acc@5 93.018 (93.018) Mem 11225MB [2024-08-26 04:38:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 1.5938 (1.1157) Acc@1 60.254 (70.819) Acc@5 85.400 (91.477) Mem 11225MB [2024-08-26 04:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 1.7842 (1.3783) Acc@1 57.812 (65.818) Acc@5 81.934 (87.593) Mem 11225MB [2024-08-26 04:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 65.847 Acc@5 87.612 [2024-08-26 04:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 65.8% [2024-08-26 04:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 65.85% [2024-08-26 04:38:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:38:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][0/625] eta 0:08:07 lr 0.001998 wd 0.0500 time 0.7797 (0.7797) data time 0.4418 (0.4418) model time 0.0000 (0.0000) loss 3.7818 (3.7818) grad_norm 1.4151 (1.4151) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][10/625] eta 0:04:14 lr 0.001998 wd 0.0500 time 0.3836 (0.4132) data time 0.0008 (0.0412) model time 0.0000 (0.0000) loss 3.0517 (3.9189) grad_norm 1.4873 (1.3254) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][20/625] eta 0:03:59 lr 0.001998 wd 0.0500 time 0.3754 (0.3966) data time 0.0011 (0.0221) model time 0.0000 (0.0000) loss 3.9069 (3.9434) grad_norm 1.4495 (1.3051) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][30/625] eta 0:03:52 lr 0.001998 wd 0.0500 time 0.3749 (0.3908) data time 0.0010 (0.0153) model time 0.0000 (0.0000) loss 4.3780 (3.9243) grad_norm 1.1730 (1.2837) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][40/625] eta 0:03:46 lr 0.001998 wd 0.0500 time 0.3822 (0.3877) data time 0.0008 (0.0119) model time 0.0000 (0.0000) loss 3.2853 (3.9390) grad_norm 1.7408 (1.3082) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][50/625] eta 0:03:41 lr 0.001998 wd 0.0500 time 0.3733 (0.3854) data time 0.0008 (0.0097) model time 0.0000 (0.0000) loss 3.1527 (3.8775) grad_norm 0.9600 (1.2745) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][60/625] eta 0:03:39 lr 0.001998 wd 0.0500 time 0.3726 (0.3879) data time 0.0009 (0.0083) model time 0.3717 (0.3994) loss 3.6766 (3.8801) grad_norm 1.2211 (1.2521) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][70/625] eta 0:03:40 lr 0.001998 wd 0.0500 time 0.3726 (0.3967) data time 0.0009 (0.0073) model time 0.3717 (0.4242) loss 3.8315 (3.8927) grad_norm 1.4292 (1.2447) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][80/625] eta 0:03:34 lr 0.001998 wd 0.0500 time 0.3718 (0.3943) data time 0.0009 (0.0066) model time 0.3709 (0.4082) loss 2.9262 (3.8753) grad_norm 1.3370 (1.2457) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][90/625] eta 0:03:31 lr 0.001998 wd 0.0500 time 0.3813 (0.3947) data time 0.0011 (0.0060) model time 0.3802 (0.4054) loss 4.5070 (3.8969) grad_norm 1.1725 (1.2396) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][100/625] eta 0:03:26 lr 0.001998 wd 0.0500 time 0.3806 (0.3929) data time 0.0011 (0.0056) model time 0.3795 (0.3994) loss 3.1342 (3.8775) grad_norm 1.2260 (1.2510) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][110/625] eta 0:03:21 lr 0.001998 wd 0.0500 time 0.3709 (0.3915) data time 0.0009 (0.0052) model time 0.3700 (0.3954) loss 5.0431 (3.8827) grad_norm 1.0384 (1.2644) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][120/625] eta 0:03:17 lr 0.001998 wd 0.0500 time 0.3721 (0.3904) data time 0.0012 (0.0048) model time 0.3709 (0.3928) loss 3.6142 (3.8771) grad_norm 0.9591 (1.2817) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][130/625] eta 0:03:12 lr 0.001998 wd 0.0500 time 0.3905 (0.3895) data time 0.0011 (0.0046) model time 0.3894 (0.3909) loss 3.8536 (3.8958) grad_norm 1.9152 (1.2953) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][140/625] eta 0:03:08 lr 0.001998 wd 0.0500 time 0.3805 (0.3887) data time 0.0011 (0.0043) model time 0.3794 (0.3894) loss 3.3491 (3.8847) grad_norm 1.6255 (1.3019) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][150/625] eta 0:03:04 lr 0.001998 wd 0.0500 time 0.3785 (0.3879) data time 0.0009 (0.0041) model time 0.3776 (0.3880) loss 3.1645 (3.8808) grad_norm 0.8705 (1.2956) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][160/625] eta 0:03:00 lr 0.001998 wd 0.0500 time 0.3760 (0.3873) data time 0.0011 (0.0039) model time 0.3749 (0.3869) loss 4.5186 (3.8985) grad_norm 1.3519 (1.2903) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][170/625] eta 0:02:55 lr 0.001998 wd 0.0500 time 0.3777 (0.3867) data time 0.0010 (0.0038) model time 0.3767 (0.3861) loss 2.7590 (3.8919) grad_norm 1.2012 (1.2886) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][180/625] eta 0:02:51 lr 0.001998 wd 0.0500 time 0.3752 (0.3861) data time 0.0009 (0.0036) model time 0.3743 (0.3852) loss 4.3016 (3.8963) grad_norm 1.0620 (1.2943) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][190/625] eta 0:02:47 lr 0.001998 wd 0.0500 time 0.3900 (0.3859) data time 0.0008 (0.0035) model time 0.3892 (0.3848) loss 4.0717 (3.8911) grad_norm 2.2017 (1.2913) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][200/625] eta 0:02:43 lr 0.001998 wd 0.0500 time 0.3679 (0.3855) data time 0.0008 (0.0034) model time 0.3671 (0.3843) loss 3.4175 (3.8742) grad_norm 3.0653 (1.2981) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][210/625] eta 0:02:40 lr 0.001997 wd 0.0500 time 0.3769 (0.3861) data time 0.0011 (0.0033) model time 0.3758 (0.3850) loss 4.2655 (3.8701) grad_norm 1.7613 (1.3042) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][220/625] eta 0:02:36 lr 0.001997 wd 0.0500 time 0.3819 (0.3857) data time 0.0013 (0.0032) model time 0.3806 (0.3846) loss 2.6621 (3.8678) grad_norm 1.1669 (1.2994) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][230/625] eta 0:02:32 lr 0.001997 wd 0.0500 time 0.3735 (0.3853) data time 0.0010 (0.0031) model time 0.3725 (0.3840) loss 4.1053 (3.8697) grad_norm 1.1222 (1.3051) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][240/625] eta 0:02:28 lr 0.001997 wd 0.0500 time 0.3742 (0.3849) data time 0.0012 (0.0030) model time 0.3731 (0.3836) loss 4.1458 (3.8616) grad_norm 1.6028 (1.3039) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][250/625] eta 0:02:24 lr 0.001997 wd 0.0500 time 0.3818 (0.3863) data time 0.0009 (0.0029) model time 0.3810 (0.3853) loss 2.6117 (3.8588) grad_norm 1.0456 (1.2989) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:39:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][260/625] eta 0:02:20 lr 0.001997 wd 0.0500 time 0.3792 (0.3860) data time 0.0008 (0.0029) model time 0.3784 (0.3849) loss 3.5157 (3.8695) grad_norm 1.3629 (1.2940) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][270/625] eta 0:02:16 lr 0.001997 wd 0.0500 time 0.3771 (0.3857) data time 0.0010 (0.0028) model time 0.3761 (0.3846) loss 3.6835 (3.8693) grad_norm 1.3185 (1.2935) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][280/625] eta 0:02:13 lr 0.001997 wd 0.0500 time 0.3591 (0.3881) data time 0.0011 (0.0028) model time 0.3581 (0.3875) loss 4.4459 (3.8641) grad_norm 1.0858 (1.2973) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][290/625] eta 0:02:09 lr 0.001997 wd 0.0500 time 0.3794 (0.3878) data time 0.0009 (0.0027) model time 0.3785 (0.3871) loss 3.0993 (3.8689) grad_norm 1.3933 (1.2999) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][300/625] eta 0:02:06 lr 0.001997 wd 0.0500 time 0.3800 (0.3883) data time 0.0011 (0.0026) model time 0.3789 (0.3877) loss 4.0357 (3.8702) grad_norm 1.0981 (1.2963) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][310/625] eta 0:02:02 lr 0.001997 wd 0.0500 time 0.3796 (0.3880) data time 0.0011 (0.0026) model time 0.3785 (0.3873) loss 4.0409 (3.8679) grad_norm 1.7976 (1.2986) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][320/625] eta 0:01:58 lr 0.001997 wd 0.0500 time 0.3767 (0.3877) data time 0.0011 (0.0025) model time 0.3756 (0.3870) loss 3.9730 (3.8667) grad_norm 1.0300 (1.3055) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][330/625] eta 0:01:54 lr 0.001997 wd 0.0500 time 0.3813 (0.3875) data time 0.0011 (0.0025) model time 0.3802 (0.3867) loss 3.4751 (3.8734) grad_norm 1.4994 (1.3061) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][340/625] eta 0:01:50 lr 0.001997 wd 0.0500 time 0.3699 (0.3871) data time 0.0011 (0.0025) model time 0.3688 (0.3863) loss 4.6226 (3.8754) grad_norm 1.0629 (1.3082) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][350/625] eta 0:01:46 lr 0.001997 wd 0.0500 time 0.3872 (0.3869) data time 0.0008 (0.0024) model time 0.3863 (0.3860) loss 4.6863 (3.8934) grad_norm 1.5726 (1.3100) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][360/625] eta 0:01:42 lr 0.001997 wd 0.0500 time 0.3847 (0.3866) data time 0.0008 (0.0024) model time 0.3839 (0.3857) loss 3.8390 (3.9028) grad_norm 0.8538 (1.3078) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][370/625] eta 0:01:38 lr 0.001997 wd 0.0500 time 0.3716 (0.3875) data time 0.0011 (0.0024) model time 0.3706 (0.3867) loss 4.4862 (3.9082) grad_norm 1.4512 (1.3042) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][380/625] eta 0:01:34 lr 0.001997 wd 0.0500 time 0.3731 (0.3873) data time 0.0012 (0.0023) model time 0.3719 (0.3864) loss 2.5745 (3.9017) grad_norm 1.2708 (1.3011) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][390/625] eta 0:01:31 lr 0.001997 wd 0.0500 time 0.3794 (0.3876) data time 0.0009 (0.0023) model time 0.3786 (0.3867) loss 4.7743 (3.9012) grad_norm 1.2833 (1.3016) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][400/625] eta 0:01:27 lr 0.001997 wd 0.0500 time 0.3824 (0.3873) data time 0.0009 (0.0023) model time 0.3815 (0.3864) loss 4.4218 (3.9032) grad_norm 1.5823 (1.3045) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][410/625] eta 0:01:23 lr 0.001997 wd 0.0500 time 0.3777 (0.3871) data time 0.0009 (0.0022) model time 0.3768 (0.3862) loss 4.8775 (3.8986) grad_norm 1.2837 (1.3082) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:40:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][420/625] eta 0:01:19 lr 0.001997 wd 0.0500 time 0.3742 (0.3869) data time 0.0012 (0.0022) model time 0.3731 (0.3859) loss 3.2866 (3.9038) grad_norm 1.8739 (1.3087) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][430/625] eta 0:01:15 lr 0.001997 wd 0.0500 time 0.3759 (0.3867) data time 0.0009 (0.0022) model time 0.3750 (0.3857) loss 4.9217 (3.9057) grad_norm 1.3037 (1.3080) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][440/625] eta 0:01:11 lr 0.001997 wd 0.0500 time 0.3791 (0.3865) data time 0.0012 (0.0022) model time 0.3779 (0.3854) loss 4.1052 (3.9087) grad_norm 1.4440 (1.3053) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][450/625] eta 0:01:07 lr 0.001997 wd 0.0500 time 0.3755 (0.3863) data time 0.0011 (0.0022) model time 0.3745 (0.3852) loss 3.4876 (3.9071) grad_norm 0.7396 (1.3030) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][460/625] eta 0:01:03 lr 0.001997 wd 0.0500 time 0.3820 (0.3865) data time 0.0011 (0.0021) model time 0.3809 (0.3855) loss 3.5280 (3.9164) grad_norm 3.4719 (1.3094) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][470/625] eta 0:00:59 lr 0.001997 wd 0.0500 time 0.3800 (0.3871) data time 0.0008 (0.0021) model time 0.3791 (0.3861) loss 4.4610 (3.9194) grad_norm 1.0673 (1.3108) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][480/625] eta 0:00:56 lr 0.001997 wd 0.0500 time 0.3761 (0.3873) data time 0.0011 (0.0021) model time 0.3750 (0.3864) loss 4.2718 (3.9128) grad_norm 1.4879 (1.3120) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][490/625] eta 0:00:52 lr 0.001997 wd 0.0500 time 0.3726 (0.3871) data time 0.0010 (0.0021) model time 0.3715 (0.3861) loss 2.8516 (3.9131) grad_norm 1.0642 (1.3075) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][500/625] eta 0:00:48 lr 0.001997 wd 0.0500 time 0.3772 (0.3869) data time 0.0011 (0.0021) model time 0.3761 (0.3859) loss 4.4449 (3.9174) grad_norm 1.0513 (1.3069) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][510/625] eta 0:00:44 lr 0.001997 wd 0.0500 time 0.3757 (0.3870) data time 0.0008 (0.0020) model time 0.3749 (0.3860) loss 4.3879 (3.9203) grad_norm 1.3070 (1.3054) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][520/625] eta 0:00:40 lr 0.001997 wd 0.0500 time 0.3716 (0.3868) data time 0.0009 (0.0020) model time 0.3708 (0.3858) loss 4.7652 (3.9192) grad_norm 0.8428 (1.3001) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][530/625] eta 0:00:36 lr 0.001997 wd 0.0500 time 0.3788 (0.3867) data time 0.0009 (0.0020) model time 0.3779 (0.3856) loss 3.7857 (3.9131) grad_norm 1.6458 (1.2979) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][540/625] eta 0:00:32 lr 0.001997 wd 0.0500 time 0.4020 (0.3865) data time 0.0011 (0.0020) model time 0.4009 (0.3855) loss 4.2161 (3.9169) grad_norm 1.0499 (1.3051) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][550/625] eta 0:00:28 lr 0.001997 wd 0.0500 time 0.3727 (0.3864) data time 0.0013 (0.0020) model time 0.3714 (0.3853) loss 2.9282 (3.9185) grad_norm 1.4318 (1.3056) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][560/625] eta 0:00:25 lr 0.001997 wd 0.0500 time 0.3739 (0.3863) data time 0.0011 (0.0020) model time 0.3728 (0.3852) loss 3.7672 (3.9169) grad_norm 0.9001 (1.3045) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:41:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][570/625] eta 0:00:21 lr 0.001997 wd 0.0500 time 0.6086 (0.3865) data time 0.0010 (0.0020) model time 0.6076 (0.3854) loss 3.6175 (3.9142) grad_norm 1.0554 (1.3030) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][580/625] eta 0:00:17 lr 0.001997 wd 0.0500 time 0.3850 (0.3868) data time 0.0009 (0.0019) model time 0.3841 (0.3857) loss 4.8996 (3.9143) grad_norm 1.9458 (1.3061) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][590/625] eta 0:00:13 lr 0.001997 wd 0.0500 time 0.3763 (0.3869) data time 0.0010 (0.0019) model time 0.3753 (0.3859) loss 3.7784 (3.9174) grad_norm 1.4008 (1.3031) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][600/625] eta 0:00:09 lr 0.001997 wd 0.0500 time 0.3983 (0.3872) data time 0.0010 (0.0019) model time 0.3973 (0.3862) loss 3.9009 (3.9119) grad_norm 1.1060 (1.3011) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][610/625] eta 0:00:05 lr 0.001997 wd 0.0500 time 0.3598 (0.3870) data time 0.0008 (0.0019) model time 0.3590 (0.3860) loss 3.0797 (3.9117) grad_norm 1.5681 (1.3008) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [26/300][620/625] eta 0:00:01 lr 0.001997 wd 0.0500 time 0.3728 (0.3867) data time 0.0006 (0.0019) model time 0.3722 (0.3857) loss 4.6403 (3.9157) grad_norm 1.1878 (1.3016) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:17 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 26 training takes 0:04:01 [2024-08-26 04:42:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:42:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:42:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.498 (0.498) Loss 0.9028 (0.9028) Acc@1 80.176 (80.176) Acc@5 94.873 (94.873) Mem 11225MB [2024-08-26 04:42:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 1.5986 (1.0723) Acc@1 62.988 (75.044) Acc@5 86.816 (93.834) Mem 11225MB [2024-08-26 04:42:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.7373 (1.3237) Acc@1 60.840 (69.954) Acc@5 85.938 (90.276) Mem 11225MB [2024-08-26 04:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 70.024 Acc@5 90.247 [2024-08-26 04:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 70.0% [2024-08-26 04:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 70.02% [2024-08-26 04:42:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:42:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:42:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.492 (0.492) Loss 0.8569 (0.8569) Acc@1 77.441 (77.441) Acc@5 93.750 (93.750) Mem 11225MB [2024-08-26 04:42:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 1.5195 (1.0613) Acc@1 61.523 (72.137) Acc@5 86.377 (92.165) Mem 11225MB [2024-08-26 04:42:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 1.7148 (1.3195) Acc@1 59.277 (67.181) Acc@5 82.764 (88.418) Mem 11225MB [2024-08-26 04:42:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 67.194 Acc@5 88.414 [2024-08-26 04:42:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 67.2% [2024-08-26 04:42:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 67.19% [2024-08-26 04:42:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:42:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:42:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][0/625] eta 0:10:20 lr 0.001997 wd 0.0500 time 0.9923 (0.9923) data time 0.4059 (0.4059) model time 0.0000 (0.0000) loss 2.5523 (2.5523) grad_norm 0.9760 (0.9760) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][10/625] eta 0:04:27 lr 0.001997 wd 0.0500 time 0.3803 (0.4345) data time 0.0008 (0.0379) model time 0.0000 (0.0000) loss 4.8597 (3.8135) grad_norm 0.9515 (1.4305) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][20/625] eta 0:04:06 lr 0.001997 wd 0.0500 time 0.3767 (0.4076) data time 0.0011 (0.0204) model time 0.0000 (0.0000) loss 3.2127 (3.7156) grad_norm 1.1301 (1.4029) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][30/625] eta 0:03:57 lr 0.001997 wd 0.0500 time 0.3898 (0.3991) data time 0.0010 (0.0142) model time 0.0000 (0.0000) loss 2.9348 (3.7325) grad_norm 1.1187 (1.3360) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][40/625] eta 0:03:50 lr 0.001997 wd 0.0500 time 0.3800 (0.3948) data time 0.0011 (0.0110) model time 0.0000 (0.0000) loss 3.5216 (3.8700) grad_norm 1.3411 (1.2710) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][50/625] eta 0:03:44 lr 0.001997 wd 0.0500 time 0.3731 (0.3912) data time 0.0011 (0.0090) model time 0.0000 (0.0000) loss 4.4923 (3.8693) grad_norm 2.0830 (1.2846) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][60/625] eta 0:03:40 lr 0.001997 wd 0.0500 time 0.3841 (0.3895) data time 0.0010 (0.0078) model time 0.3831 (0.3791) loss 3.3747 (3.8455) grad_norm 1.6559 (1.2882) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][70/625] eta 0:03:35 lr 0.001997 wd 0.0500 time 0.3985 (0.3883) data time 0.0008 (0.0069) model time 0.3976 (0.3796) loss 2.9203 (3.8544) grad_norm 1.4779 (1.2811) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:42:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][80/625] eta 0:03:31 lr 0.001997 wd 0.0500 time 0.3824 (0.3873) data time 0.0010 (0.0062) model time 0.3814 (0.3792) loss 3.5097 (3.8507) grad_norm 1.2693 (1.2814) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][90/625] eta 0:03:26 lr 0.001997 wd 0.0500 time 0.3850 (0.3863) data time 0.0010 (0.0057) model time 0.3840 (0.3787) loss 3.7043 (3.8544) grad_norm 2.0132 (1.2914) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][100/625] eta 0:03:24 lr 0.001997 wd 0.0500 time 0.3860 (0.3890) data time 0.0009 (0.0052) model time 0.3852 (0.3853) loss 4.4807 (3.8648) grad_norm 1.4696 (1.2982) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][110/625] eta 0:03:20 lr 0.001997 wd 0.0500 time 0.3805 (0.3896) data time 0.0011 (0.0049) model time 0.3795 (0.3869) loss 4.3414 (3.8530) grad_norm 1.2532 (1.3106) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][120/625] eta 0:03:16 lr 0.001997 wd 0.0500 time 0.3705 (0.3883) data time 0.0008 (0.0045) model time 0.3697 (0.3849) loss 3.1407 (3.8772) grad_norm 1.5169 (1.3092) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][130/625] eta 0:03:11 lr 0.001997 wd 0.0500 time 0.3782 (0.3878) data time 0.0009 (0.0044) model time 0.3772 (0.3841) loss 2.9141 (3.8606) grad_norm 1.2026 (1.3134) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][140/625] eta 0:03:07 lr 0.001997 wd 0.0500 time 0.3735 (0.3870) data time 0.0011 (0.0042) model time 0.3723 (0.3833) loss 3.7376 (3.8469) grad_norm 1.0735 (1.3113) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][150/625] eta 0:03:04 lr 0.001997 wd 0.0500 time 0.3825 (0.3878) data time 0.0010 (0.0040) model time 0.3815 (0.3847) loss 3.9193 (3.8568) grad_norm 1.4921 (1.3055) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][160/625] eta 0:03:00 lr 0.001997 wd 0.0500 time 0.3774 (0.3883) data time 0.0009 (0.0038) model time 0.3765 (0.3856) loss 3.4823 (3.8638) grad_norm 0.9431 (1.3006) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][170/625] eta 0:02:56 lr 0.001997 wd 0.0500 time 0.3728 (0.3876) data time 0.0008 (0.0036) model time 0.3720 (0.3847) loss 4.5495 (3.8651) grad_norm 1.3131 (1.2905) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:43:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][180/625] eta 0:02:52 lr 0.001997 wd 0.0500 time 0.3855 (0.3871) data time 0.0011 (0.0035) model time 0.3845 (0.3842) loss 3.8678 (3.8682) grad_norm 1.1183 (1.2849) loss_scale 32768.0000 (17198.6740) mem 11225MB [2024-08-26 04:43:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][190/625] eta 0:02:48 lr 0.001997 wd 0.0500 time 0.3771 (0.3879) data time 0.0008 (0.0033) model time 0.3763 (0.3855) loss 3.3765 (3.8636) grad_norm 1.0952 (1.2863) loss_scale 32768.0000 (18013.8220) mem 11225MB [2024-08-26 04:43:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][200/625] eta 0:02:44 lr 0.001997 wd 0.0500 time 0.3713 (0.3874) data time 0.0011 (0.0032) model time 0.3702 (0.3848) loss 4.0814 (3.8772) grad_norm 0.9646 (1.2798) loss_scale 32768.0000 (18747.8607) mem 11225MB [2024-08-26 04:43:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][210/625] eta 0:02:40 lr 0.001997 wd 0.0500 time 0.3735 (0.3869) data time 0.0008 (0.0031) model time 0.3727 (0.3842) loss 4.0142 (3.8800) grad_norm 1.4288 (1.2816) loss_scale 32768.0000 (19412.3223) mem 11225MB [2024-08-26 04:43:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][220/625] eta 0:02:36 lr 0.001997 wd 0.0500 time 0.3707 (0.3865) data time 0.0014 (0.0031) model time 0.3692 (0.3838) loss 4.0385 (3.8819) grad_norm 1.3360 (1.2760) loss_scale 32768.0000 (20016.6516) mem 11225MB [2024-08-26 04:43:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][230/625] eta 0:02:32 lr 0.001997 wd 0.0500 time 0.3794 (0.3861) data time 0.0008 (0.0030) model time 0.3785 (0.3834) loss 4.6162 (3.8834) grad_norm 1.2908 (1.2863) loss_scale 32768.0000 (20568.6580) mem 11225MB [2024-08-26 04:44:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][240/625] eta 0:02:28 lr 0.001997 wd 0.0500 time 0.3802 (0.3858) data time 0.0010 (0.0029) model time 0.3792 (0.3830) loss 3.9893 (3.8941) grad_norm 1.0429 (1.2950) loss_scale 32768.0000 (21074.8548) mem 11225MB [2024-08-26 04:44:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][250/625] eta 0:02:24 lr 0.001997 wd 0.0500 time 0.3790 (0.3856) data time 0.0012 (0.0029) model time 0.3778 (0.3829) loss 3.7274 (3.8819) grad_norm 1.4737 (1.2879) loss_scale 32768.0000 (21540.7171) mem 11225MB [2024-08-26 04:44:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][260/625] eta 0:02:20 lr 0.001997 wd 0.0500 time 0.3679 (0.3853) data time 0.0008 (0.0028) model time 0.3671 (0.3826) loss 4.2409 (3.8862) grad_norm 2.1110 (1.2855) loss_scale 32768.0000 (21970.8812) mem 11225MB [2024-08-26 04:44:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][270/625] eta 0:02:16 lr 0.001997 wd 0.0500 time 0.3739 (0.3850) data time 0.0008 (0.0027) model time 0.3731 (0.3823) loss 4.8620 (3.8926) grad_norm 1.2138 (1.2878) loss_scale 32768.0000 (22369.2989) mem 11225MB [2024-08-26 04:44:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][280/625] eta 0:02:12 lr 0.001997 wd 0.0500 time 0.3770 (0.3848) data time 0.0011 (0.0027) model time 0.3759 (0.3821) loss 4.2205 (3.9067) grad_norm 1.0924 (1.2878) loss_scale 32768.0000 (22739.3594) mem 11225MB [2024-08-26 04:44:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][290/625] eta 0:02:08 lr 0.001997 wd 0.0500 time 0.3745 (0.3845) data time 0.0008 (0.0026) model time 0.3737 (0.3818) loss 2.2927 (3.8861) grad_norm 1.1944 (1.2836) loss_scale 32768.0000 (23083.9863) mem 11225MB [2024-08-26 04:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][300/625] eta 0:02:04 lr 0.001997 wd 0.0500 time 0.3738 (0.3843) data time 0.0008 (0.0026) model time 0.3730 (0.3816) loss 4.3468 (3.8852) grad_norm 1.3254 (1.2846) loss_scale 32768.0000 (23405.7143) mem 11225MB [2024-08-26 04:44:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][310/625] eta 0:02:01 lr 0.001997 wd 0.0500 time 0.5751 (0.3867) data time 0.0010 (0.0025) model time 0.5741 (0.3845) loss 3.0204 (3.8831) grad_norm 1.3935 (1.2811) loss_scale 32768.0000 (23706.7524) mem 11225MB [2024-08-26 04:44:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][320/625] eta 0:01:57 lr 0.001996 wd 0.0500 time 0.3777 (0.3864) data time 0.0010 (0.0025) model time 0.3767 (0.3843) loss 3.2110 (3.8806) grad_norm 1.3067 (1.2821) loss_scale 32768.0000 (23989.0343) mem 11225MB [2024-08-26 04:44:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][330/625] eta 0:01:53 lr 0.001996 wd 0.0500 time 0.3746 (0.3862) data time 0.0011 (0.0025) model time 0.3735 (0.3840) loss 4.0719 (3.8820) grad_norm 1.6585 (1.2913) loss_scale 32768.0000 (24254.2598) mem 11225MB [2024-08-26 04:44:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][340/625] eta 0:01:50 lr 0.001996 wd 0.0500 time 0.3676 (0.3871) data time 0.0008 (0.0024) model time 0.3668 (0.3851) loss 3.3113 (3.8792) grad_norm 1.2846 (1.2943) loss_scale 32768.0000 (24503.9296) mem 11225MB [2024-08-26 04:44:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][350/625] eta 0:01:46 lr 0.001996 wd 0.0500 time 0.3563 (0.3868) data time 0.0008 (0.0024) model time 0.3555 (0.3848) loss 3.7073 (3.8831) grad_norm 1.3291 (1.2929) loss_scale 32768.0000 (24739.3732) mem 11225MB [2024-08-26 04:44:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][360/625] eta 0:01:42 lr 0.001996 wd 0.0500 time 0.3775 (0.3865) data time 0.0008 (0.0023) model time 0.3766 (0.3845) loss 4.7472 (3.8857) grad_norm 1.3153 (1.2914) loss_scale 32768.0000 (24961.7729) mem 11225MB [2024-08-26 04:44:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][370/625] eta 0:01:38 lr 0.001996 wd 0.0500 time 0.3813 (0.3864) data time 0.0011 (0.0023) model time 0.3802 (0.3844) loss 2.9880 (3.8796) grad_norm 1.0412 (1.2930) loss_scale 32768.0000 (25172.1833) mem 11225MB [2024-08-26 04:44:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][380/625] eta 0:01:34 lr 0.001996 wd 0.0500 time 0.3757 (0.3862) data time 0.0010 (0.0023) model time 0.3746 (0.3842) loss 3.7101 (3.8811) grad_norm 1.0672 (1.2978) loss_scale 32768.0000 (25371.5486) mem 11225MB [2024-08-26 04:44:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][390/625] eta 0:01:30 lr 0.001996 wd 0.0500 time 0.3767 (0.3861) data time 0.0010 (0.0023) model time 0.3756 (0.3841) loss 3.3684 (3.8855) grad_norm 1.0381 (1.2930) loss_scale 32768.0000 (25560.7161) mem 11225MB [2024-08-26 04:45:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][400/625] eta 0:01:26 lr 0.001996 wd 0.0500 time 0.3802 (0.3859) data time 0.0010 (0.0022) model time 0.3792 (0.3839) loss 4.1715 (3.8831) grad_norm 1.2355 (1.2914) loss_scale 32768.0000 (25740.4489) mem 11225MB [2024-08-26 04:45:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][410/625] eta 0:01:22 lr 0.001996 wd 0.0500 time 0.3650 (0.3858) data time 0.0009 (0.0022) model time 0.3642 (0.3838) loss 3.2264 (3.8813) grad_norm 1.2006 (1.2900) loss_scale 32768.0000 (25911.4355) mem 11225MB [2024-08-26 04:45:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][420/625] eta 0:01:19 lr 0.001996 wd 0.0500 time 0.3742 (0.3856) data time 0.0010 (0.0022) model time 0.3732 (0.3836) loss 4.1052 (3.8915) grad_norm 1.3667 (1.2906) loss_scale 32768.0000 (26074.2993) mem 11225MB [2024-08-26 04:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][430/625] eta 0:01:15 lr 0.001996 wd 0.0500 time 0.3763 (0.3854) data time 0.0008 (0.0021) model time 0.3755 (0.3834) loss 4.3968 (3.8870) grad_norm 1.2722 (1.2919) loss_scale 32768.0000 (26229.6056) mem 11225MB [2024-08-26 04:45:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][440/625] eta 0:01:11 lr 0.001996 wd 0.0500 time 0.3855 (0.3853) data time 0.0011 (0.0021) model time 0.3844 (0.3833) loss 3.3381 (3.8903) grad_norm 1.1714 (1.2953) loss_scale 32768.0000 (26377.8685) mem 11225MB [2024-08-26 04:45:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][450/625] eta 0:01:07 lr 0.001996 wd 0.0500 time 0.3752 (0.3852) data time 0.0011 (0.0021) model time 0.3741 (0.3832) loss 3.9972 (3.8906) grad_norm 1.4335 (1.2943) loss_scale 32768.0000 (26519.5565) mem 11225MB [2024-08-26 04:45:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][460/625] eta 0:01:03 lr 0.001996 wd 0.0500 time 0.3738 (0.3859) data time 0.0011 (0.0021) model time 0.3727 (0.3840) loss 4.2932 (3.8905) grad_norm 1.8180 (1.3011) loss_scale 32768.0000 (26655.0976) mem 11225MB [2024-08-26 04:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][470/625] eta 0:00:59 lr 0.001996 wd 0.0500 time 0.3795 (0.3858) data time 0.0008 (0.0021) model time 0.3787 (0.3839) loss 3.6683 (3.8920) grad_norm 1.5182 (1.3025) loss_scale 32768.0000 (26784.8832) mem 11225MB [2024-08-26 04:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][480/625] eta 0:00:55 lr 0.001996 wd 0.0500 time 0.3875 (0.3856) data time 0.0008 (0.0021) model time 0.3866 (0.3837) loss 4.7664 (3.8969) grad_norm 1.0241 (1.3028) loss_scale 32768.0000 (26909.2723) mem 11225MB [2024-08-26 04:45:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][490/625] eta 0:00:52 lr 0.001996 wd 0.0500 time 0.3737 (0.3862) data time 0.0011 (0.0020) model time 0.3726 (0.3844) loss 3.5691 (3.8909) grad_norm 1.8347 (1.2996) loss_scale 32768.0000 (27028.5947) mem 11225MB [2024-08-26 04:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][500/625] eta 0:00:48 lr 0.001996 wd 0.0500 time 0.3830 (0.3864) data time 0.0011 (0.0020) model time 0.3819 (0.3847) loss 2.7156 (3.8884) grad_norm 0.9564 (1.2999) loss_scale 32768.0000 (27143.1537) mem 11225MB [2024-08-26 04:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][510/625] eta 0:00:44 lr 0.001996 wd 0.0500 time 0.3760 (0.3862) data time 0.0008 (0.0020) model time 0.3751 (0.3845) loss 3.6790 (3.8900) grad_norm 1.2920 (1.3022) loss_scale 32768.0000 (27253.2290) mem 11225MB [2024-08-26 04:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][520/625] eta 0:00:40 lr 0.001996 wd 0.0500 time 0.3846 (0.3864) data time 0.0008 (0.0020) model time 0.3837 (0.3847) loss 4.7236 (3.8937) grad_norm 1.5573 (1.3101) loss_scale 32768.0000 (27359.0787) mem 11225MB [2024-08-26 04:45:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][530/625] eta 0:00:36 lr 0.001996 wd 0.0500 time 0.3709 (0.3863) data time 0.0010 (0.0020) model time 0.3698 (0.3845) loss 4.0269 (3.8910) grad_norm 0.8959 (1.3081) loss_scale 32768.0000 (27460.9416) mem 11225MB [2024-08-26 04:45:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][540/625] eta 0:00:32 lr 0.001996 wd 0.0500 time 0.3836 (0.3862) data time 0.0011 (0.0020) model time 0.3826 (0.3844) loss 3.4394 (3.8839) grad_norm 1.3533 (1.3043) loss_scale 32768.0000 (27559.0388) mem 11225MB [2024-08-26 04:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][550/625] eta 0:00:28 lr 0.001996 wd 0.0500 time 0.3895 (0.3861) data time 0.0009 (0.0020) model time 0.3887 (0.3843) loss 3.5905 (3.8839) grad_norm 1.3561 (1.3025) loss_scale 32768.0000 (27653.5753) mem 11225MB [2024-08-26 04:46:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][560/625] eta 0:00:25 lr 0.001996 wd 0.0500 time 0.3783 (0.3859) data time 0.0011 (0.0020) model time 0.3773 (0.3841) loss 3.5121 (3.8864) grad_norm 1.8030 (1.3041) loss_scale 32768.0000 (27744.7415) mem 11225MB [2024-08-26 04:46:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][570/625] eta 0:00:21 lr 0.001996 wd 0.0500 time 0.3959 (0.3858) data time 0.0008 (0.0020) model time 0.3951 (0.3840) loss 4.9257 (3.8833) grad_norm 1.6942 (1.3069) loss_scale 32768.0000 (27832.7145) mem 11225MB [2024-08-26 04:46:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][580/625] eta 0:00:17 lr 0.001996 wd 0.0500 time 0.3796 (0.3857) data time 0.0008 (0.0019) model time 0.3788 (0.3839) loss 3.3525 (3.8816) grad_norm 1.5185 (1.3086) loss_scale 32768.0000 (27917.6592) mem 11225MB [2024-08-26 04:46:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][590/625] eta 0:00:13 lr 0.001996 wd 0.0500 time 0.3682 (0.3856) data time 0.0011 (0.0019) model time 0.3671 (0.3838) loss 3.5884 (3.8851) grad_norm 1.4256 (1.3077) loss_scale 32768.0000 (27999.7293) mem 11225MB [2024-08-26 04:46:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][600/625] eta 0:00:09 lr 0.001996 wd 0.0500 time 0.3805 (0.3855) data time 0.0008 (0.0019) model time 0.3797 (0.3838) loss 3.9395 (3.8819) grad_norm 1.3908 (1.3057) loss_scale 32768.0000 (28079.0682) mem 11225MB [2024-08-26 04:46:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][610/625] eta 0:00:05 lr 0.001996 wd 0.0500 time 0.3643 (0.3860) data time 0.0006 (0.0019) model time 0.3637 (0.3843) loss 3.1063 (3.8799) grad_norm 0.8406 (1.3051) loss_scale 32768.0000 (28155.8101) mem 11225MB [2024-08-26 04:46:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [27/300][620/625] eta 0:00:01 lr 0.001996 wd 0.0500 time 0.3628 (0.3861) data time 0.0005 (0.0019) model time 0.3622 (0.3844) loss 3.0772 (3.8739) grad_norm 1.2049 (1.3024) loss_scale 32768.0000 (28230.0805) mem 11225MB [2024-08-26 04:46:28 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 27 training takes 0:04:01 [2024-08-26 04:46:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:46:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.484 (0.484) Loss 0.8027 (0.8027) Acc@1 82.666 (82.666) Acc@5 96.582 (96.582) Mem 11225MB [2024-08-26 04:46:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.140) Loss 1.4932 (1.0422) Acc@1 65.576 (75.763) Acc@5 87.549 (93.990) Mem 11225MB [2024-08-26 04:46:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 1.5791 (1.2792) Acc@1 63.721 (70.582) Acc@5 87.109 (90.590) Mem 11225MB [2024-08-26 04:46:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 70.587 Acc@5 90.583 [2024-08-26 04:46:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 70.6% [2024-08-26 04:46:32 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 70.59% [2024-08-26 04:46:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:46:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:46:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 0.8145 (0.8145) Acc@1 78.271 (78.271) Acc@5 94.336 (94.336) Mem 11225MB [2024-08-26 04:46:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 1.4600 (1.0163) Acc@1 62.842 (73.233) Acc@5 86.914 (92.676) Mem 11225MB [2024-08-26 04:46:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.6553 (1.2692) Acc@1 59.863 (68.276) Acc@5 83.789 (89.137) Mem 11225MB [2024-08-26 04:46:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 68.242 Acc@5 89.099 [2024-08-26 04:46:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 68.2% [2024-08-26 04:46:36 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 68.24% [2024-08-26 04:46:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:46:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:46:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][0/625] eta 0:07:39 lr 0.001996 wd 0.0500 time 0.7357 (0.7357) data time 0.3958 (0.3958) model time 0.0000 (0.0000) loss 2.7459 (2.7459) grad_norm 1.6531 (1.6531) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][10/625] eta 0:04:11 lr 0.001996 wd 0.0500 time 0.3725 (0.4084) data time 0.0011 (0.0371) model time 0.0000 (0.0000) loss 4.0824 (3.7996) grad_norm 1.1427 (1.4029) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:46:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][20/625] eta 0:04:04 lr 0.001996 wd 0.0500 time 0.3568 (0.4044) data time 0.0010 (0.0200) model time 0.0000 (0.0000) loss 2.8546 (3.8880) grad_norm 1.0211 (1.3374) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:46:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][30/625] eta 0:03:59 lr 0.001996 wd 0.0500 time 0.3760 (0.4031) data time 0.0012 (0.0139) model time 0.0000 (0.0000) loss 3.4592 (3.8887) grad_norm 1.3161 (1.3010) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:46:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][40/625] eta 0:03:52 lr 0.001996 wd 0.0500 time 0.3800 (0.3972) data time 0.0008 (0.0108) model time 0.0000 (0.0000) loss 2.9194 (3.9732) grad_norm 0.9824 (1.3331) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][50/625] eta 0:03:46 lr 0.001996 wd 0.0500 time 0.3707 (0.3937) data time 0.0008 (0.0089) model time 0.0000 (0.0000) loss 4.6402 (3.9111) grad_norm 0.9856 (1.2780) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][60/625] eta 0:03:40 lr 0.001996 wd 0.0500 time 0.3727 (0.3911) data time 0.0009 (0.0076) model time 0.3718 (0.3766) loss 3.6880 (3.8788) grad_norm 1.4893 (1.2923) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][70/625] eta 0:03:36 lr 0.001996 wd 0.0500 time 0.3845 (0.3895) data time 0.0011 (0.0067) model time 0.3834 (0.3778) loss 2.7576 (3.8698) grad_norm 1.2496 (1.2963) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][80/625] eta 0:03:31 lr 0.001996 wd 0.0500 time 0.3811 (0.3879) data time 0.0007 (0.0060) model time 0.3804 (0.3769) loss 2.7080 (3.8810) grad_norm 0.9808 (1.2869) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][90/625] eta 0:03:26 lr 0.001996 wd 0.0500 time 0.3715 (0.3867) data time 0.0011 (0.0055) model time 0.3704 (0.3767) loss 4.0455 (3.8812) grad_norm 1.0088 (1.2864) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][100/625] eta 0:03:22 lr 0.001996 wd 0.0500 time 0.3719 (0.3857) data time 0.0011 (0.0051) model time 0.3708 (0.3765) loss 3.9197 (3.8603) grad_norm 1.1743 (1.2932) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][110/625] eta 0:03:18 lr 0.001996 wd 0.0500 time 0.3694 (0.3849) data time 0.0012 (0.0047) model time 0.3682 (0.3762) loss 3.9908 (3.8556) grad_norm 2.0010 (1.3034) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][120/625] eta 0:03:13 lr 0.001996 wd 0.0500 time 0.3765 (0.3841) data time 0.0007 (0.0044) model time 0.3758 (0.3760) loss 3.8419 (3.8595) grad_norm 1.5681 (1.3011) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][130/625] eta 0:03:10 lr 0.001996 wd 0.0500 time 0.3754 (0.3849) data time 0.0010 (0.0041) model time 0.3743 (0.3781) loss 4.2892 (3.8633) grad_norm 1.0512 (1.2899) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][140/625] eta 0:03:06 lr 0.001996 wd 0.0500 time 0.3785 (0.3843) data time 0.0011 (0.0039) model time 0.3773 (0.3779) loss 4.1196 (3.8519) grad_norm 1.1699 (1.2831) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][150/625] eta 0:03:02 lr 0.001996 wd 0.0500 time 0.3831 (0.3839) data time 0.0008 (0.0038) model time 0.3822 (0.3778) loss 4.5726 (3.8617) grad_norm 1.2577 (1.2863) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][160/625] eta 0:02:58 lr 0.001996 wd 0.0500 time 0.3862 (0.3836) data time 0.0010 (0.0036) model time 0.3852 (0.3778) loss 4.0014 (3.8589) grad_norm 0.9474 (1.2840) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][170/625] eta 0:02:55 lr 0.001996 wd 0.0500 time 0.3781 (0.3865) data time 0.0014 (0.0035) model time 0.3768 (0.3822) loss 3.0152 (3.8613) grad_norm 1.0636 (1.2769) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][180/625] eta 0:02:52 lr 0.001996 wd 0.0500 time 0.3891 (0.3870) data time 0.0008 (0.0033) model time 0.3882 (0.3832) loss 4.3308 (3.8469) grad_norm 2.0172 (1.2740) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][190/625] eta 0:02:48 lr 0.001996 wd 0.0500 time 0.3743 (0.3865) data time 0.0008 (0.0032) model time 0.3735 (0.3828) loss 4.0211 (3.8440) grad_norm 1.1720 (1.2694) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][200/625] eta 0:02:44 lr 0.001996 wd 0.0500 time 0.3796 (0.3861) data time 0.0011 (0.0031) model time 0.3785 (0.3824) loss 4.2108 (3.8443) grad_norm 1.0877 (1.2732) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][210/625] eta 0:02:40 lr 0.001996 wd 0.0500 time 0.3824 (0.3859) data time 0.0010 (0.0030) model time 0.3814 (0.3823) loss 4.3170 (3.8592) grad_norm 0.9893 (1.2720) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][220/625] eta 0:02:36 lr 0.001996 wd 0.0500 time 0.3788 (0.3855) data time 0.0010 (0.0029) model time 0.3778 (0.3819) loss 4.5323 (3.8668) grad_norm 1.3718 (1.2674) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][230/625] eta 0:02:32 lr 0.001996 wd 0.0500 time 0.5907 (0.3861) data time 0.0009 (0.0029) model time 0.5898 (0.3828) loss 4.5912 (3.8750) grad_norm 1.1142 (1.2652) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][240/625] eta 0:02:28 lr 0.001996 wd 0.0500 time 0.3772 (0.3867) data time 0.0010 (0.0028) model time 0.3762 (0.3837) loss 3.6218 (3.8741) grad_norm 0.8621 (1.2683) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][250/625] eta 0:02:24 lr 0.001996 wd 0.0500 time 0.3818 (0.3864) data time 0.0011 (0.0027) model time 0.3806 (0.3833) loss 3.3904 (3.8770) grad_norm 1.1956 (1.2584) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][260/625] eta 0:02:20 lr 0.001996 wd 0.0500 time 0.3714 (0.3860) data time 0.0008 (0.0027) model time 0.3706 (0.3830) loss 3.3135 (3.8672) grad_norm 2.6241 (1.2624) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][270/625] eta 0:02:16 lr 0.001996 wd 0.0500 time 0.3792 (0.3858) data time 0.0008 (0.0026) model time 0.3784 (0.3829) loss 2.9364 (3.8648) grad_norm 1.1028 (1.2694) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][280/625] eta 0:02:13 lr 0.001996 wd 0.0500 time 0.3738 (0.3856) data time 0.0009 (0.0026) model time 0.3729 (0.3827) loss 4.4728 (3.8642) grad_norm 0.9502 (1.2673) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][290/625] eta 0:02:09 lr 0.001996 wd 0.0500 time 0.3811 (0.3867) data time 0.0010 (0.0025) model time 0.3801 (0.3841) loss 3.1769 (3.8595) grad_norm 1.3019 (1.2645) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][300/625] eta 0:02:05 lr 0.001996 wd 0.0500 time 0.3839 (0.3871) data time 0.0009 (0.0025) model time 0.3830 (0.3846) loss 2.9419 (3.8526) grad_norm 1.6055 (1.2704) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][310/625] eta 0:02:01 lr 0.001996 wd 0.0500 time 0.3827 (0.3868) data time 0.0009 (0.0024) model time 0.3818 (0.3843) loss 2.6903 (3.8554) grad_norm 1.0165 (1.2634) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 04:48:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][320/625] eta 0:01:57 lr 0.001995 wd 0.0500 time 0.3784 (0.3865) data time 0.0008 (0.0024) model time 0.3776 (0.3840) loss 4.4020 (3.8600) grad_norm 1.2634 (inf) loss_scale 16384.0000 (32257.5950) mem 11225MB [2024-08-26 04:48:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][330/625] eta 0:01:53 lr 0.001995 wd 0.0500 time 0.3867 (0.3862) data time 0.0011 (0.0023) model time 0.3857 (0.3837) loss 3.2680 (3.8616) grad_norm 0.9947 (inf) loss_scale 16384.0000 (31778.0302) mem 11225MB [2024-08-26 04:48:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][340/625] eta 0:01:50 lr 0.001995 wd 0.0500 time 0.3784 (0.3864) data time 0.0009 (0.0023) model time 0.3775 (0.3840) loss 4.4739 (3.8685) grad_norm 0.8915 (inf) loss_scale 16384.0000 (31326.5924) mem 11225MB [2024-08-26 04:48:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][350/625] eta 0:01:46 lr 0.001995 wd 0.0500 time 0.3733 (0.3861) data time 0.0011 (0.0023) model time 0.3722 (0.3837) loss 4.1154 (3.8724) grad_norm 1.2752 (inf) loss_scale 16384.0000 (30900.8775) mem 11225MB [2024-08-26 04:48:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][360/625] eta 0:01:42 lr 0.001995 wd 0.0500 time 0.3729 (0.3859) data time 0.0009 (0.0022) model time 0.3721 (0.3835) loss 3.8439 (3.8652) grad_norm 0.9959 (inf) loss_scale 16384.0000 (30498.7479) mem 11225MB [2024-08-26 04:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][370/625] eta 0:01:38 lr 0.001995 wd 0.0500 time 0.3750 (0.3857) data time 0.0010 (0.0022) model time 0.3740 (0.3833) loss 3.3774 (3.8626) grad_norm 1.6077 (inf) loss_scale 16384.0000 (30118.2965) mem 11225MB [2024-08-26 04:49:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][380/625] eta 0:01:34 lr 0.001995 wd 0.0500 time 0.3782 (0.3856) data time 0.0008 (0.0022) model time 0.3773 (0.3832) loss 2.9469 (3.8583) grad_norm 1.0229 (inf) loss_scale 16384.0000 (29757.8163) mem 11225MB [2024-08-26 04:49:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][390/625] eta 0:01:30 lr 0.001995 wd 0.0500 time 0.3821 (0.3854) data time 0.0010 (0.0022) model time 0.3811 (0.3830) loss 4.3195 (3.8670) grad_norm 1.2858 (inf) loss_scale 16384.0000 (29415.7749) mem 11225MB [2024-08-26 04:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][400/625] eta 0:01:26 lr 0.001995 wd 0.0500 time 0.3724 (0.3852) data time 0.0009 (0.0021) model time 0.3716 (0.3828) loss 4.3079 (3.8738) grad_norm 1.4456 (inf) loss_scale 16384.0000 (29090.7930) mem 11225MB [2024-08-26 04:49:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][410/625] eta 0:01:22 lr 0.001995 wd 0.0500 time 0.3833 (0.3850) data time 0.0008 (0.0021) model time 0.3825 (0.3827) loss 3.3426 (3.8760) grad_norm 1.2570 (inf) loss_scale 16384.0000 (28781.6253) mem 11225MB [2024-08-26 04:49:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][420/625] eta 0:01:19 lr 0.001995 wd 0.0500 time 0.3814 (0.3859) data time 0.0008 (0.0021) model time 0.3806 (0.3836) loss 4.7940 (3.8716) grad_norm 1.3182 (inf) loss_scale 16384.0000 (28487.1449) mem 11225MB [2024-08-26 04:49:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][430/625] eta 0:01:15 lr 0.001995 wd 0.0500 time 0.3749 (0.3857) data time 0.0011 (0.0021) model time 0.3737 (0.3834) loss 3.2957 (3.8650) grad_norm 1.2038 (inf) loss_scale 16384.0000 (28206.3295) mem 11225MB [2024-08-26 04:49:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][440/625] eta 0:01:11 lr 0.001995 wd 0.0500 time 0.3717 (0.3855) data time 0.0011 (0.0020) model time 0.3706 (0.3833) loss 4.2802 (3.8713) grad_norm 1.1094 (inf) loss_scale 16384.0000 (27938.2494) mem 11225MB [2024-08-26 04:49:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][450/625] eta 0:01:07 lr 0.001995 wd 0.0500 time 0.3740 (0.3853) data time 0.0011 (0.0020) model time 0.3729 (0.3831) loss 3.8400 (3.8750) grad_norm 1.3319 (inf) loss_scale 16384.0000 (27682.0576) mem 11225MB [2024-08-26 04:49:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][460/625] eta 0:01:03 lr 0.001995 wd 0.0500 time 0.3915 (0.3853) data time 0.0011 (0.0020) model time 0.3905 (0.3831) loss 3.4513 (3.8777) grad_norm 1.4048 (inf) loss_scale 16384.0000 (27436.9805) mem 11225MB [2024-08-26 04:49:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][470/625] eta 0:00:59 lr 0.001995 wd 0.0500 time 0.3738 (0.3851) data time 0.0011 (0.0020) model time 0.3727 (0.3829) loss 4.0228 (3.8784) grad_norm 1.1175 (inf) loss_scale 16384.0000 (27202.3100) mem 11225MB [2024-08-26 04:49:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][480/625] eta 0:00:55 lr 0.001995 wd 0.0500 time 0.3802 (0.3850) data time 0.0008 (0.0020) model time 0.3794 (0.3828) loss 3.6930 (3.8791) grad_norm 0.8640 (inf) loss_scale 16384.0000 (26977.3971) mem 11225MB [2024-08-26 04:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][490/625] eta 0:00:51 lr 0.001995 wd 0.0500 time 0.3724 (0.3849) data time 0.0010 (0.0019) model time 0.3714 (0.3827) loss 2.7721 (3.8816) grad_norm 1.7203 (inf) loss_scale 16384.0000 (26761.6456) mem 11225MB [2024-08-26 04:49:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][500/625] eta 0:00:48 lr 0.001995 wd 0.0500 time 0.5907 (0.3860) data time 0.0009 (0.0019) model time 0.5898 (0.3840) loss 3.5028 (3.8858) grad_norm 0.8744 (inf) loss_scale 16384.0000 (26554.5070) mem 11225MB [2024-08-26 04:49:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][510/625] eta 0:00:44 lr 0.001995 wd 0.0500 time 0.3806 (0.3859) data time 0.0011 (0.0019) model time 0.3795 (0.3839) loss 4.0542 (3.8863) grad_norm 1.1767 (inf) loss_scale 16384.0000 (26355.4755) mem 11225MB [2024-08-26 04:49:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][520/625] eta 0:00:40 lr 0.001995 wd 0.0500 time 0.3813 (0.3862) data time 0.0009 (0.0019) model time 0.3804 (0.3842) loss 4.5303 (3.8865) grad_norm 1.3075 (inf) loss_scale 16384.0000 (26164.0845) mem 11225MB [2024-08-26 04:50:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][530/625] eta 0:00:36 lr 0.001995 wd 0.0500 time 0.3713 (0.3860) data time 0.0009 (0.0019) model time 0.3704 (0.3840) loss 3.3380 (3.8818) grad_norm 0.8388 (inf) loss_scale 16384.0000 (25979.9021) mem 11225MB [2024-08-26 04:50:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][540/625] eta 0:00:32 lr 0.001995 wd 0.0500 time 0.3802 (0.3866) data time 0.0009 (0.0019) model time 0.3793 (0.3847) loss 4.4163 (3.8893) grad_norm 0.9331 (inf) loss_scale 16384.0000 (25802.5287) mem 11225MB [2024-08-26 04:50:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][550/625] eta 0:00:28 lr 0.001995 wd 0.0500 time 0.3752 (0.3864) data time 0.0009 (0.0019) model time 0.3743 (0.3845) loss 4.3457 (3.8904) grad_norm 1.0809 (inf) loss_scale 16384.0000 (25631.5935) mem 11225MB [2024-08-26 04:50:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][560/625] eta 0:00:25 lr 0.001995 wd 0.0500 time 0.3796 (0.3863) data time 0.0011 (0.0019) model time 0.3784 (0.3844) loss 4.2634 (3.8921) grad_norm 0.9519 (inf) loss_scale 16384.0000 (25466.7522) mem 11225MB [2024-08-26 04:50:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][570/625] eta 0:00:21 lr 0.001995 wd 0.0500 time 0.3746 (0.3861) data time 0.0009 (0.0018) model time 0.3737 (0.3842) loss 2.9785 (3.8955) grad_norm 1.8437 (inf) loss_scale 16384.0000 (25307.6848) mem 11225MB [2024-08-26 04:50:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][580/625] eta 0:00:17 lr 0.001995 wd 0.0500 time 0.3748 (0.3859) data time 0.0009 (0.0018) model time 0.3740 (0.3840) loss 2.7327 (3.8932) grad_norm 1.3998 (inf) loss_scale 16384.0000 (25154.0929) mem 11225MB [2024-08-26 04:50:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][590/625] eta 0:00:13 lr 0.001995 wd 0.0500 time 0.3781 (0.3858) data time 0.0012 (0.0018) model time 0.3769 (0.3839) loss 4.1468 (3.8952) grad_norm 1.2491 (inf) loss_scale 16384.0000 (25005.6988) mem 11225MB [2024-08-26 04:50:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][600/625] eta 0:00:09 lr 0.001995 wd 0.0500 time 0.3667 (0.3856) data time 0.0009 (0.0018) model time 0.3658 (0.3837) loss 4.3699 (3.8953) grad_norm 1.0076 (inf) loss_scale 16384.0000 (24862.2429) mem 11225MB [2024-08-26 04:50:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][610/625] eta 0:00:05 lr 0.001995 wd 0.0500 time 0.3713 (0.3854) data time 0.0006 (0.0018) model time 0.3707 (0.3835) loss 2.7519 (3.8892) grad_norm 1.8995 (inf) loss_scale 16384.0000 (24723.4828) mem 11225MB [2024-08-26 04:50:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [28/300][620/625] eta 0:00:01 lr 0.001995 wd 0.0500 time 0.3647 (0.3851) data time 0.0008 (0.0018) model time 0.3639 (0.3832) loss 3.7195 (3.8886) grad_norm 1.2271 (inf) loss_scale 16384.0000 (24589.1916) mem 11225MB [2024-08-26 04:50:38 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 28 training takes 0:04:00 [2024-08-26 04:50:38 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:50:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:50:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.8921 (0.8921) Acc@1 80.908 (80.908) Acc@5 94.971 (94.971) Mem 11225MB [2024-08-26 04:50:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 1.5000 (1.0862) Acc@1 65.918 (75.590) Acc@5 88.086 (94.003) Mem 11225MB [2024-08-26 04:50:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.6357 (1.3078) Acc@1 64.648 (70.905) Acc@5 86.084 (90.755) Mem 11225MB [2024-08-26 04:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 70.763 Acc@5 90.693 [2024-08-26 04:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 70.8% [2024-08-26 04:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 70.76% [2024-08-26 04:50:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:50:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:50:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.529 (0.529) Loss 0.7803 (0.7803) Acc@1 79.541 (79.541) Acc@5 94.678 (94.678) Mem 11225MB [2024-08-26 04:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.145) Loss 1.4062 (0.9771) Acc@1 63.721 (74.139) Acc@5 87.891 (93.173) Mem 11225MB [2024-08-26 04:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.6045 (1.2247) Acc@1 61.328 (69.280) Acc@5 84.961 (89.762) Mem 11225MB [2024-08-26 04:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 69.220 Acc@5 89.717 [2024-08-26 04:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 69.2% [2024-08-26 04:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 69.22% [2024-08-26 04:50:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:50:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:50:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][0/625] eta 0:08:06 lr 0.001995 wd 0.0500 time 0.7787 (0.7787) data time 0.4494 (0.4494) model time 0.0000 (0.0000) loss 2.3342 (2.3342) grad_norm 1.4998 (1.4998) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:50:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][10/625] eta 0:04:14 lr 0.001995 wd 0.0500 time 0.3743 (0.4139) data time 0.0013 (0.0418) model time 0.0000 (0.0000) loss 4.2344 (3.6969) grad_norm 1.2616 (1.3389) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:50:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][20/625] eta 0:04:00 lr 0.001995 wd 0.0500 time 0.3784 (0.3970) data time 0.0011 (0.0224) model time 0.0000 (0.0000) loss 4.3347 (3.7199) grad_norm 1.1300 (1.2577) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][30/625] eta 0:03:52 lr 0.001995 wd 0.0500 time 0.3726 (0.3901) data time 0.0012 (0.0156) model time 0.0000 (0.0000) loss 4.0954 (3.8486) grad_norm 2.1893 (1.3349) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][40/625] eta 0:03:46 lr 0.001995 wd 0.0500 time 0.3846 (0.3875) data time 0.0011 (0.0121) model time 0.0000 (0.0000) loss 3.5294 (3.8582) grad_norm 1.1012 (1.2996) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][50/625] eta 0:03:41 lr 0.001995 wd 0.0500 time 0.3748 (0.3854) data time 0.0010 (0.0099) model time 0.0000 (0.0000) loss 4.4379 (3.8552) grad_norm 1.3248 (1.3074) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][60/625] eta 0:03:40 lr 0.001995 wd 0.0500 time 0.3920 (0.3898) data time 0.0011 (0.0085) model time 0.3909 (0.4113) loss 4.1268 (3.9126) grad_norm 1.0061 (1.2950) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][70/625] eta 0:03:38 lr 0.001995 wd 0.0500 time 0.3816 (0.3941) data time 0.0011 (0.0075) model time 0.3805 (0.4152) loss 4.3424 (3.9092) grad_norm 1.1411 (1.2718) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][80/625] eta 0:03:34 lr 0.001995 wd 0.0500 time 0.3762 (0.3944) data time 0.0011 (0.0068) model time 0.3751 (0.4082) loss 3.5104 (3.9147) grad_norm 0.8928 (1.2738) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][90/625] eta 0:03:30 lr 0.001995 wd 0.0500 time 0.3760 (0.3925) data time 0.0008 (0.0062) model time 0.3752 (0.4002) loss 3.1646 (3.9199) grad_norm 1.6089 (1.2709) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][100/625] eta 0:03:25 lr 0.001995 wd 0.0500 time 0.3719 (0.3911) data time 0.0010 (0.0057) model time 0.3708 (0.3955) loss 3.7356 (3.9303) grad_norm 1.2573 (1.2757) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][110/625] eta 0:03:20 lr 0.001995 wd 0.0500 time 0.3851 (0.3900) data time 0.0008 (0.0053) model time 0.3842 (0.3926) loss 3.6550 (3.9296) grad_norm 1.4453 (1.2604) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][120/625] eta 0:03:16 lr 0.001995 wd 0.0500 time 0.3781 (0.3890) data time 0.0008 (0.0050) model time 0.3773 (0.3903) loss 4.0619 (3.9212) grad_norm 1.0193 (1.2619) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][130/625] eta 0:03:12 lr 0.001995 wd 0.0500 time 0.3712 (0.3883) data time 0.0010 (0.0047) model time 0.3702 (0.3888) loss 3.3267 (3.9073) grad_norm 1.0490 (1.2591) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][140/625] eta 0:03:07 lr 0.001995 wd 0.0500 time 0.3755 (0.3873) data time 0.0010 (0.0044) model time 0.3744 (0.3871) loss 3.6155 (3.9047) grad_norm 0.9298 (1.2589) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][150/625] eta 0:03:03 lr 0.001995 wd 0.0500 time 0.3759 (0.3867) data time 0.0011 (0.0042) model time 0.3748 (0.3861) loss 3.4373 (3.9048) grad_norm 2.3063 (1.2606) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][160/625] eta 0:02:59 lr 0.001995 wd 0.0500 time 0.3787 (0.3868) data time 0.0012 (0.0040) model time 0.3775 (0.3862) loss 4.0207 (3.8975) grad_norm 1.1967 (1.2627) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][170/625] eta 0:02:55 lr 0.001995 wd 0.0500 time 0.3750 (0.3863) data time 0.0008 (0.0038) model time 0.3742 (0.3854) loss 3.4544 (3.8946) grad_norm 1.3484 (1.2618) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:51:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][180/625] eta 0:02:51 lr 0.001995 wd 0.0500 time 0.3837 (0.3859) data time 0.0009 (0.0037) model time 0.3828 (0.3848) loss 4.3022 (3.9046) grad_norm 0.7644 (1.2525) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][190/625] eta 0:02:47 lr 0.001995 wd 0.0500 time 0.3754 (0.3854) data time 0.0009 (0.0036) model time 0.3746 (0.3841) loss 4.0180 (3.9043) grad_norm 1.2093 (1.2459) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][200/625] eta 0:02:43 lr 0.001995 wd 0.0500 time 0.3665 (0.3849) data time 0.0010 (0.0034) model time 0.3655 (0.3835) loss 4.0831 (3.9029) grad_norm 1.7910 (1.2514) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][210/625] eta 0:02:39 lr 0.001995 wd 0.0500 time 0.3882 (0.3846) data time 0.0011 (0.0033) model time 0.3871 (0.3832) loss 3.6384 (3.8902) grad_norm 1.1485 (1.2502) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][220/625] eta 0:02:35 lr 0.001995 wd 0.0500 time 0.3739 (0.3843) data time 0.0011 (0.0032) model time 0.3728 (0.3828) loss 3.2408 (3.8833) grad_norm 1.6303 (1.2535) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][230/625] eta 0:02:31 lr 0.001995 wd 0.0500 time 0.3800 (0.3839) data time 0.0008 (0.0031) model time 0.3791 (0.3823) loss 4.7160 (3.8864) grad_norm 1.6181 (1.2508) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][240/625] eta 0:02:27 lr 0.001995 wd 0.0500 time 0.5188 (0.3843) data time 0.0011 (0.0031) model time 0.5177 (0.3828) loss 3.9817 (3.8890) grad_norm 1.1893 (1.2463) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][250/625] eta 0:02:24 lr 0.001994 wd 0.0500 time 0.3723 (0.3846) data time 0.0011 (0.0030) model time 0.3713 (0.3832) loss 4.5108 (3.8990) grad_norm 1.0277 (1.2461) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][260/625] eta 0:02:20 lr 0.001994 wd 0.0500 time 0.3735 (0.3851) data time 0.0012 (0.0029) model time 0.3723 (0.3838) loss 3.0362 (3.9018) grad_norm 1.2547 (1.2434) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][270/625] eta 0:02:16 lr 0.001994 wd 0.0500 time 0.3779 (0.3847) data time 0.0009 (0.0029) model time 0.3770 (0.3834) loss 3.0517 (3.8919) grad_norm 1.5963 (1.2418) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][280/625] eta 0:02:12 lr 0.001994 wd 0.0500 time 0.3771 (0.3852) data time 0.0010 (0.0028) model time 0.3761 (0.3840) loss 4.2027 (3.8926) grad_norm 1.5563 (1.2563) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][290/625] eta 0:02:09 lr 0.001994 wd 0.0500 time 0.3666 (0.3857) data time 0.0011 (0.0028) model time 0.3655 (0.3845) loss 3.8790 (3.9056) grad_norm 1.5298 (1.2552) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][300/625] eta 0:02:05 lr 0.001994 wd 0.0500 time 0.3766 (0.3854) data time 0.0011 (0.0027) model time 0.3755 (0.3841) loss 3.1195 (3.8888) grad_norm 1.3901 (1.2531) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][310/625] eta 0:02:01 lr 0.001994 wd 0.0500 time 0.3734 (0.3851) data time 0.0009 (0.0026) model time 0.3726 (0.3838) loss 4.5725 (3.9038) grad_norm 2.4400 (1.2577) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][320/625] eta 0:01:57 lr 0.001994 wd 0.0500 time 0.3765 (0.3849) data time 0.0011 (0.0026) model time 0.3754 (0.3836) loss 4.0969 (3.8937) grad_norm 1.0929 (1.2608) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][330/625] eta 0:01:53 lr 0.001994 wd 0.0500 time 0.3782 (0.3846) data time 0.0010 (0.0026) model time 0.3772 (0.3833) loss 4.5168 (3.8829) grad_norm 1.1125 (1.2618) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][340/625] eta 0:01:49 lr 0.001994 wd 0.0500 time 0.3743 (0.3844) data time 0.0008 (0.0025) model time 0.3735 (0.3830) loss 4.0890 (3.8833) grad_norm 1.2150 (1.2625) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][350/625] eta 0:01:45 lr 0.001994 wd 0.0500 time 0.3802 (0.3847) data time 0.0008 (0.0025) model time 0.3793 (0.3834) loss 4.0224 (3.8873) grad_norm 0.8736 (1.2579) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][360/625] eta 0:01:42 lr 0.001994 wd 0.0500 time 0.5826 (0.3851) data time 0.0011 (0.0024) model time 0.5815 (0.3839) loss 2.7016 (3.8833) grad_norm 1.0161 (1.2605) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][370/625] eta 0:01:38 lr 0.001994 wd 0.0500 time 0.5804 (0.3861) data time 0.0009 (0.0024) model time 0.5795 (0.3850) loss 4.1302 (3.8735) grad_norm 1.7761 (1.2601) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][380/625] eta 0:01:34 lr 0.001994 wd 0.0500 time 0.3778 (0.3859) data time 0.0008 (0.0024) model time 0.3770 (0.3848) loss 2.8635 (3.8681) grad_norm 1.3411 (1.2657) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][390/625] eta 0:01:30 lr 0.001994 wd 0.0500 time 0.3747 (0.3856) data time 0.0010 (0.0023) model time 0.3737 (0.3845) loss 4.2698 (3.8717) grad_norm 0.9343 (1.2634) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][400/625] eta 0:01:26 lr 0.001994 wd 0.0500 time 0.3717 (0.3854) data time 0.0008 (0.0023) model time 0.3709 (0.3842) loss 3.9467 (3.8697) grad_norm 1.2195 (1.2608) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][410/625] eta 0:01:22 lr 0.001994 wd 0.0500 time 0.3742 (0.3852) data time 0.0009 (0.0023) model time 0.3733 (0.3839) loss 4.0961 (3.8704) grad_norm 1.0176 (1.2659) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][420/625] eta 0:01:18 lr 0.001994 wd 0.0500 time 0.3795 (0.3850) data time 0.0008 (0.0023) model time 0.3787 (0.3837) loss 3.8357 (3.8725) grad_norm 1.1434 (1.2616) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][430/625] eta 0:01:15 lr 0.001994 wd 0.0500 time 0.3810 (0.3848) data time 0.0012 (0.0022) model time 0.3798 (0.3835) loss 4.5197 (3.8792) grad_norm 1.0784 (1.2635) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][440/625] eta 0:01:11 lr 0.001994 wd 0.0500 time 0.3748 (0.3847) data time 0.0008 (0.0022) model time 0.3740 (0.3834) loss 4.6377 (3.8732) grad_norm 1.0438 (1.2665) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][450/625] eta 0:01:07 lr 0.001994 wd 0.0500 time 0.3797 (0.3845) data time 0.0011 (0.0022) model time 0.3786 (0.3832) loss 4.1729 (3.8708) grad_norm 1.9559 (1.2676) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][460/625] eta 0:01:03 lr 0.001994 wd 0.0500 time 0.3803 (0.3844) data time 0.0008 (0.0022) model time 0.3795 (0.3831) loss 4.7333 (3.8709) grad_norm 0.8685 (1.2685) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][470/625] eta 0:00:59 lr 0.001994 wd 0.0500 time 0.3797 (0.3856) data time 0.0011 (0.0022) model time 0.3787 (0.3844) loss 3.6566 (3.8683) grad_norm 1.6573 (1.2668) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][480/625] eta 0:00:55 lr 0.001994 wd 0.0500 time 0.3784 (0.3854) data time 0.0008 (0.0021) model time 0.3776 (0.3842) loss 4.3333 (3.8658) grad_norm 1.2799 (1.2678) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][490/625] eta 0:00:52 lr 0.001994 wd 0.0500 time 0.3725 (0.3852) data time 0.0012 (0.0021) model time 0.3714 (0.3840) loss 3.7447 (3.8684) grad_norm 1.1973 (1.2673) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][500/625] eta 0:00:48 lr 0.001994 wd 0.0500 time 0.3784 (0.3851) data time 0.0012 (0.0021) model time 0.3773 (0.3839) loss 4.1436 (3.8620) grad_norm 0.9498 (1.2679) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][510/625] eta 0:00:44 lr 0.001994 wd 0.0500 time 0.3775 (0.3850) data time 0.0008 (0.0021) model time 0.3767 (0.3837) loss 4.9099 (3.8613) grad_norm 1.0228 (1.2672) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][520/625] eta 0:00:40 lr 0.001994 wd 0.0500 time 0.3781 (0.3849) data time 0.0011 (0.0021) model time 0.3771 (0.3836) loss 4.1191 (3.8621) grad_norm 1.3776 (1.2643) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][530/625] eta 0:00:36 lr 0.001994 wd 0.0500 time 0.3786 (0.3848) data time 0.0008 (0.0021) model time 0.3777 (0.3835) loss 4.6450 (3.8597) grad_norm 1.3861 (1.2658) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][540/625] eta 0:00:32 lr 0.001994 wd 0.0500 time 0.3796 (0.3847) data time 0.0009 (0.0020) model time 0.3787 (0.3834) loss 4.5198 (3.8642) grad_norm 1.6150 (1.2646) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][550/625] eta 0:00:28 lr 0.001994 wd 0.0500 time 0.3804 (0.3846) data time 0.0010 (0.0020) model time 0.3794 (0.3833) loss 2.6830 (3.8600) grad_norm 1.2490 (1.2628) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][560/625] eta 0:00:24 lr 0.001994 wd 0.0500 time 0.3739 (0.3844) data time 0.0012 (0.0020) model time 0.3728 (0.3831) loss 4.0076 (3.8604) grad_norm 1.7493 (1.2605) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][570/625] eta 0:00:21 lr 0.001994 wd 0.0500 time 0.3773 (0.3846) data time 0.0009 (0.0020) model time 0.3764 (0.3834) loss 4.5560 (3.8596) grad_norm 1.7334 (1.2640) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][580/625] eta 0:00:17 lr 0.001994 wd 0.0500 time 0.5928 (0.3852) data time 0.0012 (0.0020) model time 0.5916 (0.3840) loss 3.7898 (3.8574) grad_norm 1.7742 (1.2681) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][590/625] eta 0:00:13 lr 0.001994 wd 0.0500 time 0.3855 (0.3858) data time 0.0008 (0.0020) model time 0.3847 (0.3846) loss 4.4566 (3.8617) grad_norm 1.5999 (1.2702) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][600/625] eta 0:00:09 lr 0.001994 wd 0.0500 time 0.3698 (0.3856) data time 0.0011 (0.0019) model time 0.3687 (0.3844) loss 3.6859 (3.8640) grad_norm 1.0930 (1.2706) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][610/625] eta 0:00:05 lr 0.001994 wd 0.0500 time 0.3665 (0.3855) data time 0.0006 (0.0019) model time 0.3659 (0.3843) loss 3.7046 (3.8641) grad_norm 1.0982 (1.2687) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [29/300][620/625] eta 0:00:01 lr 0.001994 wd 0.0500 time 0.3690 (0.3852) data time 0.0006 (0.0019) model time 0.3684 (0.3840) loss 3.8725 (3.8602) grad_norm 1.0559 (1.2696) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 29 training takes 0:04:00 [2024-08-26 04:54:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:54:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 0.9062 (0.9062) Acc@1 81.152 (81.152) Acc@5 95.508 (95.508) Mem 11225MB [2024-08-26 04:54:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 1.4346 (1.0677) Acc@1 67.432 (75.972) Acc@5 88.818 (93.972) Mem 11225MB [2024-08-26 04:54:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.6553 (1.2985) Acc@1 62.061 (71.201) Acc@5 86.328 (90.662) Mem 11225MB [2024-08-26 04:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 71.015 Acc@5 90.591 [2024-08-26 04:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 71.0% [2024-08-26 04:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 71.02% [2024-08-26 04:54:53 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:54:54 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 0.7490 (0.7490) Acc@1 80.273 (80.273) Acc@5 95.166 (95.166) Mem 11225MB [2024-08-26 04:54:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 1.3584 (0.9440) Acc@1 64.697 (75.009) Acc@5 88.525 (93.617) Mem 11225MB [2024-08-26 04:54:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.5625 (1.1870) Acc@1 62.646 (70.219) Acc@5 85.400 (90.265) Mem 11225MB [2024-08-26 04:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 70.108 Acc@5 90.241 [2024-08-26 04:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 70.1% [2024-08-26 04:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 70.11% [2024-08-26 04:54:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:54:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:54:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][0/625] eta 0:08:02 lr 0.001994 wd 0.0500 time 0.7713 (0.7713) data time 0.4374 (0.4374) model time 0.0000 (0.0000) loss 3.6858 (3.6858) grad_norm 2.2093 (2.2093) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][10/625] eta 0:04:14 lr 0.001994 wd 0.0500 time 0.3853 (0.4131) data time 0.0009 (0.0411) model time 0.0000 (0.0000) loss 4.5853 (3.8037) grad_norm 1.3176 (1.4210) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][20/625] eta 0:04:00 lr 0.001994 wd 0.0500 time 0.3746 (0.3971) data time 0.0008 (0.0221) model time 0.0000 (0.0000) loss 4.5287 (3.7851) grad_norm 1.1470 (1.4094) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][30/625] eta 0:03:53 lr 0.001994 wd 0.0500 time 0.3875 (0.3927) data time 0.0011 (0.0154) model time 0.0000 (0.0000) loss 3.9215 (3.6713) grad_norm 1.1785 (1.3321) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][40/625] eta 0:03:48 lr 0.001994 wd 0.0500 time 0.3951 (0.3909) data time 0.0011 (0.0119) model time 0.0000 (0.0000) loss 3.4109 (3.6797) grad_norm 0.9609 (1.2590) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][50/625] eta 0:03:43 lr 0.001994 wd 0.0500 time 0.3810 (0.3884) data time 0.0011 (0.0098) model time 0.0000 (0.0000) loss 3.8794 (3.6945) grad_norm 0.8648 (1.2466) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][60/625] eta 0:03:38 lr 0.001994 wd 0.0500 time 0.3781 (0.3870) data time 0.0009 (0.0084) model time 0.3772 (0.3792) loss 3.9315 (3.7585) grad_norm 0.8916 (1.2318) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][70/625] eta 0:03:34 lr 0.001994 wd 0.0500 time 0.4024 (0.3863) data time 0.0011 (0.0074) model time 0.4013 (0.3800) loss 3.8548 (3.7610) grad_norm 0.9025 (1.2243) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][80/625] eta 0:03:30 lr 0.001994 wd 0.0500 time 0.3857 (0.3860) data time 0.0009 (0.0066) model time 0.3848 (0.3809) loss 3.1175 (3.7707) grad_norm 1.3754 (1.2073) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][90/625] eta 0:03:26 lr 0.001994 wd 0.0500 time 0.3788 (0.3854) data time 0.0009 (0.0060) model time 0.3779 (0.3805) loss 4.3066 (3.8129) grad_norm 1.5678 (1.2096) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][100/625] eta 0:03:24 lr 0.001994 wd 0.0500 time 0.5506 (0.3889) data time 0.0011 (0.0055) model time 0.5495 (0.3883) loss 3.2119 (3.7898) grad_norm 1.3150 (1.2020) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][110/625] eta 0:03:19 lr 0.001994 wd 0.0500 time 0.3847 (0.3877) data time 0.0008 (0.0051) model time 0.3839 (0.3859) loss 4.5396 (3.8351) grad_norm 1.1326 (1.2131) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][120/625] eta 0:03:15 lr 0.001994 wd 0.0500 time 0.3828 (0.3871) data time 0.0008 (0.0048) model time 0.3820 (0.3850) loss 4.8068 (3.8653) grad_norm 1.2375 (1.2035) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][130/625] eta 0:03:11 lr 0.001994 wd 0.0500 time 0.3801 (0.3866) data time 0.0011 (0.0045) model time 0.3790 (0.3843) loss 4.0653 (3.8542) grad_norm 1.4989 (1.2123) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][140/625] eta 0:03:07 lr 0.001993 wd 0.0500 time 0.3775 (0.3859) data time 0.0011 (0.0043) model time 0.3765 (0.3834) loss 3.4060 (3.8595) grad_norm 1.2062 (1.2135) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:55:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][150/625] eta 0:03:03 lr 0.001993 wd 0.0500 time 0.3875 (0.3869) data time 0.0011 (0.0041) model time 0.3865 (0.3850) loss 3.2842 (3.8562) grad_norm 1.3354 (1.2054) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][160/625] eta 0:03:00 lr 0.001993 wd 0.0500 time 0.3852 (0.3889) data time 0.0011 (0.0039) model time 0.3841 (0.3879) loss 4.4402 (3.8578) grad_norm 2.0554 (1.2257) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][170/625] eta 0:02:56 lr 0.001993 wd 0.0500 time 0.3728 (0.3884) data time 0.0009 (0.0037) model time 0.3719 (0.3873) loss 3.5073 (3.8651) grad_norm 1.2328 (1.2425) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][180/625] eta 0:02:52 lr 0.001993 wd 0.0500 time 0.3791 (0.3879) data time 0.0008 (0.0036) model time 0.3782 (0.3866) loss 4.8934 (3.8803) grad_norm 1.1237 (1.2465) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][190/625] eta 0:02:48 lr 0.001993 wd 0.0500 time 0.3902 (0.3875) data time 0.0011 (0.0035) model time 0.3891 (0.3861) loss 4.6601 (3.8817) grad_norm 1.1354 (1.2451) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][200/625] eta 0:02:44 lr 0.001993 wd 0.0500 time 0.3768 (0.3870) data time 0.0008 (0.0033) model time 0.3760 (0.3854) loss 2.7814 (3.8537) grad_norm 0.9858 (1.2412) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][210/625] eta 0:02:40 lr 0.001993 wd 0.0500 time 0.3781 (0.3868) data time 0.0011 (0.0032) model time 0.3770 (0.3852) loss 3.0062 (3.8286) grad_norm 1.5059 (1.2358) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][220/625] eta 0:02:36 lr 0.001993 wd 0.0500 time 0.3812 (0.3865) data time 0.0010 (0.0031) model time 0.3801 (0.3848) loss 4.4518 (3.8272) grad_norm 1.0683 (1.2343) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][230/625] eta 0:02:32 lr 0.001993 wd 0.0500 time 0.3794 (0.3862) data time 0.0008 (0.0031) model time 0.3786 (0.3844) loss 4.5397 (3.8387) grad_norm 1.5381 (1.2372) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][240/625] eta 0:02:28 lr 0.001993 wd 0.0500 time 0.3750 (0.3857) data time 0.0014 (0.0030) model time 0.3737 (0.3839) loss 3.4303 (3.8451) grad_norm 1.2051 (1.2316) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][250/625] eta 0:02:24 lr 0.001993 wd 0.0500 time 0.3793 (0.3855) data time 0.0008 (0.0029) model time 0.3785 (0.3836) loss 4.6582 (3.8392) grad_norm 1.6170 (1.2342) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][260/625] eta 0:02:20 lr 0.001993 wd 0.0500 time 0.3778 (0.3852) data time 0.0011 (0.0029) model time 0.3767 (0.3833) loss 4.0444 (3.8338) grad_norm 2.1580 (1.2411) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][270/625] eta 0:02:16 lr 0.001993 wd 0.0500 time 0.3783 (0.3850) data time 0.0009 (0.0028) model time 0.3774 (0.3830) loss 4.3622 (3.8390) grad_norm 0.9932 (1.2376) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][280/625] eta 0:02:12 lr 0.001993 wd 0.0500 time 0.3770 (0.3848) data time 0.0012 (0.0027) model time 0.3758 (0.3829) loss 3.1466 (3.8347) grad_norm 1.2452 (1.2374) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][290/625] eta 0:02:08 lr 0.001993 wd 0.0500 time 0.3729 (0.3847) data time 0.0014 (0.0027) model time 0.3716 (0.3827) loss 2.6679 (3.8284) grad_norm 1.4178 (1.2409) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][300/625] eta 0:02:04 lr 0.001993 wd 0.0500 time 0.3701 (0.3844) data time 0.0012 (0.0026) model time 0.3689 (0.3825) loss 3.8102 (3.8303) grad_norm 1.2791 (1.2398) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:56:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][310/625] eta 0:02:01 lr 0.001993 wd 0.0500 time 0.3845 (0.3850) data time 0.0013 (0.0026) model time 0.3833 (0.3831) loss 3.6294 (3.8286) grad_norm 1.2971 (1.2435) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][320/625] eta 0:01:57 lr 0.001993 wd 0.0500 time 0.3756 (0.3853) data time 0.0012 (0.0025) model time 0.3744 (0.3835) loss 2.9704 (3.8228) grad_norm 1.4412 (1.2459) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][330/625] eta 0:01:53 lr 0.001993 wd 0.0500 time 0.3820 (0.3851) data time 0.0010 (0.0025) model time 0.3810 (0.3833) loss 3.1121 (3.8259) grad_norm 1.6797 (1.2484) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][340/625] eta 0:01:50 lr 0.001993 wd 0.0500 time 0.3750 (0.3860) data time 0.0011 (0.0025) model time 0.3739 (0.3845) loss 3.7189 (3.8169) grad_norm 1.4335 (1.2495) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][350/625] eta 0:01:46 lr 0.001993 wd 0.0500 time 0.3738 (0.3863) data time 0.0011 (0.0024) model time 0.3728 (0.3848) loss 3.7245 (3.8276) grad_norm 0.8703 (1.2524) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][360/625] eta 0:01:42 lr 0.001993 wd 0.0500 time 0.3851 (0.3861) data time 0.0008 (0.0024) model time 0.3842 (0.3845) loss 4.0826 (3.8283) grad_norm 1.4513 (1.2521) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][370/625] eta 0:01:38 lr 0.001993 wd 0.0500 time 0.3867 (0.3858) data time 0.0011 (0.0024) model time 0.3856 (0.3843) loss 3.7603 (3.8352) grad_norm 1.3977 (1.2498) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][380/625] eta 0:01:34 lr 0.001993 wd 0.0500 time 0.3669 (0.3856) data time 0.0008 (0.0023) model time 0.3662 (0.3840) loss 3.7218 (3.8410) grad_norm 1.0465 (1.2520) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][390/625] eta 0:01:30 lr 0.001993 wd 0.0500 time 0.3765 (0.3854) data time 0.0008 (0.0023) model time 0.3758 (0.3838) loss 3.4300 (3.8451) grad_norm 0.9851 (1.2520) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][400/625] eta 0:01:26 lr 0.001993 wd 0.0500 time 0.3797 (0.3852) data time 0.0008 (0.0023) model time 0.3789 (0.3836) loss 3.8562 (3.8519) grad_norm 1.2933 (1.2487) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][410/625] eta 0:01:22 lr 0.001993 wd 0.0500 time 0.3777 (0.3850) data time 0.0009 (0.0022) model time 0.3768 (0.3833) loss 4.5203 (3.8575) grad_norm 1.5545 (1.2488) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][420/625] eta 0:01:18 lr 0.001993 wd 0.0500 time 0.3779 (0.3848) data time 0.0009 (0.0022) model time 0.3771 (0.3832) loss 3.7747 (3.8563) grad_norm 1.2713 (1.2544) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][430/625] eta 0:01:15 lr 0.001993 wd 0.0500 time 0.3804 (0.3847) data time 0.0009 (0.0022) model time 0.3796 (0.3830) loss 2.8019 (3.8558) grad_norm 1.3227 (1.2540) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][440/625] eta 0:01:11 lr 0.001993 wd 0.0500 time 0.3784 (0.3845) data time 0.0008 (0.0022) model time 0.3776 (0.3829) loss 3.7419 (3.8565) grad_norm 0.9794 (1.2522) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][450/625] eta 0:01:07 lr 0.001993 wd 0.0500 time 0.5698 (0.3848) data time 0.0009 (0.0021) model time 0.5690 (0.3832) loss 3.2065 (3.8521) grad_norm 0.9392 (1.2514) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:57:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][460/625] eta 0:01:03 lr 0.001993 wd 0.0500 time 0.3727 (0.3851) data time 0.0008 (0.0021) model time 0.3719 (0.3835) loss 2.5760 (3.8524) grad_norm 1.0587 (1.2514) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][470/625] eta 0:00:59 lr 0.001993 wd 0.0500 time 0.3800 (0.3854) data time 0.0012 (0.0021) model time 0.3788 (0.3838) loss 3.9415 (3.8501) grad_norm 1.1571 (1.2511) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][480/625] eta 0:00:55 lr 0.001993 wd 0.0500 time 0.3774 (0.3852) data time 0.0011 (0.0021) model time 0.3764 (0.3837) loss 3.3637 (3.8499) grad_norm 0.9795 (1.2483) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][490/625] eta 0:00:52 lr 0.001993 wd 0.0500 time 0.3778 (0.3855) data time 0.0011 (0.0021) model time 0.3767 (0.3840) loss 3.7408 (3.8532) grad_norm 1.4784 (1.2476) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][500/625] eta 0:00:48 lr 0.001993 wd 0.0500 time 0.3831 (0.3858) data time 0.0011 (0.0020) model time 0.3820 (0.3843) loss 3.8878 (3.8527) grad_norm 1.0852 (1.2506) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][510/625] eta 0:00:44 lr 0.001993 wd 0.0500 time 0.3788 (0.3856) data time 0.0011 (0.0020) model time 0.3777 (0.3841) loss 3.9067 (3.8491) grad_norm 1.0959 (1.2479) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][520/625] eta 0:00:40 lr 0.001993 wd 0.0500 time 0.3753 (0.3854) data time 0.0010 (0.0020) model time 0.3743 (0.3839) loss 4.1426 (3.8486) grad_norm 1.3493 (1.2501) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][530/625] eta 0:00:36 lr 0.001993 wd 0.0500 time 0.3753 (0.3853) data time 0.0013 (0.0020) model time 0.3740 (0.3838) loss 3.3292 (3.8471) grad_norm 1.2454 (1.2534) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][540/625] eta 0:00:32 lr 0.001993 wd 0.0500 time 0.3854 (0.3851) data time 0.0011 (0.0020) model time 0.3842 (0.3836) loss 4.0225 (3.8488) grad_norm 0.9653 (1.2492) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][550/625] eta 0:00:28 lr 0.001993 wd 0.0500 time 0.4084 (0.3851) data time 0.0009 (0.0020) model time 0.4076 (0.3836) loss 4.5371 (3.8431) grad_norm 1.2434 (1.2474) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][560/625] eta 0:00:25 lr 0.001993 wd 0.0500 time 0.3751 (0.3850) data time 0.0008 (0.0019) model time 0.3743 (0.3834) loss 3.1222 (3.8421) grad_norm 0.9492 (1.2467) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][570/625] eta 0:00:21 lr 0.001993 wd 0.0500 time 0.3772 (0.3848) data time 0.0011 (0.0019) model time 0.3761 (0.3833) loss 4.1193 (3.8414) grad_norm 1.0458 (1.2496) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][580/625] eta 0:00:17 lr 0.001993 wd 0.0500 time 0.3913 (0.3848) data time 0.0008 (0.0019) model time 0.3905 (0.3833) loss 3.9263 (3.8434) grad_norm 1.1781 (1.2482) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][590/625] eta 0:00:13 lr 0.001993 wd 0.0500 time 0.3763 (0.3847) data time 0.0009 (0.0019) model time 0.3755 (0.3832) loss 3.1537 (3.8412) grad_norm 1.1452 (1.2472) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][600/625] eta 0:00:09 lr 0.001993 wd 0.0500 time 0.3771 (0.3846) data time 0.0012 (0.0019) model time 0.3759 (0.3831) loss 4.1912 (3.8401) grad_norm 1.1569 (1.2496) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][610/625] eta 0:00:05 lr 0.001993 wd 0.0500 time 0.3668 (0.3848) data time 0.0006 (0.0019) model time 0.3662 (0.3833) loss 4.1934 (3.8412) grad_norm 1.0097 (1.2522) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [30/300][620/625] eta 0:00:01 lr 0.001992 wd 0.0500 time 0.3672 (0.3848) data time 0.0005 (0.0019) model time 0.3666 (0.3833) loss 4.8372 (3.8420) grad_norm 1.1516 (1.2520) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:58:58 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 30 training takes 0:04:00 [2024-08-26 04:58:58 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 04:59:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 04:59:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.529 (0.529) Loss 0.8013 (0.8013) Acc@1 81.885 (81.885) Acc@5 96.143 (96.143) Mem 11225MB [2024-08-26 04:59:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.100 (0.146) Loss 1.3857 (0.9920) Acc@1 66.895 (76.616) Acc@5 89.355 (94.274) Mem 11225MB [2024-08-26 04:59:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 1.5859 (1.2194) Acc@1 63.379 (71.905) Acc@5 86.621 (91.171) Mem 11225MB [2024-08-26 04:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 71.781 Acc@5 91.215 [2024-08-26 04:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 71.8% [2024-08-26 04:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 71.78% [2024-08-26 04:59:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 04:59:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 04:59:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.7231 (0.7231) Acc@1 81.104 (81.104) Acc@5 95.703 (95.703) Mem 11225MB [2024-08-26 04:59:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.3203 (0.9167) Acc@1 65.820 (75.772) Acc@5 88.867 (93.914) Mem 11225MB [2024-08-26 04:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.5234 (1.1550) Acc@1 63.428 (71.005) Acc@5 86.035 (90.716) Mem 11225MB [2024-08-26 04:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 70.887 Acc@5 90.687 [2024-08-26 04:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 70.9% [2024-08-26 04:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 70.89% [2024-08-26 04:59:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 04:59:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 04:59:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][0/625] eta 0:07:34 lr 0.001992 wd 0.0500 time 0.7278 (0.7278) data time 0.3826 (0.3826) model time 0.0000 (0.0000) loss 4.6838 (4.6838) grad_norm 0.9803 (0.9803) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][10/625] eta 0:04:11 lr 0.001992 wd 0.0500 time 0.3778 (0.4092) data time 0.0011 (0.0359) model time 0.0000 (0.0000) loss 3.8779 (3.8979) grad_norm 1.3179 (1.2251) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][20/625] eta 0:04:02 lr 0.001992 wd 0.0500 time 0.5234 (0.4011) data time 0.0010 (0.0193) model time 0.0000 (0.0000) loss 2.9013 (3.8970) grad_norm 0.8943 (1.1715) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][30/625] eta 0:04:01 lr 0.001992 wd 0.0500 time 0.3743 (0.4063) data time 0.0011 (0.0135) model time 0.0000 (0.0000) loss 2.5042 (3.7476) grad_norm 1.5600 (1.2448) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][40/625] eta 0:03:53 lr 0.001992 wd 0.0500 time 0.3761 (0.3996) data time 0.0011 (0.0105) model time 0.0000 (0.0000) loss 3.6392 (3.7601) grad_norm 1.2187 (1.2223) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][50/625] eta 0:03:46 lr 0.001992 wd 0.0500 time 0.3799 (0.3947) data time 0.0011 (0.0087) model time 0.0000 (0.0000) loss 4.2281 (3.7969) grad_norm 1.8055 (1.2165) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][60/625] eta 0:03:41 lr 0.001992 wd 0.0500 time 0.3768 (0.3922) data time 0.0010 (0.0074) model time 0.3758 (0.3780) loss 4.3957 (3.7980) grad_norm 2.3089 (1.2630) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][70/625] eta 0:03:36 lr 0.001992 wd 0.0500 time 0.3746 (0.3905) data time 0.0010 (0.0066) model time 0.3735 (0.3783) loss 4.0401 (3.7770) grad_norm 1.2645 (1.2789) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][80/625] eta 0:03:31 lr 0.001992 wd 0.0500 time 0.3841 (0.3890) data time 0.0011 (0.0059) model time 0.3830 (0.3779) loss 4.0982 (3.7924) grad_norm 1.3901 (1.3066) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][90/625] eta 0:03:27 lr 0.001992 wd 0.0500 time 0.3839 (0.3878) data time 0.0012 (0.0054) model time 0.3827 (0.3777) loss 3.9283 (3.8178) grad_norm 1.4556 (1.3033) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][100/625] eta 0:03:24 lr 0.001992 wd 0.0500 time 0.3719 (0.3888) data time 0.0008 (0.0050) model time 0.3711 (0.3815) loss 3.5262 (3.8413) grad_norm 1.2531 (1.2941) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][110/625] eta 0:03:19 lr 0.001992 wd 0.0500 time 0.3731 (0.3879) data time 0.0010 (0.0046) model time 0.3721 (0.3810) loss 4.1475 (3.8725) grad_norm 1.2164 (1.3049) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][120/625] eta 0:03:15 lr 0.001992 wd 0.0500 time 0.3761 (0.3871) data time 0.0008 (0.0043) model time 0.3753 (0.3803) loss 4.5355 (3.8564) grad_norm 0.8497 (1.3006) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 04:59:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][130/625] eta 0:03:11 lr 0.001992 wd 0.0500 time 0.3735 (0.3863) data time 0.0010 (0.0041) model time 0.3725 (0.3797) loss 4.0149 (3.8734) grad_norm 1.0830 (1.2915) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][140/625] eta 0:03:07 lr 0.001992 wd 0.0500 time 0.3733 (0.3857) data time 0.0012 (0.0039) model time 0.3721 (0.3794) loss 3.6126 (3.8828) grad_norm 1.1560 (1.3235) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][150/625] eta 0:03:02 lr 0.001992 wd 0.0500 time 0.3739 (0.3850) data time 0.0011 (0.0037) model time 0.3728 (0.3789) loss 3.7091 (3.8953) grad_norm 1.1109 (1.3197) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][160/625] eta 0:02:58 lr 0.001992 wd 0.0500 time 0.3806 (0.3845) data time 0.0008 (0.0036) model time 0.3798 (0.3786) loss 3.7904 (3.8922) grad_norm 1.1176 (1.3139) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][170/625] eta 0:02:55 lr 0.001992 wd 0.0500 time 0.3796 (0.3852) data time 0.0011 (0.0034) model time 0.3785 (0.3799) loss 4.0196 (3.8990) grad_norm 0.8628 (1.3040) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][180/625] eta 0:02:51 lr 0.001992 wd 0.0500 time 0.3798 (0.3847) data time 0.0009 (0.0033) model time 0.3790 (0.3796) loss 4.6225 (3.8902) grad_norm 0.8577 (1.2959) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][190/625] eta 0:02:47 lr 0.001992 wd 0.0500 time 0.3795 (0.3844) data time 0.0008 (0.0032) model time 0.3787 (0.3795) loss 4.3816 (3.8993) grad_norm 0.9772 (1.2883) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][200/625] eta 0:02:43 lr 0.001992 wd 0.0500 time 0.3776 (0.3841) data time 0.0013 (0.0031) model time 0.3763 (0.3793) loss 3.1366 (3.8957) grad_norm 0.8329 (1.2885) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][210/625] eta 0:02:39 lr 0.001992 wd 0.0500 time 0.3772 (0.3838) data time 0.0013 (0.0030) model time 0.3760 (0.3791) loss 3.0155 (3.8937) grad_norm 1.3245 (1.2811) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][220/625] eta 0:02:35 lr 0.001992 wd 0.0500 time 0.3866 (0.3843) data time 0.0008 (0.0029) model time 0.3857 (0.3800) loss 4.4662 (3.8906) grad_norm 1.2630 (1.2742) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][230/625] eta 0:02:32 lr 0.001992 wd 0.0500 time 0.6103 (0.3850) data time 0.0010 (0.0028) model time 0.6093 (0.3810) loss 4.2867 (3.8812) grad_norm 1.4312 (1.2772) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][240/625] eta 0:02:28 lr 0.001992 wd 0.0500 time 0.3754 (0.3855) data time 0.0009 (0.0028) model time 0.3746 (0.3819) loss 4.0045 (3.8859) grad_norm 1.3346 (1.2771) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][250/625] eta 0:02:24 lr 0.001992 wd 0.0500 time 0.3724 (0.3853) data time 0.0011 (0.0027) model time 0.3713 (0.3817) loss 4.1528 (3.8902) grad_norm 1.5294 (1.2765) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][260/625] eta 0:02:20 lr 0.001992 wd 0.0500 time 0.3781 (0.3851) data time 0.0013 (0.0026) model time 0.3768 (0.3815) loss 4.3332 (3.8847) grad_norm 0.8920 (1.2746) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][270/625] eta 0:02:16 lr 0.001992 wd 0.0500 time 0.3772 (0.3848) data time 0.0009 (0.0026) model time 0.3763 (0.3813) loss 3.3394 (3.8853) grad_norm 1.7834 (1.2928) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][280/625] eta 0:02:12 lr 0.001992 wd 0.0500 time 0.3779 (0.3845) data time 0.0009 (0.0025) model time 0.3770 (0.3811) loss 3.2569 (3.8847) grad_norm 0.8439 (1.2921) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][290/625] eta 0:02:08 lr 0.001992 wd 0.0500 time 0.3738 (0.3850) data time 0.0011 (0.0025) model time 0.3727 (0.3818) loss 2.5397 (3.8817) grad_norm 0.9029 (1.2923) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][300/625] eta 0:02:05 lr 0.001992 wd 0.0500 time 0.3759 (0.3848) data time 0.0012 (0.0025) model time 0.3748 (0.3816) loss 4.3651 (3.8710) grad_norm 1.6042 (1.2926) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][310/625] eta 0:02:01 lr 0.001992 wd 0.0500 time 0.3766 (0.3846) data time 0.0008 (0.0024) model time 0.3758 (0.3814) loss 2.7366 (3.8669) grad_norm 0.9625 (1.2905) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][320/625] eta 0:01:57 lr 0.001992 wd 0.0500 time 0.3714 (0.3844) data time 0.0008 (0.0024) model time 0.3706 (0.3812) loss 4.4847 (3.8795) grad_norm 1.6607 (1.2941) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][330/625] eta 0:01:53 lr 0.001992 wd 0.0500 time 0.3757 (0.3842) data time 0.0010 (0.0024) model time 0.3746 (0.3810) loss 3.6282 (3.8859) grad_norm 1.0999 (1.2916) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][340/625] eta 0:01:49 lr 0.001992 wd 0.0500 time 0.3780 (0.3840) data time 0.0009 (0.0023) model time 0.3772 (0.3809) loss 3.4572 (3.8890) grad_norm 1.5774 (1.2907) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][350/625] eta 0:01:45 lr 0.001992 wd 0.0500 time 0.3746 (0.3837) data time 0.0009 (0.0023) model time 0.3737 (0.3806) loss 4.2286 (3.8861) grad_norm 1.4148 (1.2932) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][360/625] eta 0:01:41 lr 0.001992 wd 0.0500 time 0.3756 (0.3835) data time 0.0011 (0.0023) model time 0.3746 (0.3804) loss 3.2461 (3.8870) grad_norm 0.9242 (1.2926) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][370/625] eta 0:01:37 lr 0.001992 wd 0.0500 time 0.3796 (0.3834) data time 0.0010 (0.0022) model time 0.3786 (0.3803) loss 3.8704 (3.8880) grad_norm 1.1594 (1.2850) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][380/625] eta 0:01:33 lr 0.001992 wd 0.0500 time 0.3731 (0.3832) data time 0.0008 (0.0022) model time 0.3723 (0.3802) loss 3.8050 (3.8913) grad_norm 1.1316 (1.2830) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][390/625] eta 0:01:30 lr 0.001992 wd 0.0500 time 0.3887 (0.3831) data time 0.0010 (0.0022) model time 0.3877 (0.3802) loss 3.4562 (3.8809) grad_norm 0.7636 (1.2759) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][400/625] eta 0:01:26 lr 0.001992 wd 0.0500 time 0.3733 (0.3830) data time 0.0012 (0.0022) model time 0.3721 (0.3801) loss 3.7180 (3.8779) grad_norm 1.1393 (1.2810) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][410/625] eta 0:01:22 lr 0.001992 wd 0.0500 time 0.3761 (0.3829) data time 0.0011 (0.0021) model time 0.3750 (0.3800) loss 2.9580 (3.8814) grad_norm 1.3469 (1.2824) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][420/625] eta 0:01:18 lr 0.001992 wd 0.0500 time 0.3789 (0.3838) data time 0.0009 (0.0021) model time 0.3780 (0.3811) loss 3.1982 (3.8809) grad_norm 1.6901 (1.2811) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][430/625] eta 0:01:14 lr 0.001991 wd 0.0500 time 0.3787 (0.3842) data time 0.0012 (0.0021) model time 0.3775 (0.3816) loss 3.7412 (3.8843) grad_norm 1.7228 (1.2815) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][440/625] eta 0:01:11 lr 0.001991 wd 0.0500 time 0.3780 (0.3841) data time 0.0011 (0.0021) model time 0.3770 (0.3815) loss 2.8736 (3.8839) grad_norm 1.2013 (1.2858) loss_scale 32768.0000 (16569.7596) mem 11225MB [2024-08-26 05:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][450/625] eta 0:01:07 lr 0.001991 wd 0.0500 time 0.3814 (0.3840) data time 0.0010 (0.0020) model time 0.3803 (0.3815) loss 3.6434 (3.8787) grad_norm 1.5737 (1.2861) loss_scale 32768.0000 (16928.9224) mem 11225MB [2024-08-26 05:02:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][460/625] eta 0:01:03 lr 0.001991 wd 0.0500 time 0.3810 (0.3839) data time 0.0011 (0.0020) model time 0.3799 (0.3814) loss 4.6334 (3.8767) grad_norm 1.7373 (1.2858) loss_scale 32768.0000 (17272.5033) mem 11225MB [2024-08-26 05:02:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][470/625] eta 0:00:59 lr 0.001991 wd 0.0500 time 0.3726 (0.3838) data time 0.0013 (0.0020) model time 0.3713 (0.3813) loss 3.9260 (3.8769) grad_norm 0.9160 (1.2855) loss_scale 32768.0000 (17601.4947) mem 11225MB [2024-08-26 05:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][480/625] eta 0:00:55 lr 0.001991 wd 0.0500 time 0.3805 (0.3837) data time 0.0008 (0.0020) model time 0.3797 (0.3812) loss 4.5117 (3.8732) grad_norm 1.2436 (1.2872) loss_scale 32768.0000 (17916.8067) mem 11225MB [2024-08-26 05:02:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][490/625] eta 0:00:51 lr 0.001991 wd 0.0500 time 0.3785 (0.3837) data time 0.0009 (0.0020) model time 0.3776 (0.3812) loss 4.7483 (3.8712) grad_norm 1.7706 (1.2855) loss_scale 32768.0000 (18219.2749) mem 11225MB [2024-08-26 05:02:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][500/625] eta 0:00:47 lr 0.001991 wd 0.0500 time 0.3800 (0.3840) data time 0.0011 (0.0020) model time 0.3789 (0.3816) loss 3.6483 (3.8626) grad_norm 0.8597 (1.2801) loss_scale 32768.0000 (18509.6687) mem 11225MB [2024-08-26 05:02:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][510/625] eta 0:00:44 lr 0.001991 wd 0.0500 time 0.3905 (0.3840) data time 0.0010 (0.0020) model time 0.3894 (0.3815) loss 3.8498 (3.8588) grad_norm 1.3646 (1.2769) loss_scale 32768.0000 (18788.6967) mem 11225MB [2024-08-26 05:02:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][520/625] eta 0:00:40 lr 0.001991 wd 0.0500 time 0.3772 (0.3838) data time 0.0011 (0.0019) model time 0.3761 (0.3814) loss 3.4323 (3.8596) grad_norm 1.9256 (1.2821) loss_scale 32768.0000 (19057.0134) mem 11225MB [2024-08-26 05:02:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][530/625] eta 0:00:36 lr 0.001991 wd 0.0500 time 0.3807 (0.3838) data time 0.0011 (0.0019) model time 0.3796 (0.3814) loss 2.7021 (3.8574) grad_norm 1.1564 (1.2807) loss_scale 32768.0000 (19315.2241) mem 11225MB [2024-08-26 05:02:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][540/625] eta 0:00:32 lr 0.001991 wd 0.0500 time 0.3773 (0.3844) data time 0.0012 (0.0019) model time 0.3762 (0.3821) loss 4.1454 (3.8593) grad_norm 1.3868 (1.2787) loss_scale 32768.0000 (19563.8891) mem 11225MB [2024-08-26 05:02:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][550/625] eta 0:00:28 lr 0.001991 wd 0.0500 time 0.3816 (0.3843) data time 0.0011 (0.0019) model time 0.3805 (0.3820) loss 4.0930 (3.8628) grad_norm 1.6744 (1.2777) loss_scale 32768.0000 (19803.5281) mem 11225MB [2024-08-26 05:02:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][560/625] eta 0:00:24 lr 0.001991 wd 0.0500 time 0.3774 (0.3842) data time 0.0008 (0.0019) model time 0.3766 (0.3819) loss 4.6023 (3.8600) grad_norm 1.5727 (1.2767) loss_scale 32768.0000 (20034.6239) mem 11225MB [2024-08-26 05:02:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][570/625] eta 0:00:21 lr 0.001991 wd 0.0500 time 0.3822 (0.3841) data time 0.0009 (0.0019) model time 0.3813 (0.3818) loss 3.9927 (3.8586) grad_norm 1.1104 (1.2769) loss_scale 32768.0000 (20257.6252) mem 11225MB [2024-08-26 05:02:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][580/625] eta 0:00:17 lr 0.001991 wd 0.0500 time 0.3708 (0.3841) data time 0.0011 (0.0019) model time 0.3697 (0.3819) loss 4.3782 (3.8622) grad_norm 0.8657 (1.2785) loss_scale 32768.0000 (20472.9501) mem 11225MB [2024-08-26 05:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][590/625] eta 0:00:13 lr 0.001991 wd 0.0500 time 0.3767 (0.3840) data time 0.0010 (0.0019) model time 0.3757 (0.3818) loss 3.1505 (3.8596) grad_norm 1.2818 (1.2773) loss_scale 32768.0000 (20680.9882) mem 11225MB [2024-08-26 05:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][600/625] eta 0:00:09 lr 0.001991 wd 0.0500 time 0.3718 (0.3839) data time 0.0008 (0.0018) model time 0.3710 (0.3817) loss 3.4490 (3.8602) grad_norm 0.8926 (1.2762) loss_scale 32768.0000 (20882.1032) mem 11225MB [2024-08-26 05:03:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][610/625] eta 0:00:05 lr 0.001991 wd 0.0500 time 0.3633 (0.3840) data time 0.0008 (0.0018) model time 0.3625 (0.3818) loss 4.4484 (3.8561) grad_norm 1.6056 (1.2758) loss_scale 32768.0000 (21076.6350) mem 11225MB [2024-08-26 05:03:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [31/300][620/625] eta 0:00:01 lr 0.001991 wd 0.0500 time 0.3669 (0.3838) data time 0.0008 (0.0018) model time 0.3661 (0.3816) loss 3.1200 (3.8573) grad_norm 1.1755 (1.2743) loss_scale 32768.0000 (21264.9018) mem 11225MB [2024-08-26 05:03:07 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 31 training takes 0:03:59 [2024-08-26 05:03:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:03:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:03:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.7847 (0.7847) Acc@1 83.301 (83.301) Acc@5 96.191 (96.191) Mem 11225MB [2024-08-26 05:03:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.145) Loss 1.4238 (0.9909) Acc@1 66.553 (76.944) Acc@5 89.062 (94.425) Mem 11225MB [2024-08-26 05:03:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.5576 (1.2276) Acc@1 65.430 (72.003) Acc@5 87.256 (91.250) Mem 11225MB [2024-08-26 05:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 71.799 Acc@5 91.237 [2024-08-26 05:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 71.8% [2024-08-26 05:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 71.80% [2024-08-26 05:03:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:03:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:03:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.500 (0.500) Loss 0.7007 (0.7007) Acc@1 82.178 (82.178) Acc@5 95.703 (95.703) Mem 11225MB [2024-08-26 05:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.147) Loss 1.2852 (0.8925) Acc@1 66.699 (76.483) Acc@5 89.453 (94.185) Mem 11225MB [2024-08-26 05:03:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.4863 (1.1260) Acc@1 64.062 (71.687) Acc@5 86.816 (91.069) Mem 11225MB [2024-08-26 05:03:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 71.545 Acc@5 91.045 [2024-08-26 05:03:16 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 71.5% [2024-08-26 05:03:16 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 71.55% [2024-08-26 05:03:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:03:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][0/625] eta 0:07:44 lr 0.001991 wd 0.0500 time 0.7432 (0.7432) data time 0.4020 (0.4020) model time 0.0000 (0.0000) loss 2.8367 (2.8367) grad_norm 1.2306 (1.2306) loss_scale 32768.0000 (32768.0000) mem 11225MB [2024-08-26 05:03:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][10/625] eta 0:04:12 lr 0.001991 wd 0.0500 time 0.3762 (0.4100) data time 0.0009 (0.0376) model time 0.0000 (0.0000) loss 2.8890 (3.4140) grad_norm 1.5606 (inf) loss_scale 16384.0000 (26810.1818) mem 11225MB [2024-08-26 05:03:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][20/625] eta 0:03:59 lr 0.001991 wd 0.0500 time 0.3630 (0.3954) data time 0.0011 (0.0202) model time 0.0000 (0.0000) loss 2.9919 (3.4373) grad_norm 1.1771 (inf) loss_scale 16384.0000 (21845.3333) mem 11225MB [2024-08-26 05:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][30/625] eta 0:03:52 lr 0.001991 wd 0.0500 time 0.3778 (0.3908) data time 0.0009 (0.0142) model time 0.0000 (0.0000) loss 4.3247 (3.6073) grad_norm 1.6248 (inf) loss_scale 16384.0000 (20083.6129) mem 11225MB [2024-08-26 05:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][40/625] eta 0:03:46 lr 0.001991 wd 0.0500 time 0.3847 (0.3874) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 3.2028 (3.6261) grad_norm 1.3980 (inf) loss_scale 16384.0000 (19181.2683) mem 11225MB [2024-08-26 05:03:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][50/625] eta 0:03:41 lr 0.001991 wd 0.0500 time 0.3651 (0.3850) data time 0.0011 (0.0091) model time 0.0000 (0.0000) loss 3.3062 (3.5962) grad_norm 0.9732 (inf) loss_scale 16384.0000 (18632.7843) mem 11225MB [2024-08-26 05:03:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][60/625] eta 0:03:36 lr 0.001991 wd 0.0500 time 0.3726 (0.3835) data time 0.0011 (0.0078) model time 0.3715 (0.3747) loss 3.9300 (3.5996) grad_norm 1.3968 (inf) loss_scale 16384.0000 (18264.1311) mem 11225MB [2024-08-26 05:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][70/625] eta 0:03:36 lr 0.001991 wd 0.0500 time 0.3816 (0.3908) data time 0.0008 (0.0069) model time 0.3808 (0.4046) loss 4.0878 (3.6227) grad_norm 1.2541 (inf) loss_scale 16384.0000 (17999.3239) mem 11225MB [2024-08-26 05:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][80/625] eta 0:03:31 lr 0.001991 wd 0.0500 time 0.3715 (0.3889) data time 0.0010 (0.0062) model time 0.3706 (0.3945) loss 4.4117 (3.6810) grad_norm 2.1152 (inf) loss_scale 16384.0000 (17799.9012) mem 11225MB [2024-08-26 05:03:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][90/625] eta 0:03:28 lr 0.001991 wd 0.0500 time 0.3369 (0.3895) data time 0.0011 (0.0057) model time 0.3358 (0.3939) loss 3.9878 (3.6813) grad_norm 1.1838 (inf) loss_scale 16384.0000 (17644.3077) mem 11225MB [2024-08-26 05:03:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][100/625] eta 0:03:23 lr 0.001991 wd 0.0500 time 0.3822 (0.3884) data time 0.0008 (0.0052) model time 0.3815 (0.3905) loss 3.7832 (3.6794) grad_norm 0.9959 (inf) loss_scale 16384.0000 (17519.5248) mem 11225MB [2024-08-26 05:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][110/625] eta 0:03:19 lr 0.001991 wd 0.0500 time 0.3746 (0.3875) data time 0.0008 (0.0049) model time 0.3738 (0.3883) loss 4.2777 (3.6700) grad_norm 1.1863 (inf) loss_scale 16384.0000 (17417.2252) mem 11225MB [2024-08-26 05:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][120/625] eta 0:03:15 lr 0.001991 wd 0.0500 time 0.3785 (0.3867) data time 0.0008 (0.0046) model time 0.3776 (0.3867) loss 2.6260 (3.6830) grad_norm 1.3216 (inf) loss_scale 16384.0000 (17331.8347) mem 11225MB [2024-08-26 05:04:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][130/625] eta 0:03:11 lr 0.001991 wd 0.0500 time 0.3770 (0.3860) data time 0.0008 (0.0043) model time 0.3761 (0.3854) loss 4.4924 (3.6993) grad_norm 1.0460 (inf) loss_scale 16384.0000 (17259.4809) mem 11225MB [2024-08-26 05:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][140/625] eta 0:03:06 lr 0.001991 wd 0.0500 time 0.3791 (0.3855) data time 0.0011 (0.0041) model time 0.3780 (0.3844) loss 3.9838 (3.7081) grad_norm 1.2386 (inf) loss_scale 16384.0000 (17197.3901) mem 11225MB [2024-08-26 05:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][150/625] eta 0:03:03 lr 0.001991 wd 0.0500 time 0.3977 (0.3853) data time 0.0009 (0.0039) model time 0.3968 (0.3843) loss 4.6576 (3.7299) grad_norm 0.9236 (inf) loss_scale 16384.0000 (17143.5232) mem 11225MB [2024-08-26 05:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][160/625] eta 0:02:59 lr 0.001991 wd 0.0500 time 0.3856 (0.3851) data time 0.0009 (0.0037) model time 0.3847 (0.3838) loss 4.5220 (3.7435) grad_norm 1.0015 (inf) loss_scale 16384.0000 (17096.3478) mem 11225MB [2024-08-26 05:04:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][170/625] eta 0:02:55 lr 0.001991 wd 0.0500 time 0.3722 (0.3847) data time 0.0008 (0.0036) model time 0.3713 (0.3833) loss 4.0876 (3.7332) grad_norm 1.5759 (inf) loss_scale 16384.0000 (17054.6901) mem 11225MB [2024-08-26 05:04:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][180/625] eta 0:02:51 lr 0.001991 wd 0.0500 time 0.3791 (0.3845) data time 0.0010 (0.0035) model time 0.3780 (0.3830) loss 4.7248 (3.7465) grad_norm 1.2824 (inf) loss_scale 16384.0000 (17017.6354) mem 11225MB [2024-08-26 05:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][190/625] eta 0:02:47 lr 0.001991 wd 0.0500 time 0.3748 (0.3841) data time 0.0009 (0.0034) model time 0.3740 (0.3826) loss 4.8243 (3.7550) grad_norm 1.0775 (inf) loss_scale 16384.0000 (16984.4607) mem 11225MB [2024-08-26 05:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][200/625] eta 0:02:43 lr 0.001991 wd 0.0500 time 0.3766 (0.3839) data time 0.0011 (0.0033) model time 0.3756 (0.3822) loss 4.2237 (3.7556) grad_norm 1.5195 (inf) loss_scale 16384.0000 (16954.5871) mem 11225MB [2024-08-26 05:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][210/625] eta 0:02:39 lr 0.001991 wd 0.0500 time 0.3402 (0.3845) data time 0.0010 (0.0032) model time 0.3392 (0.3831) loss 3.2090 (3.7535) grad_norm 1.2285 (inf) loss_scale 16384.0000 (16927.5450) mem 11225MB [2024-08-26 05:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][220/625] eta 0:02:35 lr 0.001991 wd 0.0500 time 0.3958 (0.3843) data time 0.0010 (0.0031) model time 0.3948 (0.3829) loss 4.0001 (3.7603) grad_norm 1.3020 (inf) loss_scale 16384.0000 (16902.9502) mem 11225MB [2024-08-26 05:04:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][230/625] eta 0:02:31 lr 0.001990 wd 0.0500 time 0.3795 (0.3841) data time 0.0009 (0.0030) model time 0.3786 (0.3827) loss 3.1426 (3.7566) grad_norm 1.0747 (inf) loss_scale 16384.0000 (16880.4848) mem 11225MB [2024-08-26 05:04:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][240/625] eta 0:02:27 lr 0.001990 wd 0.0500 time 0.3782 (0.3839) data time 0.0011 (0.0029) model time 0.3772 (0.3823) loss 3.5086 (3.7577) grad_norm 2.0585 (inf) loss_scale 16384.0000 (16859.8838) mem 11225MB [2024-08-26 05:04:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][250/625] eta 0:02:24 lr 0.001990 wd 0.0500 time 0.5811 (0.3853) data time 0.0011 (0.0028) model time 0.5800 (0.3842) loss 3.8076 (3.7510) grad_norm 1.3697 (inf) loss_scale 16384.0000 (16840.9243) mem 11225MB [2024-08-26 05:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][260/625] eta 0:02:20 lr 0.001990 wd 0.0500 time 0.3781 (0.3851) data time 0.0008 (0.0028) model time 0.3773 (0.3839) loss 4.4772 (3.7548) grad_norm 0.9694 (inf) loss_scale 16384.0000 (16823.4176) mem 11225MB [2024-08-26 05:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][270/625] eta 0:02:16 lr 0.001990 wd 0.0500 time 0.3765 (0.3849) data time 0.0009 (0.0028) model time 0.3756 (0.3835) loss 2.7295 (3.7435) grad_norm 1.2042 (inf) loss_scale 16384.0000 (16807.2030) mem 11225MB [2024-08-26 05:05:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][280/625] eta 0:02:13 lr 0.001990 wd 0.0500 time 0.5853 (0.3861) data time 0.0010 (0.0027) model time 0.5843 (0.3850) loss 4.4698 (3.7443) grad_norm 1.5350 (inf) loss_scale 16384.0000 (16792.1423) mem 11225MB [2024-08-26 05:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][290/625] eta 0:02:09 lr 0.001990 wd 0.0500 time 0.3799 (0.3859) data time 0.0008 (0.0027) model time 0.3790 (0.3847) loss 4.3228 (3.7525) grad_norm 1.1549 (inf) loss_scale 16384.0000 (16778.1168) mem 11225MB [2024-08-26 05:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][300/625] eta 0:02:05 lr 0.001990 wd 0.0500 time 0.3784 (0.3856) data time 0.0010 (0.0026) model time 0.3774 (0.3844) loss 3.5060 (3.7584) grad_norm 1.7592 (inf) loss_scale 16384.0000 (16765.0233) mem 11225MB [2024-08-26 05:05:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][310/625] eta 0:02:01 lr 0.001990 wd 0.0500 time 0.3816 (0.3853) data time 0.0009 (0.0026) model time 0.3808 (0.3841) loss 5.1045 (3.7663) grad_norm 1.3586 (inf) loss_scale 16384.0000 (16752.7717) mem 11225MB [2024-08-26 05:05:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][320/625] eta 0:01:57 lr 0.001990 wd 0.0500 time 0.3777 (0.3851) data time 0.0010 (0.0025) model time 0.3767 (0.3838) loss 3.4832 (3.7720) grad_norm 1.4624 (inf) loss_scale 16384.0000 (16741.2835) mem 11225MB [2024-08-26 05:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][330/625] eta 0:01:53 lr 0.001990 wd 0.0500 time 0.3763 (0.3848) data time 0.0008 (0.0025) model time 0.3754 (0.3835) loss 4.0992 (3.7752) grad_norm 1.4761 (inf) loss_scale 16384.0000 (16730.4894) mem 11225MB [2024-08-26 05:05:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][340/625] eta 0:01:49 lr 0.001990 wd 0.0500 time 0.3835 (0.3846) data time 0.0008 (0.0025) model time 0.3827 (0.3832) loss 3.0254 (3.7749) grad_norm 1.2977 (inf) loss_scale 16384.0000 (16720.3284) mem 11225MB [2024-08-26 05:05:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][350/625] eta 0:01:45 lr 0.001990 wd 0.0500 time 0.3775 (0.3845) data time 0.0008 (0.0024) model time 0.3766 (0.3831) loss 4.8982 (3.7749) grad_norm 0.9699 (inf) loss_scale 16384.0000 (16710.7464) mem 11225MB [2024-08-26 05:05:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][360/625] eta 0:01:41 lr 0.001990 wd 0.0500 time 0.3772 (0.3843) data time 0.0011 (0.0024) model time 0.3761 (0.3829) loss 4.5635 (3.7771) grad_norm 1.0751 (inf) loss_scale 16384.0000 (16701.6953) mem 11225MB [2024-08-26 05:05:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][370/625] eta 0:01:38 lr 0.001990 wd 0.0500 time 0.5780 (0.3854) data time 0.0011 (0.0024) model time 0.5769 (0.3842) loss 3.6065 (3.7665) grad_norm 2.2176 (inf) loss_scale 16384.0000 (16693.1321) mem 11225MB [2024-08-26 05:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][380/625] eta 0:01:34 lr 0.001990 wd 0.0500 time 0.3791 (0.3853) data time 0.0011 (0.0023) model time 0.3780 (0.3840) loss 2.9648 (3.7651) grad_norm 1.5139 (inf) loss_scale 16384.0000 (16685.0184) mem 11225MB [2024-08-26 05:05:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][390/625] eta 0:01:30 lr 0.001990 wd 0.0500 time 0.3792 (0.3851) data time 0.0008 (0.0023) model time 0.3784 (0.3839) loss 3.9911 (3.7636) grad_norm 1.4698 (inf) loss_scale 16384.0000 (16677.3197) mem 11225MB [2024-08-26 05:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][400/625] eta 0:01:26 lr 0.001990 wd 0.0500 time 0.3767 (0.3849) data time 0.0012 (0.0023) model time 0.3755 (0.3837) loss 4.4117 (3.7665) grad_norm 1.0475 (inf) loss_scale 16384.0000 (16670.0050) mem 11225MB [2024-08-26 05:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][410/625] eta 0:01:22 lr 0.001990 wd 0.0500 time 0.3811 (0.3848) data time 0.0011 (0.0022) model time 0.3800 (0.3835) loss 4.0210 (3.7715) grad_norm 1.3772 (inf) loss_scale 16384.0000 (16663.0462) mem 11225MB [2024-08-26 05:05:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][420/625] eta 0:01:18 lr 0.001990 wd 0.0500 time 0.3865 (0.3851) data time 0.0011 (0.0022) model time 0.3854 (0.3839) loss 4.0372 (3.7754) grad_norm 0.9019 (inf) loss_scale 16384.0000 (16656.4181) mem 11225MB [2024-08-26 05:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][430/625] eta 0:01:15 lr 0.001990 wd 0.0500 time 0.3755 (0.3849) data time 0.0011 (0.0022) model time 0.3745 (0.3836) loss 2.4366 (3.7752) grad_norm 1.4280 (inf) loss_scale 16384.0000 (16650.0974) mem 11225MB [2024-08-26 05:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][440/625] eta 0:01:11 lr 0.001990 wd 0.0500 time 0.3741 (0.3848) data time 0.0011 (0.0022) model time 0.3729 (0.3836) loss 3.8613 (3.7790) grad_norm 1.0349 (inf) loss_scale 16384.0000 (16644.0635) mem 11225MB [2024-08-26 05:06:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][450/625] eta 0:01:07 lr 0.001990 wd 0.0500 time 0.3776 (0.3847) data time 0.0008 (0.0022) model time 0.3768 (0.3834) loss 4.6380 (3.7802) grad_norm 1.2259 (inf) loss_scale 16384.0000 (16638.2971) mem 11225MB [2024-08-26 05:06:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][460/625] eta 0:01:03 lr 0.001990 wd 0.0500 time 0.3836 (0.3850) data time 0.0010 (0.0021) model time 0.3825 (0.3837) loss 4.5738 (3.7846) grad_norm 1.9602 (inf) loss_scale 16384.0000 (16632.7809) mem 11225MB [2024-08-26 05:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][470/625] eta 0:00:59 lr 0.001990 wd 0.0500 time 0.3703 (0.3853) data time 0.0011 (0.0021) model time 0.3692 (0.3840) loss 3.5197 (3.7899) grad_norm 0.9979 (inf) loss_scale 16384.0000 (16627.4989) mem 11225MB [2024-08-26 05:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][480/625] eta 0:00:55 lr 0.001990 wd 0.0500 time 0.3782 (0.3852) data time 0.0008 (0.0021) model time 0.3774 (0.3839) loss 2.9173 (3.7897) grad_norm 2.0608 (inf) loss_scale 16384.0000 (16622.4366) mem 11225MB [2024-08-26 05:06:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][490/625] eta 0:00:51 lr 0.001990 wd 0.0500 time 0.3731 (0.3850) data time 0.0008 (0.0021) model time 0.3723 (0.3838) loss 4.6225 (3.7938) grad_norm 0.8892 (inf) loss_scale 16384.0000 (16617.5804) mem 11225MB [2024-08-26 05:06:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][500/625] eta 0:00:48 lr 0.001990 wd 0.0500 time 0.3883 (0.3850) data time 0.0011 (0.0021) model time 0.3872 (0.3837) loss 4.0676 (3.7896) grad_norm 1.0759 (inf) loss_scale 16384.0000 (16612.9182) mem 11225MB [2024-08-26 05:06:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][510/625] eta 0:00:44 lr 0.001990 wd 0.0500 time 0.3823 (0.3849) data time 0.0011 (0.0020) model time 0.3812 (0.3836) loss 4.1257 (3.7898) grad_norm 1.9388 (inf) loss_scale 16384.0000 (16608.4384) mem 11225MB [2024-08-26 05:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][520/625] eta 0:00:40 lr 0.001990 wd 0.0500 time 0.3749 (0.3848) data time 0.0008 (0.0020) model time 0.3741 (0.3835) loss 4.3759 (3.7910) grad_norm 1.2795 (inf) loss_scale 16384.0000 (16604.1305) mem 11225MB [2024-08-26 05:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][530/625] eta 0:00:36 lr 0.001990 wd 0.0500 time 0.3771 (0.3846) data time 0.0008 (0.0020) model time 0.3763 (0.3833) loss 3.1943 (3.7897) grad_norm 0.8005 (inf) loss_scale 16384.0000 (16599.9849) mem 11225MB [2024-08-26 05:06:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][540/625] eta 0:00:32 lr 0.001990 wd 0.0500 time 0.3915 (0.3846) data time 0.0012 (0.0020) model time 0.3903 (0.3833) loss 3.9458 (3.7914) grad_norm 1.5392 (inf) loss_scale 16384.0000 (16595.9926) mem 11225MB [2024-08-26 05:06:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][550/625] eta 0:00:28 lr 0.001990 wd 0.0500 time 0.3793 (0.3845) data time 0.0011 (0.0020) model time 0.3782 (0.3832) loss 3.8704 (3.7953) grad_norm 1.6143 (inf) loss_scale 16384.0000 (16592.1452) mem 11225MB [2024-08-26 05:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][560/625] eta 0:00:24 lr 0.001990 wd 0.0500 time 0.3828 (0.3844) data time 0.0011 (0.0020) model time 0.3816 (0.3831) loss 3.2436 (3.7931) grad_norm 0.9719 (inf) loss_scale 16384.0000 (16588.4349) mem 11225MB [2024-08-26 05:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][570/625] eta 0:00:21 lr 0.001990 wd 0.0500 time 0.3791 (0.3843) data time 0.0011 (0.0020) model time 0.3780 (0.3830) loss 3.6518 (3.7913) grad_norm 1.0109 (inf) loss_scale 16384.0000 (16584.8546) mem 11225MB [2024-08-26 05:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][580/625] eta 0:00:17 lr 0.001990 wd 0.0500 time 0.3756 (0.3853) data time 0.0014 (0.0019) model time 0.3742 (0.3840) loss 2.5955 (3.7911) grad_norm 0.9503 (inf) loss_scale 16384.0000 (16581.3976) mem 11225MB [2024-08-26 05:07:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][590/625] eta 0:00:13 lr 0.001990 wd 0.0500 time 0.3738 (0.3855) data time 0.0009 (0.0019) model time 0.3729 (0.3843) loss 4.6639 (3.7925) grad_norm 1.7746 (inf) loss_scale 16384.0000 (16578.0575) mem 11225MB [2024-08-26 05:07:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][600/625] eta 0:00:09 lr 0.001990 wd 0.0500 time 0.3805 (0.3854) data time 0.0009 (0.0019) model time 0.3796 (0.3842) loss 3.1324 (3.7944) grad_norm 1.1943 (inf) loss_scale 16384.0000 (16574.8286) mem 11225MB [2024-08-26 05:07:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][610/625] eta 0:00:05 lr 0.001990 wd 0.0500 time 0.3683 (0.3856) data time 0.0006 (0.0019) model time 0.3677 (0.3844) loss 3.1781 (3.7937) grad_norm 0.9786 (inf) loss_scale 16384.0000 (16571.7054) mem 11225MB [2024-08-26 05:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [32/300][620/625] eta 0:00:01 lr 0.001990 wd 0.0500 time 0.3631 (0.3853) data time 0.0006 (0.0019) model time 0.3625 (0.3841) loss 2.8498 (3.7908) grad_norm 0.9820 (inf) loss_scale 16384.0000 (16568.6828) mem 11225MB [2024-08-26 05:07:18 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 32 training takes 0:04:00 [2024-08-26 05:07:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:07:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:07:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.7974 (0.7974) Acc@1 82.568 (82.568) Acc@5 96.240 (96.240) Mem 11225MB [2024-08-26 05:07:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.144) Loss 1.4082 (1.0075) Acc@1 68.262 (76.673) Acc@5 89.160 (94.540) Mem 11225MB [2024-08-26 05:07:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.5010 (1.2193) Acc@1 66.211 (72.205) Acc@5 88.086 (91.662) Mem 11225MB [2024-08-26 05:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 71.899 Acc@5 91.597 [2024-08-26 05:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 71.9% [2024-08-26 05:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 71.90% [2024-08-26 05:07:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:07:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.6821 (0.6821) Acc@1 82.568 (82.568) Acc@5 95.898 (95.898) Mem 11225MB [2024-08-26 05:07:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.2539 (0.8715) Acc@1 67.432 (77.011) Acc@5 89.697 (94.416) Mem 11225MB [2024-08-26 05:07:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.4541 (1.1009) Acc@1 64.893 (72.242) Acc@5 87.061 (91.362) Mem 11225MB [2024-08-26 05:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 72.095 Acc@5 91.355 [2024-08-26 05:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 72.1% [2024-08-26 05:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 72.09% [2024-08-26 05:07:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:07:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:07:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][0/625] eta 0:07:26 lr 0.001989 wd 0.0500 time 0.7136 (0.7136) data time 0.3670 (0.3670) model time 0.0000 (0.0000) loss 4.5004 (4.5004) grad_norm 0.9758 (0.9758) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:07:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][10/625] eta 0:04:12 lr 0.001989 wd 0.0500 time 0.3847 (0.4108) data time 0.0009 (0.0344) model time 0.0000 (0.0000) loss 3.3239 (3.5802) grad_norm 1.1883 (1.2706) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:07:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][20/625] eta 0:03:59 lr 0.001989 wd 0.0500 time 0.3718 (0.3958) data time 0.0011 (0.0186) model time 0.0000 (0.0000) loss 3.6156 (3.6845) grad_norm 1.6754 (1.3241) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:07:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][30/625] eta 0:03:52 lr 0.001989 wd 0.0500 time 0.3779 (0.3902) data time 0.0010 (0.0130) model time 0.0000 (0.0000) loss 3.8728 (3.7001) grad_norm 0.9526 (1.3125) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:07:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][40/625] eta 0:03:46 lr 0.001989 wd 0.0500 time 0.3801 (0.3875) data time 0.0009 (0.0101) model time 0.0000 (0.0000) loss 2.8539 (3.7647) grad_norm 0.7737 (1.2704) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:07:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][50/625] eta 0:03:41 lr 0.001989 wd 0.0500 time 0.3731 (0.3853) data time 0.0008 (0.0083) model time 0.0000 (0.0000) loss 4.3999 (3.7963) grad_norm 1.3449 (1.2321) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:07:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][60/625] eta 0:03:36 lr 0.001989 wd 0.0500 time 0.3789 (0.3839) data time 0.0008 (0.0071) model time 0.3781 (0.3759) loss 2.5333 (3.7807) grad_norm 1.3017 (1.2592) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:07:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][70/625] eta 0:03:33 lr 0.001989 wd 0.0500 time 0.5694 (0.3855) data time 0.0008 (0.0063) model time 0.5686 (0.3847) loss 3.2183 (3.7711) grad_norm 1.2923 (1.2999) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:07:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][80/625] eta 0:03:29 lr 0.001989 wd 0.0500 time 0.3706 (0.3845) data time 0.0011 (0.0056) model time 0.3696 (0.3820) loss 3.1267 (3.7884) grad_norm 1.2329 (1.2703) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][90/625] eta 0:03:25 lr 0.001989 wd 0.0500 time 0.3754 (0.3838) data time 0.0008 (0.0051) model time 0.3746 (0.3808) loss 3.7955 (3.7738) grad_norm 1.2858 (1.2799) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][100/625] eta 0:03:23 lr 0.001989 wd 0.0500 time 0.3884 (0.3869) data time 0.0010 (0.0047) model time 0.3874 (0.3873) loss 2.7154 (3.7454) grad_norm 1.5715 (1.2856) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][110/625] eta 0:03:18 lr 0.001989 wd 0.0500 time 0.3811 (0.3859) data time 0.0011 (0.0044) model time 0.3800 (0.3853) loss 3.5798 (3.7560) grad_norm 1.2915 (1.3035) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][120/625] eta 0:03:14 lr 0.001989 wd 0.0500 time 0.3733 (0.3851) data time 0.0009 (0.0042) model time 0.3725 (0.3839) loss 2.7554 (3.7621) grad_norm 1.0724 (1.3025) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][130/625] eta 0:03:10 lr 0.001989 wd 0.0500 time 0.3808 (0.3845) data time 0.0010 (0.0039) model time 0.3798 (0.3829) loss 2.9261 (3.7811) grad_norm 1.3085 (1.2923) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][140/625] eta 0:03:06 lr 0.001989 wd 0.0500 time 0.3691 (0.3839) data time 0.0013 (0.0037) model time 0.3678 (0.3820) loss 3.3272 (3.7724) grad_norm 1.3118 (1.2896) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][150/625] eta 0:03:02 lr 0.001989 wd 0.0500 time 0.3823 (0.3836) data time 0.0010 (0.0036) model time 0.3813 (0.3815) loss 3.8856 (3.7782) grad_norm 1.0404 (1.2906) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][160/625] eta 0:02:59 lr 0.001989 wd 0.0500 time 0.3771 (0.3861) data time 0.0008 (0.0034) model time 0.3763 (0.3853) loss 4.4813 (3.7781) grad_norm 1.2345 (1.2881) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][170/625] eta 0:02:55 lr 0.001989 wd 0.0500 time 0.3768 (0.3855) data time 0.0008 (0.0033) model time 0.3759 (0.3845) loss 4.4689 (3.7922) grad_norm 1.2309 (1.2811) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][180/625] eta 0:02:51 lr 0.001989 wd 0.0500 time 0.3722 (0.3850) data time 0.0012 (0.0032) model time 0.3710 (0.3837) loss 4.0055 (3.7922) grad_norm 0.9422 (1.2669) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][190/625] eta 0:02:47 lr 0.001989 wd 0.0500 time 0.3812 (0.3857) data time 0.0008 (0.0031) model time 0.3803 (0.3846) loss 3.6400 (3.7949) grad_norm 1.0953 (1.2622) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][200/625] eta 0:02:43 lr 0.001989 wd 0.0500 time 0.3788 (0.3853) data time 0.0009 (0.0030) model time 0.3778 (0.3842) loss 3.5743 (3.8021) grad_norm 0.8550 (1.2634) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][210/625] eta 0:02:39 lr 0.001989 wd 0.0500 time 0.3810 (0.3850) data time 0.0011 (0.0029) model time 0.3799 (0.3837) loss 4.1766 (3.7973) grad_norm 1.0056 (1.2746) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][220/625] eta 0:02:35 lr 0.001989 wd 0.0500 time 0.3804 (0.3847) data time 0.0008 (0.0028) model time 0.3795 (0.3834) loss 4.7449 (3.8038) grad_norm 1.0114 (1.2743) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][230/625] eta 0:02:31 lr 0.001989 wd 0.0500 time 0.3794 (0.3844) data time 0.0008 (0.0027) model time 0.3786 (0.3830) loss 2.3484 (3.7833) grad_norm 0.7752 (1.2690) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][240/625] eta 0:02:27 lr 0.001989 wd 0.0500 time 0.3721 (0.3841) data time 0.0013 (0.0027) model time 0.3709 (0.3826) loss 3.7987 (3.8043) grad_norm 1.6277 (1.2720) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][250/625] eta 0:02:23 lr 0.001989 wd 0.0500 time 0.3779 (0.3837) data time 0.0011 (0.0026) model time 0.3768 (0.3822) loss 4.0221 (3.7974) grad_norm 1.0590 (1.2747) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][260/625] eta 0:02:19 lr 0.001989 wd 0.0500 time 0.3760 (0.3835) data time 0.0012 (0.0026) model time 0.3748 (0.3819) loss 3.7561 (3.7993) grad_norm 1.1314 (1.2676) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][270/625] eta 0:02:16 lr 0.001989 wd 0.0500 time 0.3797 (0.3832) data time 0.0008 (0.0025) model time 0.3789 (0.3816) loss 4.6607 (3.8046) grad_norm 1.2741 (1.2637) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][280/625] eta 0:02:12 lr 0.001989 wd 0.0500 time 0.3776 (0.3830) data time 0.0008 (0.0024) model time 0.3768 (0.3813) loss 3.7177 (3.8074) grad_norm 0.9368 (1.2592) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][290/625] eta 0:02:08 lr 0.001989 wd 0.0500 time 0.3780 (0.3828) data time 0.0009 (0.0024) model time 0.3772 (0.3811) loss 3.0829 (3.8074) grad_norm 1.1023 (1.2531) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][300/625] eta 0:02:04 lr 0.001989 wd 0.0500 time 0.3750 (0.3826) data time 0.0011 (0.0024) model time 0.3739 (0.3809) loss 4.4606 (3.8089) grad_norm 1.1990 (1.2558) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][310/625] eta 0:02:00 lr 0.001989 wd 0.0500 time 0.3783 (0.3838) data time 0.0012 (0.0023) model time 0.3771 (0.3823) loss 4.0460 (3.8161) grad_norm 1.2015 (1.2596) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][320/625] eta 0:01:56 lr 0.001989 wd 0.0500 time 0.3764 (0.3836) data time 0.0011 (0.0023) model time 0.3753 (0.3821) loss 4.1837 (3.8258) grad_norm 1.3816 (1.2649) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][330/625] eta 0:01:53 lr 0.001989 wd 0.0500 time 0.3731 (0.3834) data time 0.0011 (0.0023) model time 0.3720 (0.3818) loss 3.4650 (3.8296) grad_norm 1.4454 (1.2629) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][340/625] eta 0:01:49 lr 0.001989 wd 0.0500 time 0.3785 (0.3837) data time 0.0008 (0.0022) model time 0.3777 (0.3823) loss 2.7087 (3.8251) grad_norm 1.6480 (1.2635) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][350/625] eta 0:01:45 lr 0.001989 wd 0.0500 time 0.3787 (0.3847) data time 0.0008 (0.0022) model time 0.3779 (0.3834) loss 2.9677 (3.8179) grad_norm 1.2077 (1.2594) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][360/625] eta 0:01:41 lr 0.001989 wd 0.0500 time 0.3747 (0.3845) data time 0.0008 (0.0022) model time 0.3739 (0.3832) loss 4.1777 (3.8245) grad_norm 1.7333 (1.2600) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][370/625] eta 0:01:37 lr 0.001989 wd 0.0500 time 0.3758 (0.3843) data time 0.0010 (0.0021) model time 0.3747 (0.3829) loss 4.1445 (3.8278) grad_norm 1.1715 (1.2589) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][380/625] eta 0:01:34 lr 0.001988 wd 0.0500 time 0.3769 (0.3841) data time 0.0009 (0.0021) model time 0.3760 (0.3828) loss 3.0205 (3.8230) grad_norm 1.2692 (1.2571) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:09:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][390/625] eta 0:01:30 lr 0.001988 wd 0.0500 time 0.3754 (0.3839) data time 0.0011 (0.0021) model time 0.3743 (0.3826) loss 3.2697 (3.8228) grad_norm 1.5462 (1.2552) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][400/625] eta 0:01:26 lr 0.001988 wd 0.0500 time 0.3709 (0.3841) data time 0.0011 (0.0021) model time 0.3699 (0.3828) loss 3.7173 (3.8171) grad_norm 0.9117 (1.2516) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][410/625] eta 0:01:22 lr 0.001988 wd 0.0500 time 0.3793 (0.3839) data time 0.0009 (0.0020) model time 0.3784 (0.3826) loss 2.7447 (3.8109) grad_norm 1.0124 (1.2497) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][420/625] eta 0:01:18 lr 0.001988 wd 0.0500 time 0.3771 (0.3837) data time 0.0011 (0.0020) model time 0.3760 (0.3823) loss 3.5497 (3.8113) grad_norm 1.3853 (1.2535) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][430/625] eta 0:01:14 lr 0.001988 wd 0.0500 time 0.3749 (0.3835) data time 0.0014 (0.0020) model time 0.3736 (0.3821) loss 3.7748 (3.8117) grad_norm 1.2525 (1.2582) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][440/625] eta 0:01:10 lr 0.001988 wd 0.0500 time 0.3774 (0.3834) data time 0.0009 (0.0020) model time 0.3766 (0.3820) loss 4.2578 (3.8131) grad_norm 1.5079 (1.2570) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][450/625] eta 0:01:07 lr 0.001988 wd 0.0500 time 0.3853 (0.3833) data time 0.0009 (0.0020) model time 0.3844 (0.3819) loss 3.4934 (3.8036) grad_norm 1.3790 (1.2602) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][460/625] eta 0:01:03 lr 0.001988 wd 0.0500 time 0.5712 (0.3841) data time 0.0008 (0.0019) model time 0.5704 (0.3828) loss 4.7921 (3.8045) grad_norm 0.8451 (1.2584) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][470/625] eta 0:00:59 lr 0.001988 wd 0.0500 time 0.3770 (0.3844) data time 0.0011 (0.0019) model time 0.3759 (0.3831) loss 4.0174 (3.8041) grad_norm 1.3245 (1.2589) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][480/625] eta 0:00:55 lr 0.001988 wd 0.0500 time 0.3777 (0.3842) data time 0.0009 (0.0019) model time 0.3769 (0.3830) loss 3.2491 (3.8044) grad_norm 0.8890 (1.2580) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][490/625] eta 0:00:51 lr 0.001988 wd 0.0500 time 0.3825 (0.3850) data time 0.0011 (0.0019) model time 0.3815 (0.3838) loss 2.9987 (3.8038) grad_norm 1.0105 (1.2575) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][500/625] eta 0:00:48 lr 0.001988 wd 0.0500 time 0.3737 (0.3849) data time 0.0008 (0.0019) model time 0.3729 (0.3837) loss 2.7831 (3.8055) grad_norm 0.9814 (1.2550) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][510/625] eta 0:00:44 lr 0.001988 wd 0.0500 time 0.3822 (0.3848) data time 0.0011 (0.0019) model time 0.3811 (0.3836) loss 3.3834 (3.8033) grad_norm 1.4348 (1.2551) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][520/625] eta 0:00:40 lr 0.001988 wd 0.0500 time 0.3760 (0.3846) data time 0.0011 (0.0019) model time 0.3749 (0.3834) loss 3.8886 (3.8052) grad_norm 1.0281 (1.2528) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][530/625] eta 0:00:36 lr 0.001988 wd 0.0500 time 0.3728 (0.3845) data time 0.0016 (0.0018) model time 0.3713 (0.3833) loss 4.2367 (3.8060) grad_norm 1.3851 (1.2528) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][540/625] eta 0:00:32 lr 0.001988 wd 0.0500 time 0.3755 (0.3844) data time 0.0008 (0.0018) model time 0.3747 (0.3831) loss 2.5648 (3.7969) grad_norm 1.5018 (1.2532) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:10:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][550/625] eta 0:00:28 lr 0.001988 wd 0.0500 time 0.3728 (0.3842) data time 0.0011 (0.0018) model time 0.3717 (0.3830) loss 3.7883 (3.7943) grad_norm 0.9736 (1.2563) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][560/625] eta 0:00:24 lr 0.001988 wd 0.0500 time 0.3900 (0.3841) data time 0.0009 (0.0018) model time 0.3891 (0.3828) loss 4.8704 (3.7920) grad_norm 1.0282 (1.2554) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][570/625] eta 0:00:21 lr 0.001988 wd 0.0500 time 0.3881 (0.3840) data time 0.0008 (0.0018) model time 0.3872 (0.3827) loss 2.6760 (3.7889) grad_norm 1.3302 (1.2558) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][580/625] eta 0:00:17 lr 0.001988 wd 0.0500 time 0.3780 (0.3839) data time 0.0011 (0.0018) model time 0.3769 (0.3826) loss 3.8813 (3.7858) grad_norm 1.4756 (1.2558) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][590/625] eta 0:00:13 lr 0.001988 wd 0.0500 time 0.3705 (0.3840) data time 0.0011 (0.0018) model time 0.3694 (0.3827) loss 3.8991 (3.7863) grad_norm 0.9763 (1.2533) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][600/625] eta 0:00:09 lr 0.001988 wd 0.0500 time 0.3740 (0.3839) data time 0.0011 (0.0018) model time 0.3729 (0.3826) loss 4.0911 (3.7864) grad_norm 1.3919 (1.2578) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][610/625] eta 0:00:05 lr 0.001988 wd 0.0500 time 0.3626 (0.3844) data time 0.0005 (0.0018) model time 0.3620 (0.3831) loss 2.8378 (3.7854) grad_norm 1.0899 (1.2606) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [33/300][620/625] eta 0:00:01 lr 0.001988 wd 0.0500 time 0.3604 (0.3841) data time 0.0008 (0.0018) model time 0.3596 (0.3828) loss 3.8578 (3.7833) grad_norm 1.6943 (1.2657) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 33 training takes 0:03:59 [2024-08-26 05:11:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:11:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:11:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.500 (0.500) Loss 0.8291 (0.8291) Acc@1 83.740 (83.740) Acc@5 96.289 (96.289) Mem 11225MB [2024-08-26 05:11:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 1.3428 (0.9954) Acc@1 69.971 (77.464) Acc@5 89.746 (94.496) Mem 11225MB [2024-08-26 05:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.5557 (1.2295) Acc@1 65.430 (72.363) Acc@5 87.451 (91.378) Mem 11225MB [2024-08-26 05:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 72.203 Acc@5 91.339 [2024-08-26 05:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 72.2% [2024-08-26 05:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 72.20% [2024-08-26 05:11:31 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:11:32 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:11:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 0.6685 (0.6685) Acc@1 82.764 (82.764) Acc@5 96.240 (96.240) Mem 11225MB [2024-08-26 05:11:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.141) Loss 1.2275 (0.8544) Acc@1 68.262 (77.579) Acc@5 90.381 (94.624) Mem 11225MB [2024-08-26 05:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 1.4287 (1.0798) Acc@1 65.430 (72.805) Acc@5 87.500 (91.641) Mem 11225MB [2024-08-26 05:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 72.641 Acc@5 91.637 [2024-08-26 05:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 72.6% [2024-08-26 05:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 72.64% [2024-08-26 05:11:35 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:11:36 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:11:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][0/625] eta 0:07:43 lr 0.001988 wd 0.0500 time 0.7413 (0.7413) data time 0.3907 (0.3907) model time 0.0000 (0.0000) loss 4.6550 (4.6550) grad_norm 0.9202 (0.9202) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][10/625] eta 0:04:11 lr 0.001988 wd 0.0500 time 0.3824 (0.4097) data time 0.0010 (0.0365) model time 0.0000 (0.0000) loss 4.1164 (3.9999) grad_norm 1.3023 (1.1575) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][20/625] eta 0:03:59 lr 0.001988 wd 0.0500 time 0.3736 (0.3955) data time 0.0008 (0.0197) model time 0.0000 (0.0000) loss 2.3551 (3.8043) grad_norm 1.4316 (1.3305) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][30/625] eta 0:04:04 lr 0.001988 wd 0.0500 time 0.3700 (0.4102) data time 0.0009 (0.0137) model time 0.0000 (0.0000) loss 3.9679 (3.7412) grad_norm 1.1819 (1.3597) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][40/625] eta 0:03:58 lr 0.001988 wd 0.0500 time 0.3815 (0.4073) data time 0.0010 (0.0106) model time 0.0000 (0.0000) loss 4.3061 (3.7338) grad_norm 1.9158 (1.3027) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:11:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][50/625] eta 0:03:50 lr 0.001988 wd 0.0500 time 0.3770 (0.4015) data time 0.0009 (0.0088) model time 0.0000 (0.0000) loss 4.7212 (3.8069) grad_norm 0.8012 (1.2780) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][60/625] eta 0:03:44 lr 0.001988 wd 0.0500 time 0.3839 (0.3980) data time 0.0008 (0.0075) model time 0.3830 (0.3787) loss 3.8761 (3.7415) grad_norm 0.9858 (1.2682) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][70/625] eta 0:03:39 lr 0.001988 wd 0.0500 time 0.3780 (0.3952) data time 0.0009 (0.0066) model time 0.3771 (0.3779) loss 4.3548 (3.7407) grad_norm 1.0604 (1.2668) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][80/625] eta 0:03:34 lr 0.001988 wd 0.0500 time 0.3808 (0.3932) data time 0.0009 (0.0059) model time 0.3799 (0.3780) loss 3.2813 (3.7421) grad_norm 1.8651 (1.2653) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][90/625] eta 0:03:29 lr 0.001988 wd 0.0500 time 0.3970 (0.3920) data time 0.0011 (0.0054) model time 0.3958 (0.3788) loss 3.8958 (3.7328) grad_norm 1.3147 (1.2513) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][100/625] eta 0:03:25 lr 0.001988 wd 0.0500 time 0.3728 (0.3905) data time 0.0011 (0.0050) model time 0.3717 (0.3782) loss 3.3686 (3.7593) grad_norm 1.1111 (1.2506) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][110/625] eta 0:03:20 lr 0.001988 wd 0.0500 time 0.3736 (0.3890) data time 0.0009 (0.0046) model time 0.3728 (0.3773) loss 3.2246 (3.7318) grad_norm 1.3810 (1.2529) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][120/625] eta 0:03:15 lr 0.001987 wd 0.0500 time 0.3797 (0.3880) data time 0.0009 (0.0044) model time 0.3788 (0.3770) loss 3.3315 (3.7447) grad_norm 1.0068 (1.2493) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][130/625] eta 0:03:11 lr 0.001987 wd 0.0500 time 0.3647 (0.3873) data time 0.0012 (0.0041) model time 0.3635 (0.3771) loss 3.8367 (3.7435) grad_norm 1.4857 (1.2498) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][140/625] eta 0:03:07 lr 0.001987 wd 0.0500 time 0.3814 (0.3867) data time 0.0011 (0.0039) model time 0.3803 (0.3771) loss 2.7262 (3.7394) grad_norm 1.2156 (1.2529) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][150/625] eta 0:03:03 lr 0.001987 wd 0.0500 time 0.3756 (0.3860) data time 0.0011 (0.0038) model time 0.3745 (0.3769) loss 3.7433 (3.7489) grad_norm 1.1405 (1.2460) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][160/625] eta 0:02:59 lr 0.001987 wd 0.0500 time 0.3757 (0.3855) data time 0.0009 (0.0036) model time 0.3748 (0.3769) loss 4.6348 (3.7466) grad_norm 1.0076 (1.2413) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][170/625] eta 0:02:56 lr 0.001987 wd 0.0500 time 0.3749 (0.3873) data time 0.0012 (0.0034) model time 0.3736 (0.3801) loss 3.6647 (3.7566) grad_norm 1.1992 (1.2400) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][180/625] eta 0:02:52 lr 0.001987 wd 0.0500 time 0.3755 (0.3868) data time 0.0010 (0.0033) model time 0.3745 (0.3798) loss 4.0375 (3.7516) grad_norm 1.0903 (1.2498) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][190/625] eta 0:02:48 lr 0.001987 wd 0.0500 time 0.3797 (0.3863) data time 0.0011 (0.0032) model time 0.3786 (0.3796) loss 3.8854 (3.7444) grad_norm 1.6621 (1.2553) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][200/625] eta 0:02:44 lr 0.001987 wd 0.0500 time 0.3716 (0.3859) data time 0.0009 (0.0031) model time 0.3707 (0.3794) loss 4.9847 (3.7637) grad_norm 1.7036 (1.2643) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:12:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][210/625] eta 0:02:40 lr 0.001987 wd 0.0500 time 0.3818 (0.3856) data time 0.0011 (0.0030) model time 0.3808 (0.3793) loss 3.4947 (3.7544) grad_norm 1.3833 (1.2604) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][220/625] eta 0:02:36 lr 0.001987 wd 0.0500 time 0.3846 (0.3854) data time 0.0012 (0.0030) model time 0.3834 (0.3793) loss 3.1937 (3.7520) grad_norm 1.3123 (1.2680) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][230/625] eta 0:02:32 lr 0.001987 wd 0.0500 time 0.3760 (0.3850) data time 0.0008 (0.0029) model time 0.3751 (0.3792) loss 3.3683 (3.7521) grad_norm 1.0457 (1.2672) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][240/625] eta 0:02:28 lr 0.001987 wd 0.0500 time 0.3741 (0.3865) data time 0.0009 (0.0028) model time 0.3732 (0.3813) loss 2.9014 (3.7622) grad_norm 1.2488 (1.2635) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][250/625] eta 0:02:25 lr 0.001987 wd 0.0500 time 0.3843 (0.3869) data time 0.0011 (0.0027) model time 0.3832 (0.3819) loss 3.5052 (3.7623) grad_norm 1.0065 (1.2661) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][260/625] eta 0:02:21 lr 0.001987 wd 0.0500 time 0.3778 (0.3865) data time 0.0008 (0.0027) model time 0.3770 (0.3817) loss 2.5260 (3.7646) grad_norm 1.5521 (1.2659) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][270/625] eta 0:02:17 lr 0.001987 wd 0.0500 time 0.4003 (0.3863) data time 0.0011 (0.0026) model time 0.3992 (0.3816) loss 4.1665 (3.7745) grad_norm 1.1425 (1.2611) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][280/625] eta 0:02:13 lr 0.001987 wd 0.0500 time 0.3783 (0.3860) data time 0.0011 (0.0026) model time 0.3772 (0.3813) loss 3.7834 (3.7728) grad_norm 1.6153 (1.2608) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][290/625] eta 0:02:09 lr 0.001987 wd 0.0500 time 0.3738 (0.3872) data time 0.0009 (0.0026) model time 0.3729 (0.3829) loss 3.1251 (3.7606) grad_norm 0.9538 (1.2567) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][300/625] eta 0:02:05 lr 0.001987 wd 0.0500 time 0.3731 (0.3869) data time 0.0010 (0.0025) model time 0.3721 (0.3827) loss 3.3476 (3.7495) grad_norm 1.4934 (1.2559) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][310/625] eta 0:02:01 lr 0.001987 wd 0.0500 time 0.3770 (0.3866) data time 0.0009 (0.0025) model time 0.3761 (0.3824) loss 4.4230 (3.7536) grad_norm 1.1582 (1.2530) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][320/625] eta 0:01:57 lr 0.001987 wd 0.0500 time 0.3783 (0.3863) data time 0.0011 (0.0025) model time 0.3772 (0.3822) loss 4.0845 (3.7494) grad_norm 1.5494 (1.2522) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][330/625] eta 0:01:53 lr 0.001987 wd 0.0500 time 0.3738 (0.3860) data time 0.0011 (0.0024) model time 0.3728 (0.3820) loss 4.2111 (3.7499) grad_norm 1.3060 (1.2557) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][340/625] eta 0:01:49 lr 0.001987 wd 0.0500 time 0.3777 (0.3858) data time 0.0011 (0.0024) model time 0.3766 (0.3818) loss 3.6483 (3.7436) grad_norm 1.3187 (1.2563) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][350/625] eta 0:01:46 lr 0.001987 wd 0.0500 time 0.3736 (0.3857) data time 0.0010 (0.0023) model time 0.3726 (0.3817) loss 4.0159 (3.7394) grad_norm 1.0042 (1.2560) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][360/625] eta 0:01:42 lr 0.001987 wd 0.0500 time 0.3848 (0.3855) data time 0.0009 (0.0023) model time 0.3839 (0.3816) loss 4.5164 (3.7436) grad_norm 1.3954 (1.2584) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][370/625] eta 0:01:38 lr 0.001987 wd 0.0500 time 0.3801 (0.3853) data time 0.0008 (0.0023) model time 0.3793 (0.3814) loss 3.7503 (3.7386) grad_norm 1.1369 (1.2556) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][380/625] eta 0:01:34 lr 0.001987 wd 0.0500 time 0.3764 (0.3851) data time 0.0009 (0.0023) model time 0.3755 (0.3812) loss 2.4001 (3.7297) grad_norm 1.7114 (1.2572) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][390/625] eta 0:01:30 lr 0.001987 wd 0.0500 time 0.3730 (0.3850) data time 0.0012 (0.0022) model time 0.3718 (0.3812) loss 4.1104 (3.7226) grad_norm 1.6013 (1.2610) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][400/625] eta 0:01:26 lr 0.001987 wd 0.0500 time 0.3768 (0.3848) data time 0.0009 (0.0022) model time 0.3760 (0.3811) loss 2.9438 (3.7188) grad_norm 1.2908 (1.2616) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][410/625] eta 0:01:22 lr 0.001987 wd 0.0500 time 0.3840 (0.3847) data time 0.0011 (0.0022) model time 0.3829 (0.3811) loss 4.1915 (3.7156) grad_norm 1.2233 (1.2600) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][420/625] eta 0:01:19 lr 0.001987 wd 0.0500 time 0.5904 (0.3857) data time 0.0008 (0.0022) model time 0.5895 (0.3822) loss 3.1084 (3.7115) grad_norm 0.9408 (1.2613) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][430/625] eta 0:01:15 lr 0.001987 wd 0.0500 time 0.3741 (0.3860) data time 0.0010 (0.0021) model time 0.3731 (0.3826) loss 3.7566 (3.7129) grad_norm 1.1812 (1.2554) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][440/625] eta 0:01:11 lr 0.001987 wd 0.0500 time 0.3815 (0.3858) data time 0.0010 (0.0021) model time 0.3805 (0.3825) loss 2.9471 (3.7096) grad_norm 2.2411 (1.2553) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][450/625] eta 0:01:07 lr 0.001987 wd 0.0500 time 0.3791 (0.3857) data time 0.0012 (0.0021) model time 0.3780 (0.3824) loss 3.3692 (3.7124) grad_norm 1.4816 (1.2602) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][460/625] eta 0:01:03 lr 0.001986 wd 0.0500 time 0.3817 (0.3855) data time 0.0010 (0.0021) model time 0.3807 (0.3823) loss 4.3001 (3.7169) grad_norm 0.9240 (1.2589) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][470/625] eta 0:00:59 lr 0.001986 wd 0.0500 time 0.3747 (0.3853) data time 0.0011 (0.0021) model time 0.3736 (0.3821) loss 2.9575 (3.7107) grad_norm 1.3573 (1.2539) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][480/625] eta 0:00:55 lr 0.001986 wd 0.0500 time 0.3722 (0.3852) data time 0.0011 (0.0020) model time 0.3711 (0.3820) loss 2.9302 (3.7134) grad_norm 1.2357 (1.2528) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][490/625] eta 0:00:51 lr 0.001986 wd 0.0500 time 0.3826 (0.3851) data time 0.0008 (0.0020) model time 0.3818 (0.3820) loss 3.3818 (3.7157) grad_norm 1.2831 (1.2567) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][500/625] eta 0:00:48 lr 0.001986 wd 0.0500 time 0.3806 (0.3859) data time 0.0012 (0.0020) model time 0.3794 (0.3828) loss 3.3605 (3.7204) grad_norm 1.7338 (1.2560) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][510/625] eta 0:00:44 lr 0.001986 wd 0.0500 time 0.3767 (0.3857) data time 0.0011 (0.0020) model time 0.3756 (0.3828) loss 3.7693 (3.7191) grad_norm 1.5213 (1.2579) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:14:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][520/625] eta 0:00:40 lr 0.001986 wd 0.0500 time 0.3741 (0.3857) data time 0.0010 (0.0020) model time 0.3731 (0.3827) loss 4.0193 (3.7223) grad_norm 0.9756 (1.2546) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][530/625] eta 0:00:36 lr 0.001986 wd 0.0500 time 0.3824 (0.3855) data time 0.0011 (0.0020) model time 0.3813 (0.3826) loss 3.7944 (3.7175) grad_norm 1.4486 (1.2556) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][540/625] eta 0:00:32 lr 0.001986 wd 0.0500 time 0.3799 (0.3861) data time 0.0008 (0.0019) model time 0.3790 (0.3833) loss 3.1039 (3.7239) grad_norm 1.3928 (1.2562) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][550/625] eta 0:00:28 lr 0.001986 wd 0.0500 time 0.3754 (0.3863) data time 0.0011 (0.0019) model time 0.3743 (0.3835) loss 3.7064 (3.7183) grad_norm 1.5185 (1.2584) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][560/625] eta 0:00:25 lr 0.001986 wd 0.0500 time 0.3766 (0.3863) data time 0.0010 (0.0019) model time 0.3756 (0.3835) loss 3.5079 (3.7155) grad_norm 1.7499 (1.2580) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][570/625] eta 0:00:21 lr 0.001986 wd 0.0500 time 0.3778 (0.3862) data time 0.0009 (0.0019) model time 0.3769 (0.3834) loss 2.9649 (3.7107) grad_norm 0.9342 (1.2548) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][580/625] eta 0:00:17 lr 0.001986 wd 0.0500 time 0.3810 (0.3861) data time 0.0008 (0.0019) model time 0.3802 (0.3833) loss 3.6001 (3.7122) grad_norm 1.4373 (1.2527) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][590/625] eta 0:00:13 lr 0.001986 wd 0.0500 time 0.3729 (0.3859) data time 0.0010 (0.0019) model time 0.3719 (0.3832) loss 3.5669 (3.7106) grad_norm 0.8119 (1.2535) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][600/625] eta 0:00:09 lr 0.001986 wd 0.0500 time 0.3754 (0.3858) data time 0.0011 (0.0019) model time 0.3743 (0.3831) loss 4.1515 (3.7101) grad_norm 1.8129 (1.2541) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][610/625] eta 0:00:05 lr 0.001986 wd 0.0500 time 0.3663 (0.3856) data time 0.0008 (0.0019) model time 0.3655 (0.3830) loss 3.4290 (3.7118) grad_norm 1.2661 (1.2516) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [34/300][620/625] eta 0:00:01 lr 0.001986 wd 0.0500 time 0.3655 (0.3854) data time 0.0006 (0.0018) model time 0.3649 (0.3827) loss 3.3912 (3.7100) grad_norm 1.2457 (1.2511) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 34 training takes 0:04:00 [2024-08-26 05:15:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:15:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:15:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 0.8140 (0.8140) Acc@1 83.545 (83.545) Acc@5 96.533 (96.533) Mem 11225MB [2024-08-26 05:15:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.3643 (1.0143) Acc@1 68.945 (77.344) Acc@5 89.551 (94.447) Mem 11225MB [2024-08-26 05:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.5508 (1.2206) Acc@1 64.746 (72.791) Acc@5 87.109 (91.560) Mem 11225MB [2024-08-26 05:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 72.605 Acc@5 91.561 [2024-08-26 05:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 72.6% [2024-08-26 05:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 72.60% [2024-08-26 05:15:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:15:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:15:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.500 (0.500) Loss 0.6543 (0.6543) Acc@1 83.252 (83.252) Acc@5 96.484 (96.484) Mem 11225MB [2024-08-26 05:15:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.2021 (0.8392) Acc@1 68.945 (78.041) Acc@5 90.430 (94.802) Mem 11225MB [2024-08-26 05:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.4053 (1.0608) Acc@1 66.016 (73.300) Acc@5 87.744 (91.897) Mem 11225MB [2024-08-26 05:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.101 Acc@5 91.875 [2024-08-26 05:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 73.1% [2024-08-26 05:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 73.10% [2024-08-26 05:15:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:15:46 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:15:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][0/625] eta 0:08:03 lr 0.001986 wd 0.0500 time 0.7739 (0.7739) data time 0.4348 (0.4348) model time 0.0000 (0.0000) loss 3.2908 (3.2908) grad_norm 3.4722 (3.4722) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][10/625] eta 0:04:14 lr 0.001986 wd 0.0500 time 0.3730 (0.4146) data time 0.0009 (0.0405) model time 0.0000 (0.0000) loss 3.2337 (3.4850) grad_norm 1.2803 (1.3112) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][20/625] eta 0:04:00 lr 0.001986 wd 0.0500 time 0.3833 (0.3978) data time 0.0008 (0.0218) model time 0.0000 (0.0000) loss 2.9292 (3.6144) grad_norm 1.0553 (1.4184) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:15:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][30/625] eta 0:03:52 lr 0.001986 wd 0.0500 time 0.3745 (0.3906) data time 0.0008 (0.0152) model time 0.0000 (0.0000) loss 3.1416 (3.5815) grad_norm 1.0240 (1.3243) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][40/625] eta 0:03:49 lr 0.001986 wd 0.0500 time 0.3889 (0.3929) data time 0.0012 (0.0117) model time 0.0000 (0.0000) loss 3.6246 (3.6674) grad_norm 1.0385 (1.3026) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][50/625] eta 0:03:44 lr 0.001986 wd 0.0500 time 0.3829 (0.3898) data time 0.0011 (0.0097) model time 0.0000 (0.0000) loss 3.7843 (3.6924) grad_norm 1.1023 (1.2898) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][60/625] eta 0:03:40 lr 0.001986 wd 0.0500 time 0.3730 (0.3911) data time 0.0008 (0.0083) model time 0.3722 (0.3965) loss 4.3687 (3.6950) grad_norm 1.0746 (1.2551) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][70/625] eta 0:03:41 lr 0.001986 wd 0.0500 time 0.5785 (0.3983) data time 0.0009 (0.0073) model time 0.5777 (0.4190) loss 2.8082 (3.6979) grad_norm 1.4620 (1.2678) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][80/625] eta 0:03:35 lr 0.001986 wd 0.0500 time 0.3713 (0.3960) data time 0.0008 (0.0065) model time 0.3705 (0.4054) loss 3.9741 (3.7159) grad_norm 1.3616 (1.2655) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][90/625] eta 0:03:30 lr 0.001986 wd 0.0500 time 0.3778 (0.3941) data time 0.0008 (0.0059) model time 0.3770 (0.3985) loss 3.9083 (3.7393) grad_norm 1.3335 (1.2878) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][100/625] eta 0:03:26 lr 0.001986 wd 0.0500 time 0.3796 (0.3927) data time 0.0009 (0.0055) model time 0.3787 (0.3945) loss 4.6651 (3.7642) grad_norm 1.5707 (1.2878) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][110/625] eta 0:03:21 lr 0.001986 wd 0.0500 time 0.3782 (0.3913) data time 0.0011 (0.0051) model time 0.3770 (0.3914) loss 3.4594 (3.7456) grad_norm 1.0097 (1.2788) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][120/625] eta 0:03:17 lr 0.001986 wd 0.0500 time 0.3740 (0.3902) data time 0.0013 (0.0047) model time 0.3728 (0.3893) loss 4.3402 (3.7617) grad_norm 1.1066 (1.2702) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][130/625] eta 0:03:12 lr 0.001986 wd 0.0500 time 0.3758 (0.3892) data time 0.0009 (0.0045) model time 0.3749 (0.3877) loss 4.0227 (3.7527) grad_norm 0.8346 (1.2627) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:16:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][140/625] eta 0:03:08 lr 0.001986 wd 0.0500 time 0.3752 (0.3885) data time 0.0012 (0.0042) model time 0.3740 (0.3866) loss 3.1336 (3.7373) grad_norm 0.9579 (1.2576) loss_scale 32768.0000 (17429.7872) mem 11225MB [2024-08-26 05:16:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][150/625] eta 0:03:04 lr 0.001986 wd 0.0500 time 0.3734 (0.3878) data time 0.0008 (0.0040) model time 0.3725 (0.3856) loss 4.3064 (3.7640) grad_norm 0.8198 (1.2553) loss_scale 32768.0000 (18445.5629) mem 11225MB [2024-08-26 05:16:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][160/625] eta 0:03:00 lr 0.001986 wd 0.0500 time 0.3838 (0.3873) data time 0.0010 (0.0039) model time 0.3828 (0.3850) loss 4.1934 (3.7699) grad_norm 1.5042 (1.2697) loss_scale 32768.0000 (19335.1553) mem 11225MB [2024-08-26 05:16:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][170/625] eta 0:02:55 lr 0.001986 wd 0.0500 time 0.3787 (0.3868) data time 0.0011 (0.0037) model time 0.3777 (0.3844) loss 3.1726 (3.7616) grad_norm 1.1874 (1.2763) loss_scale 32768.0000 (20120.7018) mem 11225MB [2024-08-26 05:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][180/625] eta 0:02:51 lr 0.001985 wd 0.0500 time 0.3819 (0.3863) data time 0.0008 (0.0036) model time 0.3811 (0.3837) loss 2.5762 (3.7549) grad_norm 1.3053 (1.2707) loss_scale 32768.0000 (20819.4475) mem 11225MB [2024-08-26 05:17:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][190/625] eta 0:02:47 lr 0.001985 wd 0.0500 time 0.3842 (0.3858) data time 0.0011 (0.0034) model time 0.3831 (0.3832) loss 3.8764 (3.7555) grad_norm 0.9335 (1.2581) loss_scale 32768.0000 (21445.0262) mem 11225MB [2024-08-26 05:17:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][200/625] eta 0:02:43 lr 0.001985 wd 0.0500 time 0.3739 (0.3854) data time 0.0011 (0.0033) model time 0.3728 (0.3827) loss 4.2063 (3.7613) grad_norm 1.8675 (1.2696) loss_scale 32768.0000 (22008.3582) mem 11225MB [2024-08-26 05:17:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][210/625] eta 0:02:39 lr 0.001985 wd 0.0500 time 0.3737 (0.3851) data time 0.0011 (0.0032) model time 0.3726 (0.3824) loss 4.0180 (3.7583) grad_norm 1.0546 (1.2618) loss_scale 32768.0000 (22518.2938) mem 11225MB [2024-08-26 05:17:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][220/625] eta 0:02:36 lr 0.001985 wd 0.0500 time 0.3362 (0.3858) data time 0.0011 (0.0031) model time 0.3351 (0.3834) loss 4.3362 (3.7673) grad_norm 1.0395 (1.2590) loss_scale 32768.0000 (22982.0814) mem 11225MB [2024-08-26 05:17:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][230/625] eta 0:02:32 lr 0.001985 wd 0.0500 time 0.3802 (0.3855) data time 0.0008 (0.0030) model time 0.3793 (0.3831) loss 3.7904 (3.7655) grad_norm 1.0175 (1.2594) loss_scale 32768.0000 (23405.7143) mem 11225MB [2024-08-26 05:17:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][240/625] eta 0:02:28 lr 0.001985 wd 0.0500 time 0.3844 (0.3852) data time 0.0010 (0.0030) model time 0.3833 (0.3829) loss 4.2570 (3.7739) grad_norm 0.9765 (1.2655) loss_scale 32768.0000 (23794.1909) mem 11225MB [2024-08-26 05:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][250/625] eta 0:02:25 lr 0.001985 wd 0.0500 time 0.3765 (0.3867) data time 0.0011 (0.0029) model time 0.3754 (0.3848) loss 3.8970 (3.7848) grad_norm 0.8799 (1.2650) loss_scale 32768.0000 (24151.7131) mem 11225MB [2024-08-26 05:17:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][260/625] eta 0:02:21 lr 0.001985 wd 0.0500 time 0.3698 (0.3864) data time 0.0011 (0.0028) model time 0.3687 (0.3844) loss 3.5427 (3.7808) grad_norm 1.2292 (1.2606) loss_scale 32768.0000 (24481.8391) mem 11225MB [2024-08-26 05:17:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][270/625] eta 0:02:17 lr 0.001985 wd 0.0500 time 0.3818 (0.3862) data time 0.0008 (0.0028) model time 0.3811 (0.3842) loss 4.4537 (3.7842) grad_norm 1.2789 (1.2571) loss_scale 32768.0000 (24787.6015) mem 11225MB [2024-08-26 05:17:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][280/625] eta 0:02:13 lr 0.001985 wd 0.0500 time 0.3803 (0.3867) data time 0.0011 (0.0027) model time 0.3792 (0.3848) loss 4.0938 (3.7799) grad_norm 1.3272 (1.2553) loss_scale 32768.0000 (25071.6014) mem 11225MB [2024-08-26 05:17:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][290/625] eta 0:02:09 lr 0.001985 wd 0.0500 time 0.3743 (0.3871) data time 0.0008 (0.0027) model time 0.3735 (0.3854) loss 2.8044 (3.7859) grad_norm 1.0190 (1.2522) loss_scale 32768.0000 (25336.0825) mem 11225MB [2024-08-26 05:17:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][300/625] eta 0:02:05 lr 0.001985 wd 0.0500 time 0.3770 (0.3868) data time 0.0011 (0.0026) model time 0.3759 (0.3850) loss 4.0934 (3.7924) grad_norm 0.8307 (1.2489) loss_scale 32768.0000 (25582.9900) mem 11225MB [2024-08-26 05:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][310/625] eta 0:02:01 lr 0.001985 wd 0.0500 time 0.3788 (0.3865) data time 0.0011 (0.0026) model time 0.3777 (0.3847) loss 4.0249 (3.7860) grad_norm 1.4299 (1.2553) loss_scale 32768.0000 (25814.0193) mem 11225MB [2024-08-26 05:17:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][320/625] eta 0:01:57 lr 0.001985 wd 0.0500 time 0.3795 (0.3863) data time 0.0011 (0.0025) model time 0.3784 (0.3845) loss 4.0631 (3.7818) grad_norm 1.1132 (1.2536) loss_scale 32768.0000 (26030.6542) mem 11225MB [2024-08-26 05:17:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][330/625] eta 0:01:53 lr 0.001985 wd 0.0500 time 0.3745 (0.3861) data time 0.0008 (0.0025) model time 0.3738 (0.3842) loss 4.0883 (3.7818) grad_norm 1.2878 (1.2498) loss_scale 32768.0000 (26234.1994) mem 11225MB [2024-08-26 05:17:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][340/625] eta 0:01:50 lr 0.001985 wd 0.0500 time 0.3909 (0.3864) data time 0.0015 (0.0024) model time 0.3895 (0.3847) loss 3.8771 (3.7838) grad_norm 1.6030 (1.2462) loss_scale 32768.0000 (26425.8065) mem 11225MB [2024-08-26 05:18:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][350/625] eta 0:01:46 lr 0.001985 wd 0.0500 time 0.3808 (0.3862) data time 0.0009 (0.0024) model time 0.3799 (0.3845) loss 4.1885 (3.7803) grad_norm 1.0193 (1.2457) loss_scale 32768.0000 (26606.4957) mem 11225MB [2024-08-26 05:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][360/625] eta 0:01:42 lr 0.001985 wd 0.0500 time 0.3820 (0.3860) data time 0.0011 (0.0024) model time 0.3809 (0.3843) loss 4.1255 (3.7731) grad_norm 1.2259 (1.2458) loss_scale 32768.0000 (26777.1745) mem 11225MB [2024-08-26 05:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][370/625] eta 0:01:38 lr 0.001985 wd 0.0500 time 0.3818 (0.3869) data time 0.0008 (0.0023) model time 0.3810 (0.3853) loss 4.2495 (3.7679) grad_norm 0.9529 (1.2481) loss_scale 32768.0000 (26938.6523) mem 11225MB [2024-08-26 05:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][380/625] eta 0:01:34 lr 0.001985 wd 0.0500 time 0.3810 (0.3867) data time 0.0009 (0.0023) model time 0.3801 (0.3851) loss 4.9792 (3.7645) grad_norm 1.1392 (1.2499) loss_scale 32768.0000 (27091.6535) mem 11225MB [2024-08-26 05:18:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][390/625] eta 0:01:30 lr 0.001985 wd 0.0500 time 0.3760 (0.3865) data time 0.0011 (0.0023) model time 0.3749 (0.3849) loss 3.2581 (3.7697) grad_norm 1.0534 (1.2512) loss_scale 32768.0000 (27236.8286) mem 11225MB [2024-08-26 05:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][400/625] eta 0:01:26 lr 0.001985 wd 0.0500 time 0.3865 (0.3864) data time 0.0011 (0.0022) model time 0.3854 (0.3847) loss 3.7727 (3.7734) grad_norm 0.9533 (1.2549) loss_scale 32768.0000 (27374.7631) mem 11225MB [2024-08-26 05:18:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][410/625] eta 0:01:23 lr 0.001985 wd 0.0500 time 0.3794 (0.3861) data time 0.0009 (0.0022) model time 0.3785 (0.3845) loss 2.6933 (3.7696) grad_norm 1.1780 (inf) loss_scale 16384.0000 (27346.5304) mem 11225MB [2024-08-26 05:18:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][420/625] eta 0:01:19 lr 0.001985 wd 0.0500 time 0.3750 (0.3860) data time 0.0012 (0.0022) model time 0.3737 (0.3843) loss 3.3938 (3.7725) grad_norm 1.3966 (inf) loss_scale 16384.0000 (27086.1378) mem 11225MB [2024-08-26 05:18:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][430/625] eta 0:01:15 lr 0.001985 wd 0.0500 time 0.3810 (0.3857) data time 0.0008 (0.0022) model time 0.3801 (0.3840) loss 4.3485 (3.7701) grad_norm 1.0265 (inf) loss_scale 8192.0000 (26647.7587) mem 11225MB [2024-08-26 05:18:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][440/625] eta 0:01:11 lr 0.001985 wd 0.0500 time 0.3767 (0.3855) data time 0.0011 (0.0021) model time 0.3756 (0.3838) loss 3.1613 (3.7698) grad_norm 1.4966 (inf) loss_scale 8192.0000 (26229.2608) mem 11225MB [2024-08-26 05:18:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][450/625] eta 0:01:07 lr 0.001985 wd 0.0500 time 0.3797 (0.3854) data time 0.0011 (0.0021) model time 0.3786 (0.3837) loss 3.9345 (3.7679) grad_norm 1.3502 (inf) loss_scale 8192.0000 (25829.3215) mem 11225MB [2024-08-26 05:18:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][460/625] eta 0:01:03 lr 0.001985 wd 0.0500 time 0.5634 (0.3856) data time 0.0011 (0.0021) model time 0.5622 (0.3839) loss 3.3862 (3.7654) grad_norm 1.1345 (inf) loss_scale 8192.0000 (25446.7332) mem 11225MB [2024-08-26 05:18:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][470/625] eta 0:00:59 lr 0.001985 wd 0.0500 time 0.3732 (0.3862) data time 0.0011 (0.0021) model time 0.3720 (0.3846) loss 2.4185 (3.7647) grad_norm 1.0251 (inf) loss_scale 8192.0000 (25080.3907) mem 11225MB [2024-08-26 05:18:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][480/625] eta 0:00:55 lr 0.001985 wd 0.0500 time 0.3854 (0.3861) data time 0.0011 (0.0021) model time 0.3843 (0.3845) loss 3.8165 (3.7746) grad_norm 1.0375 (inf) loss_scale 8192.0000 (24729.2807) mem 11225MB [2024-08-26 05:18:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][490/625] eta 0:00:52 lr 0.001985 wd 0.0500 time 0.3716 (0.3859) data time 0.0009 (0.0020) model time 0.3707 (0.3843) loss 4.6467 (3.7778) grad_norm 1.3056 (inf) loss_scale 8192.0000 (24392.4725) mem 11225MB [2024-08-26 05:19:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][500/625] eta 0:00:48 lr 0.001984 wd 0.0500 time 0.3739 (0.3857) data time 0.0008 (0.0020) model time 0.3731 (0.3841) loss 2.9034 (3.7755) grad_norm 1.4033 (inf) loss_scale 8192.0000 (24069.1098) mem 11225MB [2024-08-26 05:19:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][510/625] eta 0:00:44 lr 0.001984 wd 0.0500 time 0.3850 (0.3856) data time 0.0011 (0.0020) model time 0.3839 (0.3839) loss 3.2575 (3.7752) grad_norm 1.1145 (inf) loss_scale 8192.0000 (23758.4031) mem 11225MB [2024-08-26 05:19:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][520/625] eta 0:00:40 lr 0.001984 wd 0.0500 time 0.3790 (0.3854) data time 0.0009 (0.0020) model time 0.3781 (0.3838) loss 3.9229 (3.7733) grad_norm 1.3355 (inf) loss_scale 8192.0000 (23459.6238) mem 11225MB [2024-08-26 05:19:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][530/625] eta 0:00:36 lr 0.001984 wd 0.0500 time 0.3784 (0.3853) data time 0.0011 (0.0020) model time 0.3773 (0.3836) loss 3.8796 (3.7736) grad_norm 1.0907 (inf) loss_scale 8192.0000 (23172.0979) mem 11225MB [2024-08-26 05:19:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][540/625] eta 0:00:32 lr 0.001984 wd 0.0500 time 0.3939 (0.3851) data time 0.0011 (0.0020) model time 0.3928 (0.3835) loss 3.8946 (3.7755) grad_norm 1.0279 (inf) loss_scale 8192.0000 (22895.2015) mem 11225MB [2024-08-26 05:19:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][550/625] eta 0:00:28 lr 0.001984 wd 0.0500 time 0.3755 (0.3854) data time 0.0011 (0.0020) model time 0.3745 (0.3838) loss 4.0736 (3.7730) grad_norm 0.9901 (inf) loss_scale 8192.0000 (22628.3557) mem 11225MB [2024-08-26 05:19:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][560/625] eta 0:00:25 lr 0.001984 wd 0.0500 time 0.3769 (0.3852) data time 0.0010 (0.0019) model time 0.3759 (0.3836) loss 2.8846 (3.7635) grad_norm 1.2001 (inf) loss_scale 8192.0000 (22371.0232) mem 11225MB [2024-08-26 05:19:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][570/625] eta 0:00:21 lr 0.001984 wd 0.0500 time 0.5895 (0.3855) data time 0.0009 (0.0019) model time 0.5887 (0.3839) loss 3.9042 (3.7632) grad_norm 1.0764 (inf) loss_scale 8192.0000 (22122.7040) mem 11225MB [2024-08-26 05:19:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][580/625] eta 0:00:17 lr 0.001984 wd 0.0500 time 0.5840 (0.3862) data time 0.0009 (0.0019) model time 0.5832 (0.3847) loss 4.1384 (3.7635) grad_norm 1.2808 (inf) loss_scale 8192.0000 (21882.9329) mem 11225MB [2024-08-26 05:19:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][590/625] eta 0:00:13 lr 0.001984 wd 0.0500 time 0.3735 (0.3864) data time 0.0010 (0.0019) model time 0.3725 (0.3849) loss 4.5223 (3.7677) grad_norm 0.9483 (inf) loss_scale 8192.0000 (21651.2758) mem 11225MB [2024-08-26 05:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][600/625] eta 0:00:09 lr 0.001984 wd 0.0500 time 0.3911 (0.3863) data time 0.0011 (0.0019) model time 0.3900 (0.3848) loss 4.0405 (3.7710) grad_norm 1.5116 (inf) loss_scale 8192.0000 (21427.3278) mem 11225MB [2024-08-26 05:19:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][610/625] eta 0:00:05 lr 0.001984 wd 0.0500 time 0.3708 (0.3861) data time 0.0006 (0.0019) model time 0.3703 (0.3846) loss 4.4242 (3.7727) grad_norm 1.0109 (inf) loss_scale 8192.0000 (21210.7103) mem 11225MB [2024-08-26 05:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [35/300][620/625] eta 0:00:01 lr 0.001984 wd 0.0500 time 0.3692 (0.3859) data time 0.0008 (0.0019) model time 0.3684 (0.3843) loss 4.1467 (3.7749) grad_norm 1.1283 (inf) loss_scale 8192.0000 (21001.0692) mem 11225MB [2024-08-26 05:19:48 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 35 training takes 0:04:01 [2024-08-26 05:19:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:19:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:19:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.492 (0.492) Loss 0.7510 (0.7510) Acc@1 82.568 (82.568) Acc@5 96.582 (96.582) Mem 11225MB [2024-08-26 05:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 1.3252 (0.9434) Acc@1 68.213 (77.441) Acc@5 90.283 (94.802) Mem 11225MB [2024-08-26 05:19:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.4355 (1.1492) Acc@1 65.576 (72.986) Acc@5 88.135 (91.939) Mem 11225MB [2024-08-26 05:19:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 72.829 Acc@5 91.929 [2024-08-26 05:19:52 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 72.8% [2024-08-26 05:19:52 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 72.83% [2024-08-26 05:19:52 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:19:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:19:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 0.6416 (0.6416) Acc@1 83.496 (83.496) Acc@5 96.533 (96.533) Mem 11225MB [2024-08-26 05:19:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.141) Loss 1.1797 (0.8256) Acc@1 69.580 (78.382) Acc@5 90.527 (94.935) Mem 11225MB [2024-08-26 05:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 1.3828 (1.0437) Acc@1 66.455 (73.768) Acc@5 87.744 (92.067) Mem 11225MB [2024-08-26 05:19:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.564 Acc@5 92.045 [2024-08-26 05:19:56 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 73.6% [2024-08-26 05:19:56 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 73.56% [2024-08-26 05:19:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:19:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][0/625] eta 0:07:48 lr 0.001984 wd 0.0500 time 0.7497 (0.7497) data time 0.4120 (0.4120) model time 0.0000 (0.0000) loss 4.3883 (4.3883) grad_norm 1.1266 (1.1266) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][10/625] eta 0:04:13 lr 0.001984 wd 0.0500 time 0.3881 (0.4125) data time 0.0010 (0.0385) model time 0.0000 (0.0000) loss 2.6269 (3.6063) grad_norm 0.8501 (1.2411) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][20/625] eta 0:04:00 lr 0.001984 wd 0.0500 time 0.3843 (0.3975) data time 0.0011 (0.0207) model time 0.0000 (0.0000) loss 4.0727 (3.5968) grad_norm 1.1885 (1.2703) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][30/625] eta 0:03:52 lr 0.001984 wd 0.0500 time 0.3744 (0.3909) data time 0.0009 (0.0144) model time 0.0000 (0.0000) loss 2.4028 (3.5358) grad_norm 0.9134 (1.2676) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][40/625] eta 0:03:49 lr 0.001984 wd 0.0500 time 0.3752 (0.3916) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 2.6543 (3.6516) grad_norm 1.0300 (1.2530) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][50/625] eta 0:03:43 lr 0.001984 wd 0.0500 time 0.3788 (0.3889) data time 0.0009 (0.0092) model time 0.0000 (0.0000) loss 5.1216 (3.7724) grad_norm 1.1855 (1.2260) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][60/625] eta 0:03:38 lr 0.001984 wd 0.0500 time 0.3732 (0.3873) data time 0.0008 (0.0078) model time 0.3723 (0.3780) loss 3.6434 (3.7375) grad_norm 1.4086 (1.2148) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][70/625] eta 0:03:34 lr 0.001984 wd 0.0500 time 0.3860 (0.3860) data time 0.0008 (0.0069) model time 0.3852 (0.3775) loss 4.5197 (3.7517) grad_norm 0.9853 (1.2319) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][80/625] eta 0:03:29 lr 0.001984 wd 0.0500 time 0.3811 (0.3850) data time 0.0008 (0.0062) model time 0.3802 (0.3773) loss 3.6274 (3.7284) grad_norm 1.4006 (1.2294) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][90/625] eta 0:03:25 lr 0.001984 wd 0.0500 time 0.3742 (0.3844) data time 0.0008 (0.0056) model time 0.3734 (0.3775) loss 4.2676 (3.7380) grad_norm 1.5981 (1.2377) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][100/625] eta 0:03:23 lr 0.001984 wd 0.0500 time 0.3887 (0.3878) data time 0.0011 (0.0052) model time 0.3876 (0.3856) loss 3.6136 (3.7815) grad_norm 1.7922 (1.2556) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][110/625] eta 0:03:19 lr 0.001984 wd 0.0500 time 0.3741 (0.3870) data time 0.0011 (0.0048) model time 0.3730 (0.3844) loss 4.1008 (3.7638) grad_norm 1.2049 (1.2557) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][120/625] eta 0:03:15 lr 0.001984 wd 0.0500 time 0.3784 (0.3864) data time 0.0009 (0.0045) model time 0.3775 (0.3835) loss 4.8471 (3.7610) grad_norm 1.1114 (1.2409) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][130/625] eta 0:03:11 lr 0.001984 wd 0.0500 time 0.3791 (0.3859) data time 0.0011 (0.0042) model time 0.3781 (0.3828) loss 4.1009 (3.7813) grad_norm 1.3875 (1.2370) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][140/625] eta 0:03:06 lr 0.001984 wd 0.0500 time 0.3651 (0.3853) data time 0.0011 (0.0040) model time 0.3639 (0.3822) loss 3.8196 (3.7670) grad_norm 0.9623 (1.2345) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][150/625] eta 0:03:03 lr 0.001984 wd 0.0500 time 0.3791 (0.3860) data time 0.0011 (0.0038) model time 0.3780 (0.3833) loss 3.4564 (3.7531) grad_norm 1.3651 (1.2365) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:20:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][160/625] eta 0:03:00 lr 0.001984 wd 0.0500 time 0.3959 (0.3879) data time 0.0011 (0.0037) model time 0.3948 (0.3863) loss 3.9978 (3.7865) grad_norm 1.4923 (1.2406) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][170/625] eta 0:02:56 lr 0.001984 wd 0.0500 time 0.3757 (0.3873) data time 0.0008 (0.0035) model time 0.3749 (0.3854) loss 3.7495 (3.7857) grad_norm 0.8383 (1.2428) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][180/625] eta 0:02:52 lr 0.001984 wd 0.0500 time 0.3799 (0.3867) data time 0.0011 (0.0034) model time 0.3788 (0.3846) loss 3.2458 (3.7905) grad_norm 1.1264 (1.2430) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][190/625] eta 0:02:47 lr 0.001983 wd 0.0500 time 0.3851 (0.3862) data time 0.0011 (0.0033) model time 0.3840 (0.3840) loss 3.2456 (3.7693) grad_norm 1.6976 (1.2393) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][200/625] eta 0:02:43 lr 0.001983 wd 0.0500 time 0.3774 (0.3858) data time 0.0009 (0.0033) model time 0.3764 (0.3835) loss 3.4889 (3.7637) grad_norm 1.7927 (1.2415) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][210/625] eta 0:02:39 lr 0.001983 wd 0.0500 time 0.3720 (0.3854) data time 0.0013 (0.0032) model time 0.3708 (0.3830) loss 4.0599 (3.7613) grad_norm 1.4206 (1.2498) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][220/625] eta 0:02:35 lr 0.001983 wd 0.0500 time 0.3720 (0.3850) data time 0.0011 (0.0031) model time 0.3708 (0.3826) loss 2.9680 (3.7469) grad_norm 1.4562 (1.2464) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][230/625] eta 0:02:31 lr 0.001983 wd 0.0500 time 0.3723 (0.3846) data time 0.0009 (0.0030) model time 0.3714 (0.3822) loss 2.5149 (3.7440) grad_norm 0.9786 (1.2424) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][240/625] eta 0:02:27 lr 0.001983 wd 0.0500 time 0.3770 (0.3843) data time 0.0011 (0.0029) model time 0.3759 (0.3818) loss 2.7136 (3.7514) grad_norm 1.0754 (1.2348) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][250/625] eta 0:02:24 lr 0.001983 wd 0.0500 time 0.3826 (0.3840) data time 0.0008 (0.0028) model time 0.3818 (0.3816) loss 2.8643 (3.7451) grad_norm 1.2179 (1.2319) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][260/625] eta 0:02:20 lr 0.001983 wd 0.0500 time 0.3851 (0.3839) data time 0.0008 (0.0028) model time 0.3843 (0.3815) loss 3.9266 (3.7352) grad_norm 1.2946 (1.2355) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][270/625] eta 0:02:16 lr 0.001983 wd 0.0500 time 0.3764 (0.3837) data time 0.0008 (0.0027) model time 0.3756 (0.3813) loss 3.9731 (3.7395) grad_norm 1.0485 (1.2346) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][280/625] eta 0:02:12 lr 0.001983 wd 0.0500 time 0.4062 (0.3835) data time 0.0008 (0.0027) model time 0.4054 (0.3811) loss 4.2868 (3.7331) grad_norm 0.9722 (1.2302) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][290/625] eta 0:02:08 lr 0.001983 wd 0.0500 time 0.3775 (0.3832) data time 0.0008 (0.0026) model time 0.3767 (0.3808) loss 4.0161 (3.7371) grad_norm 1.1248 (1.2334) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][300/625] eta 0:02:04 lr 0.001983 wd 0.0500 time 0.3717 (0.3831) data time 0.0012 (0.0026) model time 0.3705 (0.3807) loss 4.0065 (3.7390) grad_norm 1.1076 (1.2269) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:21:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][310/625] eta 0:02:01 lr 0.001983 wd 0.0500 time 0.3842 (0.3844) data time 0.0011 (0.0025) model time 0.3830 (0.3823) loss 4.2981 (3.7399) grad_norm 1.9362 (1.2290) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][320/625] eta 0:01:57 lr 0.001983 wd 0.0500 time 0.3803 (0.3844) data time 0.0011 (0.0025) model time 0.3792 (0.3824) loss 3.9635 (3.7498) grad_norm 1.7649 (1.2333) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][330/625] eta 0:01:53 lr 0.001983 wd 0.0500 time 0.3658 (0.3843) data time 0.0011 (0.0024) model time 0.3647 (0.3823) loss 2.8819 (3.7570) grad_norm 1.0109 (1.2293) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][340/625] eta 0:01:49 lr 0.001983 wd 0.0500 time 0.5552 (0.3852) data time 0.0011 (0.0024) model time 0.5541 (0.3834) loss 3.2217 (3.7464) grad_norm 1.5539 (1.2335) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][350/625] eta 0:01:45 lr 0.001983 wd 0.0500 time 0.3726 (0.3851) data time 0.0008 (0.0024) model time 0.3718 (0.3832) loss 3.1469 (3.7298) grad_norm 1.1757 (1.2376) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][360/625] eta 0:01:42 lr 0.001983 wd 0.0500 time 0.3761 (0.3849) data time 0.0009 (0.0023) model time 0.3752 (0.3830) loss 4.7592 (3.7309) grad_norm 0.8232 (1.2330) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][370/625] eta 0:01:38 lr 0.001983 wd 0.0500 time 0.3800 (0.3853) data time 0.0011 (0.0023) model time 0.3789 (0.3835) loss 3.8895 (3.7229) grad_norm 1.7240 (1.2337) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][380/625] eta 0:01:34 lr 0.001983 wd 0.0500 time 0.3897 (0.3851) data time 0.0009 (0.0023) model time 0.3888 (0.3833) loss 4.7597 (3.7337) grad_norm 1.2110 (1.2337) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][390/625] eta 0:01:30 lr 0.001983 wd 0.0500 time 0.3742 (0.3848) data time 0.0013 (0.0022) model time 0.3729 (0.3830) loss 3.8434 (3.7423) grad_norm 1.0732 (1.2326) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][400/625] eta 0:01:26 lr 0.001983 wd 0.0500 time 0.3800 (0.3847) data time 0.0013 (0.0022) model time 0.3787 (0.3829) loss 3.8156 (3.7434) grad_norm 1.6060 (1.2306) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][410/625] eta 0:01:22 lr 0.001983 wd 0.0500 time 0.3749 (0.3845) data time 0.0009 (0.0022) model time 0.3740 (0.3827) loss 4.2886 (3.7455) grad_norm 1.3811 (1.2304) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][420/625] eta 0:01:18 lr 0.001983 wd 0.0500 time 0.3782 (0.3843) data time 0.0011 (0.0022) model time 0.3771 (0.3825) loss 4.2309 (3.7419) grad_norm 1.0424 (1.2394) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][430/625] eta 0:01:14 lr 0.001983 wd 0.0500 time 0.3859 (0.3842) data time 0.0009 (0.0021) model time 0.3851 (0.3824) loss 3.9400 (3.7418) grad_norm 1.1058 (1.2405) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][440/625] eta 0:01:11 lr 0.001983 wd 0.0500 time 0.3697 (0.3840) data time 0.0011 (0.0021) model time 0.3687 (0.3822) loss 3.8440 (3.7453) grad_norm 1.1017 (1.2452) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][450/625] eta 0:01:07 lr 0.001983 wd 0.0500 time 0.3739 (0.3839) data time 0.0008 (0.0021) model time 0.3731 (0.3820) loss 4.2574 (3.7453) grad_norm 1.4312 (1.2500) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][460/625] eta 0:01:03 lr 0.001983 wd 0.0500 time 0.6173 (0.3847) data time 0.0008 (0.0021) model time 0.6165 (0.3830) loss 3.7575 (3.7439) grad_norm 0.8909 (1.2455) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:22:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][470/625] eta 0:00:59 lr 0.001983 wd 0.0500 time 0.3803 (0.3846) data time 0.0008 (0.0021) model time 0.3795 (0.3829) loss 2.8332 (3.7369) grad_norm 1.2854 (1.2455) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][480/625] eta 0:00:55 lr 0.001983 wd 0.0500 time 0.3712 (0.3845) data time 0.0011 (0.0021) model time 0.3701 (0.3827) loss 4.0926 (3.7394) grad_norm 1.3389 (1.2453) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][490/625] eta 0:00:52 lr 0.001982 wd 0.0500 time 0.3857 (0.3856) data time 0.0008 (0.0020) model time 0.3849 (0.3840) loss 3.2209 (3.7334) grad_norm 0.9715 (1.2438) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][500/625] eta 0:00:48 lr 0.001982 wd 0.0500 time 0.3777 (0.3855) data time 0.0011 (0.0020) model time 0.3766 (0.3839) loss 4.0998 (3.7333) grad_norm 0.8376 (1.2423) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][510/625] eta 0:00:44 lr 0.001982 wd 0.0500 time 0.3787 (0.3853) data time 0.0010 (0.0020) model time 0.3777 (0.3837) loss 3.1525 (3.7295) grad_norm 1.2345 (1.2429) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][520/625] eta 0:00:40 lr 0.001982 wd 0.0500 time 0.3777 (0.3852) data time 0.0011 (0.0020) model time 0.3766 (0.3836) loss 4.7522 (3.7257) grad_norm 1.0665 (1.2432) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][530/625] eta 0:00:36 lr 0.001982 wd 0.0500 time 0.3693 (0.3850) data time 0.0011 (0.0020) model time 0.3682 (0.3834) loss 2.8090 (3.7272) grad_norm 1.5263 (1.2461) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][540/625] eta 0:00:32 lr 0.001982 wd 0.0500 time 0.3757 (0.3848) data time 0.0010 (0.0020) model time 0.3746 (0.3832) loss 2.3820 (3.7230) grad_norm 1.6567 (1.2465) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][550/625] eta 0:00:28 lr 0.001982 wd 0.0500 time 0.3816 (0.3851) data time 0.0010 (0.0019) model time 0.3806 (0.3835) loss 4.0077 (3.7243) grad_norm 0.9892 (1.2452) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][560/625] eta 0:00:25 lr 0.001982 wd 0.0500 time 0.3723 (0.3849) data time 0.0008 (0.0019) model time 0.3714 (0.3833) loss 4.7342 (3.7243) grad_norm 1.0176 (1.2486) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][570/625] eta 0:00:21 lr 0.001982 wd 0.0500 time 0.3838 (0.3848) data time 0.0011 (0.0019) model time 0.3827 (0.3832) loss 3.8532 (3.7237) grad_norm 1.1765 (1.2465) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][580/625] eta 0:00:17 lr 0.001982 wd 0.0500 time 0.3833 (0.3847) data time 0.0010 (0.0019) model time 0.3823 (0.3831) loss 2.8074 (3.7222) grad_norm 1.1138 (1.2477) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][590/625] eta 0:00:13 lr 0.001982 wd 0.0500 time 0.3834 (0.3846) data time 0.0008 (0.0019) model time 0.3826 (0.3830) loss 2.6497 (3.7151) grad_norm 1.1194 (1.2495) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][600/625] eta 0:00:09 lr 0.001982 wd 0.0500 time 0.3851 (0.3846) data time 0.0011 (0.0019) model time 0.3840 (0.3830) loss 2.7376 (3.7074) grad_norm 0.8754 (1.2511) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][610/625] eta 0:00:05 lr 0.001982 wd 0.0500 time 0.3692 (0.3851) data time 0.0005 (0.0019) model time 0.3687 (0.3836) loss 3.1061 (3.7066) grad_norm 1.5692 (1.2546) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [36/300][620/625] eta 0:00:01 lr 0.001982 wd 0.0500 time 0.3672 (0.3849) data time 0.0006 (0.0019) model time 0.3667 (0.3833) loss 3.9063 (3.7093) grad_norm 1.1974 (1.2538) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:23:57 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 36 training takes 0:04:00 [2024-08-26 05:23:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:23:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:23:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.7954 (0.7954) Acc@1 83.447 (83.447) Acc@5 95.801 (95.801) Mem 11225MB [2024-08-26 05:24:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 1.3535 (0.9617) Acc@1 67.920 (78.214) Acc@5 90.430 (94.811) Mem 11225MB [2024-08-26 05:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.3838 (1.1538) Acc@1 67.139 (73.586) Acc@5 89.355 (92.132) Mem 11225MB [2024-08-26 05:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.401 Acc@5 92.053 [2024-08-26 05:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 73.4% [2024-08-26 05:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 73.40% [2024-08-26 05:24:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:24:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:24:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.6328 (0.6328) Acc@1 83.887 (83.887) Acc@5 96.582 (96.582) Mem 11225MB [2024-08-26 05:24:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.148) Loss 1.1611 (0.8140) Acc@1 70.410 (78.800) Acc@5 90.723 (95.068) Mem 11225MB [2024-08-26 05:24:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.3652 (1.0288) Acc@1 66.699 (74.172) Acc@5 87.891 (92.236) Mem 11225MB [2024-08-26 05:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.934 Acc@5 92.228 [2024-08-26 05:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 73.9% [2024-08-26 05:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 73.93% [2024-08-26 05:24:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:24:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:24:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][0/625] eta 0:07:48 lr 0.001982 wd 0.0500 time 0.7502 (0.7502) data time 0.4157 (0.4157) model time 0.0000 (0.0000) loss 3.2811 (3.2811) grad_norm 1.5752 (1.5752) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][10/625] eta 0:04:13 lr 0.001982 wd 0.0500 time 0.3778 (0.4114) data time 0.0008 (0.0390) model time 0.0000 (0.0000) loss 4.3795 (3.5747) grad_norm 0.9441 (1.3333) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][20/625] eta 0:04:04 lr 0.001982 wd 0.0500 time 0.3748 (0.4041) data time 0.0009 (0.0210) model time 0.0000 (0.0000) loss 3.7147 (3.6253) grad_norm 1.1421 (1.2921) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][30/625] eta 0:03:58 lr 0.001982 wd 0.0500 time 0.3751 (0.4012) data time 0.0011 (0.0146) model time 0.0000 (0.0000) loss 3.9047 (3.6892) grad_norm 1.0976 (1.2931) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][40/625] eta 0:03:51 lr 0.001982 wd 0.0500 time 0.3759 (0.3952) data time 0.0009 (0.0113) model time 0.0000 (0.0000) loss 3.7381 (3.7354) grad_norm 1.7666 (1.2918) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][50/625] eta 0:03:45 lr 0.001982 wd 0.0500 time 0.3831 (0.3916) data time 0.0007 (0.0093) model time 0.0000 (0.0000) loss 2.5749 (3.7085) grad_norm 1.3200 (1.2773) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][60/625] eta 0:03:39 lr 0.001982 wd 0.0500 time 0.3747 (0.3893) data time 0.0012 (0.0080) model time 0.3735 (0.3765) loss 3.4497 (3.7235) grad_norm 1.0371 (1.2758) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][70/625] eta 0:03:35 lr 0.001982 wd 0.0500 time 0.3761 (0.3877) data time 0.0010 (0.0070) model time 0.3751 (0.3765) loss 3.8452 (3.7608) grad_norm 1.1495 (1.2817) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][80/625] eta 0:03:30 lr 0.001982 wd 0.0500 time 0.3688 (0.3865) data time 0.0008 (0.0063) model time 0.3680 (0.3766) loss 4.2700 (3.7618) grad_norm 1.1734 (1.2716) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][90/625] eta 0:03:26 lr 0.001982 wd 0.0500 time 0.3793 (0.3855) data time 0.0011 (0.0057) model time 0.3782 (0.3765) loss 3.7620 (3.7569) grad_norm 1.2537 (1.2767) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][100/625] eta 0:03:22 lr 0.001982 wd 0.0500 time 0.3837 (0.3848) data time 0.0008 (0.0053) model time 0.3828 (0.3768) loss 4.3448 (3.7935) grad_norm 1.0963 (1.2696) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][110/625] eta 0:03:18 lr 0.001982 wd 0.0500 time 0.3661 (0.3859) data time 0.0011 (0.0049) model time 0.3651 (0.3799) loss 4.1606 (3.7818) grad_norm 1.2392 (1.2759) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][120/625] eta 0:03:14 lr 0.001982 wd 0.0500 time 0.3831 (0.3851) data time 0.0010 (0.0046) model time 0.3821 (0.3793) loss 3.3901 (3.7717) grad_norm 1.2651 (1.2737) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][130/625] eta 0:03:10 lr 0.001982 wd 0.0500 time 0.3744 (0.3844) data time 0.0008 (0.0043) model time 0.3736 (0.3787) loss 2.5670 (3.7503) grad_norm 0.9137 (1.2719) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][140/625] eta 0:03:06 lr 0.001982 wd 0.0500 time 0.3731 (0.3839) data time 0.0011 (0.0041) model time 0.3720 (0.3785) loss 3.9410 (3.7483) grad_norm 1.3430 (1.2632) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][150/625] eta 0:03:02 lr 0.001982 wd 0.0500 time 0.3848 (0.3838) data time 0.0008 (0.0039) model time 0.3840 (0.3787) loss 4.6877 (3.7405) grad_norm 1.1217 (1.2469) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][160/625] eta 0:02:58 lr 0.001982 wd 0.0500 time 0.3754 (0.3834) data time 0.0008 (0.0037) model time 0.3746 (0.3786) loss 4.4194 (3.7425) grad_norm 1.2404 (1.2312) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][170/625] eta 0:02:55 lr 0.001981 wd 0.0500 time 0.3669 (0.3854) data time 0.0011 (0.0036) model time 0.3659 (0.3816) loss 3.6602 (3.7359) grad_norm 1.2222 (1.2417) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][180/625] eta 0:02:51 lr 0.001981 wd 0.0500 time 0.3752 (0.3849) data time 0.0009 (0.0034) model time 0.3743 (0.3812) loss 2.9310 (3.7437) grad_norm 1.1661 (1.2593) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][190/625] eta 0:02:47 lr 0.001981 wd 0.0500 time 0.3876 (0.3849) data time 0.0011 (0.0033) model time 0.3865 (0.3813) loss 3.9493 (3.7491) grad_norm 0.8112 (1.2547) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][200/625] eta 0:02:43 lr 0.001981 wd 0.0500 time 0.3640 (0.3845) data time 0.0008 (0.0032) model time 0.3632 (0.3809) loss 2.6369 (3.7367) grad_norm 1.1041 (1.2534) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][210/625] eta 0:02:39 lr 0.001981 wd 0.0500 time 0.3764 (0.3842) data time 0.0010 (0.0032) model time 0.3754 (0.3807) loss 4.3255 (3.7327) grad_norm 1.5651 (1.2568) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][220/625] eta 0:02:35 lr 0.001981 wd 0.0500 time 0.3775 (0.3840) data time 0.0012 (0.0031) model time 0.3763 (0.3805) loss 3.0338 (3.7353) grad_norm 2.1878 (1.2637) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][230/625] eta 0:02:32 lr 0.001981 wd 0.0500 time 0.5818 (0.3852) data time 0.0008 (0.0030) model time 0.5810 (0.3822) loss 3.6348 (3.7349) grad_norm 1.2382 (1.2742) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][240/625] eta 0:02:28 lr 0.001981 wd 0.0500 time 0.3755 (0.3857) data time 0.0008 (0.0029) model time 0.3746 (0.3829) loss 3.1749 (3.7287) grad_norm 1.2807 (1.2706) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][250/625] eta 0:02:24 lr 0.001981 wd 0.0500 time 0.3763 (0.3854) data time 0.0007 (0.0029) model time 0.3755 (0.3826) loss 4.0671 (3.7272) grad_norm 1.2135 (1.2779) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][260/625] eta 0:02:20 lr 0.001981 wd 0.0500 time 0.3720 (0.3851) data time 0.0011 (0.0028) model time 0.3709 (0.3824) loss 4.2041 (3.7212) grad_norm 1.1957 (1.2778) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][270/625] eta 0:02:16 lr 0.001981 wd 0.0500 time 0.3704 (0.3848) data time 0.0010 (0.0027) model time 0.3694 (0.3820) loss 2.6538 (3.7091) grad_norm 1.2432 (1.2790) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][280/625] eta 0:02:12 lr 0.001981 wd 0.0500 time 0.3719 (0.3845) data time 0.0009 (0.0027) model time 0.3710 (0.3817) loss 4.7314 (3.7083) grad_norm 0.8606 (1.2703) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][290/625] eta 0:02:09 lr 0.001981 wd 0.0500 time 0.3749 (0.3856) data time 0.0008 (0.0026) model time 0.3741 (0.3832) loss 3.7975 (3.7056) grad_norm 1.6295 (1.2721) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][300/625] eta 0:02:05 lr 0.001981 wd 0.0500 time 0.3696 (0.3853) data time 0.0008 (0.0026) model time 0.3688 (0.3829) loss 3.8749 (3.6994) grad_norm 1.1600 (1.2661) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][310/625] eta 0:02:01 lr 0.001981 wd 0.0500 time 0.3764 (0.3851) data time 0.0008 (0.0025) model time 0.3756 (0.3826) loss 4.1813 (3.7081) grad_norm 1.7588 (1.2704) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][320/625] eta 0:01:57 lr 0.001981 wd 0.0500 time 0.3741 (0.3853) data time 0.0011 (0.0025) model time 0.3731 (0.3829) loss 3.4461 (3.7205) grad_norm 1.0913 (1.2634) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][330/625] eta 0:01:53 lr 0.001981 wd 0.0500 time 0.3759 (0.3852) data time 0.0014 (0.0024) model time 0.3745 (0.3828) loss 3.9608 (3.7223) grad_norm 1.4870 (1.2573) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][340/625] eta 0:01:49 lr 0.001981 wd 0.0500 time 0.3769 (0.3849) data time 0.0008 (0.0024) model time 0.3761 (0.3826) loss 3.7592 (3.7267) grad_norm 0.9476 (1.2536) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][350/625] eta 0:01:45 lr 0.001981 wd 0.0500 time 0.3709 (0.3847) data time 0.0010 (0.0024) model time 0.3699 (0.3824) loss 4.0886 (3.7315) grad_norm 2.1176 (1.2659) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][360/625] eta 0:01:41 lr 0.001981 wd 0.0500 time 0.3796 (0.3846) data time 0.0013 (0.0023) model time 0.3783 (0.3822) loss 3.8966 (3.7319) grad_norm 1.1614 (1.2644) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][370/625] eta 0:01:38 lr 0.001981 wd 0.0500 time 0.3801 (0.3845) data time 0.0011 (0.0023) model time 0.3790 (0.3821) loss 3.4768 (3.7319) grad_norm 1.0442 (1.2622) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][380/625] eta 0:01:34 lr 0.001981 wd 0.0500 time 0.3763 (0.3843) data time 0.0011 (0.0023) model time 0.3752 (0.3820) loss 3.0528 (3.7241) grad_norm 1.5322 (1.2604) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][390/625] eta 0:01:30 lr 0.001981 wd 0.0500 time 0.3878 (0.3843) data time 0.0011 (0.0023) model time 0.3867 (0.3820) loss 3.9797 (3.7314) grad_norm 1.0641 (1.2568) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][400/625] eta 0:01:26 lr 0.001981 wd 0.0500 time 0.3729 (0.3842) data time 0.0011 (0.0022) model time 0.3719 (0.3820) loss 3.8337 (3.7318) grad_norm 1.0177 (1.2552) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][410/625] eta 0:01:22 lr 0.001981 wd 0.0500 time 0.3818 (0.3841) data time 0.0008 (0.0022) model time 0.3810 (0.3818) loss 4.2468 (3.7407) grad_norm 1.4466 (1.2548) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][420/625] eta 0:01:18 lr 0.001981 wd 0.0500 time 0.3830 (0.3853) data time 0.0011 (0.0022) model time 0.3819 (0.3833) loss 3.6066 (3.7437) grad_norm 1.2324 (1.2577) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][430/625] eta 0:01:15 lr 0.001981 wd 0.0500 time 0.3695 (0.3852) data time 0.0008 (0.0022) model time 0.3687 (0.3831) loss 2.8589 (3.7362) grad_norm 0.9952 (1.2616) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:26:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][440/625] eta 0:01:11 lr 0.001981 wd 0.0500 time 0.3794 (0.3850) data time 0.0008 (0.0021) model time 0.3785 (0.3830) loss 3.7887 (3.7356) grad_norm 2.5111 (1.2657) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][450/625] eta 0:01:07 lr 0.001980 wd 0.0500 time 0.3938 (0.3849) data time 0.0011 (0.0021) model time 0.3927 (0.3829) loss 3.9120 (3.7376) grad_norm 1.6278 (1.2714) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][460/625] eta 0:01:03 lr 0.001980 wd 0.0500 time 0.3771 (0.3848) data time 0.0009 (0.0021) model time 0.3762 (0.3827) loss 3.7260 (3.7384) grad_norm 0.9154 (1.2701) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][470/625] eta 0:00:59 lr 0.001980 wd 0.0500 time 0.3796 (0.3846) data time 0.0008 (0.0021) model time 0.3788 (0.3826) loss 3.9683 (3.7467) grad_norm 1.2032 (1.2675) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][480/625] eta 0:00:55 lr 0.001980 wd 0.0500 time 0.3754 (0.3844) data time 0.0009 (0.0020) model time 0.3745 (0.3824) loss 4.3022 (3.7454) grad_norm 1.1825 (1.2658) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][490/625] eta 0:00:51 lr 0.001980 wd 0.0500 time 0.3769 (0.3843) data time 0.0008 (0.0020) model time 0.3761 (0.3823) loss 3.8419 (3.7453) grad_norm 1.0903 (1.2664) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][500/625] eta 0:00:48 lr 0.001980 wd 0.0500 time 0.6083 (0.3855) data time 0.0010 (0.0020) model time 0.6073 (0.3836) loss 4.5052 (3.7399) grad_norm 1.0715 (1.2617) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][510/625] eta 0:00:44 lr 0.001980 wd 0.0500 time 0.3797 (0.3852) data time 0.0008 (0.0020) model time 0.3789 (0.3834) loss 4.4427 (3.7424) grad_norm 0.9451 (1.2582) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][520/625] eta 0:00:40 lr 0.001980 wd 0.0500 time 0.3817 (0.3851) data time 0.0012 (0.0020) model time 0.3805 (0.3832) loss 4.2513 (3.7497) grad_norm 0.9914 (1.2561) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][530/625] eta 0:00:36 lr 0.001980 wd 0.0500 time 0.3695 (0.3850) data time 0.0008 (0.0020) model time 0.3687 (0.3831) loss 3.0908 (3.7406) grad_norm 1.1653 (1.2543) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][540/625] eta 0:00:32 lr 0.001980 wd 0.0500 time 0.3896 (0.3857) data time 0.0009 (0.0019) model time 0.3887 (0.3839) loss 3.0071 (3.7381) grad_norm 1.2048 (1.2537) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][550/625] eta 0:00:28 lr 0.001980 wd 0.0500 time 0.3772 (0.3856) data time 0.0011 (0.0019) model time 0.3761 (0.3838) loss 2.9519 (3.7359) grad_norm 1.9168 (1.2624) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][560/625] eta 0:00:25 lr 0.001980 wd 0.0500 time 0.3771 (0.3854) data time 0.0009 (0.0019) model time 0.3763 (0.3836) loss 3.5998 (3.7380) grad_norm 1.2360 (1.2605) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][570/625] eta 0:00:21 lr 0.001980 wd 0.0500 time 0.3788 (0.3853) data time 0.0011 (0.0019) model time 0.3777 (0.3836) loss 3.3925 (3.7386) grad_norm 1.1163 (1.2578) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][580/625] eta 0:00:17 lr 0.001980 wd 0.0500 time 0.3805 (0.3853) data time 0.0008 (0.0019) model time 0.3796 (0.3835) loss 3.2377 (3.7361) grad_norm 1.2896 (1.2556) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][590/625] eta 0:00:13 lr 0.001980 wd 0.0500 time 0.3746 (0.3852) data time 0.0008 (0.0019) model time 0.3738 (0.3834) loss 4.4621 (3.7323) grad_norm 1.3426 (1.2574) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:27:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][600/625] eta 0:00:09 lr 0.001980 wd 0.0500 time 0.3753 (0.3851) data time 0.0009 (0.0019) model time 0.3745 (0.3833) loss 3.3404 (3.7320) grad_norm 1.0495 (1.2588) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][610/625] eta 0:00:05 lr 0.001980 wd 0.0500 time 0.3666 (0.3850) data time 0.0005 (0.0019) model time 0.3661 (0.3832) loss 3.6421 (3.7327) grad_norm 1.1669 (1.2583) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [37/300][620/625] eta 0:00:01 lr 0.001980 wd 0.0500 time 0.3307 (0.3850) data time 0.0009 (0.0019) model time 0.3298 (0.3833) loss 4.0481 (3.7333) grad_norm 1.0683 (1.2559) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 37 training takes 0:04:00 [2024-08-26 05:28:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:28:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:28:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.7485 (0.7485) Acc@1 83.252 (83.252) Acc@5 96.875 (96.875) Mem 11225MB [2024-08-26 05:28:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.2363 (0.9184) Acc@1 71.338 (78.307) Acc@5 90.820 (95.011) Mem 11225MB [2024-08-26 05:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.4912 (1.1385) Acc@1 65.234 (73.551) Acc@5 87.012 (92.143) Mem 11225MB [2024-08-26 05:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.297 Acc@5 92.069 [2024-08-26 05:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 73.3% [2024-08-26 05:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.823 (0.823) Loss 0.6226 (0.6226) Acc@1 84.326 (84.326) Acc@5 96.777 (96.777) Mem 11225MB [2024-08-26 05:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.175) Loss 1.1426 (0.8029) Acc@1 70.703 (79.159) Acc@5 90.918 (95.210) Mem 11225MB [2024-08-26 05:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.136) Loss 1.3457 (1.0146) Acc@1 67.041 (74.563) Acc@5 88.184 (92.448) Mem 11225MB [2024-08-26 05:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.296 Acc@5 92.424 [2024-08-26 05:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 74.3% [2024-08-26 05:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 74.30% [2024-08-26 05:28:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:28:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][0/625] eta 0:07:41 lr 0.001980 wd 0.0500 time 0.7377 (0.7377) data time 0.3840 (0.3840) model time 0.0000 (0.0000) loss 3.7237 (3.7237) grad_norm 0.7276 (0.7276) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][10/625] eta 0:04:11 lr 0.001980 wd 0.0500 time 0.3823 (0.4090) data time 0.0008 (0.0360) model time 0.0000 (0.0000) loss 2.6260 (3.3723) grad_norm 1.3486 (1.2141) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][20/625] eta 0:03:58 lr 0.001980 wd 0.0500 time 0.3774 (0.3937) data time 0.0011 (0.0194) model time 0.0000 (0.0000) loss 3.7182 (3.5602) grad_norm 1.1673 (1.1966) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][30/625] eta 0:03:51 lr 0.001980 wd 0.0500 time 0.3764 (0.3890) data time 0.0011 (0.0135) model time 0.0000 (0.0000) loss 3.7522 (3.6196) grad_norm 1.3593 (1.2236) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][40/625] eta 0:03:46 lr 0.001980 wd 0.0500 time 0.3820 (0.3867) data time 0.0010 (0.0105) model time 0.0000 (0.0000) loss 3.8263 (3.6648) grad_norm 1.1220 (1.2158) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][50/625] eta 0:03:41 lr 0.001980 wd 0.0500 time 0.3756 (0.3848) data time 0.0009 (0.0086) model time 0.0000 (0.0000) loss 4.3996 (3.6650) grad_norm 1.2904 (1.2360) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][60/625] eta 0:03:38 lr 0.001980 wd 0.0500 time 0.5945 (0.3868) data time 0.0009 (0.0074) model time 0.5936 (0.3956) loss 4.3776 (3.6566) grad_norm 1.4160 (1.2188) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][70/625] eta 0:03:38 lr 0.001980 wd 0.0500 time 0.3781 (0.3933) data time 0.0011 (0.0067) model time 0.3770 (0.4133) loss 2.8775 (3.6903) grad_norm 1.1653 (1.2446) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][80/625] eta 0:03:33 lr 0.001980 wd 0.0500 time 0.3839 (0.3914) data time 0.0012 (0.0060) model time 0.3827 (0.4010) loss 4.1206 (3.7085) grad_norm 1.2250 (1.2578) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][90/625] eta 0:03:28 lr 0.001980 wd 0.0500 time 0.3842 (0.3899) data time 0.0010 (0.0055) model time 0.3832 (0.3951) loss 3.8575 (3.7233) grad_norm 1.1657 (1.2644) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][100/625] eta 0:03:24 lr 0.001980 wd 0.0500 time 0.3814 (0.3890) data time 0.0011 (0.0050) model time 0.3802 (0.3920) loss 3.8660 (3.7484) grad_norm 0.9570 (1.2520) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:28:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][110/625] eta 0:03:19 lr 0.001979 wd 0.0500 time 0.3730 (0.3880) data time 0.0010 (0.0047) model time 0.3720 (0.3895) loss 3.7504 (3.7326) grad_norm 1.3012 (1.2498) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][120/625] eta 0:03:15 lr 0.001979 wd 0.0500 time 0.3801 (0.3871) data time 0.0010 (0.0044) model time 0.3791 (0.3875) loss 3.3562 (3.7343) grad_norm 0.8492 (1.2429) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][130/625] eta 0:03:11 lr 0.001979 wd 0.0500 time 0.3798 (0.3865) data time 0.0011 (0.0041) model time 0.3787 (0.3863) loss 3.8751 (3.7574) grad_norm 1.1082 (1.2392) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][140/625] eta 0:03:07 lr 0.001979 wd 0.0500 time 0.3762 (0.3870) data time 0.0013 (0.0039) model time 0.3750 (0.3870) loss 4.0656 (3.7499) grad_norm 1.0610 (1.2358) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][150/625] eta 0:03:03 lr 0.001979 wd 0.0500 time 0.3765 (0.3864) data time 0.0011 (0.0037) model time 0.3754 (0.3860) loss 4.0726 (3.7580) grad_norm 0.8060 (1.2385) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][160/625] eta 0:02:59 lr 0.001979 wd 0.0500 time 0.3770 (0.3859) data time 0.0009 (0.0036) model time 0.3761 (0.3852) loss 4.8436 (3.7688) grad_norm 1.2999 (1.2506) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][170/625] eta 0:02:55 lr 0.001979 wd 0.0500 time 0.3753 (0.3854) data time 0.0011 (0.0034) model time 0.3742 (0.3845) loss 3.9680 (3.7897) grad_norm 1.1838 (1.2483) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][180/625] eta 0:02:51 lr 0.001979 wd 0.0500 time 0.3852 (0.3850) data time 0.0009 (0.0033) model time 0.3842 (0.3838) loss 3.3153 (3.7733) grad_norm 1.3619 (1.2413) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][190/625] eta 0:02:47 lr 0.001979 wd 0.0500 time 0.3804 (0.3845) data time 0.0009 (0.0032) model time 0.3795 (0.3832) loss 3.6693 (3.7852) grad_norm 0.9458 (1.2338) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][200/625] eta 0:02:43 lr 0.001979 wd 0.0500 time 0.3823 (0.3842) data time 0.0010 (0.0031) model time 0.3813 (0.3827) loss 3.9878 (3.7792) grad_norm 1.5176 (1.2264) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][210/625] eta 0:02:39 lr 0.001979 wd 0.0500 time 0.3779 (0.3840) data time 0.0009 (0.0030) model time 0.3770 (0.3825) loss 4.8417 (3.7814) grad_norm 1.2025 (1.2327) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][220/625] eta 0:02:35 lr 0.001979 wd 0.0500 time 0.3805 (0.3837) data time 0.0008 (0.0029) model time 0.3797 (0.3821) loss 2.8922 (3.7874) grad_norm 1.0290 (1.2337) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][230/625] eta 0:02:31 lr 0.001979 wd 0.0500 time 0.3802 (0.3833) data time 0.0010 (0.0028) model time 0.3792 (0.3817) loss 4.1401 (3.7927) grad_norm 1.2027 (1.2381) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][240/625] eta 0:02:27 lr 0.001979 wd 0.0500 time 0.3706 (0.3831) data time 0.0011 (0.0028) model time 0.3695 (0.3815) loss 4.0539 (3.7953) grad_norm 2.1217 (1.2440) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][250/625] eta 0:02:24 lr 0.001979 wd 0.0500 time 0.3765 (0.3845) data time 0.0009 (0.0027) model time 0.3756 (0.3833) loss 3.7959 (3.7953) grad_norm 0.9223 (1.2435) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:29:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][260/625] eta 0:02:20 lr 0.001979 wd 0.0500 time 0.3781 (0.3843) data time 0.0011 (0.0026) model time 0.3770 (0.3830) loss 4.1433 (3.7873) grad_norm 0.8110 (1.2369) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][270/625] eta 0:02:16 lr 0.001979 wd 0.0500 time 0.3824 (0.3841) data time 0.0009 (0.0026) model time 0.3816 (0.3827) loss 4.6780 (3.7830) grad_norm 1.4986 (1.2371) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][280/625] eta 0:02:12 lr 0.001979 wd 0.0500 time 0.3683 (0.3853) data time 0.0008 (0.0025) model time 0.3675 (0.3843) loss 2.9810 (3.7764) grad_norm 1.4725 (1.2365) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][290/625] eta 0:02:08 lr 0.001979 wd 0.0500 time 0.3739 (0.3850) data time 0.0009 (0.0025) model time 0.3731 (0.3839) loss 4.3609 (3.7834) grad_norm 1.4223 (1.2360) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][300/625] eta 0:02:05 lr 0.001979 wd 0.0500 time 0.3774 (0.3847) data time 0.0011 (0.0024) model time 0.3762 (0.3835) loss 3.9947 (3.7837) grad_norm 1.2257 (1.2377) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][310/625] eta 0:02:01 lr 0.001979 wd 0.0500 time 0.3824 (0.3845) data time 0.0010 (0.0024) model time 0.3814 (0.3833) loss 3.9000 (3.7927) grad_norm 0.9768 (1.2383) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][320/625] eta 0:01:57 lr 0.001979 wd 0.0500 time 0.3785 (0.3843) data time 0.0009 (0.0024) model time 0.3777 (0.3831) loss 4.1101 (3.7967) grad_norm 1.2570 (1.2364) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][330/625] eta 0:01:53 lr 0.001979 wd 0.0500 time 0.3775 (0.3846) data time 0.0008 (0.0023) model time 0.3767 (0.3834) loss 3.9126 (3.7878) grad_norm 0.7564 (1.2384) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][340/625] eta 0:01:49 lr 0.001979 wd 0.0500 time 0.3858 (0.3845) data time 0.0011 (0.0023) model time 0.3847 (0.3832) loss 3.6271 (3.7827) grad_norm 1.0615 (1.2350) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][350/625] eta 0:01:45 lr 0.001979 wd 0.0500 time 0.3742 (0.3842) data time 0.0010 (0.0023) model time 0.3733 (0.3830) loss 3.6210 (3.7809) grad_norm 1.5212 (1.2358) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][360/625] eta 0:01:41 lr 0.001979 wd 0.0500 time 0.3783 (0.3841) data time 0.0010 (0.0022) model time 0.3773 (0.3828) loss 3.8487 (3.7803) grad_norm 1.2394 (1.2375) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][370/625] eta 0:01:38 lr 0.001979 wd 0.0500 time 0.3823 (0.3851) data time 0.0010 (0.0022) model time 0.3812 (0.3840) loss 4.0557 (3.7733) grad_norm 1.7152 (1.2401) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][380/625] eta 0:01:34 lr 0.001979 wd 0.0500 time 0.3716 (0.3849) data time 0.0011 (0.0022) model time 0.3705 (0.3838) loss 3.4360 (3.7656) grad_norm 1.3270 (1.2405) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][390/625] eta 0:01:30 lr 0.001978 wd 0.0500 time 0.3822 (0.3847) data time 0.0009 (0.0022) model time 0.3813 (0.3835) loss 2.1158 (3.7517) grad_norm 1.2050 (1.2499) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][400/625] eta 0:01:26 lr 0.001978 wd 0.0500 time 0.3729 (0.3846) data time 0.0011 (0.0021) model time 0.3717 (0.3834) loss 3.7987 (3.7449) grad_norm 1.2484 (1.2467) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][410/625] eta 0:01:22 lr 0.001978 wd 0.0500 time 0.3791 (0.3844) data time 0.0009 (0.0021) model time 0.3782 (0.3832) loss 4.2400 (3.7407) grad_norm 1.2442 (1.2443) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:30:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][420/625] eta 0:01:18 lr 0.001978 wd 0.0500 time 0.3787 (0.3843) data time 0.0009 (0.0021) model time 0.3778 (0.3830) loss 4.5936 (3.7373) grad_norm 1.7932 (1.2468) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][430/625] eta 0:01:14 lr 0.001978 wd 0.0500 time 0.3815 (0.3842) data time 0.0012 (0.0021) model time 0.3803 (0.3829) loss 4.0750 (3.7335) grad_norm 0.9363 (1.2460) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][440/625] eta 0:01:11 lr 0.001978 wd 0.0500 time 0.3719 (0.3840) data time 0.0009 (0.0020) model time 0.3711 (0.3828) loss 4.1209 (3.7304) grad_norm 1.0577 (1.2468) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][450/625] eta 0:01:07 lr 0.001978 wd 0.0500 time 0.3744 (0.3843) data time 0.0011 (0.0020) model time 0.3734 (0.3831) loss 3.9974 (3.7303) grad_norm 1.0019 (1.2464) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][460/625] eta 0:01:03 lr 0.001978 wd 0.0500 time 0.5686 (0.3846) data time 0.0013 (0.0020) model time 0.5673 (0.3834) loss 3.8105 (3.7274) grad_norm 1.3003 (1.2433) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][470/625] eta 0:00:59 lr 0.001978 wd 0.0500 time 0.3815 (0.3848) data time 0.0009 (0.0020) model time 0.3807 (0.3837) loss 3.5824 (3.7225) grad_norm 1.3962 (1.2502) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][480/625] eta 0:00:55 lr 0.001978 wd 0.0500 time 0.3716 (0.3847) data time 0.0010 (0.0020) model time 0.3705 (0.3835) loss 3.6370 (3.7194) grad_norm 1.1876 (1.2480) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][490/625] eta 0:00:51 lr 0.001978 wd 0.0500 time 0.3702 (0.3845) data time 0.0011 (0.0020) model time 0.3691 (0.3833) loss 3.2830 (3.7257) grad_norm 1.6966 (1.2485) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][500/625] eta 0:00:48 lr 0.001978 wd 0.0500 time 0.3776 (0.3844) data time 0.0009 (0.0019) model time 0.3767 (0.3831) loss 3.3829 (3.7262) grad_norm 1.3047 (1.2475) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][510/625] eta 0:00:44 lr 0.001978 wd 0.0500 time 0.3761 (0.3842) data time 0.0010 (0.0019) model time 0.3751 (0.3829) loss 3.8114 (3.7256) grad_norm 1.4330 (1.2487) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][520/625] eta 0:00:40 lr 0.001978 wd 0.0500 time 0.3734 (0.3841) data time 0.0010 (0.0019) model time 0.3725 (0.3828) loss 2.9477 (3.7170) grad_norm 1.2199 (1.2509) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][530/625] eta 0:00:36 lr 0.001978 wd 0.0500 time 0.3750 (0.3839) data time 0.0009 (0.0019) model time 0.3741 (0.3826) loss 4.5888 (3.7210) grad_norm 0.7732 (1.2480) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][540/625] eta 0:00:32 lr 0.001978 wd 0.0500 time 0.3837 (0.3838) data time 0.0008 (0.0019) model time 0.3829 (0.3825) loss 4.2810 (3.7240) grad_norm 1.0503 (1.2440) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:31:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][550/625] eta 0:00:28 lr 0.001978 wd 0.0500 time 0.3728 (0.3837) data time 0.0011 (0.0019) model time 0.3717 (0.3823) loss 3.6590 (3.7226) grad_norm 1.3234 (1.2420) loss_scale 16384.0000 (8266.3376) mem 11225MB [2024-08-26 05:31:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][560/625] eta 0:00:24 lr 0.001978 wd 0.0500 time 0.3735 (0.3835) data time 0.0009 (0.0019) model time 0.3726 (0.3822) loss 4.2586 (3.7232) grad_norm 1.6694 (1.2427) loss_scale 16384.0000 (8411.0374) mem 11225MB [2024-08-26 05:31:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][570/625] eta 0:00:21 lr 0.001978 wd 0.0500 time 0.3871 (0.3834) data time 0.0010 (0.0019) model time 0.3861 (0.3821) loss 3.7336 (3.7203) grad_norm 1.0984 (1.2428) loss_scale 16384.0000 (8550.6690) mem 11225MB [2024-08-26 05:32:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][580/625] eta 0:00:17 lr 0.001978 wd 0.0500 time 0.5776 (0.3848) data time 0.0008 (0.0018) model time 0.5768 (0.3836) loss 3.2352 (3.7205) grad_norm 1.6613 (1.2424) loss_scale 16384.0000 (8685.4940) mem 11225MB [2024-08-26 05:32:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][590/625] eta 0:00:13 lr 0.001978 wd 0.0500 time 0.3720 (0.3846) data time 0.0011 (0.0018) model time 0.3709 (0.3834) loss 3.5002 (3.7174) grad_norm 0.8413 (1.2440) loss_scale 16384.0000 (8815.7563) mem 11225MB [2024-08-26 05:32:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][600/625] eta 0:00:09 lr 0.001978 wd 0.0500 time 0.3798 (0.3846) data time 0.0009 (0.0018) model time 0.3790 (0.3834) loss 4.1674 (3.7160) grad_norm 1.6161 (1.2425) loss_scale 16384.0000 (8941.6839) mem 11225MB [2024-08-26 05:32:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][610/625] eta 0:00:05 lr 0.001978 wd 0.0500 time 0.3689 (0.3844) data time 0.0008 (0.0018) model time 0.3682 (0.3832) loss 4.1738 (3.7199) grad_norm 1.7356 (1.2425) loss_scale 16384.0000 (9063.4894) mem 11225MB [2024-08-26 05:32:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [38/300][620/625] eta 0:00:01 lr 0.001978 wd 0.0500 time 0.3663 (0.3842) data time 0.0008 (0.0018) model time 0.3656 (0.3829) loss 4.0501 (3.7194) grad_norm 1.3326 (1.2429) loss_scale 16384.0000 (9181.3720) mem 11225MB [2024-08-26 05:32:16 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 38 training takes 0:04:00 [2024-08-26 05:32:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:32:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:32:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.519 (0.519) Loss 0.7524 (0.7524) Acc@1 83.984 (83.984) Acc@5 96.631 (96.631) Mem 11225MB [2024-08-26 05:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.148) Loss 1.3633 (0.9355) Acc@1 68.701 (78.405) Acc@5 89.648 (95.059) Mem 11225MB [2024-08-26 05:32:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 1.4229 (1.1553) Acc@1 66.895 (73.665) Acc@5 89.014 (92.167) Mem 11225MB [2024-08-26 05:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.494 Acc@5 92.061 [2024-08-26 05:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 73.5% [2024-08-26 05:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 73.49% [2024-08-26 05:32:21 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:32:22 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.503 (0.503) Loss 0.6138 (0.6138) Acc@1 84.277 (84.277) Acc@5 96.924 (96.924) Mem 11225MB [2024-08-26 05:32:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.1270 (0.7932) Acc@1 71.240 (79.519) Acc@5 91.357 (95.392) Mem 11225MB [2024-08-26 05:32:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.098 (0.119) Loss 1.3281 (1.0023) Acc@1 67.480 (74.919) Acc@5 88.428 (92.676) Mem 11225MB [2024-08-26 05:32:25 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.636 Acc@5 92.622 [2024-08-26 05:32:25 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 74.6% [2024-08-26 05:32:25 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 74.64% [2024-08-26 05:32:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:32:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:32:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][0/625] eta 0:07:52 lr 0.001978 wd 0.0500 time 0.7553 (0.7553) data time 0.3979 (0.3979) model time 0.0000 (0.0000) loss 3.6420 (3.6420) grad_norm 1.0676 (1.0676) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:32:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][10/625] eta 0:04:14 lr 0.001978 wd 0.0500 time 0.3828 (0.4130) data time 0.0008 (0.0372) model time 0.0000 (0.0000) loss 4.4980 (3.8039) grad_norm 1.1220 (1.5455) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:32:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][20/625] eta 0:04:00 lr 0.001978 wd 0.0500 time 0.3782 (0.3983) data time 0.0082 (0.0204) model time 0.0000 (0.0000) loss 4.4567 (3.7281) grad_norm 1.1423 (1.3382) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:32:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][30/625] eta 0:03:53 lr 0.001977 wd 0.0500 time 0.3787 (0.3926) data time 0.0011 (0.0142) model time 0.0000 (0.0000) loss 3.6911 (3.7258) grad_norm 1.4288 (1.2491) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:32:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][40/625] eta 0:03:47 lr 0.001977 wd 0.0500 time 0.3880 (0.3895) data time 0.0012 (0.0112) model time 0.0000 (0.0000) loss 3.7292 (3.6972) grad_norm 1.0756 (1.3018) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:32:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][50/625] eta 0:03:42 lr 0.001977 wd 0.0500 time 0.3853 (0.3877) data time 0.0009 (0.0092) model time 0.0000 (0.0000) loss 2.5501 (3.6903) grad_norm 1.1055 (1.2748) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:32:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][60/625] eta 0:03:38 lr 0.001977 wd 0.0500 time 0.3804 (0.3866) data time 0.0008 (0.0079) model time 0.3796 (0.3802) loss 2.7846 (3.6960) grad_norm 0.9413 (1.2597) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:32:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][70/625] eta 0:03:33 lr 0.001977 wd 0.0500 time 0.3785 (0.3854) data time 0.0011 (0.0069) model time 0.3774 (0.3785) loss 4.1221 (3.7055) grad_norm 1.2560 (1.2326) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:32:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][80/625] eta 0:03:29 lr 0.001977 wd 0.0500 time 0.3762 (0.3845) data time 0.0009 (0.0062) model time 0.3753 (0.3780) loss 2.9905 (3.7202) grad_norm 1.3855 (1.2274) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:33:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][90/625] eta 0:03:25 lr 0.001977 wd 0.0500 time 0.3823 (0.3838) data time 0.0008 (0.0057) model time 0.3814 (0.3778) loss 3.9312 (3.7082) grad_norm 1.4772 (1.2317) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:33:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][100/625] eta 0:03:24 lr 0.001977 wd 0.0500 time 0.3800 (0.3898) data time 0.0008 (0.0052) model time 0.3792 (0.3910) loss 2.7981 (3.7075) grad_norm 0.9745 (1.2379) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:33:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][110/625] eta 0:03:20 lr 0.001977 wd 0.0500 time 0.3779 (0.3890) data time 0.0011 (0.0048) model time 0.3767 (0.3890) loss 2.8598 (3.7192) grad_norm 1.1648 (1.2338) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][120/625] eta 0:03:16 lr 0.001977 wd 0.0500 time 0.3769 (0.3883) data time 0.0009 (0.0045) model time 0.3761 (0.3876) loss 3.2442 (3.7204) grad_norm 1.7803 (1.2328) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][130/625] eta 0:03:11 lr 0.001977 wd 0.0500 time 0.3738 (0.3873) data time 0.0011 (0.0043) model time 0.3727 (0.3859) loss 3.9538 (3.7286) grad_norm 2.2254 (1.2460) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][140/625] eta 0:03:07 lr 0.001977 wd 0.0500 time 0.3791 (0.3867) data time 0.0011 (0.0041) model time 0.3780 (0.3849) loss 3.9724 (3.7041) grad_norm 1.1232 (1.2436) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:33:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][150/625] eta 0:03:04 lr 0.001977 wd 0.0500 time 0.5836 (0.3886) data time 0.0009 (0.0039) model time 0.5827 (0.3879) loss 2.7066 (3.7084) grad_norm 1.0826 (1.2386) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 05:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][160/625] eta 0:03:01 lr 0.001977 wd 0.0500 time 0.3766 (0.3893) data time 0.0011 (0.0037) model time 0.3755 (0.3889) loss 3.5101 (3.7066) grad_norm 1.2581 (inf) loss_scale 8192.0000 (15875.1801) mem 11225MB [2024-08-26 05:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][170/625] eta 0:02:56 lr 0.001977 wd 0.0500 time 0.3713 (0.3886) data time 0.0008 (0.0036) model time 0.3704 (0.3878) loss 2.9012 (3.7069) grad_norm 1.6911 (inf) loss_scale 8192.0000 (15425.8713) mem 11225MB [2024-08-26 05:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][180/625] eta 0:02:52 lr 0.001977 wd 0.0500 time 0.3773 (0.3880) data time 0.0009 (0.0034) model time 0.3764 (0.3870) loss 3.8120 (3.7063) grad_norm 1.0180 (inf) loss_scale 8192.0000 (15026.2099) mem 11225MB [2024-08-26 05:33:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][190/625] eta 0:02:48 lr 0.001977 wd 0.0500 time 0.3725 (0.3876) data time 0.0008 (0.0033) model time 0.3717 (0.3864) loss 3.7252 (3.7181) grad_norm 1.1914 (inf) loss_scale 8192.0000 (14668.3979) mem 11225MB [2024-08-26 05:33:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][200/625] eta 0:02:44 lr 0.001977 wd 0.0500 time 0.3648 (0.3871) data time 0.0011 (0.0032) model time 0.3637 (0.3858) loss 3.6216 (3.7180) grad_norm 1.3864 (inf) loss_scale 8192.0000 (14346.1891) mem 11225MB [2024-08-26 05:33:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][210/625] eta 0:02:40 lr 0.001977 wd 0.0500 time 0.3811 (0.3867) data time 0.0009 (0.0031) model time 0.3802 (0.3852) loss 4.5862 (3.7153) grad_norm 0.9395 (inf) loss_scale 8192.0000 (14054.5213) mem 11225MB [2024-08-26 05:33:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][220/625] eta 0:02:36 lr 0.001977 wd 0.0500 time 0.3761 (0.3863) data time 0.0009 (0.0030) model time 0.3752 (0.3847) loss 4.2983 (3.7192) grad_norm 1.0718 (inf) loss_scale 8192.0000 (13789.2489) mem 11225MB [2024-08-26 05:33:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][230/625] eta 0:02:32 lr 0.001977 wd 0.0500 time 0.3633 (0.3860) data time 0.0008 (0.0029) model time 0.3625 (0.3843) loss 2.6316 (3.7185) grad_norm 1.0649 (inf) loss_scale 8192.0000 (13546.9437) mem 11225MB [2024-08-26 05:33:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][240/625] eta 0:02:28 lr 0.001977 wd 0.0500 time 0.3750 (0.3857) data time 0.0009 (0.0029) model time 0.3741 (0.3840) loss 2.9199 (3.7174) grad_norm 1.2109 (inf) loss_scale 8192.0000 (13324.7469) mem 11225MB [2024-08-26 05:34:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][250/625] eta 0:02:24 lr 0.001977 wd 0.0500 time 0.3819 (0.3855) data time 0.0008 (0.0028) model time 0.3811 (0.3838) loss 3.9698 (3.7286) grad_norm 1.0730 (inf) loss_scale 8192.0000 (13120.2550) mem 11225MB [2024-08-26 05:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][260/625] eta 0:02:20 lr 0.001977 wd 0.0500 time 0.3687 (0.3852) data time 0.0011 (0.0028) model time 0.3676 (0.3834) loss 3.9823 (3.7335) grad_norm 0.9424 (inf) loss_scale 8192.0000 (12931.4330) mem 11225MB [2024-08-26 05:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][270/625] eta 0:02:16 lr 0.001977 wd 0.0500 time 0.3750 (0.3855) data time 0.0009 (0.0027) model time 0.3742 (0.3839) loss 4.2459 (3.7271) grad_norm 1.4160 (inf) loss_scale 8192.0000 (12756.5461) mem 11225MB [2024-08-26 05:34:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][280/625] eta 0:02:12 lr 0.001977 wd 0.0500 time 0.3768 (0.3852) data time 0.0011 (0.0027) model time 0.3757 (0.3835) loss 3.8635 (3.7203) grad_norm 1.3241 (inf) loss_scale 8192.0000 (12594.1068) mem 11225MB [2024-08-26 05:34:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][290/625] eta 0:02:08 lr 0.001976 wd 0.0500 time 0.3773 (0.3850) data time 0.0011 (0.0026) model time 0.3762 (0.3832) loss 3.5989 (3.7190) grad_norm 1.5956 (inf) loss_scale 8192.0000 (12442.8316) mem 11225MB [2024-08-26 05:34:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][300/625] eta 0:02:05 lr 0.001976 wd 0.0500 time 0.3716 (0.3848) data time 0.0008 (0.0025) model time 0.3707 (0.3830) loss 3.0024 (3.7171) grad_norm 1.0640 (inf) loss_scale 8192.0000 (12301.6080) mem 11225MB [2024-08-26 05:34:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][310/625] eta 0:02:01 lr 0.001976 wd 0.0500 time 0.3772 (0.3863) data time 0.0009 (0.0025) model time 0.3763 (0.3848) loss 4.7815 (3.7211) grad_norm 0.8697 (inf) loss_scale 8192.0000 (12169.4662) mem 11225MB [2024-08-26 05:34:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][320/625] eta 0:01:57 lr 0.001976 wd 0.0500 time 0.3770 (0.3861) data time 0.0012 (0.0025) model time 0.3758 (0.3846) loss 4.7189 (3.7280) grad_norm 2.1054 (inf) loss_scale 8192.0000 (12045.5576) mem 11225MB [2024-08-26 05:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][330/625] eta 0:01:53 lr 0.001976 wd 0.0500 time 0.3713 (0.3858) data time 0.0011 (0.0024) model time 0.3702 (0.3843) loss 4.3125 (3.7225) grad_norm 1.3196 (inf) loss_scale 8192.0000 (11929.1360) mem 11225MB [2024-08-26 05:34:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][340/625] eta 0:01:50 lr 0.001976 wd 0.0500 time 0.3685 (0.3867) data time 0.0011 (0.0024) model time 0.3674 (0.3854) loss 3.9033 (3.7218) grad_norm 1.2098 (inf) loss_scale 8192.0000 (11819.5425) mem 11225MB [2024-08-26 05:34:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][350/625] eta 0:01:46 lr 0.001976 wd 0.0500 time 0.3689 (0.3864) data time 0.0011 (0.0024) model time 0.3678 (0.3850) loss 4.1591 (3.7271) grad_norm 0.9523 (inf) loss_scale 8192.0000 (11716.1937) mem 11225MB [2024-08-26 05:34:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][360/625] eta 0:01:42 lr 0.001976 wd 0.0500 time 0.3911 (0.3862) data time 0.0008 (0.0023) model time 0.3903 (0.3848) loss 3.4607 (3.7219) grad_norm 1.1738 (inf) loss_scale 8192.0000 (11618.5706) mem 11225MB [2024-08-26 05:34:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][370/625] eta 0:01:38 lr 0.001976 wd 0.0500 time 0.3793 (0.3860) data time 0.0012 (0.0023) model time 0.3781 (0.3845) loss 3.5918 (3.7281) grad_norm 1.2459 (inf) loss_scale 8192.0000 (11526.2102) mem 11225MB [2024-08-26 05:34:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][380/625] eta 0:01:34 lr 0.001976 wd 0.0500 time 0.3734 (0.3857) data time 0.0008 (0.0023) model time 0.3726 (0.3842) loss 4.4140 (3.7202) grad_norm 1.3337 (inf) loss_scale 8192.0000 (11438.6982) mem 11225MB [2024-08-26 05:34:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][390/625] eta 0:01:30 lr 0.001976 wd 0.0500 time 0.3740 (0.3855) data time 0.0011 (0.0023) model time 0.3729 (0.3839) loss 3.8695 (3.7214) grad_norm 1.3489 (inf) loss_scale 8192.0000 (11355.6624) mem 11225MB [2024-08-26 05:35:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][400/625] eta 0:01:26 lr 0.001976 wd 0.0500 time 0.3535 (0.3853) data time 0.0014 (0.0022) model time 0.3520 (0.3838) loss 3.1859 (3.7113) grad_norm 1.2303 (inf) loss_scale 8192.0000 (11276.7681) mem 11225MB [2024-08-26 05:35:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][410/625] eta 0:01:22 lr 0.001976 wd 0.0500 time 0.3766 (0.3852) data time 0.0011 (0.0022) model time 0.3755 (0.3836) loss 3.7520 (3.7100) grad_norm 1.2853 (inf) loss_scale 8192.0000 (11201.7129) mem 11225MB [2024-08-26 05:35:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][420/625] eta 0:01:18 lr 0.001976 wd 0.0500 time 0.3814 (0.3851) data time 0.0008 (0.0022) model time 0.3807 (0.3835) loss 4.6896 (3.7089) grad_norm 1.2368 (inf) loss_scale 8192.0000 (11130.2233) mem 11225MB [2024-08-26 05:35:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][430/625] eta 0:01:15 lr 0.001976 wd 0.0500 time 0.3744 (0.3849) data time 0.0008 (0.0022) model time 0.3735 (0.3834) loss 3.4114 (3.7156) grad_norm 1.2580 (inf) loss_scale 8192.0000 (11062.0510) mem 11225MB [2024-08-26 05:35:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][440/625] eta 0:01:11 lr 0.001976 wd 0.0500 time 0.3732 (0.3847) data time 0.0009 (0.0021) model time 0.3723 (0.3831) loss 4.4282 (3.7211) grad_norm 0.8578 (inf) loss_scale 8192.0000 (10996.9705) mem 11225MB [2024-08-26 05:35:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][450/625] eta 0:01:07 lr 0.001976 wd 0.0500 time 0.3780 (0.3846) data time 0.0010 (0.0021) model time 0.3770 (0.3830) loss 4.1862 (3.7239) grad_norm 1.1563 (inf) loss_scale 8192.0000 (10934.7761) mem 11225MB [2024-08-26 05:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][460/625] eta 0:01:03 lr 0.001976 wd 0.0500 time 0.3747 (0.3853) data time 0.0008 (0.0021) model time 0.3738 (0.3838) loss 3.3716 (3.7189) grad_norm 0.9232 (inf) loss_scale 8192.0000 (10875.2798) mem 11225MB [2024-08-26 05:35:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][470/625] eta 0:00:59 lr 0.001976 wd 0.0500 time 0.3749 (0.3852) data time 0.0011 (0.0021) model time 0.3738 (0.3837) loss 3.9733 (3.7115) grad_norm 1.0251 (inf) loss_scale 8192.0000 (10818.3100) mem 11225MB [2024-08-26 05:35:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][480/625] eta 0:00:55 lr 0.001976 wd 0.0500 time 0.3786 (0.3855) data time 0.0009 (0.0021) model time 0.3777 (0.3840) loss 3.8549 (3.7093) grad_norm 1.1256 (inf) loss_scale 8192.0000 (10763.7089) mem 11225MB [2024-08-26 05:35:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][490/625] eta 0:00:52 lr 0.001976 wd 0.0500 time 0.3762 (0.3864) data time 0.0008 (0.0020) model time 0.3754 (0.3851) loss 4.3320 (3.7146) grad_norm 1.1651 (inf) loss_scale 8192.0000 (10711.3320) mem 11225MB [2024-08-26 05:35:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][500/625] eta 0:00:48 lr 0.001976 wd 0.0500 time 0.3815 (0.3862) data time 0.0011 (0.0020) model time 0.3805 (0.3848) loss 3.6985 (3.7063) grad_norm 1.0938 (inf) loss_scale 8192.0000 (10661.0459) mem 11225MB [2024-08-26 05:35:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][510/625] eta 0:00:44 lr 0.001976 wd 0.0500 time 0.3749 (0.3860) data time 0.0010 (0.0020) model time 0.3739 (0.3846) loss 4.2074 (3.7068) grad_norm 1.1508 (inf) loss_scale 8192.0000 (10612.7280) mem 11225MB [2024-08-26 05:35:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][520/625] eta 0:00:40 lr 0.001976 wd 0.0500 time 0.3774 (0.3859) data time 0.0009 (0.0020) model time 0.3765 (0.3845) loss 2.6851 (3.7084) grad_norm 1.1138 (inf) loss_scale 8192.0000 (10566.2649) mem 11225MB [2024-08-26 05:35:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][530/625] eta 0:00:36 lr 0.001976 wd 0.0500 time 0.3735 (0.3857) data time 0.0008 (0.0020) model time 0.3727 (0.3843) loss 4.5106 (3.7068) grad_norm 1.7310 (inf) loss_scale 8192.0000 (10521.5518) mem 11225MB [2024-08-26 05:35:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][540/625] eta 0:00:32 lr 0.001976 wd 0.0500 time 0.3755 (0.3856) data time 0.0008 (0.0020) model time 0.3746 (0.3842) loss 2.9377 (3.7000) grad_norm 0.9548 (inf) loss_scale 8192.0000 (10478.4917) mem 11225MB [2024-08-26 05:35:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][550/625] eta 0:00:28 lr 0.001975 wd 0.0500 time 0.3778 (0.3855) data time 0.0012 (0.0019) model time 0.3767 (0.3841) loss 3.0751 (3.6973) grad_norm 1.5012 (inf) loss_scale 8192.0000 (10436.9946) mem 11225MB [2024-08-26 05:36:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][560/625] eta 0:00:25 lr 0.001975 wd 0.0500 time 0.3808 (0.3854) data time 0.0011 (0.0019) model time 0.3797 (0.3840) loss 3.6617 (3.7012) grad_norm 0.8653 (inf) loss_scale 8192.0000 (10396.9768) mem 11225MB [2024-08-26 05:36:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][570/625] eta 0:00:21 lr 0.001975 wd 0.0500 time 0.3788 (0.3853) data time 0.0011 (0.0019) model time 0.3777 (0.3839) loss 2.9580 (3.7039) grad_norm 1.1011 (inf) loss_scale 8192.0000 (10358.3608) mem 11225MB [2024-08-26 05:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][580/625] eta 0:00:17 lr 0.001975 wd 0.0500 time 0.3725 (0.3851) data time 0.0010 (0.0019) model time 0.3715 (0.3837) loss 3.9065 (3.7030) grad_norm 0.9528 (inf) loss_scale 8192.0000 (10321.0740) mem 11225MB [2024-08-26 05:36:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][590/625] eta 0:00:13 lr 0.001975 wd 0.0500 time 0.3715 (0.3850) data time 0.0009 (0.0019) model time 0.3707 (0.3835) loss 4.3302 (3.7052) grad_norm 1.6571 (inf) loss_scale 8192.0000 (10285.0491) mem 11225MB [2024-08-26 05:36:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][600/625] eta 0:00:09 lr 0.001975 wd 0.0500 time 0.3812 (0.3848) data time 0.0009 (0.0019) model time 0.3803 (0.3834) loss 3.4243 (3.7047) grad_norm 0.9305 (inf) loss_scale 8192.0000 (10250.2230) mem 11225MB [2024-08-26 05:36:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][610/625] eta 0:00:05 lr 0.001975 wd 0.0500 time 0.3662 (0.3854) data time 0.0008 (0.0019) model time 0.3654 (0.3840) loss 3.7329 (3.7035) grad_norm 1.2247 (inf) loss_scale 8192.0000 (10216.5368) mem 11225MB [2024-08-26 05:36:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [39/300][620/625] eta 0:00:01 lr 0.001975 wd 0.0500 time 0.3679 (0.3851) data time 0.0006 (0.0019) model time 0.3673 (0.3837) loss 3.4722 (3.7059) grad_norm 1.2820 (inf) loss_scale 8192.0000 (10183.9356) mem 11225MB [2024-08-26 05:36:26 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 39 training takes 0:04:00 [2024-08-26 05:36:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:36:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:36:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.7402 (0.7402) Acc@1 82.422 (82.422) Acc@5 96.826 (96.826) Mem 11225MB [2024-08-26 05:36:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 1.2227 (0.9069) Acc@1 69.629 (78.840) Acc@5 90.967 (95.188) Mem 11225MB [2024-08-26 05:36:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.3984 (1.1122) Acc@1 66.260 (74.105) Acc@5 88.184 (92.327) Mem 11225MB [2024-08-26 05:36:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.912 Acc@5 92.306 [2024-08-26 05:36:31 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 73.9% [2024-08-26 05:36:31 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 73.91% [2024-08-26 05:36:31 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:36:32 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:36:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 0.6069 (0.6069) Acc@1 84.473 (84.473) Acc@5 97.021 (97.021) Mem 11225MB [2024-08-26 05:36:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.142) Loss 1.1143 (0.7856) Acc@1 71.484 (79.719) Acc@5 91.455 (95.481) Mem 11225MB [2024-08-26 05:36:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 1.3105 (0.9914) Acc@1 67.432 (75.151) Acc@5 88.770 (92.806) Mem 11225MB [2024-08-26 05:36:35 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.904 Acc@5 92.774 [2024-08-26 05:36:35 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 74.9% [2024-08-26 05:36:35 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 74.90% [2024-08-26 05:36:35 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:36:36 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:36:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][0/625] eta 0:07:52 lr 0.001975 wd 0.0500 time 0.7558 (0.7558) data time 0.4085 (0.4085) model time 0.0000 (0.0000) loss 3.7154 (3.7154) grad_norm 1.0113 (1.0113) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:36:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][10/625] eta 0:04:14 lr 0.001975 wd 0.0500 time 0.3897 (0.4146) data time 0.0008 (0.0382) model time 0.0000 (0.0000) loss 3.6317 (3.6299) grad_norm 1.0175 (1.3149) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:36:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][20/625] eta 0:04:00 lr 0.001975 wd 0.0500 time 0.3790 (0.3977) data time 0.0008 (0.0205) model time 0.0000 (0.0000) loss 4.4036 (3.7292) grad_norm 1.1745 (1.3745) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:36:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][30/625] eta 0:04:00 lr 0.001975 wd 0.0500 time 0.3821 (0.4045) data time 0.0008 (0.0142) model time 0.0000 (0.0000) loss 4.2471 (3.6654) grad_norm 1.3295 (1.3082) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:36:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][40/625] eta 0:03:52 lr 0.001975 wd 0.0500 time 0.4003 (0.3980) data time 0.0011 (0.0111) model time 0.0000 (0.0000) loss 3.0085 (3.6042) grad_norm 0.9757 (1.2731) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:36:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][50/625] eta 0:03:46 lr 0.001975 wd 0.0500 time 0.3736 (0.3938) data time 0.0009 (0.0091) model time 0.0000 (0.0000) loss 3.2738 (3.6294) grad_norm 1.6624 (1.2801) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][60/625] eta 0:03:41 lr 0.001975 wd 0.0500 time 0.3790 (0.3912) data time 0.0013 (0.0078) model time 0.3777 (0.3772) loss 4.0330 (3.6591) grad_norm 0.8426 (1.2719) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][70/625] eta 0:03:36 lr 0.001975 wd 0.0500 time 0.3821 (0.3894) data time 0.0008 (0.0068) model time 0.3813 (0.3773) loss 4.7492 (3.7148) grad_norm 1.2381 (1.2656) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][80/625] eta 0:03:31 lr 0.001975 wd 0.0500 time 0.3775 (0.3882) data time 0.0008 (0.0061) model time 0.3767 (0.3777) loss 3.8418 (3.6945) grad_norm 1.6930 (1.2699) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][90/625] eta 0:03:27 lr 0.001975 wd 0.0500 time 0.3766 (0.3871) data time 0.0011 (0.0056) model time 0.3755 (0.3775) loss 3.4683 (3.6814) grad_norm 2.1280 (1.2784) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][100/625] eta 0:03:22 lr 0.001975 wd 0.0500 time 0.3802 (0.3863) data time 0.0008 (0.0052) model time 0.3794 (0.3776) loss 3.9833 (3.6696) grad_norm 0.9892 (1.2777) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][110/625] eta 0:03:18 lr 0.001975 wd 0.0500 time 0.3787 (0.3857) data time 0.0008 (0.0048) model time 0.3779 (0.3778) loss 4.0201 (3.6632) grad_norm 1.9055 (1.2752) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][120/625] eta 0:03:14 lr 0.001975 wd 0.0500 time 0.3733 (0.3853) data time 0.0008 (0.0045) model time 0.3725 (0.3781) loss 4.3742 (3.6905) grad_norm 1.0930 (1.2832) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][130/625] eta 0:03:10 lr 0.001975 wd 0.0500 time 0.3788 (0.3850) data time 0.0010 (0.0042) model time 0.3778 (0.3783) loss 3.3462 (3.6628) grad_norm 1.0608 (1.2893) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][140/625] eta 0:03:06 lr 0.001975 wd 0.0500 time 0.3798 (0.3846) data time 0.0019 (0.0040) model time 0.3779 (0.3782) loss 3.7746 (3.6626) grad_norm 1.2966 (1.2805) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][150/625] eta 0:03:02 lr 0.001975 wd 0.0500 time 0.3819 (0.3842) data time 0.0008 (0.0038) model time 0.3811 (0.3782) loss 3.2919 (3.6665) grad_norm 1.1976 (1.2675) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][160/625] eta 0:02:59 lr 0.001975 wd 0.0500 time 0.3759 (0.3851) data time 0.0008 (0.0037) model time 0.3751 (0.3799) loss 2.7380 (3.6623) grad_norm 0.9700 (1.2673) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][170/625] eta 0:02:56 lr 0.001975 wd 0.0500 time 0.3799 (0.3880) data time 0.0008 (0.0035) model time 0.3790 (0.3844) loss 2.4536 (3.6624) grad_norm 0.9175 (1.2597) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][180/625] eta 0:02:52 lr 0.001974 wd 0.0500 time 0.3806 (0.3875) data time 0.0008 (0.0034) model time 0.3799 (0.3838) loss 4.1904 (3.6729) grad_norm 1.2032 (1.2576) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][190/625] eta 0:02:48 lr 0.001974 wd 0.0500 time 0.3787 (0.3870) data time 0.0008 (0.0033) model time 0.3779 (0.3834) loss 4.5614 (3.6758) grad_norm 1.0864 (1.2545) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][200/625] eta 0:02:44 lr 0.001974 wd 0.0500 time 0.3748 (0.3866) data time 0.0008 (0.0032) model time 0.3739 (0.3830) loss 4.0863 (3.6704) grad_norm 1.3527 (1.2821) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:37:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][210/625] eta 0:02:40 lr 0.001974 wd 0.0500 time 0.3747 (0.3862) data time 0.0008 (0.0031) model time 0.3738 (0.3826) loss 2.3329 (3.6579) grad_norm 0.9380 (1.2786) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][220/625] eta 0:02:36 lr 0.001974 wd 0.0500 time 0.3889 (0.3860) data time 0.0009 (0.0030) model time 0.3881 (0.3825) loss 4.3098 (3.6680) grad_norm 1.0048 (1.2697) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][230/625] eta 0:02:32 lr 0.001974 wd 0.0500 time 0.3743 (0.3856) data time 0.0011 (0.0029) model time 0.3732 (0.3822) loss 4.0462 (3.6598) grad_norm 1.9444 (1.2662) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][240/625] eta 0:02:28 lr 0.001974 wd 0.0500 time 0.3857 (0.3870) data time 0.0009 (0.0028) model time 0.3848 (0.3841) loss 2.5906 (3.6691) grad_norm 1.1805 (1.2705) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][250/625] eta 0:02:24 lr 0.001974 wd 0.0500 time 0.3737 (0.3866) data time 0.0009 (0.0028) model time 0.3728 (0.3837) loss 2.5119 (3.6722) grad_norm 1.4688 (1.2708) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][260/625] eta 0:02:21 lr 0.001974 wd 0.0500 time 0.3756 (0.3863) data time 0.0009 (0.0027) model time 0.3747 (0.3834) loss 4.2357 (3.6718) grad_norm 0.7946 (1.2694) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][270/625] eta 0:02:17 lr 0.001974 wd 0.0500 time 0.3845 (0.3860) data time 0.0008 (0.0027) model time 0.3837 (0.3831) loss 4.4850 (3.6785) grad_norm 0.8651 (1.2690) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][280/625] eta 0:02:13 lr 0.001974 wd 0.0500 time 0.3831 (0.3863) data time 0.0011 (0.0026) model time 0.3820 (0.3835) loss 3.2801 (3.6792) grad_norm 1.3180 (1.2685) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][290/625] eta 0:02:09 lr 0.001974 wd 0.0500 time 0.3846 (0.3875) data time 0.0009 (0.0026) model time 0.3837 (0.3850) loss 2.6872 (3.6828) grad_norm 1.2822 (1.2722) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][300/625] eta 0:02:05 lr 0.001974 wd 0.0500 time 0.3747 (0.3871) data time 0.0009 (0.0025) model time 0.3738 (0.3846) loss 3.9678 (3.6797) grad_norm 1.3053 (1.2844) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][310/625] eta 0:02:01 lr 0.001974 wd 0.0500 time 0.3765 (0.3868) data time 0.0010 (0.0025) model time 0.3754 (0.3843) loss 3.5213 (3.6843) grad_norm 1.1929 (1.2810) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][320/625] eta 0:01:57 lr 0.001974 wd 0.0500 time 0.3872 (0.3866) data time 0.0011 (0.0024) model time 0.3861 (0.3841) loss 3.3880 (3.6852) grad_norm 1.0929 (1.2750) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][330/625] eta 0:01:53 lr 0.001974 wd 0.0500 time 0.3741 (0.3864) data time 0.0008 (0.0024) model time 0.3733 (0.3839) loss 4.6165 (3.6936) grad_norm 1.3458 (1.2748) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][340/625] eta 0:01:50 lr 0.001974 wd 0.0500 time 0.3875 (0.3861) data time 0.0008 (0.0023) model time 0.3867 (0.3836) loss 2.9639 (3.6964) grad_norm 1.3537 (1.2767) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][350/625] eta 0:01:46 lr 0.001974 wd 0.0500 time 0.3691 (0.3858) data time 0.0008 (0.0023) model time 0.3683 (0.3833) loss 4.9806 (3.6949) grad_norm 1.1050 (1.2730) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][360/625] eta 0:01:42 lr 0.001974 wd 0.0500 time 0.3855 (0.3856) data time 0.0009 (0.0023) model time 0.3847 (0.3831) loss 2.9127 (3.6910) grad_norm 0.9118 (1.2717) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][370/625] eta 0:01:38 lr 0.001974 wd 0.0500 time 0.3823 (0.3854) data time 0.0011 (0.0022) model time 0.3811 (0.3830) loss 2.7708 (3.6845) grad_norm 1.3635 (1.2699) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][380/625] eta 0:01:34 lr 0.001974 wd 0.0500 time 0.3680 (0.3852) data time 0.0011 (0.0022) model time 0.3670 (0.3827) loss 4.0102 (3.6737) grad_norm 1.1890 (1.2681) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][390/625] eta 0:01:30 lr 0.001974 wd 0.0500 time 0.3827 (0.3850) data time 0.0011 (0.0022) model time 0.3816 (0.3826) loss 3.7092 (3.6762) grad_norm 1.2626 (1.2686) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][400/625] eta 0:01:26 lr 0.001974 wd 0.0500 time 0.3819 (0.3848) data time 0.0009 (0.0022) model time 0.3810 (0.3824) loss 3.1386 (3.6706) grad_norm 1.0862 (1.2655) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][410/625] eta 0:01:22 lr 0.001974 wd 0.0500 time 0.3773 (0.3847) data time 0.0011 (0.0021) model time 0.3762 (0.3822) loss 3.4789 (3.6712) grad_norm 0.9549 (1.2637) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][420/625] eta 0:01:19 lr 0.001973 wd 0.0500 time 0.5838 (0.3855) data time 0.0008 (0.0021) model time 0.5830 (0.3833) loss 4.2618 (3.6704) grad_norm 1.2702 (1.2637) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][430/625] eta 0:01:15 lr 0.001973 wd 0.0500 time 0.3884 (0.3855) data time 0.0008 (0.0021) model time 0.3875 (0.3832) loss 2.7731 (3.6621) grad_norm 0.9072 (1.2615) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][440/625] eta 0:01:11 lr 0.001973 wd 0.0500 time 0.3793 (0.3852) data time 0.0008 (0.0021) model time 0.3785 (0.3830) loss 4.4822 (3.6627) grad_norm 1.1837 (1.2648) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][450/625] eta 0:01:07 lr 0.001973 wd 0.0500 time 0.3835 (0.3851) data time 0.0011 (0.0021) model time 0.3824 (0.3828) loss 3.6936 (3.6683) grad_norm 1.4561 (1.2650) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][460/625] eta 0:01:03 lr 0.001973 wd 0.0500 time 0.3786 (0.3849) data time 0.0011 (0.0020) model time 0.3775 (0.3827) loss 2.6334 (3.6575) grad_norm 1.4141 (1.2598) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][470/625] eta 0:00:59 lr 0.001973 wd 0.0500 time 0.3758 (0.3848) data time 0.0011 (0.0020) model time 0.3747 (0.3825) loss 3.4211 (3.6623) grad_norm 1.2136 (1.2580) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][480/625] eta 0:00:55 lr 0.001973 wd 0.0500 time 0.3755 (0.3847) data time 0.0008 (0.0020) model time 0.3747 (0.3825) loss 4.1769 (3.6621) grad_norm 1.1656 (1.2543) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][490/625] eta 0:00:51 lr 0.001973 wd 0.0500 time 0.3771 (0.3849) data time 0.0011 (0.0020) model time 0.3760 (0.3827) loss 3.1642 (3.6617) grad_norm 1.2057 (1.2553) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][500/625] eta 0:00:48 lr 0.001973 wd 0.0500 time 0.3747 (0.3859) data time 0.0008 (0.0020) model time 0.3739 (0.3839) loss 2.6919 (3.6618) grad_norm 1.0553 (1.2534) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][510/625] eta 0:00:44 lr 0.001973 wd 0.0500 time 0.3723 (0.3857) data time 0.0009 (0.0019) model time 0.3713 (0.3837) loss 3.1923 (3.6630) grad_norm 1.4236 (1.2532) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:39:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][520/625] eta 0:00:40 lr 0.001973 wd 0.0500 time 0.3730 (0.3856) data time 0.0012 (0.0019) model time 0.3719 (0.3836) loss 4.1514 (3.6656) grad_norm 1.4379 (1.2550) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][530/625] eta 0:00:36 lr 0.001973 wd 0.0500 time 0.3769 (0.3855) data time 0.0011 (0.0019) model time 0.3759 (0.3834) loss 4.0413 (3.6740) grad_norm 1.0888 (1.2558) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][540/625] eta 0:00:32 lr 0.001973 wd 0.0500 time 0.3848 (0.3861) data time 0.0009 (0.0019) model time 0.3839 (0.3842) loss 4.0711 (3.6710) grad_norm 1.1775 (1.2574) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][550/625] eta 0:00:28 lr 0.001973 wd 0.0500 time 0.3790 (0.3860) data time 0.0011 (0.0019) model time 0.3778 (0.3840) loss 3.8963 (3.6668) grad_norm 1.2773 (1.2580) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][560/625] eta 0:00:25 lr 0.001973 wd 0.0500 time 0.3803 (0.3859) data time 0.0009 (0.0019) model time 0.3794 (0.3840) loss 3.5126 (3.6675) grad_norm 1.3333 (1.2580) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][570/625] eta 0:00:21 lr 0.001973 wd 0.0500 time 0.3704 (0.3858) data time 0.0009 (0.0019) model time 0.3695 (0.3838) loss 4.1015 (3.6693) grad_norm 0.9392 (1.2560) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][580/625] eta 0:00:17 lr 0.001973 wd 0.0500 time 0.3825 (0.3857) data time 0.0012 (0.0019) model time 0.3813 (0.3837) loss 4.2144 (3.6716) grad_norm 0.9603 (1.2528) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][590/625] eta 0:00:13 lr 0.001973 wd 0.0500 time 0.3856 (0.3856) data time 0.0009 (0.0018) model time 0.3847 (0.3836) loss 4.7807 (3.6773) grad_norm 1.2993 (1.2536) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][600/625] eta 0:00:09 lr 0.001973 wd 0.0500 time 0.3797 (0.3855) data time 0.0008 (0.0018) model time 0.3789 (0.3835) loss 3.6976 (3.6773) grad_norm 1.1570 (1.2526) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][610/625] eta 0:00:05 lr 0.001973 wd 0.0500 time 0.3733 (0.3854) data time 0.0008 (0.0018) model time 0.3725 (0.3834) loss 3.8245 (3.6795) grad_norm 1.0281 (1.2498) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [40/300][620/625] eta 0:00:01 lr 0.001973 wd 0.0500 time 0.3688 (0.3851) data time 0.0006 (0.0018) model time 0.3682 (0.3832) loss 2.5953 (3.6754) grad_norm 2.1243 (1.2494) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 40 training takes 0:04:00 [2024-08-26 05:40:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:40:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:40:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.504 (0.504) Loss 0.7837 (0.7837) Acc@1 83.154 (83.154) Acc@5 96.582 (96.582) Mem 11225MB [2024-08-26 05:40:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.142) Loss 1.2578 (0.9233) Acc@1 70.508 (78.369) Acc@5 90.576 (94.971) Mem 11225MB [2024-08-26 05:40:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.3945 (1.1175) Acc@1 66.406 (73.972) Acc@5 88.721 (92.334) Mem 11225MB [2024-08-26 05:40:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.682 Acc@5 92.278 [2024-08-26 05:40:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 73.7% [2024-08-26 05:40:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.845 (0.845) Loss 0.6016 (0.6016) Acc@1 84.473 (84.473) Acc@5 97.266 (97.266) Mem 11225MB [2024-08-26 05:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.178) Loss 1.0996 (0.7781) Acc@1 71.826 (79.949) Acc@5 91.650 (95.597) Mem 11225MB [2024-08-26 05:40:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.138) Loss 1.3008 (0.9812) Acc@1 67.627 (75.381) Acc@5 89.014 (92.973) Mem 11225MB [2024-08-26 05:40:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.126 Acc@5 92.944 [2024-08-26 05:40:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 75.1% [2024-08-26 05:40:44 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 75.13% [2024-08-26 05:40:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:40:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:40:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][0/625] eta 0:07:39 lr 0.001973 wd 0.0500 time 0.7345 (0.7345) data time 0.3961 (0.3961) model time 0.0000 (0.0000) loss 3.3297 (3.3297) grad_norm 1.1475 (1.1475) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][10/625] eta 0:04:13 lr 0.001973 wd 0.0500 time 0.3758 (0.4124) data time 0.0009 (0.0370) model time 0.0000 (0.0000) loss 3.3810 (3.5601) grad_norm 1.2643 (1.2238) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][20/625] eta 0:04:01 lr 0.001973 wd 0.0500 time 0.3755 (0.3985) data time 0.0008 (0.0199) model time 0.0000 (0.0000) loss 4.5929 (3.6752) grad_norm 1.0710 (1.1769) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:40:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][30/625] eta 0:03:53 lr 0.001973 wd 0.0500 time 0.3805 (0.3924) data time 0.0008 (0.0138) model time 0.0000 (0.0000) loss 4.4758 (3.8223) grad_norm 1.1195 (1.2147) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][40/625] eta 0:03:47 lr 0.001972 wd 0.0500 time 0.3859 (0.3893) data time 0.0011 (0.0107) model time 0.0000 (0.0000) loss 4.2548 (3.7209) grad_norm 1.1167 (1.1962) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][50/625] eta 0:03:42 lr 0.001972 wd 0.0500 time 0.3779 (0.3870) data time 0.0009 (0.0089) model time 0.0000 (0.0000) loss 4.2149 (3.7506) grad_norm 1.3100 (1.1860) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][60/625] eta 0:03:39 lr 0.001972 wd 0.0500 time 0.5875 (0.3886) data time 0.0008 (0.0076) model time 0.5867 (0.3961) loss 3.6452 (3.6754) grad_norm 1.3292 (1.1843) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][70/625] eta 0:03:42 lr 0.001972 wd 0.0500 time 0.5998 (0.4001) data time 0.0011 (0.0067) model time 0.5987 (0.4327) loss 3.4659 (3.6673) grad_norm 1.4747 (1.2343) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][80/625] eta 0:03:36 lr 0.001972 wd 0.0500 time 0.3690 (0.3973) data time 0.0008 (0.0060) model time 0.3682 (0.4138) loss 4.2536 (3.6661) grad_norm 1.1098 (1.2234) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][90/625] eta 0:03:31 lr 0.001972 wd 0.0500 time 0.3817 (0.3952) data time 0.0009 (0.0054) model time 0.3809 (0.4047) loss 3.5918 (3.6554) grad_norm 1.3312 (1.2328) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][100/625] eta 0:03:26 lr 0.001972 wd 0.0500 time 0.3816 (0.3935) data time 0.0009 (0.0050) model time 0.3807 (0.3992) loss 3.8522 (3.6308) grad_norm 1.0372 (1.2238) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][110/625] eta 0:03:21 lr 0.001972 wd 0.0500 time 0.3631 (0.3921) data time 0.0010 (0.0047) model time 0.3621 (0.3953) loss 4.0995 (3.6608) grad_norm 1.4562 (1.2383) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][120/625] eta 0:03:17 lr 0.001972 wd 0.0500 time 0.3756 (0.3909) data time 0.0008 (0.0044) model time 0.3748 (0.3927) loss 3.3213 (3.6404) grad_norm 1.1589 (1.2390) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][130/625] eta 0:03:13 lr 0.001972 wd 0.0500 time 0.3889 (0.3901) data time 0.0011 (0.0041) model time 0.3879 (0.3909) loss 3.7135 (3.6463) grad_norm 1.2181 (1.2372) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][140/625] eta 0:03:08 lr 0.001972 wd 0.0500 time 0.3734 (0.3894) data time 0.0009 (0.0039) model time 0.3726 (0.3897) loss 3.7819 (3.6513) grad_norm 1.2122 (1.2303) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][150/625] eta 0:03:04 lr 0.001972 wd 0.0500 time 0.3869 (0.3888) data time 0.0008 (0.0037) model time 0.3861 (0.3887) loss 3.4878 (3.6602) grad_norm 1.0195 (1.2171) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][160/625] eta 0:03:00 lr 0.001972 wd 0.0500 time 0.3977 (0.3883) data time 0.0011 (0.0036) model time 0.3966 (0.3878) loss 3.6801 (3.6569) grad_norm 1.3201 (1.2179) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][170/625] eta 0:02:56 lr 0.001972 wd 0.0500 time 0.3697 (0.3876) data time 0.0009 (0.0034) model time 0.3688 (0.3868) loss 3.5043 (3.6404) grad_norm 1.2183 (1.2284) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][180/625] eta 0:02:52 lr 0.001972 wd 0.0500 time 0.3847 (0.3873) data time 0.0011 (0.0033) model time 0.3835 (0.3863) loss 4.0260 (3.6378) grad_norm 1.8767 (1.2296) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:41:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][190/625] eta 0:02:48 lr 0.001972 wd 0.0500 time 0.3753 (0.3867) data time 0.0012 (0.0032) model time 0.3742 (0.3855) loss 3.7378 (3.6376) grad_norm 1.2672 (1.2333) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][200/625] eta 0:02:44 lr 0.001972 wd 0.0500 time 0.3744 (0.3863) data time 0.0010 (0.0031) model time 0.3734 (0.3849) loss 4.1601 (3.6418) grad_norm 1.3247 (1.2302) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][210/625] eta 0:02:40 lr 0.001972 wd 0.0500 time 0.3700 (0.3868) data time 0.0008 (0.0030) model time 0.3692 (0.3856) loss 4.2350 (3.6362) grad_norm 1.2006 (1.2316) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][220/625] eta 0:02:36 lr 0.001972 wd 0.0500 time 0.3908 (0.3864) data time 0.0011 (0.0029) model time 0.3897 (0.3851) loss 2.8257 (3.6409) grad_norm 1.3331 (1.2366) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][230/625] eta 0:02:32 lr 0.001972 wd 0.0500 time 0.3659 (0.3860) data time 0.0008 (0.0028) model time 0.3650 (0.3846) loss 3.4699 (3.6422) grad_norm 0.8412 (1.2368) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][240/625] eta 0:02:28 lr 0.001972 wd 0.0500 time 0.3758 (0.3857) data time 0.0008 (0.0027) model time 0.3750 (0.3842) loss 3.3547 (3.6331) grad_norm 0.9999 (1.2274) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][250/625] eta 0:02:25 lr 0.001972 wd 0.0500 time 0.5347 (0.3880) data time 0.0010 (0.0027) model time 0.5337 (0.3872) loss 3.4877 (3.6356) grad_norm 1.6153 (1.2308) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][260/625] eta 0:02:21 lr 0.001972 wd 0.0500 time 0.3734 (0.3877) data time 0.0008 (0.0026) model time 0.3726 (0.3867) loss 4.5300 (3.6455) grad_norm 1.2817 (1.2425) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][270/625] eta 0:02:17 lr 0.001972 wd 0.0500 time 0.3750 (0.3872) data time 0.0010 (0.0026) model time 0.3740 (0.3862) loss 3.5780 (3.6523) grad_norm 1.0500 (1.2422) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][280/625] eta 0:02:13 lr 0.001971 wd 0.0500 time 0.3768 (0.3878) data time 0.0008 (0.0026) model time 0.3759 (0.3868) loss 2.3183 (3.6418) grad_norm 1.0477 (1.2428) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][290/625] eta 0:02:09 lr 0.001971 wd 0.0500 time 0.3748 (0.3875) data time 0.0011 (0.0025) model time 0.3737 (0.3864) loss 3.9377 (3.6456) grad_norm 0.9815 (1.2378) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][300/625] eta 0:02:05 lr 0.001971 wd 0.0500 time 0.3833 (0.3872) data time 0.0011 (0.0025) model time 0.3822 (0.3861) loss 3.8670 (3.6475) grad_norm 1.1489 (1.2356) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][310/625] eta 0:02:01 lr 0.001971 wd 0.0500 time 0.3750 (0.3869) data time 0.0009 (0.0024) model time 0.3741 (0.3858) loss 2.5332 (3.6453) grad_norm 1.0277 (1.2370) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][320/625] eta 0:01:57 lr 0.001971 wd 0.0500 time 0.3739 (0.3867) data time 0.0009 (0.0024) model time 0.3730 (0.3855) loss 4.5432 (3.6528) grad_norm 1.0208 (1.2406) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][330/625] eta 0:01:54 lr 0.001971 wd 0.0500 time 0.3877 (0.3865) data time 0.0009 (0.0024) model time 0.3869 (0.3853) loss 2.9270 (3.6569) grad_norm 0.9492 (1.2422) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:42:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][340/625] eta 0:01:50 lr 0.001971 wd 0.0500 time 0.3815 (0.3864) data time 0.0009 (0.0023) model time 0.3806 (0.3851) loss 3.1432 (3.6547) grad_norm 1.1172 (1.2423) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][350/625] eta 0:01:46 lr 0.001971 wd 0.0500 time 0.3772 (0.3863) data time 0.0010 (0.0023) model time 0.3762 (0.3850) loss 3.4842 (3.6547) grad_norm 1.2075 (1.2406) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][360/625] eta 0:01:42 lr 0.001971 wd 0.0500 time 0.3765 (0.3862) data time 0.0009 (0.0023) model time 0.3756 (0.3849) loss 3.1266 (3.6507) grad_norm 1.2792 (1.2387) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][370/625] eta 0:01:38 lr 0.001971 wd 0.0500 time 0.5873 (0.3877) data time 0.0011 (0.0022) model time 0.5862 (0.3867) loss 2.9988 (3.6532) grad_norm 1.4184 (1.2470) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][380/625] eta 0:01:34 lr 0.001971 wd 0.0500 time 0.3774 (0.3875) data time 0.0008 (0.0022) model time 0.3766 (0.3865) loss 4.0848 (3.6445) grad_norm 0.8491 (1.2439) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][390/625] eta 0:01:31 lr 0.001971 wd 0.0500 time 0.3811 (0.3879) data time 0.0008 (0.0022) model time 0.3803 (0.3869) loss 4.7215 (3.6469) grad_norm 1.2925 (1.2390) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][400/625] eta 0:01:27 lr 0.001971 wd 0.0500 time 0.3999 (0.3878) data time 0.0009 (0.0021) model time 0.3990 (0.3867) loss 3.9997 (3.6509) grad_norm 0.8829 (1.2416) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][410/625] eta 0:01:23 lr 0.001971 wd 0.0500 time 0.3775 (0.3875) data time 0.0009 (0.0021) model time 0.3766 (0.3865) loss 4.6309 (3.6586) grad_norm 1.1599 (1.2443) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][420/625] eta 0:01:19 lr 0.001971 wd 0.0500 time 0.3859 (0.3873) data time 0.0010 (0.0021) model time 0.3848 (0.3862) loss 3.7928 (3.6515) grad_norm 1.0656 (1.2422) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][430/625] eta 0:01:15 lr 0.001971 wd 0.0500 time 0.3927 (0.3872) data time 0.0009 (0.0021) model time 0.3918 (0.3861) loss 2.2575 (3.6497) grad_norm 1.2827 (1.2401) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][440/625] eta 0:01:11 lr 0.001971 wd 0.0500 time 0.3713 (0.3870) data time 0.0009 (0.0021) model time 0.3704 (0.3858) loss 3.2008 (3.6464) grad_norm 1.2104 (1.2401) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][450/625] eta 0:01:07 lr 0.001971 wd 0.0500 time 0.3815 (0.3869) data time 0.0014 (0.0020) model time 0.3801 (0.3857) loss 3.8847 (3.6475) grad_norm 1.1922 (1.2413) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][460/625] eta 0:01:03 lr 0.001971 wd 0.0500 time 0.3783 (0.3867) data time 0.0009 (0.0021) model time 0.3774 (0.3855) loss 4.3024 (3.6525) grad_norm 1.1787 (1.2474) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][470/625] eta 0:00:59 lr 0.001971 wd 0.0500 time 0.3738 (0.3870) data time 0.0011 (0.0021) model time 0.3727 (0.3857) loss 3.9443 (3.6488) grad_norm 0.8138 (1.2449) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][480/625] eta 0:00:56 lr 0.001971 wd 0.0500 time 0.3780 (0.3868) data time 0.0011 (0.0020) model time 0.3769 (0.3856) loss 2.6145 (3.6455) grad_norm 1.0174 (1.2434) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][490/625] eta 0:00:52 lr 0.001971 wd 0.0500 time 0.3954 (0.3867) data time 0.0008 (0.0020) model time 0.3945 (0.3854) loss 4.7587 (3.6463) grad_norm 1.3799 (1.2449) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:43:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][500/625] eta 0:00:48 lr 0.001971 wd 0.0500 time 0.3834 (0.3865) data time 0.0008 (0.0020) model time 0.3826 (0.3852) loss 2.5776 (3.6444) grad_norm 1.2882 (1.2456) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][510/625] eta 0:00:44 lr 0.001970 wd 0.0500 time 0.3764 (0.3867) data time 0.0008 (0.0020) model time 0.3756 (0.3854) loss 4.2151 (3.6492) grad_norm 1.2536 (1.2463) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][520/625] eta 0:00:40 lr 0.001970 wd 0.0500 time 0.3750 (0.3866) data time 0.0014 (0.0020) model time 0.3736 (0.3853) loss 3.0232 (3.6549) grad_norm 1.1316 (1.2470) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][530/625] eta 0:00:36 lr 0.001970 wd 0.0500 time 0.3654 (0.3864) data time 0.0011 (0.0020) model time 0.3642 (0.3851) loss 3.6706 (3.6576) grad_norm 2.7529 (1.2509) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][540/625] eta 0:00:32 lr 0.001970 wd 0.0500 time 0.3749 (0.3863) data time 0.0008 (0.0020) model time 0.3741 (0.3850) loss 2.9883 (3.6507) grad_norm 1.2389 (1.2537) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][550/625] eta 0:00:28 lr 0.001970 wd 0.0500 time 0.3743 (0.3861) data time 0.0011 (0.0019) model time 0.3732 (0.3848) loss 3.8891 (3.6491) grad_norm 1.5368 (1.2532) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][560/625] eta 0:00:25 lr 0.001970 wd 0.0500 time 0.3741 (0.3860) data time 0.0011 (0.0019) model time 0.3730 (0.3846) loss 3.6752 (3.6492) grad_norm 1.3965 (1.2520) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][570/625] eta 0:00:21 lr 0.001970 wd 0.0500 time 0.3727 (0.3858) data time 0.0008 (0.0019) model time 0.3719 (0.3845) loss 2.6969 (3.6554) grad_norm 1.6067 (1.2502) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][580/625] eta 0:00:17 lr 0.001970 wd 0.0500 time 0.3868 (0.3868) data time 0.0012 (0.0019) model time 0.3855 (0.3855) loss 3.1988 (3.6605) grad_norm 1.3645 (1.2520) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][590/625] eta 0:00:13 lr 0.001970 wd 0.0500 time 0.3707 (0.3870) data time 0.0011 (0.0019) model time 0.3696 (0.3857) loss 4.0814 (3.6659) grad_norm 1.3921 (1.2553) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][600/625] eta 0:00:09 lr 0.001970 wd 0.0500 time 0.3766 (0.3868) data time 0.0011 (0.0019) model time 0.3755 (0.3856) loss 3.1531 (3.6691) grad_norm 1.0171 (1.2539) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][610/625] eta 0:00:05 lr 0.001970 wd 0.0500 time 0.3712 (0.3867) data time 0.0008 (0.0019) model time 0.3703 (0.3854) loss 2.8261 (3.6648) grad_norm 1.3540 (1.2530) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [41/300][620/625] eta 0:00:01 lr 0.001970 wd 0.0500 time 0.3660 (0.3864) data time 0.0006 (0.0019) model time 0.3654 (0.3851) loss 4.1042 (3.6623) grad_norm 1.6597 (1.2546) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:44:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 41 training takes 0:04:01 [2024-08-26 05:44:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:44:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:44:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.573 (0.573) Loss 0.6899 (0.6899) Acc@1 84.229 (84.229) Acc@5 96.729 (96.729) Mem 11225MB [2024-08-26 05:44:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.150) Loss 1.2539 (0.8956) Acc@1 70.117 (79.022) Acc@5 91.162 (95.166) Mem 11225MB [2024-08-26 05:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.124) Loss 1.4395 (1.1007) Acc@1 66.260 (74.444) Acc@5 88.281 (92.492) Mem 11225MB [2024-08-26 05:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.236 Acc@5 92.474 [2024-08-26 05:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 74.2% [2024-08-26 05:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 74.24% [2024-08-26 05:44:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:44:52 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:44:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.498 (0.498) Loss 0.5957 (0.5957) Acc@1 85.010 (85.010) Acc@5 97.412 (97.412) Mem 11225MB [2024-08-26 05:44:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 1.0869 (0.7712) Acc@1 72.021 (80.202) Acc@5 91.748 (95.659) Mem 11225MB [2024-08-26 05:44:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.2891 (0.9722) Acc@1 67.627 (75.623) Acc@5 88.916 (93.090) Mem 11225MB [2024-08-26 05:44:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.348 Acc@5 93.054 [2024-08-26 05:44:55 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 75.3% [2024-08-26 05:44:55 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 75.35% [2024-08-26 05:44:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:44:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:44:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][0/625] eta 0:08:02 lr 0.001970 wd 0.0500 time 0.7715 (0.7715) data time 0.4317 (0.4317) model time 0.0000 (0.0000) loss 3.6897 (3.6897) grad_norm 1.1468 (1.1468) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][10/625] eta 0:04:13 lr 0.001970 wd 0.0500 time 0.3913 (0.4130) data time 0.0009 (0.0402) model time 0.0000 (0.0000) loss 4.5945 (3.4982) grad_norm 1.2242 (1.2393) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][20/625] eta 0:04:04 lr 0.001970 wd 0.0500 time 0.3713 (0.4049) data time 0.0009 (0.0216) model time 0.0000 (0.0000) loss 3.9533 (3.4773) grad_norm 1.0440 (1.1768) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][30/625] eta 0:03:55 lr 0.001970 wd 0.0500 time 0.3762 (0.3963) data time 0.0012 (0.0151) model time 0.0000 (0.0000) loss 3.6463 (3.6045) grad_norm 1.3588 (1.1950) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][40/625] eta 0:03:49 lr 0.001970 wd 0.0500 time 0.3839 (0.3922) data time 0.0011 (0.0118) model time 0.0000 (0.0000) loss 3.6630 (3.6262) grad_norm 1.1138 (1.1875) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][50/625] eta 0:03:43 lr 0.001970 wd 0.0500 time 0.3768 (0.3892) data time 0.0010 (0.0097) model time 0.0000 (0.0000) loss 3.2493 (3.6517) grad_norm 1.0189 (1.2223) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][60/625] eta 0:03:38 lr 0.001970 wd 0.0500 time 0.3767 (0.3873) data time 0.0011 (0.0084) model time 0.3756 (0.3755) loss 3.4710 (3.6425) grad_norm 1.5621 (1.2668) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][70/625] eta 0:03:34 lr 0.001970 wd 0.0500 time 0.3712 (0.3862) data time 0.0008 (0.0074) model time 0.3703 (0.3770) loss 4.0512 (3.6597) grad_norm 0.8917 (1.2463) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][80/625] eta 0:03:29 lr 0.001970 wd 0.0500 time 0.3762 (0.3853) data time 0.0009 (0.0066) model time 0.3753 (0.3772) loss 3.7006 (3.6783) grad_norm 1.3875 (1.2470) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][90/625] eta 0:03:25 lr 0.001970 wd 0.0500 time 0.3778 (0.3843) data time 0.0010 (0.0060) model time 0.3768 (0.3769) loss 3.6718 (3.6766) grad_norm 1.1112 (1.2496) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][100/625] eta 0:03:22 lr 0.001970 wd 0.0500 time 0.3832 (0.3855) data time 0.0010 (0.0055) model time 0.3822 (0.3805) loss 3.7912 (3.6616) grad_norm 1.1709 (1.2368) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][110/625] eta 0:03:18 lr 0.001970 wd 0.0500 time 0.3752 (0.3853) data time 0.0011 (0.0051) model time 0.3741 (0.3807) loss 4.1165 (3.6401) grad_norm 1.4971 (1.2419) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][120/625] eta 0:03:14 lr 0.001969 wd 0.0500 time 0.3832 (0.3847) data time 0.0011 (0.0048) model time 0.3821 (0.3802) loss 2.6150 (3.6487) grad_norm 1.0729 (1.2400) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][130/625] eta 0:03:10 lr 0.001969 wd 0.0500 time 0.3782 (0.3843) data time 0.0012 (0.0045) model time 0.3770 (0.3799) loss 2.6226 (3.6421) grad_norm 0.9639 (1.2297) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][140/625] eta 0:03:06 lr 0.001969 wd 0.0500 time 0.3786 (0.3838) data time 0.0009 (0.0043) model time 0.3778 (0.3795) loss 2.8140 (3.6379) grad_norm 1.2907 (1.2363) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][150/625] eta 0:03:02 lr 0.001969 wd 0.0500 time 0.3784 (0.3834) data time 0.0010 (0.0041) model time 0.3774 (0.3792) loss 3.8098 (3.6603) grad_norm 1.2011 (1.2412) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:45:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][160/625] eta 0:02:59 lr 0.001969 wd 0.0500 time 0.3734 (0.3868) data time 0.0010 (0.0039) model time 0.3724 (0.3844) loss 4.0493 (3.6691) grad_norm 1.8379 (1.2412) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][170/625] eta 0:02:55 lr 0.001969 wd 0.0500 time 0.3784 (0.3862) data time 0.0011 (0.0037) model time 0.3774 (0.3838) loss 3.8532 (3.6538) grad_norm 1.0222 (1.2464) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][180/625] eta 0:02:51 lr 0.001969 wd 0.0500 time 0.3854 (0.3858) data time 0.0011 (0.0036) model time 0.3843 (0.3833) loss 3.0909 (3.6410) grad_norm 1.3705 (1.2432) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][190/625] eta 0:02:47 lr 0.001969 wd 0.0500 time 0.3844 (0.3855) data time 0.0008 (0.0035) model time 0.3836 (0.3829) loss 4.8158 (3.6477) grad_norm 1.3087 (1.2400) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][200/625] eta 0:02:44 lr 0.001969 wd 0.0500 time 0.3788 (0.3860) data time 0.0011 (0.0034) model time 0.3777 (0.3837) loss 3.6489 (3.6534) grad_norm 1.0896 (1.2351) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][210/625] eta 0:02:40 lr 0.001969 wd 0.0500 time 0.3729 (0.3856) data time 0.0011 (0.0033) model time 0.3718 (0.3832) loss 3.1553 (3.6428) grad_norm 0.8815 (1.2322) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][220/625] eta 0:02:36 lr 0.001969 wd 0.0500 time 0.3845 (0.3853) data time 0.0011 (0.0032) model time 0.3835 (0.3830) loss 3.8276 (3.6324) grad_norm 0.8871 (1.2287) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][230/625] eta 0:02:32 lr 0.001969 wd 0.0500 time 0.3740 (0.3851) data time 0.0008 (0.0031) model time 0.3732 (0.3827) loss 3.7224 (3.6212) grad_norm 1.1340 (1.2363) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][240/625] eta 0:02:28 lr 0.001969 wd 0.0500 time 0.3733 (0.3847) data time 0.0011 (0.0030) model time 0.3723 (0.3824) loss 3.8898 (3.6227) grad_norm 0.8537 (1.2322) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][250/625] eta 0:02:24 lr 0.001969 wd 0.0500 time 0.3729 (0.3843) data time 0.0011 (0.0029) model time 0.3717 (0.3819) loss 3.8812 (3.6120) grad_norm 1.1668 (1.2337) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][260/625] eta 0:02:20 lr 0.001969 wd 0.0500 time 0.3761 (0.3841) data time 0.0011 (0.0028) model time 0.3750 (0.3817) loss 2.6492 (3.6129) grad_norm 0.9700 (1.2408) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][270/625] eta 0:02:16 lr 0.001969 wd 0.0500 time 0.3829 (0.3839) data time 0.0009 (0.0028) model time 0.3820 (0.3815) loss 3.9576 (3.6180) grad_norm 1.2569 (1.2390) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:46:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][280/625] eta 0:02:12 lr 0.001969 wd 0.0500 time 0.3771 (0.3836) data time 0.0011 (0.0027) model time 0.3759 (0.3812) loss 3.8349 (3.6208) grad_norm 0.9524 (1.2342) loss_scale 16384.0000 (8337.7651) mem 11225MB [2024-08-26 05:46:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][290/625] eta 0:02:08 lr 0.001969 wd 0.0500 time 0.3731 (0.3834) data time 0.0013 (0.0027) model time 0.3718 (0.3810) loss 3.6542 (3.6214) grad_norm 0.8855 (1.2305) loss_scale 16384.0000 (8614.2680) mem 11225MB [2024-08-26 05:46:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][300/625] eta 0:02:04 lr 0.001969 wd 0.0500 time 0.3815 (0.3832) data time 0.0008 (0.0026) model time 0.3807 (0.3808) loss 2.5626 (3.6266) grad_norm 1.0831 (1.2263) loss_scale 16384.0000 (8872.3987) mem 11225MB [2024-08-26 05:46:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][310/625] eta 0:02:00 lr 0.001969 wd 0.0500 time 0.3759 (0.3837) data time 0.0011 (0.0026) model time 0.3748 (0.3815) loss 3.8223 (3.6283) grad_norm 1.1224 (1.2242) loss_scale 16384.0000 (9113.9293) mem 11225MB [2024-08-26 05:47:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][320/625] eta 0:01:57 lr 0.001969 wd 0.0500 time 0.3835 (0.3841) data time 0.0010 (0.0025) model time 0.3824 (0.3820) loss 2.9140 (3.6339) grad_norm 1.2168 (1.2334) loss_scale 16384.0000 (9340.4112) mem 11225MB [2024-08-26 05:47:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][330/625] eta 0:01:53 lr 0.001969 wd 0.0500 time 0.3766 (0.3839) data time 0.0008 (0.0025) model time 0.3758 (0.3818) loss 4.2852 (3.6353) grad_norm 0.9692 (1.2340) loss_scale 16384.0000 (9553.2085) mem 11225MB [2024-08-26 05:47:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][340/625] eta 0:01:49 lr 0.001968 wd 0.0500 time 0.3791 (0.3844) data time 0.0011 (0.0024) model time 0.3780 (0.3823) loss 4.0271 (3.6287) grad_norm 1.2058 (1.2380) loss_scale 16384.0000 (9753.5249) mem 11225MB [2024-08-26 05:47:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][350/625] eta 0:01:45 lr 0.001968 wd 0.0500 time 0.3752 (0.3848) data time 0.0008 (0.0024) model time 0.3744 (0.3828) loss 4.3300 (3.6341) grad_norm 0.9761 (1.2425) loss_scale 16384.0000 (9942.4274) mem 11225MB [2024-08-26 05:47:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][360/625] eta 0:01:41 lr 0.001968 wd 0.0500 time 0.3765 (0.3846) data time 0.0011 (0.0024) model time 0.3754 (0.3826) loss 3.9943 (3.6402) grad_norm 1.2410 (1.2423) loss_scale 16384.0000 (10120.8643) mem 11225MB [2024-08-26 05:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][370/625] eta 0:01:38 lr 0.001968 wd 0.0500 time 0.3762 (0.3844) data time 0.0011 (0.0023) model time 0.3752 (0.3824) loss 2.1161 (3.6402) grad_norm 0.9322 (1.2408) loss_scale 16384.0000 (10289.6819) mem 11225MB [2024-08-26 05:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][380/625] eta 0:01:34 lr 0.001968 wd 0.0500 time 0.3751 (0.3842) data time 0.0009 (0.0023) model time 0.3742 (0.3823) loss 2.5919 (3.6333) grad_norm 0.9021 (1.2365) loss_scale 16384.0000 (10449.6378) mem 11225MB [2024-08-26 05:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][390/625] eta 0:01:30 lr 0.001968 wd 0.0500 time 0.3812 (0.3840) data time 0.0010 (0.0023) model time 0.3802 (0.3821) loss 3.4576 (3.6296) grad_norm 1.2341 (1.2327) loss_scale 16384.0000 (10601.4118) mem 11225MB [2024-08-26 05:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][400/625] eta 0:01:26 lr 0.001968 wd 0.0500 time 0.3766 (0.3839) data time 0.0012 (0.0022) model time 0.3754 (0.3820) loss 3.4400 (3.6343) grad_norm 1.6523 (1.2312) loss_scale 16384.0000 (10745.6160) mem 11225MB [2024-08-26 05:47:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][410/625] eta 0:01:22 lr 0.001968 wd 0.0500 time 0.3786 (0.3837) data time 0.0009 (0.0022) model time 0.3777 (0.3818) loss 3.4964 (3.6354) grad_norm 0.9508 (1.2320) loss_scale 16384.0000 (10882.8029) mem 11225MB [2024-08-26 05:47:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][420/625] eta 0:01:18 lr 0.001968 wd 0.0500 time 0.3751 (0.3836) data time 0.0011 (0.0022) model time 0.3741 (0.3817) loss 3.7992 (3.6321) grad_norm 1.0398 (1.2298) loss_scale 16384.0000 (11013.4727) mem 11225MB [2024-08-26 05:47:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][430/625] eta 0:01:14 lr 0.001968 wd 0.0500 time 0.3777 (0.3834) data time 0.0009 (0.0022) model time 0.3768 (0.3815) loss 3.9015 (3.6310) grad_norm 0.8608 (1.2293) loss_scale 16384.0000 (11138.0789) mem 11225MB [2024-08-26 05:47:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][440/625] eta 0:01:10 lr 0.001968 wd 0.0500 time 0.3723 (0.3833) data time 0.0009 (0.0021) model time 0.3714 (0.3814) loss 4.8612 (3.6413) grad_norm 1.3433 (1.2279) loss_scale 16384.0000 (11257.0340) mem 11225MB [2024-08-26 05:47:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][450/625] eta 0:01:07 lr 0.001968 wd 0.0500 time 0.3766 (0.3832) data time 0.0008 (0.0021) model time 0.3757 (0.3813) loss 4.2985 (3.6414) grad_norm 1.2520 (1.2299) loss_scale 16384.0000 (11370.7140) mem 11225MB [2024-08-26 05:47:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][460/625] eta 0:01:03 lr 0.001968 wd 0.0500 time 0.5842 (0.3844) data time 0.0008 (0.0021) model time 0.5834 (0.3827) loss 3.8121 (3.6446) grad_norm 1.3690 (1.2291) loss_scale 16384.0000 (11479.4620) mem 11225MB [2024-08-26 05:47:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][470/625] eta 0:00:59 lr 0.001968 wd 0.0500 time 0.3745 (0.3843) data time 0.0011 (0.0021) model time 0.3734 (0.3825) loss 4.1219 (3.6444) grad_norm 1.0384 (1.2275) loss_scale 16384.0000 (11583.5924) mem 11225MB [2024-08-26 05:48:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][480/625] eta 0:00:55 lr 0.001968 wd 0.0500 time 0.3874 (0.3842) data time 0.0010 (0.0020) model time 0.3864 (0.3824) loss 4.0832 (3.6457) grad_norm 1.3889 (1.2300) loss_scale 16384.0000 (11683.3929) mem 11225MB [2024-08-26 05:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][490/625] eta 0:00:51 lr 0.001968 wd 0.0500 time 0.3795 (0.3849) data time 0.0011 (0.0020) model time 0.3784 (0.3833) loss 3.9118 (3.6484) grad_norm 1.3178 (1.2320) loss_scale 16384.0000 (11779.1283) mem 11225MB [2024-08-26 05:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][500/625] eta 0:00:48 lr 0.001968 wd 0.0500 time 0.3742 (0.3847) data time 0.0011 (0.0020) model time 0.3731 (0.3831) loss 2.3282 (3.6465) grad_norm 1.4111 (1.2326) loss_scale 16384.0000 (11871.0419) mem 11225MB [2024-08-26 05:48:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][510/625] eta 0:00:44 lr 0.001968 wd 0.0500 time 0.3806 (0.3846) data time 0.0010 (0.0020) model time 0.3796 (0.3829) loss 3.7855 (3.6477) grad_norm 1.0615 (1.2353) loss_scale 16384.0000 (11959.3581) mem 11225MB [2024-08-26 05:48:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][520/625] eta 0:00:40 lr 0.001968 wd 0.0500 time 0.3797 (0.3844) data time 0.0010 (0.0020) model time 0.3787 (0.3828) loss 3.8702 (3.6473) grad_norm 0.8690 (1.2331) loss_scale 16384.0000 (12044.2841) mem 11225MB [2024-08-26 05:48:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][530/625] eta 0:00:36 lr 0.001968 wd 0.0500 time 0.3791 (0.3847) data time 0.0008 (0.0020) model time 0.3784 (0.3830) loss 3.1018 (3.6490) grad_norm 1.0314 (1.2340) loss_scale 16384.0000 (12126.0113) mem 11225MB [2024-08-26 05:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][540/625] eta 0:00:32 lr 0.001968 wd 0.0500 time 0.3842 (0.3846) data time 0.0008 (0.0020) model time 0.3834 (0.3829) loss 3.8894 (3.6445) grad_norm 1.0114 (1.2324) loss_scale 16384.0000 (12204.7172) mem 11225MB [2024-08-26 05:48:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][550/625] eta 0:00:28 lr 0.001968 wd 0.0500 time 0.3778 (0.3844) data time 0.0008 (0.0019) model time 0.3770 (0.3828) loss 2.7257 (3.6475) grad_norm 1.0981 (1.2420) loss_scale 16384.0000 (12280.5662) mem 11225MB [2024-08-26 05:48:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][560/625] eta 0:00:24 lr 0.001968 wd 0.0500 time 0.3730 (0.3843) data time 0.0011 (0.0019) model time 0.3719 (0.3826) loss 3.7801 (3.6438) grad_norm 1.0469 (1.2399) loss_scale 16384.0000 (12353.7112) mem 11225MB [2024-08-26 05:48:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][570/625] eta 0:00:21 lr 0.001967 wd 0.0500 time 0.3721 (0.3841) data time 0.0011 (0.0019) model time 0.3710 (0.3825) loss 3.7936 (3.6414) grad_norm 1.2254 (1.2395) loss_scale 16384.0000 (12424.2942) mem 11225MB [2024-08-26 05:48:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][580/625] eta 0:00:17 lr 0.001967 wd 0.0500 time 0.3765 (0.3840) data time 0.0008 (0.0019) model time 0.3757 (0.3823) loss 4.0742 (3.6371) grad_norm 1.0518 (1.2403) loss_scale 16384.0000 (12492.4475) mem 11225MB [2024-08-26 05:48:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][590/625] eta 0:00:13 lr 0.001967 wd 0.0500 time 0.3772 (0.3838) data time 0.0008 (0.0019) model time 0.3764 (0.3822) loss 2.8320 (3.6363) grad_norm 1.0075 (1.2385) loss_scale 16384.0000 (12558.2944) mem 11225MB [2024-08-26 05:48:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][600/625] eta 0:00:09 lr 0.001967 wd 0.0500 time 0.3795 (0.3838) data time 0.0010 (0.0019) model time 0.3785 (0.3821) loss 3.8001 (3.6400) grad_norm 1.8078 (1.2417) loss_scale 16384.0000 (12621.9501) mem 11225MB [2024-08-26 05:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][610/625] eta 0:00:05 lr 0.001967 wd 0.0500 time 0.3685 (0.3843) data time 0.0008 (0.0019) model time 0.3677 (0.3827) loss 3.9172 (3.6419) grad_norm 1.2193 (inf) loss_scale 8192.0000 (12549.4468) mem 11225MB [2024-08-26 05:48:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [42/300][620/625] eta 0:00:01 lr 0.001967 wd 0.0500 time 0.3677 (0.3840) data time 0.0008 (0.0019) model time 0.3669 (0.3824) loss 3.7443 (3.6431) grad_norm 0.9925 (inf) loss_scale 8192.0000 (12479.2786) mem 11225MB [2024-08-26 05:48:56 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 42 training takes 0:03:59 [2024-08-26 05:48:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:48:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:48:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.531 (0.531) Loss 0.7769 (0.7769) Acc@1 83.691 (83.691) Acc@5 96.680 (96.680) Mem 11225MB [2024-08-26 05:48:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.146) Loss 1.3262 (0.9503) Acc@1 68.506 (78.391) Acc@5 90.869 (95.162) Mem 11225MB [2024-08-26 05:49:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.4678 (1.1521) Acc@1 66.895 (74.082) Acc@5 89.062 (92.485) Mem 11225MB [2024-08-26 05:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 73.840 Acc@5 92.368 [2024-08-26 05:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 73.8% [2024-08-26 05:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.868 (0.868) Loss 0.5923 (0.5923) Acc@1 85.254 (85.254) Acc@5 97.559 (97.559) Mem 11225MB [2024-08-26 05:49:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.178) Loss 1.0752 (0.7656) Acc@1 72.607 (80.407) Acc@5 92.041 (95.770) Mem 11225MB [2024-08-26 05:49:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.2764 (0.9643) Acc@1 67.871 (75.858) Acc@5 88.965 (93.208) Mem 11225MB [2024-08-26 05:49:04 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.592 Acc@5 93.182 [2024-08-26 05:49:04 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 75.6% [2024-08-26 05:49:04 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 75.59% [2024-08-26 05:49:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:49:05 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:49:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][0/625] eta 0:07:34 lr 0.001967 wd 0.0500 time 0.7273 (0.7273) data time 0.3864 (0.3864) model time 0.0000 (0.0000) loss 2.4918 (2.4918) grad_norm 0.8780 (0.8780) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][10/625] eta 0:04:11 lr 0.001967 wd 0.0500 time 0.3689 (0.4093) data time 0.0010 (0.0362) model time 0.0000 (0.0000) loss 4.3899 (3.6994) grad_norm 1.3672 (1.1023) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][20/625] eta 0:03:58 lr 0.001967 wd 0.0500 time 0.3813 (0.3950) data time 0.0011 (0.0195) model time 0.0000 (0.0000) loss 4.0923 (3.5605) grad_norm 0.9756 (1.2168) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][30/625] eta 0:04:03 lr 0.001967 wd 0.0500 time 0.3842 (0.4097) data time 0.0009 (0.0135) model time 0.0000 (0.0000) loss 3.3237 (3.4468) grad_norm 0.9760 (1.2040) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][40/625] eta 0:03:55 lr 0.001967 wd 0.0500 time 0.3747 (0.4018) data time 0.0011 (0.0105) model time 0.0000 (0.0000) loss 3.2868 (3.5240) grad_norm 1.5499 (1.2081) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][50/625] eta 0:03:48 lr 0.001967 wd 0.0500 time 0.3724 (0.3968) data time 0.0011 (0.0087) model time 0.0000 (0.0000) loss 2.4406 (3.5114) grad_norm 2.3898 (1.2253) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][60/625] eta 0:03:42 lr 0.001967 wd 0.0500 time 0.3771 (0.3938) data time 0.0012 (0.0075) model time 0.3759 (0.3774) loss 4.4259 (3.5832) grad_norm 0.8791 (1.2387) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][70/625] eta 0:03:37 lr 0.001967 wd 0.0500 time 0.3830 (0.3916) data time 0.0011 (0.0066) model time 0.3820 (0.3773) loss 3.8246 (3.5645) grad_norm 0.9678 (1.2398) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][80/625] eta 0:03:32 lr 0.001967 wd 0.0500 time 0.3845 (0.3900) data time 0.0010 (0.0059) model time 0.3835 (0.3773) loss 4.0978 (3.5550) grad_norm 0.9601 (1.2379) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][90/625] eta 0:03:27 lr 0.001967 wd 0.0500 time 0.3717 (0.3886) data time 0.0013 (0.0054) model time 0.3704 (0.3770) loss 3.7251 (3.5717) grad_norm 1.4624 (1.2585) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][100/625] eta 0:03:23 lr 0.001967 wd 0.0500 time 0.3754 (0.3874) data time 0.0009 (0.0050) model time 0.3745 (0.3767) loss 3.3235 (3.5760) grad_norm 1.9752 (1.2758) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][110/625] eta 0:03:19 lr 0.001967 wd 0.0500 time 0.3739 (0.3866) data time 0.0009 (0.0046) model time 0.3731 (0.3768) loss 2.8646 (3.5873) grad_norm 1.1269 (1.2708) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][120/625] eta 0:03:14 lr 0.001967 wd 0.0500 time 0.3795 (0.3861) data time 0.0011 (0.0043) model time 0.3785 (0.3772) loss 3.5780 (3.6002) grad_norm 1.6615 (1.2626) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][130/625] eta 0:03:10 lr 0.001967 wd 0.0500 time 0.3772 (0.3856) data time 0.0008 (0.0041) model time 0.3764 (0.3773) loss 2.5780 (3.5781) grad_norm 1.4846 (1.2664) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:49:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][140/625] eta 0:03:06 lr 0.001967 wd 0.0500 time 0.3740 (0.3849) data time 0.0011 (0.0039) model time 0.3729 (0.3770) loss 3.9970 (3.5873) grad_norm 2.4474 (1.2913) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][150/625] eta 0:03:03 lr 0.001967 wd 0.0500 time 0.3790 (0.3859) data time 0.0008 (0.0037) model time 0.3781 (0.3792) loss 4.8626 (3.6055) grad_norm 1.0771 (1.2968) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][160/625] eta 0:02:59 lr 0.001966 wd 0.0500 time 0.3797 (0.3855) data time 0.0008 (0.0035) model time 0.3789 (0.3791) loss 4.0992 (3.6096) grad_norm 1.5093 (1.2982) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][170/625] eta 0:02:56 lr 0.001966 wd 0.0500 time 0.5516 (0.3872) data time 0.0011 (0.0034) model time 0.5505 (0.3821) loss 3.8986 (3.6149) grad_norm 0.7544 (1.2963) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][180/625] eta 0:02:52 lr 0.001966 wd 0.0500 time 0.3763 (0.3867) data time 0.0009 (0.0033) model time 0.3753 (0.3816) loss 3.8990 (3.6255) grad_norm 0.7956 (1.2888) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][190/625] eta 0:02:48 lr 0.001966 wd 0.0500 time 0.3857 (0.3863) data time 0.0010 (0.0032) model time 0.3846 (0.3814) loss 3.0903 (3.6209) grad_norm 1.1030 (1.2910) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][200/625] eta 0:02:43 lr 0.001966 wd 0.0500 time 0.3748 (0.3858) data time 0.0008 (0.0031) model time 0.3740 (0.3810) loss 3.4044 (3.6235) grad_norm 1.2241 (1.2804) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][210/625] eta 0:02:39 lr 0.001966 wd 0.0500 time 0.3871 (0.3855) data time 0.0011 (0.0030) model time 0.3860 (0.3807) loss 3.3095 (3.6325) grad_norm 0.6778 (1.2726) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][220/625] eta 0:02:36 lr 0.001966 wd 0.0500 time 0.3790 (0.3852) data time 0.0010 (0.0029) model time 0.3781 (0.3806) loss 3.2684 (3.6294) grad_norm 1.1990 (1.2687) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][230/625] eta 0:02:32 lr 0.001966 wd 0.0500 time 0.3808 (0.3850) data time 0.0009 (0.0028) model time 0.3798 (0.3805) loss 4.0101 (3.6352) grad_norm 1.0502 (1.2650) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][240/625] eta 0:02:29 lr 0.001966 wd 0.0500 time 0.3795 (0.3873) data time 0.0010 (0.0027) model time 0.3784 (0.3836) loss 3.2651 (3.6274) grad_norm 1.2966 (1.2653) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][250/625] eta 0:02:25 lr 0.001966 wd 0.0500 time 0.3722 (0.3869) data time 0.0010 (0.0027) model time 0.3712 (0.3833) loss 2.8755 (3.6272) grad_norm 0.9494 (1.2597) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][260/625] eta 0:02:21 lr 0.001966 wd 0.0500 time 0.3724 (0.3867) data time 0.0009 (0.0026) model time 0.3716 (0.3832) loss 4.1598 (3.6158) grad_norm 1.8995 (1.2601) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][270/625] eta 0:02:17 lr 0.001966 wd 0.0500 time 0.3799 (0.3865) data time 0.0008 (0.0026) model time 0.3791 (0.3830) loss 4.3388 (3.6096) grad_norm 1.9277 (1.2651) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][280/625] eta 0:02:13 lr 0.001966 wd 0.0500 time 0.3807 (0.3863) data time 0.0009 (0.0025) model time 0.3798 (0.3828) loss 2.7232 (3.6168) grad_norm 0.9541 (1.2736) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:50:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][290/625] eta 0:02:09 lr 0.001966 wd 0.0500 time 0.5874 (0.3874) data time 0.0011 (0.0025) model time 0.5864 (0.3843) loss 3.0142 (3.6203) grad_norm 1.0421 (1.2692) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][300/625] eta 0:02:05 lr 0.001966 wd 0.0500 time 0.3766 (0.3873) data time 0.0011 (0.0024) model time 0.3755 (0.3842) loss 4.0837 (3.6141) grad_norm 1.3974 (1.2679) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][310/625] eta 0:02:01 lr 0.001966 wd 0.0500 time 0.3773 (0.3871) data time 0.0008 (0.0024) model time 0.3765 (0.3840) loss 3.4209 (3.6169) grad_norm 1.0425 (1.2704) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][320/625] eta 0:01:57 lr 0.001966 wd 0.0500 time 0.3744 (0.3868) data time 0.0011 (0.0024) model time 0.3734 (0.3837) loss 3.6386 (3.6219) grad_norm 1.8417 (1.2767) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][330/625] eta 0:01:54 lr 0.001966 wd 0.0500 time 0.3794 (0.3866) data time 0.0011 (0.0023) model time 0.3783 (0.3835) loss 4.0170 (3.6265) grad_norm 1.5680 (1.2757) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][340/625] eta 0:01:50 lr 0.001966 wd 0.0500 time 0.3872 (0.3863) data time 0.0010 (0.0023) model time 0.3862 (0.3833) loss 4.5963 (3.6278) grad_norm 1.0139 (1.2735) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][350/625] eta 0:01:46 lr 0.001966 wd 0.0500 time 0.3772 (0.3861) data time 0.0011 (0.0022) model time 0.3760 (0.3831) loss 3.2183 (3.6230) grad_norm 1.1769 (1.2721) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][360/625] eta 0:01:42 lr 0.001966 wd 0.0500 time 0.3748 (0.3864) data time 0.0009 (0.0022) model time 0.3739 (0.3835) loss 3.1190 (3.6201) grad_norm 1.3559 (1.2713) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][370/625] eta 0:01:38 lr 0.001966 wd 0.0500 time 0.3786 (0.3862) data time 0.0011 (0.0022) model time 0.3775 (0.3833) loss 3.4628 (3.6222) grad_norm 1.6731 (1.2716) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][380/625] eta 0:01:34 lr 0.001965 wd 0.0500 time 0.3764 (0.3860) data time 0.0008 (0.0022) model time 0.3756 (0.3832) loss 4.5737 (3.6275) grad_norm 1.3036 (1.2710) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][390/625] eta 0:01:30 lr 0.001965 wd 0.0500 time 0.3764 (0.3859) data time 0.0010 (0.0021) model time 0.3754 (0.3831) loss 4.0920 (3.6219) grad_norm 0.8805 (1.2703) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][400/625] eta 0:01:26 lr 0.001965 wd 0.0500 time 0.3786 (0.3857) data time 0.0011 (0.0021) model time 0.3776 (0.3830) loss 3.5995 (3.6169) grad_norm 1.1691 (1.2695) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][410/625] eta 0:01:22 lr 0.001965 wd 0.0500 time 0.3773 (0.3856) data time 0.0011 (0.0021) model time 0.3762 (0.3828) loss 4.0936 (3.6185) grad_norm 1.1914 (1.2705) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][420/625] eta 0:01:19 lr 0.001965 wd 0.0500 time 0.3779 (0.3868) data time 0.0009 (0.0021) model time 0.3770 (0.3843) loss 3.4350 (3.6132) grad_norm 1.0462 (1.2692) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][430/625] eta 0:01:15 lr 0.001965 wd 0.0500 time 0.3776 (0.3867) data time 0.0011 (0.0021) model time 0.3764 (0.3842) loss 3.8794 (3.6163) grad_norm 1.0826 (1.2663) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][440/625] eta 0:01:11 lr 0.001965 wd 0.0500 time 0.3856 (0.3865) data time 0.0011 (0.0020) model time 0.3845 (0.3840) loss 3.8826 (3.6154) grad_norm 1.0927 (1.2662) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:51:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][450/625] eta 0:01:07 lr 0.001965 wd 0.0500 time 0.3758 (0.3863) data time 0.0011 (0.0020) model time 0.3747 (0.3838) loss 3.8726 (3.6161) grad_norm 1.2235 (1.2619) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][460/625] eta 0:01:03 lr 0.001965 wd 0.0500 time 0.3737 (0.3862) data time 0.0009 (0.0020) model time 0.3728 (0.3837) loss 3.5983 (3.6121) grad_norm 1.0510 (1.2628) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][470/625] eta 0:00:59 lr 0.001965 wd 0.0500 time 0.3795 (0.3860) data time 0.0008 (0.0020) model time 0.3787 (0.3836) loss 3.2685 (3.6135) grad_norm 0.9557 (1.2629) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][480/625] eta 0:00:55 lr 0.001965 wd 0.0500 time 0.3762 (0.3858) data time 0.0010 (0.0020) model time 0.3752 (0.3834) loss 4.1406 (3.6167) grad_norm 1.0738 (1.2633) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][490/625] eta 0:00:52 lr 0.001965 wd 0.0500 time 0.3810 (0.3857) data time 0.0008 (0.0020) model time 0.3801 (0.3833) loss 2.1954 (3.6136) grad_norm 1.1166 (1.2637) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][500/625] eta 0:00:48 lr 0.001965 wd 0.0500 time 0.3846 (0.3865) data time 0.0008 (0.0019) model time 0.3838 (0.3841) loss 4.3255 (3.6181) grad_norm 1.5071 (1.2633) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][510/625] eta 0:00:44 lr 0.001965 wd 0.0500 time 0.3780 (0.3864) data time 0.0009 (0.0019) model time 0.3771 (0.3841) loss 4.3130 (3.6151) grad_norm 0.8973 (1.2627) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][520/625] eta 0:00:40 lr 0.001965 wd 0.0500 time 0.3854 (0.3863) data time 0.0010 (0.0019) model time 0.3844 (0.3840) loss 2.9607 (3.6141) grad_norm 1.1517 (1.2604) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][530/625] eta 0:00:36 lr 0.001965 wd 0.0500 time 0.3841 (0.3862) data time 0.0011 (0.0019) model time 0.3830 (0.3839) loss 3.1088 (3.6122) grad_norm 1.3733 (1.2585) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][540/625] eta 0:00:32 lr 0.001965 wd 0.0500 time 0.3712 (0.3871) data time 0.0010 (0.0019) model time 0.3701 (0.3849) loss 3.8107 (3.6143) grad_norm 1.3495 (1.2576) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][550/625] eta 0:00:29 lr 0.001965 wd 0.0500 time 0.3830 (0.3870) data time 0.0011 (0.0019) model time 0.3820 (0.3848) loss 4.1847 (3.6125) grad_norm 1.0681 (1.2554) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][560/625] eta 0:00:25 lr 0.001965 wd 0.0500 time 0.3730 (0.3868) data time 0.0008 (0.0019) model time 0.3721 (0.3846) loss 2.8060 (3.6080) grad_norm 1.1600 (1.2582) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][570/625] eta 0:00:21 lr 0.001965 wd 0.0500 time 0.3835 (0.3867) data time 0.0008 (0.0018) model time 0.3827 (0.3845) loss 4.6993 (3.6105) grad_norm 1.4399 (1.2553) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][580/625] eta 0:00:17 lr 0.001965 wd 0.0500 time 0.3727 (0.3865) data time 0.0010 (0.0018) model time 0.3716 (0.3844) loss 3.9147 (3.6117) grad_norm 0.9969 (1.2575) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][590/625] eta 0:00:13 lr 0.001964 wd 0.0500 time 0.3905 (0.3864) data time 0.0010 (0.0018) model time 0.3895 (0.3843) loss 2.8689 (3.6141) grad_norm 1.3400 (1.2567) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:52:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][600/625] eta 0:00:09 lr 0.001964 wd 0.0500 time 0.3749 (0.3863) data time 0.0009 (0.0018) model time 0.3741 (0.3842) loss 3.3219 (3.6151) grad_norm 1.7534 (1.2578) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][610/625] eta 0:00:05 lr 0.001964 wd 0.0500 time 0.3707 (0.3861) data time 0.0008 (0.0018) model time 0.3699 (0.3840) loss 4.0995 (3.6172) grad_norm 0.9639 (1.2586) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [43/300][620/625] eta 0:00:01 lr 0.001964 wd 0.0500 time 0.3662 (0.3859) data time 0.0006 (0.0018) model time 0.3656 (0.3837) loss 3.4963 (3.6167) grad_norm 2.1001 (1.2613) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 43 training takes 0:04:01 [2024-08-26 05:53:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:53:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:53:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 0.7500 (0.7500) Acc@1 83.203 (83.203) Acc@5 96.094 (96.094) Mem 11225MB [2024-08-26 05:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.147) Loss 1.2373 (0.8892) Acc@1 70.410 (79.133) Acc@5 91.846 (95.397) Mem 11225MB [2024-08-26 05:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.4473 (1.0929) Acc@1 66.309 (74.484) Acc@5 88.525 (92.778) Mem 11225MB [2024-08-26 05:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.318 Acc@5 92.732 [2024-08-26 05:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 74.3% [2024-08-26 05:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 74.32% [2024-08-26 05:53:10 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:53:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:53:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.5884 (0.5884) Acc@1 85.449 (85.449) Acc@5 97.559 (97.559) Mem 11225MB [2024-08-26 05:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.142) Loss 1.0645 (0.7599) Acc@1 72.900 (80.682) Acc@5 92.139 (95.823) Mem 11225MB [2024-08-26 05:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.2676 (0.9568) Acc@1 67.871 (76.121) Acc@5 89.209 (93.334) Mem 11225MB [2024-08-26 05:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.858 Acc@5 93.312 [2024-08-26 05:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 75.9% [2024-08-26 05:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 75.86% [2024-08-26 05:53:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:53:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:53:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][0/625] eta 0:08:11 lr 0.001964 wd 0.0500 time 0.7869 (0.7869) data time 0.4484 (0.4484) model time 0.0000 (0.0000) loss 2.6761 (2.6761) grad_norm 1.2969 (1.2969) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][10/625] eta 0:04:16 lr 0.001964 wd 0.0500 time 0.3814 (0.4169) data time 0.0008 (0.0419) model time 0.0000 (0.0000) loss 2.6019 (3.4844) grad_norm 1.1609 (1.3089) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][20/625] eta 0:04:00 lr 0.001964 wd 0.0500 time 0.3744 (0.3982) data time 0.0009 (0.0225) model time 0.0000 (0.0000) loss 2.7535 (3.5695) grad_norm 2.1495 (1.3294) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][30/625] eta 0:03:53 lr 0.001964 wd 0.0500 time 0.3739 (0.3918) data time 0.0008 (0.0156) model time 0.0000 (0.0000) loss 3.5165 (3.5978) grad_norm 1.1852 (1.3367) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][40/625] eta 0:03:47 lr 0.001964 wd 0.0500 time 0.3918 (0.3886) data time 0.0008 (0.0121) model time 0.0000 (0.0000) loss 4.6483 (3.6797) grad_norm 1.1067 (1.2822) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][50/625] eta 0:03:42 lr 0.001964 wd 0.0500 time 0.3720 (0.3861) data time 0.0011 (0.0100) model time 0.0000 (0.0000) loss 2.3604 (3.7047) grad_norm 1.6970 (1.2652) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][60/625] eta 0:03:37 lr 0.001964 wd 0.0500 time 0.3723 (0.3850) data time 0.0011 (0.0085) model time 0.3712 (0.3780) loss 3.8357 (3.7040) grad_norm 1.5365 (1.2759) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][70/625] eta 0:03:39 lr 0.001964 wd 0.0500 time 0.3770 (0.3960) data time 0.0012 (0.0075) model time 0.3758 (0.4200) loss 3.8583 (3.6793) grad_norm 1.0229 (1.2668) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][80/625] eta 0:03:34 lr 0.001964 wd 0.0500 time 0.3785 (0.3938) data time 0.0011 (0.0067) model time 0.3774 (0.4058) loss 3.7631 (3.6537) grad_norm 0.8455 (1.2425) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][90/625] eta 0:03:29 lr 0.001964 wd 0.0500 time 0.3800 (0.3920) data time 0.0012 (0.0061) model time 0.3788 (0.3983) loss 3.9656 (3.6676) grad_norm 0.9761 (1.2624) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][100/625] eta 0:03:25 lr 0.001964 wd 0.0500 time 0.3823 (0.3907) data time 0.0010 (0.0056) model time 0.3813 (0.3943) loss 3.5656 (3.6655) grad_norm 1.6202 (1.2589) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][110/625] eta 0:03:20 lr 0.001964 wd 0.0500 time 0.3716 (0.3894) data time 0.0008 (0.0052) model time 0.3707 (0.3910) loss 2.3909 (3.6411) grad_norm 1.2477 (1.2543) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][120/625] eta 0:03:16 lr 0.001964 wd 0.0500 time 0.3762 (0.3885) data time 0.0008 (0.0049) model time 0.3755 (0.3890) loss 3.0508 (3.6655) grad_norm 1.5291 (1.2491) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][130/625] eta 0:03:11 lr 0.001964 wd 0.0500 time 0.3789 (0.3876) data time 0.0011 (0.0046) model time 0.3779 (0.3874) loss 3.1229 (3.6550) grad_norm 1.2220 (1.2451) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][140/625] eta 0:03:07 lr 0.001964 wd 0.0500 time 0.3757 (0.3869) data time 0.0011 (0.0043) model time 0.3746 (0.3861) loss 3.1970 (3.6605) grad_norm 1.2885 (1.2532) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][150/625] eta 0:03:03 lr 0.001964 wd 0.0500 time 0.3745 (0.3863) data time 0.0011 (0.0041) model time 0.3734 (0.3852) loss 3.3181 (3.6650) grad_norm 1.2166 (1.2573) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][160/625] eta 0:03:00 lr 0.001964 wd 0.0500 time 0.6213 (0.3873) data time 0.0010 (0.0040) model time 0.6203 (0.3867) loss 3.5300 (3.6687) grad_norm 1.4633 (1.2539) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][170/625] eta 0:02:55 lr 0.001964 wd 0.0500 time 0.3771 (0.3866) data time 0.0008 (0.0038) model time 0.3763 (0.3857) loss 2.4574 (3.6681) grad_norm 1.2846 (1.2482) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][180/625] eta 0:02:51 lr 0.001963 wd 0.0500 time 0.3825 (0.3862) data time 0.0010 (0.0036) model time 0.3815 (0.3850) loss 3.8950 (3.6753) grad_norm 1.6202 (1.2704) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][190/625] eta 0:02:47 lr 0.001963 wd 0.0500 time 0.3759 (0.3857) data time 0.0010 (0.0035) model time 0.3749 (0.3845) loss 3.8708 (3.6867) grad_norm 1.3687 (1.2731) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][200/625] eta 0:02:43 lr 0.001963 wd 0.0500 time 0.3640 (0.3853) data time 0.0009 (0.0034) model time 0.3631 (0.3839) loss 4.3828 (3.7012) grad_norm 1.2073 (1.2670) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][210/625] eta 0:02:39 lr 0.001963 wd 0.0500 time 0.3932 (0.3849) data time 0.0010 (0.0033) model time 0.3921 (0.3834) loss 3.1402 (3.7088) grad_norm 1.1280 (1.2666) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][220/625] eta 0:02:35 lr 0.001963 wd 0.0500 time 0.3750 (0.3845) data time 0.0008 (0.0032) model time 0.3742 (0.3829) loss 2.9337 (3.7116) grad_norm 0.9088 (1.2692) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][230/625] eta 0:02:31 lr 0.001963 wd 0.0500 time 0.3749 (0.3842) data time 0.0008 (0.0031) model time 0.3741 (0.3825) loss 3.9313 (3.7154) grad_norm 1.1647 (1.2650) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][240/625] eta 0:02:27 lr 0.001963 wd 0.0500 time 0.3744 (0.3840) data time 0.0010 (0.0030) model time 0.3735 (0.3822) loss 3.0782 (3.7037) grad_norm 0.8061 (1.2616) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][250/625] eta 0:02:24 lr 0.001963 wd 0.0500 time 0.5836 (0.3861) data time 0.0009 (0.0029) model time 0.5827 (0.3849) loss 4.4039 (3.7124) grad_norm 1.1300 (1.2612) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][260/625] eta 0:02:20 lr 0.001963 wd 0.0500 time 0.3731 (0.3858) data time 0.0009 (0.0029) model time 0.3722 (0.3846) loss 2.4954 (3.7031) grad_norm 1.3030 (1.2626) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][270/625] eta 0:02:16 lr 0.001963 wd 0.0500 time 0.3771 (0.3856) data time 0.0012 (0.0028) model time 0.3760 (0.3844) loss 4.2111 (3.7138) grad_norm 1.6293 (1.2676) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][280/625] eta 0:02:14 lr 0.001963 wd 0.0500 time 0.3328 (0.3887) data time 0.0011 (0.0027) model time 0.3317 (0.3881) loss 4.0250 (3.7088) grad_norm 1.0219 (1.2698) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][290/625] eta 0:02:10 lr 0.001963 wd 0.0500 time 0.3796 (0.3883) data time 0.0010 (0.0027) model time 0.3786 (0.3876) loss 3.8719 (3.7019) grad_norm 1.0579 (1.2647) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][300/625] eta 0:02:06 lr 0.001963 wd 0.0500 time 0.3788 (0.3880) data time 0.0008 (0.0026) model time 0.3780 (0.3872) loss 2.8313 (3.6944) grad_norm 1.2808 (1.2615) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][310/625] eta 0:02:02 lr 0.001963 wd 0.0500 time 0.3674 (0.3876) data time 0.0011 (0.0026) model time 0.3663 (0.3867) loss 3.8374 (3.6910) grad_norm 0.7280 (1.2561) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][320/625] eta 0:01:58 lr 0.001963 wd 0.0500 time 0.3748 (0.3872) data time 0.0010 (0.0026) model time 0.3737 (0.3863) loss 4.0100 (3.6918) grad_norm 1.2159 (1.2523) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][330/625] eta 0:01:54 lr 0.001963 wd 0.0500 time 0.3808 (0.3870) data time 0.0008 (0.0025) model time 0.3800 (0.3860) loss 4.5341 (3.6969) grad_norm 1.1298 (1.2487) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][340/625] eta 0:01:50 lr 0.001963 wd 0.0500 time 0.3861 (0.3868) data time 0.0008 (0.0025) model time 0.3853 (0.3858) loss 2.3208 (3.6954) grad_norm 1.4356 (1.2480) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][350/625] eta 0:01:46 lr 0.001963 wd 0.0500 time 0.3816 (0.3866) data time 0.0008 (0.0024) model time 0.3807 (0.3855) loss 3.6000 (3.6991) grad_norm 1.0076 (1.2434) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][360/625] eta 0:01:42 lr 0.001963 wd 0.0500 time 0.3810 (0.3864) data time 0.0011 (0.0024) model time 0.3799 (0.3853) loss 3.2886 (3.6999) grad_norm 1.1009 (1.2402) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][370/625] eta 0:01:38 lr 0.001963 wd 0.0500 time 0.3805 (0.3878) data time 0.0009 (0.0024) model time 0.3796 (0.3869) loss 2.6891 (3.6980) grad_norm 1.1484 (1.2433) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][380/625] eta 0:01:34 lr 0.001963 wd 0.0500 time 0.3798 (0.3876) data time 0.0012 (0.0023) model time 0.3785 (0.3866) loss 2.3799 (3.6937) grad_norm 1.0288 (1.2463) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][390/625] eta 0:01:31 lr 0.001962 wd 0.0500 time 0.3838 (0.3874) data time 0.0011 (0.0023) model time 0.3827 (0.3864) loss 4.0423 (3.6894) grad_norm 2.1787 (1.2457) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][400/625] eta 0:01:27 lr 0.001962 wd 0.0500 time 0.3831 (0.3871) data time 0.0011 (0.0023) model time 0.3820 (0.3861) loss 3.4293 (3.6835) grad_norm 1.0774 (1.2475) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][410/625] eta 0:01:23 lr 0.001962 wd 0.0500 time 0.3820 (0.3870) data time 0.0011 (0.0023) model time 0.3809 (0.3859) loss 3.4269 (3.6783) grad_norm 1.3571 (1.2479) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:55:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][420/625] eta 0:01:19 lr 0.001962 wd 0.0500 time 0.3742 (0.3867) data time 0.0011 (0.0022) model time 0.3731 (0.3857) loss 4.0425 (3.6814) grad_norm 1.0168 (1.2480) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][430/625] eta 0:01:15 lr 0.001962 wd 0.0500 time 0.3790 (0.3865) data time 0.0011 (0.0022) model time 0.3780 (0.3854) loss 2.6761 (3.6822) grad_norm 1.1058 (1.2472) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][440/625] eta 0:01:11 lr 0.001962 wd 0.0500 time 0.3747 (0.3864) data time 0.0008 (0.0022) model time 0.3739 (0.3852) loss 4.7463 (3.6857) grad_norm 1.0020 (1.2461) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][450/625] eta 0:01:07 lr 0.001962 wd 0.0500 time 0.3720 (0.3861) data time 0.0009 (0.0022) model time 0.3712 (0.3850) loss 2.6112 (3.6871) grad_norm 1.1367 (1.2515) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][460/625] eta 0:01:03 lr 0.001962 wd 0.0500 time 0.5246 (0.3863) data time 0.0008 (0.0021) model time 0.5238 (0.3851) loss 3.2328 (3.6851) grad_norm 1.2125 (1.2520) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][470/625] eta 0:00:59 lr 0.001962 wd 0.0500 time 0.3739 (0.3869) data time 0.0008 (0.0021) model time 0.3731 (0.3859) loss 2.4036 (3.6840) grad_norm 1.3446 (1.2550) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][480/625] eta 0:00:56 lr 0.001962 wd 0.0500 time 0.3741 (0.3867) data time 0.0008 (0.0021) model time 0.3734 (0.3856) loss 2.6508 (3.6847) grad_norm 1.0399 (1.2530) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][490/625] eta 0:00:52 lr 0.001962 wd 0.0500 time 0.3765 (0.3870) data time 0.0012 (0.0021) model time 0.3753 (0.3859) loss 3.5159 (3.6708) grad_norm 0.9696 (1.2510) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][500/625] eta 0:00:48 lr 0.001962 wd 0.0500 time 0.3825 (0.3868) data time 0.0011 (0.0021) model time 0.3815 (0.3858) loss 4.4343 (3.6740) grad_norm 1.5510 (1.2498) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][510/625] eta 0:00:44 lr 0.001962 wd 0.0500 time 0.3839 (0.3867) data time 0.0009 (0.0020) model time 0.3831 (0.3856) loss 2.2207 (3.6689) grad_norm 1.1630 (1.2517) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][520/625] eta 0:00:40 lr 0.001962 wd 0.0500 time 0.3714 (0.3864) data time 0.0009 (0.0020) model time 0.3706 (0.3853) loss 2.6571 (3.6693) grad_norm 1.3368 (1.2486) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][530/625] eta 0:00:36 lr 0.001962 wd 0.0500 time 0.3757 (0.3863) data time 0.0011 (0.0020) model time 0.3746 (0.3852) loss 3.7109 (3.6701) grad_norm 0.8496 (1.2501) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][540/625] eta 0:00:32 lr 0.001962 wd 0.0500 time 0.3827 (0.3861) data time 0.0009 (0.0020) model time 0.3818 (0.3850) loss 2.6668 (3.6691) grad_norm 0.9594 (1.2473) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][550/625] eta 0:00:28 lr 0.001962 wd 0.0500 time 0.3797 (0.3860) data time 0.0008 (0.0020) model time 0.3789 (0.3848) loss 3.0469 (3.6690) grad_norm 1.0100 (1.2431) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][560/625] eta 0:00:25 lr 0.001962 wd 0.0500 time 0.3761 (0.3859) data time 0.0009 (0.0020) model time 0.3752 (0.3847) loss 4.2803 (3.6660) grad_norm 1.5026 (1.2453) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][570/625] eta 0:00:21 lr 0.001962 wd 0.0500 time 0.3765 (0.3857) data time 0.0008 (0.0019) model time 0.3756 (0.3846) loss 2.9138 (3.6588) grad_norm 1.0356 (1.2522) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][580/625] eta 0:00:17 lr 0.001962 wd 0.0500 time 0.5818 (0.3866) data time 0.0009 (0.0019) model time 0.5809 (0.3855) loss 3.9773 (3.6538) grad_norm 1.3472 (1.2529) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][590/625] eta 0:00:13 lr 0.001961 wd 0.0500 time 0.3775 (0.3869) data time 0.0008 (0.0019) model time 0.3766 (0.3858) loss 4.3375 (3.6562) grad_norm 1.4084 (1.2522) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][600/625] eta 0:00:09 lr 0.001961 wd 0.0500 time 0.3815 (0.3867) data time 0.0008 (0.0019) model time 0.3806 (0.3856) loss 3.8280 (3.6553) grad_norm 1.4561 (1.2540) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][610/625] eta 0:00:05 lr 0.001961 wd 0.0500 time 0.3662 (0.3865) data time 0.0009 (0.0019) model time 0.3653 (0.3854) loss 3.6103 (3.6553) grad_norm 1.8470 (1.2549) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [44/300][620/625] eta 0:00:01 lr 0.001961 wd 0.0500 time 0.3692 (0.3863) data time 0.0006 (0.0019) model time 0.3686 (0.3851) loss 2.7221 (3.6538) grad_norm 1.3262 (1.2558) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:17 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 44 training takes 0:04:01 [2024-08-26 05:57:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 05:57:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 05:57:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.512 (0.512) Loss 0.7061 (0.7061) Acc@1 83.154 (83.154) Acc@5 96.826 (96.826) Mem 11225MB [2024-08-26 05:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.2129 (0.8682) Acc@1 71.240 (79.119) Acc@5 90.967 (95.415) Mem 11225MB [2024-08-26 05:57:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.4365 (1.0702) Acc@1 66.748 (74.793) Acc@5 88.574 (92.883) Mem 11225MB [2024-08-26 05:57:21 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.508 Acc@5 92.764 [2024-08-26 05:57:21 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 74.5% [2024-08-26 05:57:21 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 74.51% [2024-08-26 05:57:21 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 05:57:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 05:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.491 (0.491) Loss 0.5854 (0.5854) Acc@1 85.400 (85.400) Acc@5 97.656 (97.656) Mem 11225MB [2024-08-26 05:57:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.140) Loss 1.0566 (0.7554) Acc@1 72.949 (80.855) Acc@5 92.432 (95.938) Mem 11225MB [2024-08-26 05:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.117) Loss 1.2598 (0.9506) Acc@1 68.115 (76.309) Acc@5 89.453 (93.490) Mem 11225MB [2024-08-26 05:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.032 Acc@5 93.460 [2024-08-26 05:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 76.0% [2024-08-26 05:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 76.03% [2024-08-26 05:57:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 05:57:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 05:57:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][0/625] eta 0:07:59 lr 0.001961 wd 0.0500 time 0.7673 (0.7673) data time 0.4087 (0.4087) model time 0.0000 (0.0000) loss 3.5421 (3.5421) grad_norm 0.8026 (0.8026) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][10/625] eta 0:04:13 lr 0.001961 wd 0.0500 time 0.3773 (0.4114) data time 0.0011 (0.0382) model time 0.0000 (0.0000) loss 3.6207 (3.7666) grad_norm 1.5082 (1.3404) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][20/625] eta 0:03:58 lr 0.001961 wd 0.0500 time 0.3767 (0.3948) data time 0.0008 (0.0206) model time 0.0000 (0.0000) loss 2.8796 (3.7386) grad_norm 1.5773 (1.3216) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][30/625] eta 0:03:51 lr 0.001961 wd 0.0500 time 0.3723 (0.3886) data time 0.0008 (0.0143) model time 0.0000 (0.0000) loss 3.9169 (3.6655) grad_norm 1.0349 (1.2610) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][40/625] eta 0:03:45 lr 0.001961 wd 0.0500 time 0.3739 (0.3858) data time 0.0009 (0.0111) model time 0.0000 (0.0000) loss 3.8008 (3.5929) grad_norm 1.0771 (1.2577) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][50/625] eta 0:03:40 lr 0.001961 wd 0.0500 time 0.3749 (0.3838) data time 0.0009 (0.0091) model time 0.0000 (0.0000) loss 4.3883 (3.6226) grad_norm 1.4563 (1.2822) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][60/625] eta 0:03:36 lr 0.001961 wd 0.0500 time 0.3736 (0.3826) data time 0.0012 (0.0078) model time 0.3724 (0.3751) loss 4.3589 (3.5802) grad_norm 0.9430 (1.2773) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][70/625] eta 0:03:31 lr 0.001961 wd 0.0500 time 0.3911 (0.3818) data time 0.0011 (0.0069) model time 0.3900 (0.3755) loss 3.6172 (3.6046) grad_norm 1.1293 (1.2789) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][80/625] eta 0:03:27 lr 0.001961 wd 0.0500 time 0.3743 (0.3813) data time 0.0008 (0.0062) model time 0.3735 (0.3759) loss 3.4831 (3.5678) grad_norm 0.9091 (1.2862) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][90/625] eta 0:03:23 lr 0.001961 wd 0.0500 time 0.3742 (0.3808) data time 0.0011 (0.0056) model time 0.3731 (0.3757) loss 4.2071 (3.5789) grad_norm 1.3021 (1.2778) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][100/625] eta 0:03:21 lr 0.001961 wd 0.0500 time 0.3757 (0.3843) data time 0.0016 (0.0052) model time 0.3740 (0.3836) loss 3.8071 (3.5877) grad_norm 0.9001 (1.2728) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][110/625] eta 0:03:18 lr 0.001961 wd 0.0500 time 0.5493 (0.3851) data time 0.0010 (0.0048) model time 0.5483 (0.3851) loss 2.8457 (3.6033) grad_norm 1.3246 (1.2676) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][120/625] eta 0:03:14 lr 0.001961 wd 0.0500 time 0.3767 (0.3844) data time 0.0011 (0.0045) model time 0.3756 (0.3837) loss 3.8408 (3.6023) grad_norm 1.4377 (1.2567) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][130/625] eta 0:03:09 lr 0.001961 wd 0.0500 time 0.3725 (0.3837) data time 0.0008 (0.0043) model time 0.3717 (0.3824) loss 4.1633 (3.6143) grad_norm 1.3376 (1.2689) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][140/625] eta 0:03:05 lr 0.001961 wd 0.0500 time 0.3711 (0.3832) data time 0.0008 (0.0040) model time 0.3703 (0.3817) loss 2.9731 (3.6108) grad_norm 0.9576 (1.2879) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][150/625] eta 0:03:01 lr 0.001961 wd 0.0500 time 0.3820 (0.3827) data time 0.0008 (0.0038) model time 0.3812 (0.3810) loss 2.5779 (3.6147) grad_norm 1.1230 (1.2849) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][160/625] eta 0:02:59 lr 0.001961 wd 0.0500 time 0.5947 (0.3863) data time 0.0009 (0.0037) model time 0.5938 (0.3863) loss 4.0166 (3.6166) grad_norm 0.9367 (1.2841) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][170/625] eta 0:02:55 lr 0.001960 wd 0.0500 time 0.3689 (0.3856) data time 0.0008 (0.0035) model time 0.3681 (0.3852) loss 2.3507 (3.5961) grad_norm 1.2542 (1.2791) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][180/625] eta 0:02:51 lr 0.001960 wd 0.0500 time 0.3759 (0.3852) data time 0.0012 (0.0034) model time 0.3748 (0.3845) loss 3.1156 (3.5950) grad_norm 1.7408 (1.2961) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][190/625] eta 0:02:47 lr 0.001960 wd 0.0500 time 0.3828 (0.3848) data time 0.0008 (0.0033) model time 0.3819 (0.3840) loss 3.0670 (3.6068) grad_norm 1.2147 (1.3020) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][200/625] eta 0:02:43 lr 0.001960 wd 0.0500 time 0.3777 (0.3844) data time 0.0011 (0.0032) model time 0.3766 (0.3834) loss 3.9520 (3.6151) grad_norm 1.2528 (1.2980) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][210/625] eta 0:02:39 lr 0.001960 wd 0.0500 time 0.3762 (0.3840) data time 0.0008 (0.0031) model time 0.3753 (0.3829) loss 4.4719 (3.6225) grad_norm 1.0903 (1.2893) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][220/625] eta 0:02:35 lr 0.001960 wd 0.0500 time 0.3734 (0.3836) data time 0.0012 (0.0030) model time 0.3722 (0.3824) loss 3.6490 (3.6231) grad_norm 1.0369 (1.2780) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][230/625] eta 0:02:31 lr 0.001960 wd 0.0500 time 0.3782 (0.3833) data time 0.0011 (0.0029) model time 0.3771 (0.3820) loss 3.3650 (3.6219) grad_norm 1.0576 (1.2741) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:58:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][240/625] eta 0:02:27 lr 0.001960 wd 0.0500 time 0.3744 (0.3830) data time 0.0008 (0.0028) model time 0.3736 (0.3817) loss 3.8823 (3.6201) grad_norm 1.2464 (1.2721) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][250/625] eta 0:02:23 lr 0.001960 wd 0.0500 time 0.3783 (0.3827) data time 0.0008 (0.0028) model time 0.3774 (0.3813) loss 2.8977 (3.6297) grad_norm 1.4644 (1.2775) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][260/625] eta 0:02:19 lr 0.001960 wd 0.0500 time 0.3682 (0.3825) data time 0.0009 (0.0027) model time 0.3674 (0.3811) loss 3.1112 (3.6247) grad_norm 1.0905 (1.2735) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][270/625] eta 0:02:15 lr 0.001960 wd 0.0500 time 0.3732 (0.3823) data time 0.0011 (0.0027) model time 0.3721 (0.3808) loss 3.6909 (3.6210) grad_norm 1.0747 (1.2781) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][280/625] eta 0:02:11 lr 0.001960 wd 0.0500 time 0.3814 (0.3822) data time 0.0008 (0.0026) model time 0.3806 (0.3807) loss 3.8859 (3.6070) grad_norm 1.0050 (1.2775) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][290/625] eta 0:02:08 lr 0.001960 wd 0.0500 time 0.3810 (0.3821) data time 0.0008 (0.0025) model time 0.3802 (0.3806) loss 4.1497 (3.6080) grad_norm 0.9865 (1.2698) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][300/625] eta 0:02:04 lr 0.001960 wd 0.0500 time 0.3817 (0.3819) data time 0.0008 (0.0025) model time 0.3809 (0.3804) loss 3.9789 (3.6172) grad_norm 2.9198 (1.2735) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][310/625] eta 0:02:00 lr 0.001960 wd 0.0500 time 0.3746 (0.3835) data time 0.0011 (0.0025) model time 0.3735 (0.3823) loss 2.5299 (3.6107) grad_norm 1.2913 (1.2760) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][320/625] eta 0:01:56 lr 0.001960 wd 0.0500 time 0.3743 (0.3833) data time 0.0011 (0.0024) model time 0.3732 (0.3820) loss 2.5144 (3.6068) grad_norm 0.9527 (1.2776) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][330/625] eta 0:01:53 lr 0.001960 wd 0.0500 time 0.3821 (0.3837) data time 0.0011 (0.0024) model time 0.3810 (0.3826) loss 3.1923 (3.6123) grad_norm 0.9920 (1.2754) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][340/625] eta 0:01:49 lr 0.001960 wd 0.0500 time 0.5868 (0.3849) data time 0.0012 (0.0023) model time 0.5856 (0.3839) loss 3.3177 (3.6125) grad_norm 1.7150 (1.2735) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][350/625] eta 0:01:45 lr 0.001960 wd 0.0500 time 0.3730 (0.3852) data time 0.0012 (0.0023) model time 0.3718 (0.3843) loss 3.7456 (3.6102) grad_norm 1.4039 (1.2747) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][360/625] eta 0:01:42 lr 0.001960 wd 0.0500 time 0.3957 (0.3851) data time 0.0011 (0.0023) model time 0.3946 (0.3841) loss 3.3086 (3.6074) grad_norm 1.0583 (1.2773) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][370/625] eta 0:01:38 lr 0.001959 wd 0.0500 time 0.3797 (0.3849) data time 0.0011 (0.0023) model time 0.3786 (0.3839) loss 2.4992 (3.6005) grad_norm 1.6508 (1.2724) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][380/625] eta 0:01:34 lr 0.001959 wd 0.0500 time 0.3718 (0.3847) data time 0.0009 (0.0022) model time 0.3709 (0.3836) loss 3.4509 (3.6091) grad_norm 1.2014 (1.2725) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 05:59:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][390/625] eta 0:01:30 lr 0.001959 wd 0.0500 time 0.3790 (0.3845) data time 0.0013 (0.0022) model time 0.3777 (0.3835) loss 4.2511 (3.6138) grad_norm 0.9882 (1.2773) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][400/625] eta 0:01:26 lr 0.001959 wd 0.0500 time 0.3762 (0.3843) data time 0.0011 (0.0022) model time 0.3750 (0.3833) loss 4.1001 (3.6173) grad_norm 0.9407 (1.2736) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][410/625] eta 0:01:22 lr 0.001959 wd 0.0500 time 0.3799 (0.3843) data time 0.0011 (0.0021) model time 0.3788 (0.3832) loss 2.7092 (3.6142) grad_norm 1.8789 (1.2714) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][420/625] eta 0:01:18 lr 0.001959 wd 0.0500 time 0.3862 (0.3841) data time 0.0009 (0.0021) model time 0.3853 (0.3830) loss 4.5903 (3.6218) grad_norm 1.1032 (1.2778) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][430/625] eta 0:01:14 lr 0.001959 wd 0.0500 time 0.3732 (0.3840) data time 0.0009 (0.0021) model time 0.3724 (0.3829) loss 3.6450 (3.6320) grad_norm 1.0071 (1.2713) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][440/625] eta 0:01:11 lr 0.001959 wd 0.0500 time 0.3752 (0.3839) data time 0.0010 (0.0021) model time 0.3742 (0.3827) loss 3.3905 (3.6284) grad_norm 0.8473 (1.2668) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][450/625] eta 0:01:07 lr 0.001959 wd 0.0500 time 0.3754 (0.3837) data time 0.0008 (0.0021) model time 0.3746 (0.3826) loss 2.5687 (3.6271) grad_norm 1.2382 (1.2636) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][460/625] eta 0:01:03 lr 0.001959 wd 0.0500 time 0.3800 (0.3845) data time 0.0008 (0.0021) model time 0.3793 (0.3834) loss 3.8671 (3.6178) grad_norm 1.0722 (1.2655) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][470/625] eta 0:00:59 lr 0.001959 wd 0.0500 time 0.3807 (0.3848) data time 0.0010 (0.0020) model time 0.3796 (0.3838) loss 3.6970 (3.6164) grad_norm 1.3976 (1.2646) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][480/625] eta 0:00:55 lr 0.001959 wd 0.0500 time 0.3707 (0.3847) data time 0.0008 (0.0020) model time 0.3698 (0.3836) loss 3.1899 (3.6158) grad_norm 1.2840 (1.2630) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][490/625] eta 0:00:52 lr 0.001959 wd 0.0500 time 0.3755 (0.3852) data time 0.0009 (0.0020) model time 0.3746 (0.3842) loss 4.0718 (3.6200) grad_norm 1.0834 (1.2630) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][500/625] eta 0:00:48 lr 0.001959 wd 0.0500 time 0.3731 (0.3851) data time 0.0009 (0.0020) model time 0.3722 (0.3840) loss 4.1939 (3.6227) grad_norm 1.1716 (1.2624) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][510/625] eta 0:00:44 lr 0.001959 wd 0.0500 time 0.3964 (0.3854) data time 0.0011 (0.0020) model time 0.3954 (0.3844) loss 2.7583 (3.6249) grad_norm 1.5132 (1.2619) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][520/625] eta 0:00:40 lr 0.001959 wd 0.0500 time 0.3737 (0.3852) data time 0.0010 (0.0020) model time 0.3726 (0.3842) loss 3.1123 (3.6250) grad_norm 1.0966 (1.2639) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][530/625] eta 0:00:36 lr 0.001959 wd 0.0500 time 0.3745 (0.3851) data time 0.0011 (0.0019) model time 0.3734 (0.3841) loss 4.1634 (3.6259) grad_norm 1.5270 (1.2621) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][540/625] eta 0:00:32 lr 0.001959 wd 0.0500 time 0.3873 (0.3850) data time 0.0011 (0.0019) model time 0.3862 (0.3840) loss 3.2661 (3.6303) grad_norm 1.3967 (1.2628) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:00:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][550/625] eta 0:00:28 lr 0.001959 wd 0.0500 time 0.3907 (0.3849) data time 0.0010 (0.0019) model time 0.3896 (0.3839) loss 3.7263 (3.6347) grad_norm 1.8634 (1.2667) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][560/625] eta 0:00:25 lr 0.001959 wd 0.0500 time 0.3754 (0.3848) data time 0.0008 (0.0019) model time 0.3746 (0.3837) loss 3.8917 (3.6363) grad_norm 1.1667 (1.2677) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][570/625] eta 0:00:21 lr 0.001958 wd 0.0500 time 0.3730 (0.3847) data time 0.0008 (0.0019) model time 0.3722 (0.3836) loss 4.1456 (3.6367) grad_norm 1.1749 (1.2651) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][580/625] eta 0:00:17 lr 0.001958 wd 0.0500 time 0.3815 (0.3846) data time 0.0008 (0.0019) model time 0.3807 (0.3835) loss 4.3543 (3.6277) grad_norm 1.6985 (1.2656) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][590/625] eta 0:00:13 lr 0.001958 wd 0.0500 time 0.3742 (0.3845) data time 0.0008 (0.0019) model time 0.3734 (0.3834) loss 3.5997 (3.6258) grad_norm 1.3428 (1.2647) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][600/625] eta 0:00:09 lr 0.001958 wd 0.0500 time 0.3786 (0.3844) data time 0.0009 (0.0018) model time 0.3777 (0.3833) loss 2.3395 (3.6249) grad_norm 1.5174 (1.2646) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][610/625] eta 0:00:05 lr 0.001958 wd 0.0500 time 0.3676 (0.3849) data time 0.0005 (0.0018) model time 0.3671 (0.3839) loss 4.6797 (3.6248) grad_norm 0.9153 (1.2685) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [45/300][620/625] eta 0:00:01 lr 0.001958 wd 0.0500 time 0.3641 (0.3847) data time 0.0008 (0.0018) model time 0.3633 (0.3836) loss 3.8221 (3.6247) grad_norm 1.3435 (1.2683) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 45 training takes 0:04:00 [2024-08-26 06:01:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:01:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:01:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.576 (0.576) Loss 0.6973 (0.6973) Acc@1 84.277 (84.277) Acc@5 96.094 (96.094) Mem 11225MB [2024-08-26 06:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.148) Loss 1.1748 (0.8554) Acc@1 71.436 (79.474) Acc@5 92.090 (95.534) Mem 11225MB [2024-08-26 06:01:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.123) Loss 1.3428 (1.0534) Acc@1 68.164 (75.056) Acc@5 89.258 (92.899) Mem 11225MB [2024-08-26 06:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.826 Acc@5 92.824 [2024-08-26 06:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 74.8% [2024-08-26 06:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 74.83% [2024-08-26 06:01:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 06:01:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 06:01:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.513 (0.513) Loss 0.5825 (0.5825) Acc@1 85.693 (85.693) Acc@5 97.754 (97.754) Mem 11225MB [2024-08-26 06:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 1.0479 (0.7518) Acc@1 73.096 (81.024) Acc@5 92.529 (96.005) Mem 11225MB [2024-08-26 06:01:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 1.2559 (0.9451) Acc@1 68.359 (76.525) Acc@5 89.795 (93.576) Mem 11225MB [2024-08-26 06:01:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.234 Acc@5 93.534 [2024-08-26 06:01:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 76.2% [2024-08-26 06:01:36 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 76.23% [2024-08-26 06:01:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:01:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][0/625] eta 0:08:22 lr 0.001958 wd 0.0500 time 0.8042 (0.8042) data time 0.4532 (0.4532) model time 0.0000 (0.0000) loss 3.2673 (3.2673) grad_norm 1.4166 (1.4166) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][10/625] eta 0:04:16 lr 0.001958 wd 0.0500 time 0.3805 (0.4177) data time 0.0011 (0.0430) model time 0.0000 (0.0000) loss 3.4021 (3.5932) grad_norm 1.2311 (1.1457) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][20/625] eta 0:04:00 lr 0.001958 wd 0.0500 time 0.3744 (0.3980) data time 0.0011 (0.0231) model time 0.0000 (0.0000) loss 3.2716 (3.5335) grad_norm 1.0181 (1.2633) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][30/625] eta 0:04:00 lr 0.001958 wd 0.0500 time 0.3693 (0.4041) data time 0.0008 (0.0160) model time 0.0000 (0.0000) loss 2.9185 (3.4956) grad_norm 1.2317 (1.2457) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][40/625] eta 0:03:52 lr 0.001958 wd 0.0500 time 0.3752 (0.3970) data time 0.0011 (0.0124) model time 0.0000 (0.0000) loss 4.0354 (3.5853) grad_norm 1.1906 (1.2210) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][50/625] eta 0:03:45 lr 0.001958 wd 0.0500 time 0.3690 (0.3926) data time 0.0012 (0.0102) model time 0.0000 (0.0000) loss 4.1295 (3.6028) grad_norm 1.6631 (1.2506) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][60/625] eta 0:03:40 lr 0.001958 wd 0.0500 time 0.3746 (0.3900) data time 0.0011 (0.0089) model time 0.3736 (0.3751) loss 3.3678 (3.6142) grad_norm 1.0700 (1.2745) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:02:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][70/625] eta 0:03:36 lr 0.001958 wd 0.0500 time 0.3760 (0.3896) data time 0.0011 (0.0078) model time 0.3749 (0.3805) loss 3.9580 (3.5692) grad_norm 1.3631 (1.2759) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:02:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][80/625] eta 0:03:31 lr 0.001958 wd 0.0500 time 0.3605 (0.3881) data time 0.0008 (0.0070) model time 0.3596 (0.3791) loss 3.7391 (3.5587) grad_norm 1.1502 (1.2601) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][90/625] eta 0:03:26 lr 0.001958 wd 0.0500 time 0.3771 (0.3868) data time 0.0011 (0.0063) model time 0.3760 (0.3780) loss 2.3079 (3.5269) grad_norm 1.1063 (1.2544) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:02:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][100/625] eta 0:03:22 lr 0.001958 wd 0.0500 time 0.3940 (0.3861) data time 0.0008 (0.0058) model time 0.3932 (0.3781) loss 3.9742 (3.5483) grad_norm 2.4711 (1.2590) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:02:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][110/625] eta 0:03:18 lr 0.001958 wd 0.0500 time 0.3736 (0.3853) data time 0.0011 (0.0054) model time 0.3725 (0.3778) loss 3.0753 (3.5596) grad_norm 1.3139 (1.2652) loss_scale 16384.0000 (8930.0180) mem 11225MB [2024-08-26 06:02:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][120/625] eta 0:03:14 lr 0.001958 wd 0.0500 time 0.3804 (0.3849) data time 0.0009 (0.0050) model time 0.3795 (0.3780) loss 4.3124 (3.5928) grad_norm 1.0782 (1.2584) loss_scale 16384.0000 (9546.0496) mem 11225MB [2024-08-26 06:02:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][130/625] eta 0:03:10 lr 0.001958 wd 0.0500 time 0.3814 (0.3843) data time 0.0009 (0.0047) model time 0.3805 (0.3778) loss 3.6959 (3.5905) grad_norm 1.3292 (1.2662) loss_scale 16384.0000 (10068.0305) mem 11225MB [2024-08-26 06:02:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][140/625] eta 0:03:06 lr 0.001957 wd 0.0500 time 0.3772 (0.3839) data time 0.0011 (0.0045) model time 0.3761 (0.3777) loss 2.4810 (3.6015) grad_norm 0.9719 (1.2630) loss_scale 16384.0000 (10515.9716) mem 11225MB [2024-08-26 06:02:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][150/625] eta 0:03:02 lr 0.001957 wd 0.0500 time 0.3763 (0.3834) data time 0.0010 (0.0043) model time 0.3753 (0.3775) loss 4.1921 (3.6104) grad_norm 1.4235 (1.2534) loss_scale 16384.0000 (10904.5828) mem 11225MB [2024-08-26 06:02:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][160/625] eta 0:02:58 lr 0.001957 wd 0.0500 time 0.3758 (0.3831) data time 0.0009 (0.0041) model time 0.3749 (0.3775) loss 2.6984 (3.6101) grad_norm 1.0679 (1.2469) loss_scale 16384.0000 (11244.9193) mem 11225MB [2024-08-26 06:02:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][170/625] eta 0:02:55 lr 0.001957 wd 0.0500 time 0.3806 (0.3863) data time 0.0010 (0.0039) model time 0.3795 (0.3824) loss 3.5212 (3.6149) grad_norm 1.1398 (1.2405) loss_scale 16384.0000 (11545.4503) mem 11225MB [2024-08-26 06:02:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][180/625] eta 0:02:51 lr 0.001957 wd 0.0500 time 0.3862 (0.3859) data time 0.0013 (0.0038) model time 0.3850 (0.3820) loss 3.4063 (3.6218) grad_norm 1.6350 (1.2418) loss_scale 16384.0000 (11812.7735) mem 11225MB [2024-08-26 06:02:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][190/625] eta 0:02:47 lr 0.001957 wd 0.0500 time 0.3776 (0.3856) data time 0.0012 (0.0036) model time 0.3764 (0.3819) loss 3.8576 (3.6228) grad_norm 1.0580 (1.2445) loss_scale 16384.0000 (12052.1047) mem 11225MB [2024-08-26 06:02:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][200/625] eta 0:02:43 lr 0.001957 wd 0.0500 time 0.3796 (0.3853) data time 0.0009 (0.0035) model time 0.3788 (0.3816) loss 2.7346 (3.6205) grad_norm 1.4452 (1.2440) loss_scale 16384.0000 (12267.6219) mem 11225MB [2024-08-26 06:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][210/625] eta 0:02:39 lr 0.001957 wd 0.0500 time 0.3865 (0.3849) data time 0.0008 (0.0034) model time 0.3857 (0.3812) loss 4.3149 (3.6132) grad_norm 1.1835 (1.2430) loss_scale 16384.0000 (12462.7109) mem 11225MB [2024-08-26 06:03:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][220/625] eta 0:02:35 lr 0.001957 wd 0.0500 time 0.3770 (0.3845) data time 0.0012 (0.0033) model time 0.3757 (0.3808) loss 3.7224 (3.6123) grad_norm 1.1409 (1.2382) loss_scale 16384.0000 (12640.1448) mem 11225MB [2024-08-26 06:03:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][230/625] eta 0:02:32 lr 0.001957 wd 0.0500 time 0.5984 (0.3850) data time 0.0008 (0.0032) model time 0.5975 (0.3816) loss 4.2512 (3.6200) grad_norm 1.3152 (1.2405) loss_scale 16384.0000 (12802.2165) mem 11225MB [2024-08-26 06:03:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][240/625] eta 0:02:28 lr 0.001957 wd 0.0500 time 0.3790 (0.3857) data time 0.0010 (0.0031) model time 0.3780 (0.3826) loss 3.1577 (3.6286) grad_norm 1.9511 (1.2407) loss_scale 16384.0000 (12950.8382) mem 11225MB [2024-08-26 06:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][250/625] eta 0:02:24 lr 0.001957 wd 0.0500 time 0.3746 (0.3853) data time 0.0011 (0.0031) model time 0.3735 (0.3822) loss 3.8589 (3.6329) grad_norm 1.2836 (1.2406) loss_scale 16384.0000 (13087.6175) mem 11225MB [2024-08-26 06:03:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][260/625] eta 0:02:20 lr 0.001957 wd 0.0500 time 0.3752 (0.3851) data time 0.0012 (0.0030) model time 0.3740 (0.3820) loss 4.2354 (3.6305) grad_norm 1.1409 (1.2488) loss_scale 16384.0000 (13213.9157) mem 11225MB [2024-08-26 06:03:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][270/625] eta 0:02:16 lr 0.001957 wd 0.0500 time 0.3715 (0.3847) data time 0.0008 (0.0029) model time 0.3707 (0.3817) loss 4.0606 (3.6413) grad_norm 1.0311 (1.2448) loss_scale 16384.0000 (13330.8930) mem 11225MB [2024-08-26 06:03:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][280/625] eta 0:02:12 lr 0.001957 wd 0.0500 time 0.3796 (0.3852) data time 0.0008 (0.0028) model time 0.3787 (0.3823) loss 2.5757 (3.6327) grad_norm 1.3993 (1.2540) loss_scale 16384.0000 (13439.5445) mem 11225MB [2024-08-26 06:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][290/625] eta 0:02:09 lr 0.001957 wd 0.0500 time 0.3779 (0.3863) data time 0.0008 (0.0028) model time 0.3771 (0.3837) loss 3.8564 (3.6360) grad_norm 1.5864 (1.2590) loss_scale 16384.0000 (13540.7285) mem 11225MB [2024-08-26 06:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][300/625] eta 0:02:05 lr 0.001957 wd 0.0500 time 0.3727 (0.3860) data time 0.0008 (0.0027) model time 0.3719 (0.3834) loss 3.8823 (3.6357) grad_norm 1.5209 (1.2598) loss_scale 16384.0000 (13635.1894) mem 11225MB [2024-08-26 06:03:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][310/625] eta 0:02:01 lr 0.001957 wd 0.0500 time 0.3706 (0.3856) data time 0.0012 (0.0027) model time 0.3694 (0.3831) loss 3.4464 (3.6378) grad_norm 1.1554 (1.2587) loss_scale 16384.0000 (13723.5756) mem 11225MB [2024-08-26 06:03:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][320/625] eta 0:01:57 lr 0.001957 wd 0.0500 time 0.3780 (0.3853) data time 0.0008 (0.0026) model time 0.3772 (0.3828) loss 4.1694 (3.6398) grad_norm 1.1506 (1.2497) loss_scale 16384.0000 (13806.4548) mem 11225MB [2024-08-26 06:03:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][330/625] eta 0:01:53 lr 0.001956 wd 0.0500 time 0.3762 (0.3851) data time 0.0011 (0.0026) model time 0.3751 (0.3826) loss 3.8523 (3.6430) grad_norm 1.0036 (1.2507) loss_scale 16384.0000 (13884.3263) mem 11225MB [2024-08-26 06:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][340/625] eta 0:01:49 lr 0.001956 wd 0.0500 time 0.3691 (0.3849) data time 0.0011 (0.0025) model time 0.3680 (0.3823) loss 3.2287 (3.6369) grad_norm 1.7539 (1.2550) loss_scale 16384.0000 (13957.6305) mem 11225MB [2024-08-26 06:03:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][350/625] eta 0:01:45 lr 0.001956 wd 0.0500 time 0.3802 (0.3847) data time 0.0010 (0.0025) model time 0.3791 (0.3821) loss 4.3395 (3.6291) grad_norm 1.5299 (1.2589) loss_scale 16384.0000 (14026.7578) mem 11225MB [2024-08-26 06:03:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][360/625] eta 0:01:41 lr 0.001956 wd 0.0500 time 0.3711 (0.3845) data time 0.0008 (0.0025) model time 0.3703 (0.3819) loss 2.7757 (3.6241) grad_norm 0.9300 (1.2597) loss_scale 16384.0000 (14092.0554) mem 11225MB [2024-08-26 06:03:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][370/625] eta 0:01:37 lr 0.001956 wd 0.0500 time 0.3758 (0.3843) data time 0.0011 (0.0024) model time 0.3747 (0.3817) loss 3.5796 (3.6244) grad_norm 1.7589 (1.2607) loss_scale 16384.0000 (14153.8329) mem 11225MB [2024-08-26 06:04:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][380/625] eta 0:01:34 lr 0.001956 wd 0.0500 time 0.3846 (0.3841) data time 0.0008 (0.0024) model time 0.3837 (0.3816) loss 4.2696 (3.6233) grad_norm 1.0896 (1.2648) loss_scale 16384.0000 (14212.3675) mem 11225MB [2024-08-26 06:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][390/625] eta 0:01:30 lr 0.001956 wd 0.0500 time 0.3759 (0.3839) data time 0.0011 (0.0024) model time 0.3748 (0.3815) loss 3.3849 (3.6220) grad_norm 1.0745 (1.2652) loss_scale 16384.0000 (14267.9079) mem 11225MB [2024-08-26 06:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][400/625] eta 0:01:26 lr 0.001956 wd 0.0500 time 0.3799 (0.3838) data time 0.0010 (0.0023) model time 0.3788 (0.3813) loss 3.7846 (3.6274) grad_norm 1.1039 (1.2604) loss_scale 16384.0000 (14320.6783) mem 11225MB [2024-08-26 06:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][410/625] eta 0:01:22 lr 0.001956 wd 0.0500 time 0.3724 (0.3837) data time 0.0009 (0.0023) model time 0.3715 (0.3812) loss 3.4763 (3.6212) grad_norm 1.0938 (1.2660) loss_scale 16384.0000 (14370.8808) mem 11225MB [2024-08-26 06:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][420/625] eta 0:01:18 lr 0.001956 wd 0.0500 time 0.3741 (0.3845) data time 0.0008 (0.0023) model time 0.3733 (0.3822) loss 4.1541 (3.6214) grad_norm 1.1581 (1.2675) loss_scale 16384.0000 (14418.6983) mem 11225MB [2024-08-26 06:04:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][430/625] eta 0:01:14 lr 0.001956 wd 0.0500 time 0.3734 (0.3843) data time 0.0011 (0.0023) model time 0.3723 (0.3820) loss 3.9810 (3.6265) grad_norm 1.1847 (1.2682) loss_scale 16384.0000 (14464.2970) mem 11225MB [2024-08-26 06:04:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][440/625] eta 0:01:11 lr 0.001956 wd 0.0500 time 0.3674 (0.3842) data time 0.0011 (0.0022) model time 0.3662 (0.3819) loss 3.7450 (3.6263) grad_norm 1.0033 (1.2660) loss_scale 16384.0000 (14507.8277) mem 11225MB [2024-08-26 06:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][450/625] eta 0:01:07 lr 0.001956 wd 0.0500 time 0.3872 (0.3840) data time 0.0008 (0.0022) model time 0.3864 (0.3818) loss 3.5406 (3.6257) grad_norm 0.9212 (1.2660) loss_scale 16384.0000 (14549.4279) mem 11225MB [2024-08-26 06:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][460/625] eta 0:01:03 lr 0.001956 wd 0.0500 time 0.3765 (0.3843) data time 0.0014 (0.0022) model time 0.3751 (0.3821) loss 3.6244 (3.6281) grad_norm 1.8614 (1.2668) loss_scale 16384.0000 (14589.2234) mem 11225MB [2024-08-26 06:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][470/625] eta 0:00:59 lr 0.001956 wd 0.0500 time 0.3757 (0.3841) data time 0.0010 (0.0022) model time 0.3746 (0.3819) loss 2.5430 (3.6252) grad_norm 1.2340 (1.2663) loss_scale 16384.0000 (14627.3291) mem 11225MB [2024-08-26 06:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][480/625] eta 0:00:55 lr 0.001956 wd 0.0500 time 0.3810 (0.3841) data time 0.0008 (0.0021) model time 0.3801 (0.3819) loss 2.7304 (3.6233) grad_norm 0.9806 (1.2657) loss_scale 16384.0000 (14663.8503) mem 11225MB [2024-08-26 06:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][490/625] eta 0:00:51 lr 0.001956 wd 0.0500 time 0.3739 (0.3840) data time 0.0009 (0.0021) model time 0.3729 (0.3818) loss 2.8145 (3.6204) grad_norm 1.5248 (1.2700) loss_scale 16384.0000 (14698.8839) mem 11225MB [2024-08-26 06:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][500/625] eta 0:00:48 lr 0.001956 wd 0.0500 time 0.3801 (0.3847) data time 0.0010 (0.0021) model time 0.3790 (0.3826) loss 3.8327 (3.6214) grad_norm 1.0828 (1.2663) loss_scale 16384.0000 (14732.5190) mem 11225MB [2024-08-26 06:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][510/625] eta 0:00:44 lr 0.001956 wd 0.0500 time 0.3988 (0.3846) data time 0.0011 (0.0021) model time 0.3978 (0.3825) loss 3.3101 (3.6162) grad_norm 0.8831 (1.2633) loss_scale 16384.0000 (14764.8376) mem 11225MB [2024-08-26 06:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][520/625] eta 0:00:40 lr 0.001955 wd 0.0500 time 0.3748 (0.3844) data time 0.0008 (0.0021) model time 0.3740 (0.3824) loss 2.8158 (3.6141) grad_norm 1.1765 (1.2632) loss_scale 16384.0000 (14795.9155) mem 11225MB [2024-08-26 06:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][530/625] eta 0:00:36 lr 0.001955 wd 0.0500 time 0.3787 (0.3843) data time 0.0011 (0.0020) model time 0.3777 (0.3822) loss 3.9825 (3.6142) grad_norm 1.7638 (1.2614) loss_scale 16384.0000 (14825.8230) mem 11225MB [2024-08-26 06:05:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][540/625] eta 0:00:32 lr 0.001955 wd 0.0500 time 0.3821 (0.3850) data time 0.0008 (0.0020) model time 0.3813 (0.3830) loss 4.3317 (3.6178) grad_norm 0.8814 (1.2651) loss_scale 16384.0000 (14854.6248) mem 11225MB [2024-08-26 06:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][550/625] eta 0:00:28 lr 0.001955 wd 0.0500 time 0.3729 (0.3848) data time 0.0009 (0.0020) model time 0.3721 (0.3829) loss 3.9830 (3.6143) grad_norm 1.0134 (1.2614) loss_scale 16384.0000 (14882.3811) mem 11225MB [2024-08-26 06:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][560/625] eta 0:00:25 lr 0.001955 wd 0.0500 time 0.3854 (0.3847) data time 0.0010 (0.0020) model time 0.3844 (0.3828) loss 2.6954 (3.6173) grad_norm 1.2719 (1.2629) loss_scale 16384.0000 (14909.1480) mem 11225MB [2024-08-26 06:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][570/625] eta 0:00:21 lr 0.001955 wd 0.0500 time 0.3628 (0.3846) data time 0.0011 (0.0020) model time 0.3617 (0.3826) loss 3.9729 (3.6189) grad_norm 1.0158 (1.2640) loss_scale 16384.0000 (14934.9772) mem 11225MB [2024-08-26 06:05:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][580/625] eta 0:00:17 lr 0.001955 wd 0.0500 time 0.3719 (0.3848) data time 0.0011 (0.0020) model time 0.3708 (0.3829) loss 3.5741 (3.6207) grad_norm 1.6239 (1.2615) loss_scale 16384.0000 (14959.9174) mem 11225MB [2024-08-26 06:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][590/625] eta 0:00:13 lr 0.001955 wd 0.0500 time 0.3802 (0.3847) data time 0.0008 (0.0020) model time 0.3794 (0.3828) loss 3.2145 (3.6229) grad_norm 1.0166 (1.2610) loss_scale 16384.0000 (14984.0135) mem 11225MB [2024-08-26 06:05:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][600/625] eta 0:00:09 lr 0.001955 wd 0.0500 time 0.3730 (0.3845) data time 0.0011 (0.0019) model time 0.3719 (0.3826) loss 3.1817 (3.6241) grad_norm 1.5204 (1.2634) loss_scale 16384.0000 (15007.3078) mem 11225MB [2024-08-26 06:05:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][610/625] eta 0:00:05 lr 0.001955 wd 0.0500 time 0.3708 (0.3844) data time 0.0005 (0.0019) model time 0.3703 (0.3825) loss 4.2830 (3.6195) grad_norm 1.5842 (1.2647) loss_scale 16384.0000 (15029.8396) mem 11225MB [2024-08-26 06:05:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [46/300][620/625] eta 0:00:01 lr 0.001955 wd 0.0500 time 0.3694 (0.3842) data time 0.0006 (0.0019) model time 0.3689 (0.3823) loss 4.2505 (3.6219) grad_norm 1.0329 (1.2644) loss_scale 16384.0000 (15051.6457) mem 11225MB [2024-08-26 06:05:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 46 training takes 0:04:00 [2024-08-26 06:05:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:05:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:05:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.7227 (0.7227) Acc@1 83.105 (83.105) Acc@5 97.363 (97.363) Mem 11225MB [2024-08-26 06:05:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 1.2266 (0.9044) Acc@1 70.654 (79.150) Acc@5 91.553 (95.517) Mem 11225MB [2024-08-26 06:05:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.4043 (1.0924) Acc@1 68.115 (74.953) Acc@5 88.428 (92.778) Mem 11225MB [2024-08-26 06:05:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.670 Acc@5 92.776 [2024-08-26 06:05:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 74.7% [2024-08-26 06:05:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.965 (0.965) Loss 0.5820 (0.5820) Acc@1 85.889 (85.889) Acc@5 97.705 (97.705) Mem 11225MB [2024-08-26 06:05:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.187) Loss 1.0371 (0.7483) Acc@1 73.486 (81.161) Acc@5 92.627 (96.058) Mem 11225MB [2024-08-26 06:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.142) Loss 1.2441 (0.9394) Acc@1 68.799 (76.744) Acc@5 89.893 (93.685) Mem 11225MB [2024-08-26 06:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.460 Acc@5 93.656 [2024-08-26 06:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 76.5% [2024-08-26 06:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 76.46% [2024-08-26 06:05:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:05:46 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:05:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][0/625] eta 0:08:12 lr 0.001955 wd 0.0500 time 0.7882 (0.7882) data time 0.4522 (0.4522) model time 0.0000 (0.0000) loss 2.9002 (2.9002) grad_norm 1.0928 (1.0928) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:05:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][10/625] eta 0:04:14 lr 0.001955 wd 0.0500 time 0.3792 (0.4145) data time 0.0010 (0.0421) model time 0.0000 (0.0000) loss 3.7945 (3.9080) grad_norm 1.1588 (1.1429) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][20/625] eta 0:03:59 lr 0.001955 wd 0.0500 time 0.3782 (0.3965) data time 0.0011 (0.0226) model time 0.0000 (0.0000) loss 3.8088 (3.7432) grad_norm 1.4901 (1.1535) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:05:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][30/625] eta 0:03:52 lr 0.001955 wd 0.0500 time 0.3771 (0.3903) data time 0.0009 (0.0157) model time 0.0000 (0.0000) loss 2.7818 (3.6804) grad_norm 1.2799 (1.1790) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:06:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][40/625] eta 0:03:46 lr 0.001955 wd 0.0500 time 0.3806 (0.3871) data time 0.0011 (0.0122) model time 0.0000 (0.0000) loss 3.6553 (3.6698) grad_norm 1.3715 (1.1987) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:06:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][50/625] eta 0:03:41 lr 0.001955 wd 0.0500 time 0.3779 (0.3847) data time 0.0009 (0.0100) model time 0.0000 (0.0000) loss 2.7700 (3.6451) grad_norm 1.1381 (inf) loss_scale 8192.0000 (15580.8627) mem 11225MB [2024-08-26 06:06:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][60/625] eta 0:03:39 lr 0.001955 wd 0.0500 time 0.5801 (0.3893) data time 0.0011 (0.0085) model time 0.5790 (0.4114) loss 3.8024 (3.6459) grad_norm 1.2645 (inf) loss_scale 8192.0000 (14369.5738) mem 11225MB [2024-08-26 06:06:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][70/625] eta 0:03:39 lr 0.001955 wd 0.0500 time 0.5427 (0.3953) data time 0.0011 (0.0075) model time 0.5416 (0.4213) loss 3.4224 (3.6176) grad_norm 0.8962 (inf) loss_scale 8192.0000 (13499.4930) mem 11225MB [2024-08-26 06:06:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][80/625] eta 0:03:34 lr 0.001954 wd 0.0500 time 0.3767 (0.3931) data time 0.0015 (0.0067) model time 0.3752 (0.4063) loss 2.9133 (3.6034) grad_norm 1.0240 (inf) loss_scale 8192.0000 (12844.2469) mem 11225MB [2024-08-26 06:06:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][90/625] eta 0:03:29 lr 0.001954 wd 0.0500 time 0.3740 (0.3915) data time 0.0007 (0.0061) model time 0.3733 (0.3990) loss 3.9657 (3.6190) grad_norm 1.7808 (inf) loss_scale 8192.0000 (12333.0110) mem 11225MB [2024-08-26 06:06:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][100/625] eta 0:03:24 lr 0.001954 wd 0.0500 time 0.3730 (0.3898) data time 0.0009 (0.0056) model time 0.3721 (0.3939) loss 3.8908 (3.6098) grad_norm 1.0123 (inf) loss_scale 8192.0000 (11923.0099) mem 11225MB [2024-08-26 06:06:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][110/625] eta 0:03:20 lr 0.001954 wd 0.0500 time 0.3753 (0.3898) data time 0.0009 (0.0052) model time 0.3744 (0.3931) loss 4.0059 (3.6318) grad_norm 1.5979 (inf) loss_scale 8192.0000 (11586.8829) mem 11225MB [2024-08-26 06:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][120/625] eta 0:03:16 lr 0.001954 wd 0.0500 time 0.3816 (0.3890) data time 0.0010 (0.0048) model time 0.3805 (0.3909) loss 3.6302 (3.6437) grad_norm 1.3854 (inf) loss_scale 8192.0000 (11306.3140) mem 11225MB [2024-08-26 06:06:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][130/625] eta 0:03:12 lr 0.001954 wd 0.0500 time 0.3706 (0.3880) data time 0.0008 (0.0046) model time 0.3698 (0.3890) loss 4.6070 (3.6409) grad_norm 1.2894 (inf) loss_scale 8192.0000 (11068.5802) mem 11225MB [2024-08-26 06:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][140/625] eta 0:03:07 lr 0.001954 wd 0.0500 time 0.3787 (0.3873) data time 0.0009 (0.0043) model time 0.3778 (0.3876) loss 4.5319 (3.6556) grad_norm 1.6088 (inf) loss_scale 8192.0000 (10864.5674) mem 11225MB [2024-08-26 06:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][150/625] eta 0:03:03 lr 0.001954 wd 0.0500 time 0.3768 (0.3867) data time 0.0013 (0.0041) model time 0.3755 (0.3867) loss 3.8959 (3.6684) grad_norm 1.2149 (inf) loss_scale 8192.0000 (10687.5762) mem 11225MB [2024-08-26 06:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][160/625] eta 0:02:59 lr 0.001954 wd 0.0500 time 0.3795 (0.3861) data time 0.0009 (0.0039) model time 0.3786 (0.3856) loss 4.3751 (3.6601) grad_norm 1.6670 (inf) loss_scale 8192.0000 (10532.5714) mem 11225MB [2024-08-26 06:06:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][170/625] eta 0:02:55 lr 0.001954 wd 0.0500 time 0.3754 (0.3856) data time 0.0008 (0.0038) model time 0.3746 (0.3849) loss 4.2400 (3.6737) grad_norm 1.0122 (inf) loss_scale 8192.0000 (10395.6959) mem 11225MB [2024-08-26 06:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][180/625] eta 0:02:51 lr 0.001954 wd 0.0500 time 0.3860 (0.3851) data time 0.0010 (0.0036) model time 0.3849 (0.3842) loss 3.4311 (3.6699) grad_norm 2.4671 (inf) loss_scale 8192.0000 (10273.9448) mem 11225MB [2024-08-26 06:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][190/625] eta 0:02:47 lr 0.001954 wd 0.0500 time 0.3811 (0.3848) data time 0.0010 (0.0035) model time 0.3801 (0.3837) loss 2.9167 (3.6430) grad_norm 1.1029 (inf) loss_scale 8192.0000 (10164.9424) mem 11225MB [2024-08-26 06:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][200/625] eta 0:02:43 lr 0.001954 wd 0.0500 time 0.3789 (0.3844) data time 0.0010 (0.0034) model time 0.3778 (0.3832) loss 3.7294 (3.6383) grad_norm 1.3216 (inf) loss_scale 8192.0000 (10066.7861) mem 11225MB [2024-08-26 06:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][210/625] eta 0:02:39 lr 0.001954 wd 0.0500 time 0.3726 (0.3840) data time 0.0012 (0.0033) model time 0.3714 (0.3827) loss 3.6787 (3.6375) grad_norm 1.1388 (inf) loss_scale 8192.0000 (9977.9336) mem 11225MB [2024-08-26 06:07:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][220/625] eta 0:02:35 lr 0.001954 wd 0.0500 time 0.3766 (0.3838) data time 0.0011 (0.0032) model time 0.3755 (0.3823) loss 3.7942 (3.6355) grad_norm 1.1072 (inf) loss_scale 8192.0000 (9897.1222) mem 11225MB [2024-08-26 06:07:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][230/625] eta 0:02:31 lr 0.001954 wd 0.0500 time 0.3740 (0.3835) data time 0.0011 (0.0031) model time 0.3729 (0.3820) loss 4.0538 (3.6415) grad_norm 1.0703 (inf) loss_scale 8192.0000 (9823.3074) mem 11225MB [2024-08-26 06:07:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][240/625] eta 0:02:27 lr 0.001954 wd 0.0500 time 0.3787 (0.3833) data time 0.0009 (0.0030) model time 0.3778 (0.3817) loss 4.0152 (3.6375) grad_norm 1.1596 (inf) loss_scale 8192.0000 (9755.6183) mem 11225MB [2024-08-26 06:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][250/625] eta 0:02:24 lr 0.001954 wd 0.0500 time 0.3746 (0.3853) data time 0.0009 (0.0029) model time 0.3737 (0.3843) loss 2.9891 (3.6324) grad_norm 1.5315 (inf) loss_scale 8192.0000 (9693.3227) mem 11225MB [2024-08-26 06:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][260/625] eta 0:02:20 lr 0.001954 wd 0.0500 time 0.3951 (0.3850) data time 0.0012 (0.0029) model time 0.3939 (0.3839) loss 3.1132 (3.6259) grad_norm 1.0890 (inf) loss_scale 8192.0000 (9635.8008) mem 11225MB [2024-08-26 06:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][270/625] eta 0:02:16 lr 0.001953 wd 0.0500 time 0.3712 (0.3848) data time 0.0008 (0.0028) model time 0.3704 (0.3836) loss 2.2635 (3.6225) grad_norm 1.1080 (inf) loss_scale 8192.0000 (9582.5240) mem 11225MB [2024-08-26 06:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][280/625] eta 0:02:13 lr 0.001953 wd 0.0500 time 0.5918 (0.3860) data time 0.0008 (0.0027) model time 0.5910 (0.3851) loss 2.9453 (3.6201) grad_norm 0.9894 (inf) loss_scale 8192.0000 (9533.0391) mem 11225MB [2024-08-26 06:07:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][290/625] eta 0:02:09 lr 0.001953 wd 0.0500 time 0.3904 (0.3862) data time 0.0009 (0.0027) model time 0.3895 (0.3854) loss 3.4260 (3.6227) grad_norm 0.8593 (inf) loss_scale 8192.0000 (9486.9553) mem 11225MB [2024-08-26 06:07:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][300/625] eta 0:02:05 lr 0.001953 wd 0.0500 time 0.3748 (0.3859) data time 0.0008 (0.0026) model time 0.3740 (0.3850) loss 2.8592 (3.6215) grad_norm 0.8046 (inf) loss_scale 8192.0000 (9443.9336) mem 11225MB [2024-08-26 06:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][310/625] eta 0:02:01 lr 0.001953 wd 0.0500 time 0.3734 (0.3856) data time 0.0009 (0.0026) model time 0.3725 (0.3847) loss 3.6674 (3.6203) grad_norm 1.1862 (inf) loss_scale 8192.0000 (9403.6785) mem 11225MB [2024-08-26 06:07:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][320/625] eta 0:01:57 lr 0.001953 wd 0.0500 time 0.3833 (0.3854) data time 0.0011 (0.0025) model time 0.3822 (0.3844) loss 2.8908 (3.6207) grad_norm 1.1985 (inf) loss_scale 8192.0000 (9365.9315) mem 11225MB [2024-08-26 06:07:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][330/625] eta 0:01:53 lr 0.001953 wd 0.0500 time 0.3716 (0.3851) data time 0.0009 (0.0025) model time 0.3707 (0.3840) loss 2.4756 (3.6130) grad_norm 1.7173 (inf) loss_scale 8192.0000 (9330.4653) mem 11225MB [2024-08-26 06:07:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][340/625] eta 0:01:49 lr 0.001953 wd 0.0500 time 0.3770 (0.3849) data time 0.0011 (0.0025) model time 0.3760 (0.3838) loss 3.4803 (3.6151) grad_norm 1.3051 (inf) loss_scale 8192.0000 (9297.0792) mem 11225MB [2024-08-26 06:08:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][350/625] eta 0:01:45 lr 0.001953 wd 0.0500 time 0.3750 (0.3847) data time 0.0009 (0.0024) model time 0.3741 (0.3835) loss 3.8571 (3.6164) grad_norm 1.1543 (inf) loss_scale 8192.0000 (9265.5954) mem 11225MB [2024-08-26 06:08:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][360/625] eta 0:01:41 lr 0.001953 wd 0.0500 time 0.3762 (0.3845) data time 0.0008 (0.0024) model time 0.3755 (0.3833) loss 4.2053 (3.6239) grad_norm 1.1062 (inf) loss_scale 8192.0000 (9235.8560) mem 11225MB [2024-08-26 06:08:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][370/625] eta 0:01:38 lr 0.001953 wd 0.0500 time 0.3825 (0.3854) data time 0.0010 (0.0024) model time 0.3815 (0.3844) loss 4.0861 (3.6296) grad_norm 1.4031 (inf) loss_scale 8192.0000 (9207.7197) mem 11225MB [2024-08-26 06:08:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][380/625] eta 0:01:34 lr 0.001953 wd 0.0500 time 0.3786 (0.3852) data time 0.0011 (0.0023) model time 0.3776 (0.3841) loss 3.5915 (3.6255) grad_norm 0.8580 (inf) loss_scale 8192.0000 (9181.0604) mem 11225MB [2024-08-26 06:08:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][390/625] eta 0:01:30 lr 0.001953 wd 0.0500 time 0.3743 (0.3850) data time 0.0011 (0.0023) model time 0.3733 (0.3839) loss 3.3536 (3.6272) grad_norm 1.0431 (inf) loss_scale 8192.0000 (9155.7647) mem 11225MB [2024-08-26 06:08:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][400/625] eta 0:01:26 lr 0.001953 wd 0.0500 time 0.3807 (0.3849) data time 0.0011 (0.0023) model time 0.3796 (0.3838) loss 3.8069 (3.6273) grad_norm 1.3662 (inf) loss_scale 8192.0000 (9131.7307) mem 11225MB [2024-08-26 06:08:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][410/625] eta 0:01:22 lr 0.001953 wd 0.0500 time 0.3765 (0.3852) data time 0.0012 (0.0022) model time 0.3753 (0.3841) loss 4.0257 (3.6340) grad_norm 1.1860 (inf) loss_scale 8192.0000 (9108.8662) mem 11225MB [2024-08-26 06:08:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][420/625] eta 0:01:18 lr 0.001953 wd 0.0500 time 0.3706 (0.3849) data time 0.0008 (0.0022) model time 0.3698 (0.3838) loss 3.1289 (3.6297) grad_norm 1.4569 (inf) loss_scale 8192.0000 (9087.0879) mem 11225MB [2024-08-26 06:08:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][430/625] eta 0:01:15 lr 0.001953 wd 0.0500 time 0.3781 (0.3848) data time 0.0009 (0.0022) model time 0.3772 (0.3836) loss 3.2613 (3.6218) grad_norm 0.8846 (inf) loss_scale 8192.0000 (9066.3202) mem 11225MB [2024-08-26 06:08:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][440/625] eta 0:01:11 lr 0.001953 wd 0.0500 time 0.3798 (0.3846) data time 0.0011 (0.0022) model time 0.3787 (0.3834) loss 3.5902 (3.6254) grad_norm 1.0358 (inf) loss_scale 8192.0000 (9046.4943) mem 11225MB [2024-08-26 06:08:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][450/625] eta 0:01:07 lr 0.001953 wd 0.0500 time 0.3800 (0.3845) data time 0.0010 (0.0021) model time 0.3789 (0.3833) loss 3.5047 (3.6265) grad_norm 1.3582 (inf) loss_scale 8192.0000 (9027.5477) mem 11225MB [2024-08-26 06:08:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][460/625] eta 0:01:03 lr 0.001952 wd 0.0500 time 0.3731 (0.3848) data time 0.0009 (0.0021) model time 0.3723 (0.3837) loss 4.5609 (3.6293) grad_norm 1.5590 (inf) loss_scale 8192.0000 (9009.4230) mem 11225MB [2024-08-26 06:08:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][470/625] eta 0:00:59 lr 0.001952 wd 0.0500 time 0.3720 (0.3856) data time 0.0011 (0.0021) model time 0.3709 (0.3846) loss 4.0525 (3.6294) grad_norm 1.2009 (inf) loss_scale 8192.0000 (8992.0679) mem 11225MB [2024-08-26 06:08:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][480/625] eta 0:00:55 lr 0.001952 wd 0.0500 time 0.3788 (0.3855) data time 0.0008 (0.0021) model time 0.3780 (0.3844) loss 3.7484 (3.6233) grad_norm 1.1862 (inf) loss_scale 8192.0000 (8975.4345) mem 11225MB [2024-08-26 06:08:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][490/625] eta 0:00:52 lr 0.001952 wd 0.0500 time 0.3762 (0.3853) data time 0.0011 (0.0021) model time 0.3751 (0.3842) loss 4.1596 (3.6284) grad_norm 0.9278 (inf) loss_scale 8192.0000 (8959.4786) mem 11225MB [2024-08-26 06:08:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][500/625] eta 0:00:48 lr 0.001952 wd 0.0500 time 0.3722 (0.3852) data time 0.0008 (0.0020) model time 0.3714 (0.3841) loss 4.5703 (3.6347) grad_norm 1.0068 (inf) loss_scale 8192.0000 (8944.1597) mem 11225MB [2024-08-26 06:09:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][510/625] eta 0:00:44 lr 0.001952 wd 0.0500 time 0.3741 (0.3850) data time 0.0008 (0.0020) model time 0.3733 (0.3839) loss 3.3153 (3.6320) grad_norm 1.2072 (inf) loss_scale 8192.0000 (8929.4403) mem 11225MB [2024-08-26 06:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][520/625] eta 0:00:40 lr 0.001952 wd 0.0500 time 0.3685 (0.3849) data time 0.0011 (0.0020) model time 0.3674 (0.3837) loss 2.6969 (3.6295) grad_norm 1.4589 (inf) loss_scale 8192.0000 (8915.2860) mem 11225MB [2024-08-26 06:09:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][530/625] eta 0:00:36 lr 0.001952 wd 0.0500 time 0.3739 (0.3847) data time 0.0010 (0.0020) model time 0.3729 (0.3836) loss 2.8443 (3.6333) grad_norm 1.5524 (inf) loss_scale 8192.0000 (8901.6648) mem 11225MB [2024-08-26 06:09:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][540/625] eta 0:00:32 lr 0.001952 wd 0.0500 time 0.3746 (0.3845) data time 0.0009 (0.0020) model time 0.3737 (0.3834) loss 3.4460 (3.6347) grad_norm 1.0466 (inf) loss_scale 8192.0000 (8888.5471) mem 11225MB [2024-08-26 06:09:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][550/625] eta 0:00:28 lr 0.001952 wd 0.0500 time 0.3767 (0.3844) data time 0.0011 (0.0020) model time 0.3756 (0.3832) loss 2.8385 (3.6358) grad_norm 0.9566 (inf) loss_scale 8192.0000 (8875.9056) mem 11225MB [2024-08-26 06:09:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][560/625] eta 0:00:24 lr 0.001952 wd 0.0500 time 0.3724 (0.3842) data time 0.0008 (0.0019) model time 0.3716 (0.3831) loss 3.2857 (3.6347) grad_norm 1.1530 (inf) loss_scale 8192.0000 (8863.7148) mem 11225MB [2024-08-26 06:09:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][570/625] eta 0:00:21 lr 0.001952 wd 0.0500 time 0.3715 (0.3841) data time 0.0009 (0.0019) model time 0.3706 (0.3829) loss 4.6102 (3.6334) grad_norm 1.3489 (inf) loss_scale 8192.0000 (8851.9510) mem 11225MB [2024-08-26 06:09:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][580/625] eta 0:00:17 lr 0.001952 wd 0.0500 time 0.3793 (0.3851) data time 0.0011 (0.0019) model time 0.3783 (0.3840) loss 3.6509 (3.6311) grad_norm 1.7674 (inf) loss_scale 8192.0000 (8840.5921) mem 11225MB [2024-08-26 06:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][590/625] eta 0:00:13 lr 0.001952 wd 0.0500 time 0.3814 (0.3856) data time 0.0011 (0.0019) model time 0.3803 (0.3846) loss 3.9100 (3.6317) grad_norm 1.2834 (inf) loss_scale 8192.0000 (8829.6176) mem 11225MB [2024-08-26 06:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][600/625] eta 0:00:09 lr 0.001952 wd 0.0500 time 0.3792 (0.3855) data time 0.0009 (0.0019) model time 0.3784 (0.3845) loss 2.9686 (3.6318) grad_norm 1.3239 (inf) loss_scale 8192.0000 (8819.0083) mem 11225MB [2024-08-26 06:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][610/625] eta 0:00:05 lr 0.001952 wd 0.0500 time 0.3668 (0.3854) data time 0.0008 (0.0019) model time 0.3660 (0.3843) loss 3.6833 (3.6352) grad_norm 1.1422 (inf) loss_scale 8192.0000 (8808.7463) mem 11225MB [2024-08-26 06:09:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [47/300][620/625] eta 0:00:01 lr 0.001952 wd 0.0500 time 0.3677 (0.3854) data time 0.0008 (0.0019) model time 0.3669 (0.3843) loss 3.2479 (3.6331) grad_norm 1.6790 (inf) loss_scale 8192.0000 (8798.8148) mem 11225MB [2024-08-26 06:09:46 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 47 training takes 0:04:00 [2024-08-26 06:09:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:09:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:09:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.512 (0.512) Loss 0.8003 (0.8003) Acc@1 83.594 (83.594) Acc@5 96.338 (96.338) Mem 11225MB [2024-08-26 06:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 1.2422 (0.9477) Acc@1 71.826 (79.310) Acc@5 91.455 (95.241) Mem 11225MB [2024-08-26 06:09:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.4443 (1.1409) Acc@1 67.139 (74.888) Acc@5 89.453 (92.608) Mem 11225MB [2024-08-26 06:09:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.690 Acc@5 92.594 [2024-08-26 06:09:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 74.7% [2024-08-26 06:09:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.825 (0.825) Loss 0.5801 (0.5801) Acc@1 85.986 (85.986) Acc@5 97.656 (97.656) Mem 11225MB [2024-08-26 06:09:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.175) Loss 1.0322 (0.7455) Acc@1 74.121 (81.396) Acc@5 92.725 (96.072) Mem 11225MB [2024-08-26 06:09:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 1.2383 (0.9344) Acc@1 69.092 (76.946) Acc@5 90.039 (93.762) Mem 11225MB [2024-08-26 06:09:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.669 Acc@5 93.728 [2024-08-26 06:09:54 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 76.7% [2024-08-26 06:09:54 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 76.67% [2024-08-26 06:09:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:09:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][0/625] eta 0:08:36 lr 0.001952 wd 0.0500 time 0.8271 (0.8271) data time 0.4815 (0.4815) model time 0.0000 (0.0000) loss 4.1443 (4.1443) grad_norm 1.1955 (1.1955) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][10/625] eta 0:04:17 lr 0.001951 wd 0.0500 time 0.3802 (0.4187) data time 0.0012 (0.0448) model time 0.0000 (0.0000) loss 3.6995 (3.5687) grad_norm 0.9642 (1.1395) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][20/625] eta 0:04:01 lr 0.001951 wd 0.0500 time 0.3685 (0.3985) data time 0.0011 (0.0240) model time 0.0000 (0.0000) loss 3.0540 (3.6187) grad_norm 1.9365 (1.1711) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][30/625] eta 0:03:52 lr 0.001951 wd 0.0500 time 0.3746 (0.3912) data time 0.0008 (0.0167) model time 0.0000 (0.0000) loss 4.5067 (3.6780) grad_norm 1.3702 (1.1624) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][40/625] eta 0:03:47 lr 0.001951 wd 0.0500 time 0.4087 (0.3889) data time 0.0011 (0.0129) model time 0.0000 (0.0000) loss 4.0882 (3.6610) grad_norm 0.8461 (1.1561) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][50/625] eta 0:03:42 lr 0.001951 wd 0.0500 time 0.3760 (0.3867) data time 0.0008 (0.0106) model time 0.0000 (0.0000) loss 3.5406 (3.7068) grad_norm 1.7531 (1.1865) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][60/625] eta 0:03:37 lr 0.001951 wd 0.0500 time 0.3785 (0.3853) data time 0.0009 (0.0090) model time 0.3776 (0.3773) loss 4.4145 (3.7482) grad_norm 1.3843 (1.2137) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][70/625] eta 0:03:33 lr 0.001951 wd 0.0500 time 0.4115 (0.3844) data time 0.0010 (0.0079) model time 0.4105 (0.3774) loss 4.1520 (3.7322) grad_norm 0.8713 (1.2062) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][80/625] eta 0:03:29 lr 0.001951 wd 0.0500 time 0.3901 (0.3837) data time 0.0009 (0.0071) model time 0.3892 (0.3775) loss 3.5221 (3.7014) grad_norm 1.1322 (1.2197) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][90/625] eta 0:03:25 lr 0.001951 wd 0.0500 time 0.3663 (0.3842) data time 0.0011 (0.0064) model time 0.3652 (0.3798) loss 3.0707 (3.6777) grad_norm 1.0456 (1.2436) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][100/625] eta 0:03:24 lr 0.001951 wd 0.0500 time 0.3801 (0.3891) data time 0.0008 (0.0059) model time 0.3793 (0.3903) loss 4.0339 (3.6974) grad_norm 1.1134 (1.2298) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][110/625] eta 0:03:19 lr 0.001951 wd 0.0500 time 0.3786 (0.3879) data time 0.0011 (0.0055) model time 0.3775 (0.3877) loss 3.8317 (3.6932) grad_norm 1.7529 (1.2300) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][120/625] eta 0:03:15 lr 0.001951 wd 0.0500 time 0.3870 (0.3869) data time 0.0008 (0.0051) model time 0.3862 (0.3859) loss 4.3617 (3.6783) grad_norm 1.1397 (1.2395) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][130/625] eta 0:03:11 lr 0.001951 wd 0.0500 time 0.3727 (0.3861) data time 0.0011 (0.0048) model time 0.3717 (0.3846) loss 2.7882 (3.6750) grad_norm 1.0784 (1.2476) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][140/625] eta 0:03:06 lr 0.001951 wd 0.0500 time 0.3793 (0.3855) data time 0.0009 (0.0046) model time 0.3785 (0.3837) loss 3.6161 (3.6717) grad_norm 1.1903 (1.2478) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][150/625] eta 0:03:03 lr 0.001951 wd 0.0500 time 0.5832 (0.3864) data time 0.0009 (0.0043) model time 0.5824 (0.3852) loss 2.3922 (3.6645) grad_norm 0.8711 (1.2372) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:10:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][160/625] eta 0:03:00 lr 0.001951 wd 0.0500 time 0.3865 (0.3875) data time 0.0010 (0.0042) model time 0.3854 (0.3867) loss 2.3175 (3.6483) grad_norm 1.7044 (1.2453) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][170/625] eta 0:02:56 lr 0.001951 wd 0.0500 time 0.3750 (0.3869) data time 0.0009 (0.0040) model time 0.3741 (0.3858) loss 2.9133 (3.6549) grad_norm 1.0239 (1.2438) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][180/625] eta 0:02:51 lr 0.001951 wd 0.0500 time 0.3775 (0.3864) data time 0.0008 (0.0038) model time 0.3767 (0.3852) loss 3.5764 (3.6547) grad_norm 1.0932 (1.2427) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][190/625] eta 0:02:47 lr 0.001950 wd 0.0500 time 0.3734 (0.3859) data time 0.0009 (0.0037) model time 0.3724 (0.3845) loss 4.4374 (3.6488) grad_norm 1.0969 (1.2364) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][200/625] eta 0:02:43 lr 0.001950 wd 0.0500 time 0.3830 (0.3856) data time 0.0009 (0.0036) model time 0.3821 (0.3841) loss 2.6434 (3.6484) grad_norm 1.3815 (1.2359) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][210/625] eta 0:02:39 lr 0.001950 wd 0.0500 time 0.3799 (0.3852) data time 0.0011 (0.0034) model time 0.3788 (0.3836) loss 2.9848 (3.6513) grad_norm 1.2395 (1.2334) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][220/625] eta 0:02:35 lr 0.001950 wd 0.0500 time 0.3755 (0.3849) data time 0.0008 (0.0033) model time 0.3747 (0.3831) loss 3.1437 (3.6446) grad_norm 1.2610 (1.2316) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][230/625] eta 0:02:31 lr 0.001950 wd 0.0500 time 0.3761 (0.3847) data time 0.0011 (0.0032) model time 0.3751 (0.3829) loss 3.7705 (3.6523) grad_norm 1.1029 (1.2286) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][240/625] eta 0:02:28 lr 0.001950 wd 0.0500 time 0.3806 (0.3845) data time 0.0013 (0.0032) model time 0.3793 (0.3827) loss 3.8850 (3.6502) grad_norm 1.2411 (1.2258) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][250/625] eta 0:02:24 lr 0.001950 wd 0.0500 time 0.3748 (0.3841) data time 0.0011 (0.0031) model time 0.3737 (0.3823) loss 4.2433 (3.6350) grad_norm 1.1041 (1.2234) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][260/625] eta 0:02:20 lr 0.001950 wd 0.0500 time 0.3782 (0.3840) data time 0.0011 (0.0030) model time 0.3771 (0.3822) loss 3.8056 (3.6402) grad_norm 0.9395 (1.2317) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][270/625] eta 0:02:16 lr 0.001950 wd 0.0500 time 0.3737 (0.3838) data time 0.0008 (0.0030) model time 0.3729 (0.3819) loss 2.5034 (3.6389) grad_norm 1.1183 (1.2300) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][280/625] eta 0:02:12 lr 0.001950 wd 0.0500 time 0.3712 (0.3836) data time 0.0009 (0.0029) model time 0.3703 (0.3818) loss 3.2890 (3.6414) grad_norm 1.0832 (1.2355) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][290/625] eta 0:02:08 lr 0.001950 wd 0.0500 time 0.3718 (0.3835) data time 0.0010 (0.0028) model time 0.3708 (0.3816) loss 3.5053 (3.6506) grad_norm 0.9711 (1.2297) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][300/625] eta 0:02:04 lr 0.001950 wd 0.0500 time 0.3765 (0.3839) data time 0.0009 (0.0028) model time 0.3757 (0.3822) loss 4.4047 (3.6467) grad_norm 1.3977 (1.2292) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][310/625] eta 0:02:01 lr 0.001950 wd 0.0500 time 0.3758 (0.3851) data time 0.0011 (0.0027) model time 0.3747 (0.3836) loss 3.9871 (3.6378) grad_norm 1.2890 (1.2256) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:11:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][320/625] eta 0:01:57 lr 0.001950 wd 0.0500 time 0.3772 (0.3850) data time 0.0008 (0.0027) model time 0.3764 (0.3835) loss 4.3770 (3.6412) grad_norm 1.3773 (1.2280) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][330/625] eta 0:01:53 lr 0.001950 wd 0.0500 time 0.3819 (0.3848) data time 0.0011 (0.0026) model time 0.3808 (0.3833) loss 2.9445 (3.6301) grad_norm 0.9064 (1.2275) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][340/625] eta 0:01:50 lr 0.001950 wd 0.0500 time 0.3739 (0.3862) data time 0.0012 (0.0026) model time 0.3727 (0.3849) loss 3.2878 (3.6366) grad_norm 2.2538 (1.2312) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][350/625] eta 0:01:46 lr 0.001950 wd 0.0500 time 0.3741 (0.3860) data time 0.0008 (0.0026) model time 0.3732 (0.3847) loss 3.5025 (3.6337) grad_norm 0.7889 (1.2296) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][360/625] eta 0:01:42 lr 0.001950 wd 0.0500 time 0.3805 (0.3858) data time 0.0011 (0.0025) model time 0.3794 (0.3844) loss 3.5803 (3.6288) grad_norm 1.2826 (1.2308) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][370/625] eta 0:01:38 lr 0.001949 wd 0.0500 time 0.3764 (0.3856) data time 0.0012 (0.0025) model time 0.3752 (0.3842) loss 3.5220 (3.6302) grad_norm 1.5557 (1.2527) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][380/625] eta 0:01:34 lr 0.001949 wd 0.0500 time 0.3768 (0.3853) data time 0.0010 (0.0025) model time 0.3758 (0.3839) loss 3.7010 (3.6237) grad_norm 0.9754 (1.2469) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][390/625] eta 0:01:30 lr 0.001949 wd 0.0500 time 0.3787 (0.3851) data time 0.0008 (0.0024) model time 0.3779 (0.3837) loss 3.1587 (3.6172) grad_norm 1.2421 (1.2476) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][400/625] eta 0:01:26 lr 0.001949 wd 0.0500 time 0.3734 (0.3850) data time 0.0010 (0.0024) model time 0.3724 (0.3835) loss 4.4244 (3.6229) grad_norm 1.0715 (1.2463) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][410/625] eta 0:01:22 lr 0.001949 wd 0.0500 time 0.3785 (0.3848) data time 0.0012 (0.0024) model time 0.3773 (0.3834) loss 2.5910 (3.6194) grad_norm 1.0047 (1.2427) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][420/625] eta 0:01:18 lr 0.001949 wd 0.0500 time 0.3781 (0.3847) data time 0.0010 (0.0023) model time 0.3771 (0.3832) loss 3.9307 (3.6152) grad_norm 1.2129 (1.2392) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][430/625] eta 0:01:14 lr 0.001949 wd 0.0500 time 0.3799 (0.3845) data time 0.0008 (0.0023) model time 0.3791 (0.3830) loss 4.5549 (3.6114) grad_norm 0.9536 (1.2425) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][440/625] eta 0:01:11 lr 0.001949 wd 0.0500 time 0.3785 (0.3843) data time 0.0010 (0.0023) model time 0.3775 (0.3828) loss 3.9448 (3.6110) grad_norm 1.1901 (1.2427) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][450/625] eta 0:01:07 lr 0.001949 wd 0.0500 time 0.3756 (0.3841) data time 0.0010 (0.0023) model time 0.3746 (0.3826) loss 3.9801 (3.6150) grad_norm 1.0383 (1.2418) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][460/625] eta 0:01:03 lr 0.001949 wd 0.0500 time 0.3801 (0.3849) data time 0.0008 (0.0022) model time 0.3793 (0.3834) loss 2.7834 (3.6110) grad_norm 1.2825 (1.2427) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:12:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][470/625] eta 0:00:59 lr 0.001949 wd 0.0500 time 0.3818 (0.3848) data time 0.0011 (0.0022) model time 0.3807 (0.3833) loss 4.0315 (3.6049) grad_norm 1.1453 (1.2413) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][480/625] eta 0:00:55 lr 0.001949 wd 0.0500 time 0.3747 (0.3850) data time 0.0008 (0.0022) model time 0.3739 (0.3836) loss 2.9219 (3.6050) grad_norm 0.9905 (1.2429) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][490/625] eta 0:00:52 lr 0.001949 wd 0.0500 time 0.5765 (0.3857) data time 0.0009 (0.0022) model time 0.5756 (0.3844) loss 3.5282 (3.6075) grad_norm 1.3593 (1.2413) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][500/625] eta 0:00:48 lr 0.001949 wd 0.0500 time 0.3750 (0.3855) data time 0.0008 (0.0022) model time 0.3741 (0.3842) loss 4.0995 (3.6045) grad_norm 1.5802 (1.2419) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][510/625] eta 0:00:44 lr 0.001949 wd 0.0500 time 0.3744 (0.3854) data time 0.0010 (0.0021) model time 0.3734 (0.3840) loss 4.3623 (3.6052) grad_norm 1.6059 (1.2433) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][520/625] eta 0:00:40 lr 0.001949 wd 0.0500 time 0.3598 (0.3852) data time 0.0011 (0.0021) model time 0.3587 (0.3838) loss 3.7107 (3.6012) grad_norm 1.1350 (1.2421) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][530/625] eta 0:00:36 lr 0.001949 wd 0.0500 time 0.3717 (0.3849) data time 0.0011 (0.0021) model time 0.3706 (0.3836) loss 3.1946 (3.5965) grad_norm 1.0072 (1.2413) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][540/625] eta 0:00:32 lr 0.001949 wd 0.0500 time 0.3720 (0.3848) data time 0.0011 (0.0021) model time 0.3709 (0.3834) loss 4.2223 (3.5976) grad_norm 1.0587 (1.2384) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][550/625] eta 0:00:28 lr 0.001948 wd 0.0500 time 0.3774 (0.3847) data time 0.0008 (0.0021) model time 0.3766 (0.3832) loss 3.7204 (3.5967) grad_norm 1.1123 (1.2364) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][560/625] eta 0:00:24 lr 0.001948 wd 0.0500 time 0.3811 (0.3845) data time 0.0008 (0.0021) model time 0.3803 (0.3831) loss 4.3678 (3.5931) grad_norm 1.1697 (1.2351) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][570/625] eta 0:00:21 lr 0.001948 wd 0.0500 time 0.3715 (0.3844) data time 0.0008 (0.0020) model time 0.3708 (0.3829) loss 4.1674 (3.5895) grad_norm 0.8789 (1.2336) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][580/625] eta 0:00:17 lr 0.001948 wd 0.0500 time 0.3848 (0.3843) data time 0.0010 (0.0020) model time 0.3838 (0.3828) loss 4.6386 (3.5892) grad_norm 1.1575 (1.2320) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][590/625] eta 0:00:13 lr 0.001948 wd 0.0500 time 0.3797 (0.3841) data time 0.0012 (0.0020) model time 0.3785 (0.3827) loss 3.8288 (3.5891) grad_norm 1.5595 (1.2350) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][600/625] eta 0:00:09 lr 0.001948 wd 0.0500 time 0.3940 (0.3844) data time 0.0011 (0.0020) model time 0.3929 (0.3830) loss 2.4380 (3.5932) grad_norm 1.2424 (1.2341) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][610/625] eta 0:00:05 lr 0.001948 wd 0.0500 time 0.3708 (0.3849) data time 0.0008 (0.0020) model time 0.3700 (0.3836) loss 3.1581 (3.5907) grad_norm 0.9867 (1.2362) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [48/300][620/625] eta 0:00:01 lr 0.001948 wd 0.0500 time 0.3658 (0.3847) data time 0.0006 (0.0020) model time 0.3652 (0.3833) loss 2.9434 (3.5901) grad_norm 0.9527 (1.2377) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:13:55 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 48 training takes 0:04:00 [2024-08-26 06:13:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:13:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.492 (0.492) Loss 0.7354 (0.7354) Acc@1 84.131 (84.131) Acc@5 96.191 (96.191) Mem 11225MB [2024-08-26 06:13:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.2100 (0.8906) Acc@1 71.436 (79.057) Acc@5 90.869 (95.064) Mem 11225MB [2024-08-26 06:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.3721 (1.0812) Acc@1 67.725 (74.681) Acc@5 88.623 (92.657) Mem 11225MB [2024-08-26 06:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.488 Acc@5 92.630 [2024-08-26 06:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 74.5% [2024-08-26 06:14:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.889 (0.889) Loss 0.5786 (0.5786) Acc@1 85.889 (85.889) Acc@5 97.656 (97.656) Mem 11225MB [2024-08-26 06:14:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.182) Loss 1.0244 (0.7427) Acc@1 74.316 (81.472) Acc@5 93.066 (96.156) Mem 11225MB [2024-08-26 06:14:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.141) Loss 1.2334 (0.9301) Acc@1 69.189 (77.121) Acc@5 90.234 (93.855) Mem 11225MB [2024-08-26 06:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.829 Acc@5 93.830 [2024-08-26 06:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 76.8% [2024-08-26 06:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 76.83% [2024-08-26 06:14:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:14:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:14:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][0/625] eta 0:08:01 lr 0.001948 wd 0.0500 time 0.7708 (0.7708) data time 0.3946 (0.3946) model time 0.0000 (0.0000) loss 3.9168 (3.9168) grad_norm 1.4805 (1.4805) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][10/625] eta 0:04:15 lr 0.001948 wd 0.0500 time 0.3842 (0.4149) data time 0.0009 (0.0372) model time 0.0000 (0.0000) loss 3.8184 (3.7901) grad_norm 0.9758 (1.2171) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][20/625] eta 0:04:01 lr 0.001948 wd 0.0500 time 0.3714 (0.3984) data time 0.0010 (0.0200) model time 0.0000 (0.0000) loss 2.9877 (3.7892) grad_norm 1.5106 (1.2117) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][30/625] eta 0:04:00 lr 0.001948 wd 0.0500 time 0.3719 (0.4035) data time 0.0012 (0.0139) model time 0.0000 (0.0000) loss 3.8624 (3.8551) grad_norm 1.6326 (1.2741) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][40/625] eta 0:03:52 lr 0.001948 wd 0.0500 time 0.3819 (0.3975) data time 0.0008 (0.0108) model time 0.0000 (0.0000) loss 3.8234 (3.7830) grad_norm 1.1726 (1.2425) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][50/625] eta 0:03:46 lr 0.001948 wd 0.0500 time 0.3727 (0.3940) data time 0.0008 (0.0089) model time 0.0000 (0.0000) loss 4.5536 (3.7682) grad_norm 1.1359 (1.2469) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][60/625] eta 0:03:40 lr 0.001948 wd 0.0500 time 0.3776 (0.3911) data time 0.0011 (0.0076) model time 0.3765 (0.3754) loss 3.0996 (3.7293) grad_norm 1.0106 (1.2514) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][70/625] eta 0:03:36 lr 0.001948 wd 0.0500 time 0.3842 (0.3896) data time 0.0009 (0.0067) model time 0.3833 (0.3772) loss 2.8365 (3.7283) grad_norm 0.9542 (1.2922) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][80/625] eta 0:03:31 lr 0.001948 wd 0.0500 time 0.3726 (0.3881) data time 0.0009 (0.0060) model time 0.3717 (0.3769) loss 3.6820 (3.7069) grad_norm 1.1407 (1.2851) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][90/625] eta 0:03:27 lr 0.001948 wd 0.0500 time 0.3849 (0.3870) data time 0.0010 (0.0055) model time 0.3838 (0.3769) loss 3.4528 (3.6848) grad_norm 1.1650 (1.2780) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][100/625] eta 0:03:22 lr 0.001947 wd 0.0500 time 0.3809 (0.3863) data time 0.0009 (0.0051) model time 0.3799 (0.3772) loss 4.4082 (3.7033) grad_norm 1.6623 (1.2752) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][110/625] eta 0:03:18 lr 0.001947 wd 0.0500 time 0.3662 (0.3856) data time 0.0012 (0.0047) model time 0.3650 (0.3773) loss 2.9999 (3.6816) grad_norm 1.3931 (1.2744) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][120/625] eta 0:03:15 lr 0.001947 wd 0.0500 time 0.3366 (0.3861) data time 0.0009 (0.0044) model time 0.3358 (0.3792) loss 4.6015 (3.6720) grad_norm 1.0585 (1.2782) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][130/625] eta 0:03:10 lr 0.001947 wd 0.0500 time 0.3849 (0.3855) data time 0.0012 (0.0042) model time 0.3837 (0.3788) loss 2.6510 (3.6324) grad_norm 0.9203 (1.2730) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:14:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][140/625] eta 0:03:06 lr 0.001947 wd 0.0500 time 0.3733 (0.3848) data time 0.0009 (0.0040) model time 0.3724 (0.3784) loss 4.1048 (3.6192) grad_norm 0.8992 (1.2597) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][150/625] eta 0:03:02 lr 0.001947 wd 0.0500 time 0.3717 (0.3843) data time 0.0013 (0.0038) model time 0.3704 (0.3781) loss 2.9919 (3.6084) grad_norm 1.1455 (1.2554) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][160/625] eta 0:02:58 lr 0.001947 wd 0.0500 time 0.3872 (0.3840) data time 0.0008 (0.0036) model time 0.3864 (0.3781) loss 3.5104 (3.6164) grad_norm 1.3292 (1.2610) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][170/625] eta 0:02:55 lr 0.001947 wd 0.0500 time 0.3723 (0.3848) data time 0.0009 (0.0035) model time 0.3714 (0.3798) loss 3.9595 (3.6209) grad_norm 1.4946 (1.2585) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][180/625] eta 0:02:51 lr 0.001947 wd 0.0500 time 0.3769 (0.3853) data time 0.0013 (0.0034) model time 0.3756 (0.3807) loss 3.8621 (3.6286) grad_norm 1.3939 (1.2625) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][190/625] eta 0:02:47 lr 0.001947 wd 0.0500 time 0.3740 (0.3848) data time 0.0013 (0.0032) model time 0.3727 (0.3803) loss 3.6428 (3.6375) grad_norm 1.6144 (1.2662) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][200/625] eta 0:02:43 lr 0.001947 wd 0.0500 time 0.3672 (0.3844) data time 0.0011 (0.0031) model time 0.3661 (0.3799) loss 2.3039 (3.6271) grad_norm 0.8255 (1.2533) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][210/625] eta 0:02:39 lr 0.001947 wd 0.0500 time 0.3798 (0.3841) data time 0.0008 (0.0030) model time 0.3789 (0.3797) loss 3.2397 (3.6263) grad_norm 1.0851 (1.2459) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][220/625] eta 0:02:35 lr 0.001947 wd 0.0500 time 0.3818 (0.3839) data time 0.0010 (0.0030) model time 0.3808 (0.3796) loss 3.5354 (3.6366) grad_norm 0.8774 (1.2390) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][230/625] eta 0:02:31 lr 0.001947 wd 0.0500 time 0.5927 (0.3845) data time 0.0008 (0.0029) model time 0.5919 (0.3806) loss 2.9518 (3.6257) grad_norm 1.2826 (1.2457) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][240/625] eta 0:02:28 lr 0.001947 wd 0.0500 time 0.3924 (0.3861) data time 0.0009 (0.0028) model time 0.3915 (0.3828) loss 3.2379 (3.6204) grad_norm 1.3169 (1.2535) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][250/625] eta 0:02:24 lr 0.001947 wd 0.0500 time 0.3748 (0.3857) data time 0.0011 (0.0027) model time 0.3737 (0.3824) loss 4.1534 (3.6162) grad_norm 1.6628 (1.2530) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][260/625] eta 0:02:20 lr 0.001947 wd 0.0500 time 0.3772 (0.3855) data time 0.0010 (0.0027) model time 0.3762 (0.3822) loss 2.9458 (3.6079) grad_norm 2.8600 (1.2713) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][270/625] eta 0:02:16 lr 0.001947 wd 0.0500 time 0.3781 (0.3852) data time 0.0008 (0.0026) model time 0.3773 (0.3820) loss 3.3638 (3.6114) grad_norm 1.2283 (1.2723) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][280/625] eta 0:02:12 lr 0.001946 wd 0.0500 time 0.3768 (0.3849) data time 0.0008 (0.0026) model time 0.3760 (0.3817) loss 2.8719 (3.6014) grad_norm 1.1874 (1.2713) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][290/625] eta 0:02:09 lr 0.001946 wd 0.0500 time 0.3742 (0.3854) data time 0.0011 (0.0025) model time 0.3731 (0.3824) loss 2.9922 (3.5990) grad_norm 1.3587 (1.2719) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][300/625] eta 0:02:05 lr 0.001946 wd 0.0500 time 0.3827 (0.3858) data time 0.0009 (0.0025) model time 0.3819 (0.3829) loss 3.2756 (3.5982) grad_norm 1.1614 (1.2776) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][310/625] eta 0:02:01 lr 0.001946 wd 0.0500 time 0.3840 (0.3860) data time 0.0009 (0.0024) model time 0.3831 (0.3832) loss 3.2159 (3.5991) grad_norm 1.1456 (1.2775) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][320/625] eta 0:01:57 lr 0.001946 wd 0.0500 time 0.3746 (0.3857) data time 0.0009 (0.0024) model time 0.3737 (0.3830) loss 2.8649 (3.5899) grad_norm 1.2137 (1.2761) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][330/625] eta 0:01:53 lr 0.001946 wd 0.0500 time 0.3716 (0.3854) data time 0.0008 (0.0023) model time 0.3708 (0.3827) loss 2.4250 (3.5824) grad_norm 0.9118 (1.2721) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][340/625] eta 0:01:49 lr 0.001946 wd 0.0500 time 0.3752 (0.3852) data time 0.0009 (0.0023) model time 0.3743 (0.3825) loss 4.4601 (3.5779) grad_norm 0.9153 (1.2674) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][350/625] eta 0:01:45 lr 0.001946 wd 0.0500 time 0.3760 (0.3849) data time 0.0012 (0.0023) model time 0.3748 (0.3822) loss 3.2549 (3.5818) grad_norm 1.0088 (1.2717) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][360/625] eta 0:01:41 lr 0.001946 wd 0.0500 time 0.3819 (0.3847) data time 0.0008 (0.0022) model time 0.3810 (0.3820) loss 2.5550 (3.5709) grad_norm 1.7859 (1.2740) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][370/625] eta 0:01:38 lr 0.001946 wd 0.0500 time 0.3774 (0.3844) data time 0.0012 (0.0022) model time 0.3762 (0.3817) loss 2.3089 (3.5689) grad_norm 1.0770 (1.2717) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][380/625] eta 0:01:34 lr 0.001946 wd 0.0500 time 0.3794 (0.3843) data time 0.0008 (0.0022) model time 0.3786 (0.3816) loss 3.8635 (3.5662) grad_norm 1.3883 (1.2708) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][390/625] eta 0:01:30 lr 0.001946 wd 0.0500 time 0.3823 (0.3842) data time 0.0010 (0.0022) model time 0.3813 (0.3815) loss 3.2652 (3.5729) grad_norm 1.4285 (1.2718) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][400/625] eta 0:01:26 lr 0.001946 wd 0.0500 time 0.3853 (0.3840) data time 0.0008 (0.0022) model time 0.3845 (0.3814) loss 3.4315 (3.5735) grad_norm 1.4322 (1.2738) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][410/625] eta 0:01:22 lr 0.001946 wd 0.0500 time 0.3734 (0.3839) data time 0.0010 (0.0021) model time 0.3724 (0.3812) loss 2.8242 (3.5723) grad_norm 1.6984 (1.2712) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][420/625] eta 0:01:19 lr 0.001946 wd 0.0500 time 0.3718 (0.3855) data time 0.0008 (0.0021) model time 0.3709 (0.3832) loss 2.7478 (3.5630) grad_norm 1.1969 (1.2729) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][430/625] eta 0:01:15 lr 0.001946 wd 0.0500 time 0.3812 (0.3858) data time 0.0011 (0.0021) model time 0.3801 (0.3835) loss 2.8467 (3.5605) grad_norm 0.9579 (1.2685) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][440/625] eta 0:01:11 lr 0.001946 wd 0.0500 time 0.3758 (0.3856) data time 0.0012 (0.0021) model time 0.3746 (0.3833) loss 3.7332 (3.5590) grad_norm 1.1077 (1.2657) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][450/625] eta 0:01:07 lr 0.001945 wd 0.0500 time 0.3925 (0.3854) data time 0.0010 (0.0020) model time 0.3915 (0.3831) loss 3.6537 (3.5648) grad_norm 0.9731 (1.2646) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][460/625] eta 0:01:03 lr 0.001945 wd 0.0500 time 0.3720 (0.3852) data time 0.0009 (0.0020) model time 0.3712 (0.3829) loss 4.4631 (3.5708) grad_norm 1.1356 (1.2652) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][470/625] eta 0:00:59 lr 0.001945 wd 0.0500 time 0.3816 (0.3850) data time 0.0008 (0.0020) model time 0.3807 (0.3828) loss 3.4490 (3.5719) grad_norm 0.8629 (1.2624) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][480/625] eta 0:00:55 lr 0.001945 wd 0.0500 time 0.3837 (0.3849) data time 0.0008 (0.0020) model time 0.3829 (0.3826) loss 3.9545 (3.5733) grad_norm 1.6903 (1.2596) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][490/625] eta 0:00:51 lr 0.001945 wd 0.0500 time 0.3817 (0.3848) data time 0.0008 (0.0020) model time 0.3808 (0.3825) loss 4.4141 (3.5757) grad_norm 1.0756 (1.2565) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][500/625] eta 0:00:48 lr 0.001945 wd 0.0500 time 0.5893 (0.3855) data time 0.0011 (0.0019) model time 0.5882 (0.3833) loss 3.0692 (3.5770) grad_norm 1.0816 (1.2545) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][510/625] eta 0:00:44 lr 0.001945 wd 0.0500 time 0.3721 (0.3853) data time 0.0009 (0.0019) model time 0.3712 (0.3832) loss 3.8184 (3.5783) grad_norm 1.1197 (1.2538) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][520/625] eta 0:00:40 lr 0.001945 wd 0.0500 time 0.3741 (0.3852) data time 0.0011 (0.0019) model time 0.3730 (0.3830) loss 3.2407 (3.5792) grad_norm 2.1332 (1.2553) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][530/625] eta 0:00:36 lr 0.001945 wd 0.0500 time 0.3817 (0.3850) data time 0.0010 (0.0019) model time 0.3807 (0.3829) loss 4.3912 (3.5794) grad_norm 1.5744 (1.2582) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][540/625] eta 0:00:32 lr 0.001945 wd 0.0500 time 0.3851 (0.3860) data time 0.0011 (0.0019) model time 0.3839 (0.3840) loss 3.3422 (3.5813) grad_norm 1.3213 (1.2562) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][550/625] eta 0:00:28 lr 0.001945 wd 0.0500 time 0.3801 (0.3859) data time 0.0008 (0.0019) model time 0.3793 (0.3839) loss 3.3290 (3.5817) grad_norm 1.5169 (1.2581) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][560/625] eta 0:00:25 lr 0.001945 wd 0.0500 time 0.3750 (0.3858) data time 0.0010 (0.0019) model time 0.3740 (0.3838) loss 3.7303 (3.5840) grad_norm 1.3817 (1.2595) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][570/625] eta 0:00:21 lr 0.001945 wd 0.0500 time 0.3769 (0.3856) data time 0.0008 (0.0019) model time 0.3760 (0.3836) loss 3.3006 (3.5815) grad_norm 1.2455 (1.2589) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][580/625] eta 0:00:17 lr 0.001945 wd 0.0500 time 0.3781 (0.3855) data time 0.0008 (0.0018) model time 0.3772 (0.3835) loss 4.4932 (3.5858) grad_norm 1.0683 (1.2620) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][590/625] eta 0:00:13 lr 0.001945 wd 0.0500 time 0.3820 (0.3854) data time 0.0008 (0.0018) model time 0.3812 (0.3834) loss 4.4717 (3.5852) grad_norm 1.5689 (1.2621) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][600/625] eta 0:00:09 lr 0.001945 wd 0.0500 time 0.3744 (0.3853) data time 0.0010 (0.0018) model time 0.3734 (0.3833) loss 2.8278 (3.5819) grad_norm 1.1903 (1.2669) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:17:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][610/625] eta 0:00:05 lr 0.001945 wd 0.0500 time 0.3676 (0.3852) data time 0.0005 (0.0018) model time 0.3670 (0.3832) loss 2.5272 (3.5832) grad_norm 1.0696 (1.2677) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [49/300][620/625] eta 0:00:01 lr 0.001944 wd 0.0500 time 0.3694 (0.3849) data time 0.0008 (0.0018) model time 0.3686 (0.3829) loss 3.8901 (3.5802) grad_norm 1.3769 (1.2665) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:04 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 49 training takes 0:04:00 [2024-08-26 06:18:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:18:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 0.6865 (0.6865) Acc@1 84.961 (84.961) Acc@5 97.070 (97.070) Mem 11225MB [2024-08-26 06:18:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 1.1562 (0.8645) Acc@1 72.998 (79.812) Acc@5 92.285 (95.486) Mem 11225MB [2024-08-26 06:18:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.3447 (1.0636) Acc@1 69.141 (75.312) Acc@5 90.234 (93.001) Mem 11225MB [2024-08-26 06:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.106 Acc@5 92.922 [2024-08-26 06:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.1% [2024-08-26 06:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 75.11% [2024-08-26 06:18:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 06:18:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 06:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.517 (0.517) Loss 0.5776 (0.5776) Acc@1 86.035 (86.035) Acc@5 97.656 (97.656) Mem 11225MB [2024-08-26 06:18:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.146) Loss 1.0195 (0.7404) Acc@1 74.902 (81.610) Acc@5 93.262 (96.214) Mem 11225MB [2024-08-26 06:18:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.2236 (0.9261) Acc@1 69.531 (77.300) Acc@5 90.332 (93.952) Mem 11225MB [2024-08-26 06:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.007 Acc@5 93.924 [2024-08-26 06:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 77.0% [2024-08-26 06:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.01% [2024-08-26 06:18:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:18:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][0/625] eta 0:07:37 lr 0.001944 wd 0.0500 time 0.7323 (0.7323) data time 0.3832 (0.3832) model time 0.0000 (0.0000) loss 2.8897 (2.8897) grad_norm 1.0190 (1.0190) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][10/625] eta 0:04:12 lr 0.001944 wd 0.0500 time 0.3799 (0.4100) data time 0.0011 (0.0360) model time 0.0000 (0.0000) loss 3.8722 (3.7455) grad_norm 0.9863 (1.1260) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][20/625] eta 0:03:59 lr 0.001944 wd 0.0500 time 0.3762 (0.3958) data time 0.0011 (0.0194) model time 0.0000 (0.0000) loss 4.0242 (3.7413) grad_norm 1.0484 (1.1535) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][30/625] eta 0:03:51 lr 0.001944 wd 0.0500 time 0.3769 (0.3898) data time 0.0009 (0.0135) model time 0.0000 (0.0000) loss 3.3333 (3.7367) grad_norm 1.1994 (1.2011) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][40/625] eta 0:03:46 lr 0.001944 wd 0.0500 time 0.3825 (0.3872) data time 0.0010 (0.0105) model time 0.0000 (0.0000) loss 4.1072 (3.6936) grad_norm 2.3173 (1.1995) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][50/625] eta 0:03:41 lr 0.001944 wd 0.0500 time 0.3714 (0.3846) data time 0.0009 (0.0086) model time 0.0000 (0.0000) loss 2.2026 (3.5853) grad_norm 0.9860 (1.2027) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][60/625] eta 0:03:38 lr 0.001944 wd 0.0500 time 0.3739 (0.3875) data time 0.0008 (0.0074) model time 0.3731 (0.4008) loss 3.7224 (3.5912) grad_norm 1.2528 (1.2073) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][70/625] eta 0:03:38 lr 0.001944 wd 0.0500 time 0.3810 (0.3940) data time 0.0012 (0.0065) model time 0.3798 (0.4167) loss 3.8500 (3.6410) grad_norm 0.9646 (1.2226) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][80/625] eta 0:03:33 lr 0.001944 wd 0.0500 time 0.3713 (0.3919) data time 0.0008 (0.0059) model time 0.3705 (0.4032) loss 2.8947 (3.6157) grad_norm 1.1616 (1.2480) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][90/625] eta 0:03:28 lr 0.001944 wd 0.0500 time 0.3758 (0.3904) data time 0.0013 (0.0053) model time 0.3745 (0.3966) loss 4.0189 (3.6316) grad_norm 1.2943 (1.2371) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][100/625] eta 0:03:24 lr 0.001944 wd 0.0500 time 0.3873 (0.3892) data time 0.0008 (0.0049) model time 0.3864 (0.3927) loss 4.0716 (3.6622) grad_norm 1.3798 (1.2269) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:18:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][110/625] eta 0:03:19 lr 0.001944 wd 0.0500 time 0.3782 (0.3882) data time 0.0010 (0.0046) model time 0.3771 (0.3902) loss 3.6529 (3.6302) grad_norm 1.5870 (1.2103) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:19:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][120/625] eta 0:03:16 lr 0.001944 wd 0.0500 time 0.3831 (0.3889) data time 0.0010 (0.0043) model time 0.3821 (0.3910) loss 3.8452 (3.6237) grad_norm 1.0448 (1.2200) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:19:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][130/625] eta 0:03:12 lr 0.001944 wd 0.0500 time 0.3766 (0.3879) data time 0.0010 (0.0041) model time 0.3756 (0.3889) loss 3.5407 (3.6256) grad_norm 1.4526 (1.2239) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:19:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][140/625] eta 0:03:07 lr 0.001944 wd 0.0500 time 0.3781 (0.3870) data time 0.0008 (0.0038) model time 0.3772 (0.3873) loss 3.4648 (3.6115) grad_norm 1.1770 (1.2293) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:19:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][150/625] eta 0:03:03 lr 0.001944 wd 0.0500 time 0.3790 (0.3866) data time 0.0009 (0.0037) model time 0.3782 (0.3865) loss 3.6333 (3.6131) grad_norm 1.2504 (1.2401) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:19:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][160/625] eta 0:02:59 lr 0.001944 wd 0.0500 time 0.3958 (0.3862) data time 0.0008 (0.0035) model time 0.3950 (0.3859) loss 2.8968 (3.5919) grad_norm 0.9662 (1.2393) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:19:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][170/625] eta 0:02:55 lr 0.001943 wd 0.0500 time 0.3613 (0.3857) data time 0.0009 (0.0034) model time 0.3604 (0.3850) loss 4.0759 (3.6027) grad_norm 1.2892 (1.2448) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:19:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][180/625] eta 0:02:51 lr 0.001943 wd 0.0500 time 0.3887 (0.3854) data time 0.0008 (0.0032) model time 0.3879 (0.3845) loss 4.5966 (3.6182) grad_norm 1.9895 (1.2600) loss_scale 16384.0000 (8644.5967) mem 11225MB [2024-08-26 06:19:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][190/625] eta 0:02:47 lr 0.001943 wd 0.0500 time 0.3825 (0.3851) data time 0.0008 (0.0031) model time 0.3817 (0.3841) loss 3.4375 (3.6160) grad_norm 1.5736 (1.2641) loss_scale 16384.0000 (9049.8010) mem 11225MB [2024-08-26 06:19:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][200/625] eta 0:02:43 lr 0.001943 wd 0.0500 time 0.3737 (0.3848) data time 0.0009 (0.0030) model time 0.3728 (0.3837) loss 4.3031 (3.6205) grad_norm 1.2409 (1.2594) loss_scale 16384.0000 (9414.6866) mem 11225MB [2024-08-26 06:19:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][210/625] eta 0:02:39 lr 0.001943 wd 0.0500 time 0.3802 (0.3845) data time 0.0011 (0.0029) model time 0.3791 (0.3833) loss 3.7267 (3.6263) grad_norm 3.0028 (1.2633) loss_scale 16384.0000 (9744.9858) mem 11225MB [2024-08-26 06:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][220/625] eta 0:02:35 lr 0.001943 wd 0.0500 time 0.3716 (0.3841) data time 0.0009 (0.0029) model time 0.3708 (0.3828) loss 2.8259 (3.6154) grad_norm 1.7581 (1.2736) loss_scale 16384.0000 (10045.3937) mem 11225MB [2024-08-26 06:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][230/625] eta 0:02:31 lr 0.001943 wd 0.0500 time 0.3786 (0.3838) data time 0.0008 (0.0028) model time 0.3778 (0.3824) loss 3.4741 (3.6054) grad_norm 1.0567 (1.2715) loss_scale 16384.0000 (10319.7922) mem 11225MB [2024-08-26 06:19:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][240/625] eta 0:02:27 lr 0.001943 wd 0.0500 time 0.3808 (0.3843) data time 0.0011 (0.0027) model time 0.3798 (0.3831) loss 3.8965 (3.5995) grad_norm 1.2546 (1.2648) loss_scale 16384.0000 (10571.4191) mem 11225MB [2024-08-26 06:19:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][250/625] eta 0:02:24 lr 0.001943 wd 0.0500 time 0.3773 (0.3857) data time 0.0008 (0.0026) model time 0.3765 (0.3848) loss 3.4501 (3.5962) grad_norm 1.0775 (1.2669) loss_scale 16384.0000 (10802.9960) mem 11225MB [2024-08-26 06:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][260/625] eta 0:02:20 lr 0.001943 wd 0.0500 time 0.3758 (0.3863) data time 0.0011 (0.0026) model time 0.3748 (0.3856) loss 4.0286 (3.5998) grad_norm 1.5218 (1.2640) loss_scale 16384.0000 (11016.8276) mem 11225MB [2024-08-26 06:19:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][270/625] eta 0:02:17 lr 0.001943 wd 0.0500 time 0.3850 (0.3861) data time 0.0009 (0.0025) model time 0.3841 (0.3853) loss 2.6622 (3.6023) grad_norm 0.8472 (1.2616) loss_scale 16384.0000 (11214.8782) mem 11225MB [2024-08-26 06:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][280/625] eta 0:02:14 lr 0.001943 wd 0.0500 time 0.3771 (0.3889) data time 0.0011 (0.0025) model time 0.3761 (0.3887) loss 3.8786 (3.6054) grad_norm 1.9657 (1.2624) loss_scale 16384.0000 (11398.8327) mem 11225MB [2024-08-26 06:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][290/625] eta 0:02:10 lr 0.001943 wd 0.0500 time 0.3790 (0.3887) data time 0.0012 (0.0024) model time 0.3778 (0.3884) loss 3.4283 (3.5938) grad_norm 1.2246 (1.2724) loss_scale 16384.0000 (11570.1443) mem 11225MB [2024-08-26 06:20:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][300/625] eta 0:02:06 lr 0.001943 wd 0.0500 time 0.3837 (0.3884) data time 0.0011 (0.0024) model time 0.3826 (0.3880) loss 3.7351 (3.5964) grad_norm 0.7924 (1.2687) loss_scale 16384.0000 (11730.0731) mem 11225MB [2024-08-26 06:20:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][310/625] eta 0:02:02 lr 0.001943 wd 0.0500 time 0.3724 (0.3881) data time 0.0012 (0.0024) model time 0.3711 (0.3876) loss 3.1454 (3.5924) grad_norm 0.8874 (1.2631) loss_scale 16384.0000 (11879.7170) mem 11225MB [2024-08-26 06:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][320/625] eta 0:01:58 lr 0.001943 wd 0.0500 time 0.3751 (0.3877) data time 0.0011 (0.0023) model time 0.3739 (0.3872) loss 3.6405 (3.5877) grad_norm 1.3023 (1.2648) loss_scale 16384.0000 (12020.0374) mem 11225MB [2024-08-26 06:20:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][330/625] eta 0:01:54 lr 0.001942 wd 0.0500 time 0.3787 (0.3874) data time 0.0011 (0.0023) model time 0.3776 (0.3868) loss 4.1939 (3.5924) grad_norm 1.3222 (1.2722) loss_scale 16384.0000 (12151.8792) mem 11225MB [2024-08-26 06:20:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][340/625] eta 0:01:50 lr 0.001942 wd 0.0500 time 0.3758 (0.3872) data time 0.0012 (0.0023) model time 0.3746 (0.3865) loss 4.0898 (3.5896) grad_norm 1.5376 (1.2692) loss_scale 16384.0000 (12275.9883) mem 11225MB [2024-08-26 06:20:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][350/625] eta 0:01:46 lr 0.001942 wd 0.0500 time 0.3779 (0.3869) data time 0.0008 (0.0022) model time 0.3770 (0.3861) loss 2.8924 (3.5867) grad_norm 1.0516 (1.2655) loss_scale 16384.0000 (12393.0256) mem 11225MB [2024-08-26 06:20:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][360/625] eta 0:01:42 lr 0.001942 wd 0.0500 time 0.3882 (0.3867) data time 0.0013 (0.0022) model time 0.3869 (0.3859) loss 3.5685 (3.5859) grad_norm 1.1881 (1.2638) loss_scale 16384.0000 (12503.5789) mem 11225MB [2024-08-26 06:20:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][370/625] eta 0:01:38 lr 0.001942 wd 0.0500 time 0.5795 (0.3882) data time 0.0008 (0.0022) model time 0.5787 (0.3876) loss 4.5061 (3.5954) grad_norm 1.2110 (1.2606) loss_scale 16384.0000 (12608.1725) mem 11225MB [2024-08-26 06:20:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][380/625] eta 0:01:35 lr 0.001942 wd 0.0500 time 0.3900 (0.3879) data time 0.0008 (0.0022) model time 0.3891 (0.3873) loss 4.2903 (3.6007) grad_norm 0.9399 (1.2582) loss_scale 16384.0000 (12707.2756) mem 11225MB [2024-08-26 06:20:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][390/625] eta 0:01:31 lr 0.001942 wd 0.0500 time 0.3796 (0.3877) data time 0.0008 (0.0022) model time 0.3788 (0.3870) loss 4.1737 (3.5959) grad_norm 1.2857 (1.2630) loss_scale 16384.0000 (12801.3095) mem 11225MB [2024-08-26 06:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][400/625] eta 0:01:27 lr 0.001942 wd 0.0500 time 0.3825 (0.3875) data time 0.0013 (0.0021) model time 0.3812 (0.3867) loss 3.9603 (3.5934) grad_norm 1.0372 (1.2661) loss_scale 16384.0000 (12890.6534) mem 11225MB [2024-08-26 06:20:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][410/625] eta 0:01:23 lr 0.001942 wd 0.0500 time 0.3867 (0.3873) data time 0.0009 (0.0021) model time 0.3859 (0.3864) loss 2.8514 (3.5869) grad_norm 1.5586 (1.2677) loss_scale 16384.0000 (12975.6496) mem 11225MB [2024-08-26 06:20:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][420/625] eta 0:01:19 lr 0.001942 wd 0.0500 time 0.3791 (0.3870) data time 0.0008 (0.0021) model time 0.3783 (0.3862) loss 3.0712 (3.5937) grad_norm 1.0283 (1.2652) loss_scale 16384.0000 (13056.6081) mem 11225MB [2024-08-26 06:21:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][430/625] eta 0:01:15 lr 0.001942 wd 0.0500 time 0.3700 (0.3869) data time 0.0011 (0.0021) model time 0.3688 (0.3860) loss 3.3769 (3.5959) grad_norm 0.9195 (1.2626) loss_scale 16384.0000 (13133.8097) mem 11225MB [2024-08-26 06:21:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][440/625] eta 0:01:11 lr 0.001942 wd 0.0500 time 0.4514 (0.3869) data time 0.0008 (0.0021) model time 0.4506 (0.3860) loss 4.4310 (3.5970) grad_norm 1.1104 (1.2613) loss_scale 16384.0000 (13207.5102) mem 11225MB [2024-08-26 06:21:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][450/625] eta 0:01:07 lr 0.001942 wd 0.0500 time 0.3781 (0.3871) data time 0.0010 (0.0020) model time 0.3771 (0.3863) loss 3.4831 (3.6011) grad_norm 2.3669 (1.2636) loss_scale 16384.0000 (13277.9424) mem 11225MB [2024-08-26 06:21:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][460/625] eta 0:01:03 lr 0.001942 wd 0.0500 time 0.5899 (0.3878) data time 0.0013 (0.0020) model time 0.5887 (0.3870) loss 3.4199 (3.6077) grad_norm 1.2109 (1.2617) loss_scale 16384.0000 (13345.3189) mem 11225MB [2024-08-26 06:21:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][470/625] eta 0:01:00 lr 0.001942 wd 0.0500 time 0.3750 (0.3880) data time 0.0011 (0.0020) model time 0.3739 (0.3872) loss 3.9107 (3.6151) grad_norm 0.9999 (1.2621) loss_scale 16384.0000 (13409.8344) mem 11225MB [2024-08-26 06:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][480/625] eta 0:00:56 lr 0.001942 wd 0.0500 time 0.3717 (0.3879) data time 0.0011 (0.0020) model time 0.3706 (0.3870) loss 3.6203 (3.6203) grad_norm 1.0440 (1.2651) loss_scale 16384.0000 (13471.6674) mem 11225MB [2024-08-26 06:21:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][490/625] eta 0:00:52 lr 0.001942 wd 0.0500 time 0.3748 (0.3877) data time 0.0008 (0.0020) model time 0.3740 (0.3868) loss 2.9083 (3.6119) grad_norm 1.0162 (1.2649) loss_scale 16384.0000 (13530.9817) mem 11225MB [2024-08-26 06:21:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][500/625] eta 0:00:48 lr 0.001941 wd 0.0500 time 0.3743 (0.3875) data time 0.0008 (0.0020) model time 0.3735 (0.3866) loss 3.7237 (3.6087) grad_norm 1.0727 (1.2607) loss_scale 16384.0000 (13587.9281) mem 11225MB [2024-08-26 06:21:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][510/625] eta 0:00:44 lr 0.001941 wd 0.0500 time 0.3746 (0.3872) data time 0.0009 (0.0020) model time 0.3738 (0.3863) loss 4.0615 (3.6120) grad_norm 1.4248 (1.2633) loss_scale 16384.0000 (13642.6458) mem 11225MB [2024-08-26 06:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][520/625] eta 0:00:40 lr 0.001941 wd 0.0500 time 0.3790 (0.3871) data time 0.0011 (0.0019) model time 0.3779 (0.3861) loss 3.4306 (3.6072) grad_norm 1.3202 (1.2618) loss_scale 16384.0000 (13695.2630) mem 11225MB [2024-08-26 06:21:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][530/625] eta 0:00:36 lr 0.001941 wd 0.0500 time 0.3829 (0.3869) data time 0.0008 (0.0019) model time 0.3821 (0.3859) loss 4.4432 (3.6151) grad_norm 1.0783 (1.2623) loss_scale 16384.0000 (13745.8983) mem 11225MB [2024-08-26 06:21:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][540/625] eta 0:00:32 lr 0.001941 wd 0.0500 time 0.3736 (0.3867) data time 0.0008 (0.0019) model time 0.3727 (0.3858) loss 4.1015 (3.6181) grad_norm 1.3422 (1.2610) loss_scale 16384.0000 (13794.6617) mem 11225MB [2024-08-26 06:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][550/625] eta 0:00:28 lr 0.001941 wd 0.0500 time 0.3798 (0.3866) data time 0.0011 (0.0019) model time 0.3787 (0.3856) loss 3.6199 (3.6191) grad_norm 1.0572 (1.2623) loss_scale 16384.0000 (13841.6552) mem 11225MB [2024-08-26 06:21:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][560/625] eta 0:00:25 lr 0.001941 wd 0.0500 time 0.3948 (0.3865) data time 0.0011 (0.0019) model time 0.3938 (0.3855) loss 3.4348 (3.6204) grad_norm 1.6541 (1.2641) loss_scale 16384.0000 (13886.9733) mem 11225MB [2024-08-26 06:21:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][570/625] eta 0:00:21 lr 0.001941 wd 0.0500 time 0.5873 (0.3867) data time 0.0011 (0.0019) model time 0.5862 (0.3857) loss 3.7137 (3.6189) grad_norm 0.9479 (1.2609) loss_scale 16384.0000 (13930.7040) mem 11225MB [2024-08-26 06:21:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][580/625] eta 0:00:17 lr 0.001941 wd 0.0500 time 0.3728 (0.3876) data time 0.0011 (0.0019) model time 0.3717 (0.3866) loss 3.5915 (3.6205) grad_norm 1.2216 (1.2584) loss_scale 16384.0000 (13972.9294) mem 11225MB [2024-08-26 06:22:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][590/625] eta 0:00:13 lr 0.001941 wd 0.0500 time 0.3738 (0.3877) data time 0.0009 (0.0019) model time 0.3729 (0.3868) loss 3.6573 (3.6235) grad_norm 1.8055 (1.2632) loss_scale 16384.0000 (14013.7259) mem 11225MB [2024-08-26 06:22:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][600/625] eta 0:00:09 lr 0.001941 wd 0.0500 time 0.3812 (0.3876) data time 0.0011 (0.0018) model time 0.3802 (0.3866) loss 3.8888 (3.6248) grad_norm 1.0332 (1.2640) loss_scale 16384.0000 (14053.1647) mem 11225MB [2024-08-26 06:22:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][610/625] eta 0:00:05 lr 0.001941 wd 0.0500 time 0.3701 (0.3874) data time 0.0008 (0.0018) model time 0.3693 (0.3864) loss 3.7538 (3.6219) grad_norm 1.4464 (1.2626) loss_scale 16384.0000 (14091.3126) mem 11225MB [2024-08-26 06:22:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [50/300][620/625] eta 0:00:01 lr 0.001941 wd 0.0500 time 0.3720 (0.3871) data time 0.0008 (0.0018) model time 0.3712 (0.3861) loss 3.7511 (3.6192) grad_norm 1.8282 (1.2629) loss_scale 16384.0000 (14128.2319) mem 11225MB [2024-08-26 06:22:16 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 50 training takes 0:04:01 [2024-08-26 06:22:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:22:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.7065 (0.7065) Acc@1 84.375 (84.375) Acc@5 96.631 (96.631) Mem 11225MB [2024-08-26 06:22:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.1689 (0.8660) Acc@1 72.900 (79.785) Acc@5 92.041 (95.597) Mem 11225MB [2024-08-26 06:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.3613 (1.0648) Acc@1 67.139 (75.456) Acc@5 89.893 (93.059) Mem 11225MB [2024-08-26 06:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.152 Acc@5 93.038 [2024-08-26 06:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.2% [2024-08-26 06:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 75.15% [2024-08-26 06:22:20 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 06:22:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 06:22:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.491 (0.491) Loss 0.5752 (0.5752) Acc@1 86.133 (86.133) Acc@5 97.559 (97.559) Mem 11225MB [2024-08-26 06:22:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.0127 (0.7376) Acc@1 75.000 (81.765) Acc@5 93.262 (96.236) Mem 11225MB [2024-08-26 06:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 1.2168 (0.9218) Acc@1 69.775 (77.493) Acc@5 90.625 (94.015) Mem 11225MB [2024-08-26 06:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.187 Acc@5 94.002 [2024-08-26 06:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 77.2% [2024-08-26 06:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.19% [2024-08-26 06:22:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:22:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:22:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][0/625] eta 0:08:15 lr 0.001941 wd 0.0500 time 0.7931 (0.7931) data time 0.4583 (0.4583) model time 0.0000 (0.0000) loss 4.4397 (4.4397) grad_norm 1.3973 (1.3973) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:22:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][10/625] eta 0:04:14 lr 0.001941 wd 0.0500 time 0.3783 (0.4146) data time 0.0008 (0.0427) model time 0.0000 (0.0000) loss 4.3090 (3.8895) grad_norm 1.2146 (1.3057) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:22:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][20/625] eta 0:03:59 lr 0.001941 wd 0.0500 time 0.3744 (0.3966) data time 0.0011 (0.0229) model time 0.0000 (0.0000) loss 3.2594 (3.8174) grad_norm 1.1883 (1.2751) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:22:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][30/625] eta 0:03:51 lr 0.001941 wd 0.0500 time 0.3781 (0.3898) data time 0.0010 (0.0161) model time 0.0000 (0.0000) loss 3.7195 (3.7567) grad_norm 1.3251 (1.2814) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:22:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][40/625] eta 0:03:46 lr 0.001940 wd 0.0500 time 0.3711 (0.3867) data time 0.0012 (0.0124) model time 0.0000 (0.0000) loss 3.6480 (3.7744) grad_norm 1.2519 (1.2826) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:22:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][50/625] eta 0:03:41 lr 0.001940 wd 0.0500 time 0.3768 (0.3847) data time 0.0009 (0.0102) model time 0.0000 (0.0000) loss 3.8758 (3.7065) grad_norm 1.1259 (1.2470) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:22:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][60/625] eta 0:03:36 lr 0.001940 wd 0.0500 time 0.3792 (0.3836) data time 0.0008 (0.0087) model time 0.3783 (0.3771) loss 4.2280 (3.6849) grad_norm 1.7048 (1.2584) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:22:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][70/625] eta 0:03:32 lr 0.001940 wd 0.0500 time 0.3791 (0.3829) data time 0.0008 (0.0076) model time 0.3783 (0.3774) loss 4.2301 (3.6668) grad_norm 1.2495 (1.2452) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:22:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][80/625] eta 0:03:29 lr 0.001940 wd 0.0500 time 0.3710 (0.3845) data time 0.0009 (0.0068) model time 0.3701 (0.3831) loss 4.3029 (3.6898) grad_norm 1.4760 (1.2396) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:23:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][90/625] eta 0:03:25 lr 0.001940 wd 0.0500 time 0.3774 (0.3837) data time 0.0008 (0.0062) model time 0.3765 (0.3813) loss 3.0955 (3.7086) grad_norm 1.5170 (1.2577) loss_scale 16384.0000 (16384.0000) mem 11225MB [2024-08-26 06:23:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][100/625] eta 0:03:23 lr 0.001940 wd 0.0500 time 0.3726 (0.3885) data time 0.0011 (0.0057) model time 0.3715 (0.3912) loss 3.4335 (3.7021) grad_norm 1.1906 (inf) loss_scale 8192.0000 (16221.7822) mem 11225MB [2024-08-26 06:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][110/625] eta 0:03:19 lr 0.001940 wd 0.0500 time 0.3815 (0.3874) data time 0.0011 (0.0053) model time 0.3804 (0.3887) loss 2.9256 (3.6761) grad_norm 1.1063 (inf) loss_scale 8192.0000 (15498.3784) mem 11225MB [2024-08-26 06:23:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][120/625] eta 0:03:15 lr 0.001940 wd 0.0500 time 0.3792 (0.3866) data time 0.0009 (0.0049) model time 0.3783 (0.3869) loss 3.7180 (3.6659) grad_norm 1.9243 (inf) loss_scale 8192.0000 (14894.5455) mem 11225MB [2024-08-26 06:23:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][130/625] eta 0:03:11 lr 0.001940 wd 0.0500 time 0.3797 (0.3859) data time 0.0011 (0.0047) model time 0.3786 (0.3856) loss 3.5615 (3.6710) grad_norm 1.3537 (inf) loss_scale 8192.0000 (14382.9008) mem 11225MB [2024-08-26 06:23:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][140/625] eta 0:03:06 lr 0.001940 wd 0.0500 time 0.3891 (0.3855) data time 0.0009 (0.0044) model time 0.3882 (0.3849) loss 4.0536 (3.6346) grad_norm 0.9523 (inf) loss_scale 8192.0000 (13943.8298) mem 11225MB [2024-08-26 06:23:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][150/625] eta 0:03:04 lr 0.001940 wd 0.0500 time 0.3770 (0.3876) data time 0.0010 (0.0042) model time 0.3760 (0.3880) loss 3.6054 (3.6394) grad_norm 1.3023 (inf) loss_scale 8192.0000 (13562.9139) mem 11225MB [2024-08-26 06:23:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][160/625] eta 0:03:00 lr 0.001940 wd 0.0500 time 0.3779 (0.3880) data time 0.0011 (0.0040) model time 0.3768 (0.3885) loss 2.7933 (3.6175) grad_norm 1.2873 (inf) loss_scale 8192.0000 (13229.3168) mem 11225MB [2024-08-26 06:23:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][170/625] eta 0:02:56 lr 0.001940 wd 0.0500 time 0.3781 (0.3874) data time 0.0011 (0.0038) model time 0.3771 (0.3874) loss 3.4548 (3.6133) grad_norm 1.2670 (inf) loss_scale 8192.0000 (12934.7368) mem 11225MB [2024-08-26 06:23:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][180/625] eta 0:02:52 lr 0.001940 wd 0.0500 time 0.3738 (0.3869) data time 0.0008 (0.0037) model time 0.3729 (0.3866) loss 3.7974 (3.6146) grad_norm 1.5936 (inf) loss_scale 8192.0000 (12672.7072) mem 11225MB [2024-08-26 06:23:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][190/625] eta 0:02:48 lr 0.001940 wd 0.0500 time 0.3749 (0.3866) data time 0.0010 (0.0036) model time 0.3739 (0.3861) loss 4.0213 (3.6153) grad_norm 1.3696 (inf) loss_scale 8192.0000 (12438.1152) mem 11225MB [2024-08-26 06:23:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][200/625] eta 0:02:44 lr 0.001939 wd 0.0500 time 0.3773 (0.3861) data time 0.0012 (0.0034) model time 0.3761 (0.3854) loss 3.7981 (3.6310) grad_norm 1.2131 (inf) loss_scale 8192.0000 (12226.8657) mem 11225MB [2024-08-26 06:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][210/625] eta 0:02:40 lr 0.001939 wd 0.0500 time 0.3863 (0.3858) data time 0.0008 (0.0033) model time 0.3854 (0.3850) loss 4.6441 (3.6429) grad_norm 0.9365 (inf) loss_scale 8192.0000 (12035.6398) mem 11225MB [2024-08-26 06:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][220/625] eta 0:02:36 lr 0.001939 wd 0.0500 time 0.3770 (0.3855) data time 0.0011 (0.0032) model time 0.3759 (0.3846) loss 3.9517 (3.6432) grad_norm 1.8139 (inf) loss_scale 8192.0000 (11861.7195) mem 11225MB [2024-08-26 06:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][230/625] eta 0:02:32 lr 0.001939 wd 0.0500 time 0.3748 (0.3851) data time 0.0008 (0.0032) model time 0.3739 (0.3841) loss 2.9666 (3.6451) grad_norm 1.0654 (inf) loss_scale 8192.0000 (11702.8571) mem 11225MB [2024-08-26 06:23:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][240/625] eta 0:02:28 lr 0.001939 wd 0.0500 time 0.3765 (0.3850) data time 0.0008 (0.0031) model time 0.3757 (0.3839) loss 2.5358 (3.6371) grad_norm 1.8234 (inf) loss_scale 8192.0000 (11557.1784) mem 11225MB [2024-08-26 06:24:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][250/625] eta 0:02:24 lr 0.001939 wd 0.0500 time 0.3748 (0.3847) data time 0.0010 (0.0030) model time 0.3738 (0.3835) loss 3.3543 (3.6323) grad_norm 0.8694 (inf) loss_scale 8192.0000 (11423.1076) mem 11225MB [2024-08-26 06:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][260/625] eta 0:02:20 lr 0.001939 wd 0.0500 time 0.3841 (0.3845) data time 0.0008 (0.0029) model time 0.3832 (0.3832) loss 2.5104 (3.6078) grad_norm 1.5558 (inf) loss_scale 8192.0000 (11299.3103) mem 11225MB [2024-08-26 06:24:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][270/625] eta 0:02:16 lr 0.001939 wd 0.0500 time 0.3818 (0.3842) data time 0.0009 (0.0029) model time 0.3809 (0.3829) loss 3.7261 (3.5957) grad_norm 1.2477 (inf) loss_scale 8192.0000 (11184.6494) mem 11225MB [2024-08-26 06:24:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][280/625] eta 0:02:12 lr 0.001939 wd 0.0500 time 0.3774 (0.3840) data time 0.0011 (0.0028) model time 0.3763 (0.3827) loss 3.9149 (3.5887) grad_norm 1.0038 (inf) loss_scale 8192.0000 (11078.1495) mem 11225MB [2024-08-26 06:24:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][290/625] eta 0:02:08 lr 0.001939 wd 0.0500 time 0.3380 (0.3845) data time 0.0011 (0.0028) model time 0.3369 (0.3833) loss 3.1577 (3.5902) grad_norm 1.1441 (inf) loss_scale 8192.0000 (10978.9691) mem 11225MB [2024-08-26 06:24:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][300/625] eta 0:02:04 lr 0.001939 wd 0.0500 time 0.3850 (0.3843) data time 0.0009 (0.0027) model time 0.3842 (0.3830) loss 3.7016 (3.5874) grad_norm 1.3541 (inf) loss_scale 8192.0000 (10886.3787) mem 11225MB [2024-08-26 06:24:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][310/625] eta 0:02:01 lr 0.001939 wd 0.0500 time 0.3746 (0.3862) data time 0.0011 (0.0027) model time 0.3735 (0.3853) loss 3.4396 (3.5873) grad_norm 0.9145 (inf) loss_scale 8192.0000 (10799.7428) mem 11225MB [2024-08-26 06:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][320/625] eta 0:01:57 lr 0.001939 wd 0.0500 time 0.3787 (0.3860) data time 0.0011 (0.0026) model time 0.3777 (0.3850) loss 3.9775 (3.5846) grad_norm 1.0017 (inf) loss_scale 8192.0000 (10718.5047) mem 11225MB [2024-08-26 06:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][330/625] eta 0:01:53 lr 0.001939 wd 0.0500 time 0.3764 (0.3857) data time 0.0011 (0.0026) model time 0.3753 (0.3847) loss 3.5027 (3.5867) grad_norm 1.2625 (inf) loss_scale 8192.0000 (10642.1752) mem 11225MB [2024-08-26 06:24:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][340/625] eta 0:01:50 lr 0.001939 wd 0.0500 time 0.3778 (0.3866) data time 0.0009 (0.0025) model time 0.3769 (0.3858) loss 2.7695 (3.5943) grad_norm 1.9786 (inf) loss_scale 8192.0000 (10570.3226) mem 11225MB [2024-08-26 06:24:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][350/625] eta 0:01:46 lr 0.001939 wd 0.0500 time 0.3723 (0.3869) data time 0.0008 (0.0025) model time 0.3715 (0.3861) loss 3.8168 (3.5954) grad_norm 1.0784 (inf) loss_scale 8192.0000 (10502.5641) mem 11225MB [2024-08-26 06:24:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][360/625] eta 0:01:42 lr 0.001939 wd 0.0500 time 0.3734 (0.3866) data time 0.0008 (0.0025) model time 0.3726 (0.3858) loss 4.0621 (3.5998) grad_norm 1.4327 (inf) loss_scale 8192.0000 (10438.5596) mem 11225MB [2024-08-26 06:24:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][370/625] eta 0:01:38 lr 0.001938 wd 0.0500 time 0.3765 (0.3864) data time 0.0008 (0.0024) model time 0.3757 (0.3855) loss 3.7178 (3.5971) grad_norm 1.5799 (inf) loss_scale 8192.0000 (10378.0054) mem 11225MB [2024-08-26 06:24:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][380/625] eta 0:01:34 lr 0.001938 wd 0.0500 time 0.3728 (0.3863) data time 0.0009 (0.0024) model time 0.3719 (0.3853) loss 2.0145 (3.6005) grad_norm 0.9198 (inf) loss_scale 8192.0000 (10320.6299) mem 11225MB [2024-08-26 06:24:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][390/625] eta 0:01:30 lr 0.001938 wd 0.0500 time 0.3770 (0.3860) data time 0.0009 (0.0024) model time 0.3761 (0.3850) loss 3.6805 (3.5995) grad_norm 0.9173 (inf) loss_scale 8192.0000 (10266.1893) mem 11225MB [2024-08-26 06:25:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][400/625] eta 0:01:26 lr 0.001938 wd 0.0500 time 0.3742 (0.3858) data time 0.0008 (0.0023) model time 0.3734 (0.3848) loss 3.1156 (3.5899) grad_norm 0.9728 (inf) loss_scale 8192.0000 (10214.4638) mem 11225MB [2024-08-26 06:25:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][410/625] eta 0:01:22 lr 0.001938 wd 0.0500 time 0.3769 (0.3857) data time 0.0011 (0.0023) model time 0.3758 (0.3847) loss 3.8676 (3.5925) grad_norm 1.2367 (inf) loss_scale 8192.0000 (10165.2555) mem 11225MB [2024-08-26 06:25:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][420/625] eta 0:01:19 lr 0.001938 wd 0.0500 time 0.3824 (0.3856) data time 0.0011 (0.0023) model time 0.3812 (0.3846) loss 2.6344 (3.5953) grad_norm 0.8900 (inf) loss_scale 8192.0000 (10118.3848) mem 11225MB [2024-08-26 06:25:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][430/625] eta 0:01:15 lr 0.001938 wd 0.0500 time 0.3812 (0.3854) data time 0.0008 (0.0023) model time 0.3804 (0.3843) loss 4.5069 (3.5995) grad_norm 1.1728 (inf) loss_scale 8192.0000 (10073.6891) mem 11225MB [2024-08-26 06:25:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][440/625] eta 0:01:11 lr 0.001938 wd 0.0500 time 0.3771 (0.3853) data time 0.0011 (0.0022) model time 0.3761 (0.3841) loss 3.3960 (3.5993) grad_norm 1.4741 (inf) loss_scale 8192.0000 (10031.0204) mem 11225MB [2024-08-26 06:25:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][450/625] eta 0:01:07 lr 0.001938 wd 0.0500 time 0.3739 (0.3851) data time 0.0011 (0.0022) model time 0.3727 (0.3840) loss 3.6420 (3.5886) grad_norm 1.4468 (inf) loss_scale 8192.0000 (9990.2439) mem 11225MB [2024-08-26 06:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][460/625] eta 0:01:03 lr 0.001938 wd 0.0500 time 0.3705 (0.3858) data time 0.0011 (0.0022) model time 0.3694 (0.3847) loss 3.3591 (3.5881) grad_norm 1.1116 (inf) loss_scale 8192.0000 (9951.2364) mem 11225MB [2024-08-26 06:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][470/625] eta 0:00:59 lr 0.001938 wd 0.0500 time 0.3754 (0.3860) data time 0.0008 (0.0022) model time 0.3745 (0.3850) loss 3.9734 (3.5902) grad_norm 1.8706 (inf) loss_scale 8192.0000 (9913.8854) mem 11225MB [2024-08-26 06:25:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][480/625] eta 0:00:55 lr 0.001938 wd 0.0500 time 0.3770 (0.3861) data time 0.0011 (0.0021) model time 0.3759 (0.3851) loss 3.5568 (3.5922) grad_norm 1.4160 (inf) loss_scale 8192.0000 (9878.0873) mem 11225MB [2024-08-26 06:25:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][490/625] eta 0:00:52 lr 0.001938 wd 0.0500 time 0.3796 (0.3872) data time 0.0012 (0.0021) model time 0.3784 (0.3863) loss 3.3520 (3.5959) grad_norm 1.1695 (inf) loss_scale 8192.0000 (9843.7475) mem 11225MB [2024-08-26 06:25:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][500/625] eta 0:00:48 lr 0.001938 wd 0.0500 time 0.3771 (0.3870) data time 0.0011 (0.0021) model time 0.3761 (0.3861) loss 3.7553 (3.5935) grad_norm 1.1368 (inf) loss_scale 8192.0000 (9810.7784) mem 11225MB [2024-08-26 06:25:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][510/625] eta 0:00:44 lr 0.001938 wd 0.0500 time 0.3752 (0.3868) data time 0.0011 (0.0021) model time 0.3741 (0.3859) loss 3.2424 (3.5947) grad_norm 1.3185 (inf) loss_scale 8192.0000 (9779.0998) mem 11225MB [2024-08-26 06:25:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][520/625] eta 0:00:40 lr 0.001938 wd 0.0500 time 0.3725 (0.3866) data time 0.0011 (0.0021) model time 0.3714 (0.3856) loss 2.6733 (3.5937) grad_norm 1.2812 (inf) loss_scale 8192.0000 (9748.6372) mem 11225MB [2024-08-26 06:25:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][530/625] eta 0:00:36 lr 0.001937 wd 0.0500 time 0.3716 (0.3865) data time 0.0011 (0.0020) model time 0.3705 (0.3855) loss 3.8027 (3.5952) grad_norm 1.1830 (inf) loss_scale 8192.0000 (9719.3220) mem 11225MB [2024-08-26 06:25:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][540/625] eta 0:00:32 lr 0.001937 wd 0.0500 time 0.3852 (0.3863) data time 0.0012 (0.0020) model time 0.3840 (0.3853) loss 3.0644 (3.5920) grad_norm 2.2700 (inf) loss_scale 8192.0000 (9691.0906) mem 11225MB [2024-08-26 06:25:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][550/625] eta 0:00:28 lr 0.001937 wd 0.0500 time 0.3779 (0.3861) data time 0.0009 (0.0020) model time 0.3770 (0.3851) loss 4.0016 (3.5919) grad_norm 1.3509 (inf) loss_scale 8192.0000 (9663.8838) mem 11225MB [2024-08-26 06:26:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][560/625] eta 0:00:25 lr 0.001937 wd 0.0500 time 0.3789 (0.3860) data time 0.0009 (0.0020) model time 0.3780 (0.3850) loss 2.9164 (3.5904) grad_norm 1.1325 (inf) loss_scale 8192.0000 (9637.6471) mem 11225MB [2024-08-26 06:26:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][570/625] eta 0:00:21 lr 0.001937 wd 0.0500 time 0.3737 (0.3859) data time 0.0011 (0.0020) model time 0.3726 (0.3848) loss 4.0253 (3.5880) grad_norm 1.5395 (inf) loss_scale 8192.0000 (9612.3292) mem 11225MB [2024-08-26 06:26:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][580/625] eta 0:00:17 lr 0.001937 wd 0.0500 time 0.3786 (0.3857) data time 0.0012 (0.0020) model time 0.3774 (0.3846) loss 3.9300 (3.5919) grad_norm 1.5234 (inf) loss_scale 8192.0000 (9587.8830) mem 11225MB [2024-08-26 06:26:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][590/625] eta 0:00:13 lr 0.001937 wd 0.0500 time 0.3706 (0.3855) data time 0.0009 (0.0020) model time 0.3697 (0.3844) loss 3.5827 (3.5942) grad_norm 1.1517 (inf) loss_scale 8192.0000 (9564.2640) mem 11225MB [2024-08-26 06:26:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][600/625] eta 0:00:09 lr 0.001937 wd 0.0500 time 0.3785 (0.3857) data time 0.0010 (0.0019) model time 0.3775 (0.3846) loss 3.2895 (3.5913) grad_norm 1.2664 (inf) loss_scale 8192.0000 (9541.4309) mem 11225MB [2024-08-26 06:26:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][610/625] eta 0:00:05 lr 0.001937 wd 0.0500 time 0.3682 (0.3866) data time 0.0006 (0.0019) model time 0.3676 (0.3856) loss 4.3957 (3.5923) grad_norm 0.7969 (inf) loss_scale 8192.0000 (9519.3453) mem 11225MB [2024-08-26 06:26:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [51/300][620/625] eta 0:00:01 lr 0.001937 wd 0.0500 time 0.3649 (0.3863) data time 0.0005 (0.0019) model time 0.3644 (0.3852) loss 3.7633 (3.5866) grad_norm 1.2570 (inf) loss_scale 8192.0000 (9497.9710) mem 11225MB [2024-08-26 06:26:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 51 training takes 0:04:01 [2024-08-26 06:26:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:26:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.507 (0.507) Loss 0.7559 (0.7559) Acc@1 82.373 (82.373) Acc@5 96.582 (96.582) Mem 11225MB [2024-08-26 06:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 1.2129 (0.8809) Acc@1 71.240 (79.581) Acc@5 91.455 (95.494) Mem 11225MB [2024-08-26 06:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.3877 (1.0831) Acc@1 68.018 (75.119) Acc@5 89.600 (92.929) Mem 11225MB [2024-08-26 06:26:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 74.972 Acc@5 92.896 [2024-08-26 06:26:31 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.0% [2024-08-26 06:26:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.887 (0.887) Loss 0.5728 (0.5728) Acc@1 86.230 (86.230) Acc@5 97.559 (97.559) Mem 11225MB [2024-08-26 06:26:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.183) Loss 1.0078 (0.7353) Acc@1 75.146 (81.823) Acc@5 93.311 (96.276) Mem 11225MB [2024-08-26 06:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.140) Loss 1.2090 (0.9184) Acc@1 69.873 (77.569) Acc@5 90.771 (94.117) Mem 11225MB [2024-08-26 06:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.273 Acc@5 94.100 [2024-08-26 06:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 77.3% [2024-08-26 06:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.27% [2024-08-26 06:26:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:26:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:26:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][0/625] eta 0:07:40 lr 0.001937 wd 0.0500 time 0.7362 (0.7362) data time 0.3860 (0.3860) model time 0.0000 (0.0000) loss 2.6547 (2.6547) grad_norm 1.2413 (1.2413) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:26:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][10/625] eta 0:04:11 lr 0.001937 wd 0.0500 time 0.3871 (0.4096) data time 0.0008 (0.0361) model time 0.0000 (0.0000) loss 2.4681 (3.3152) grad_norm 1.0400 (1.3386) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:26:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][20/625] eta 0:04:04 lr 0.001937 wd 0.0500 time 0.3773 (0.4048) data time 0.0009 (0.0195) model time 0.0000 (0.0000) loss 4.2374 (3.5186) grad_norm 1.9864 (1.4289) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:26:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][30/625] eta 0:04:03 lr 0.001937 wd 0.0500 time 0.3804 (0.4089) data time 0.0009 (0.0135) model time 0.0000 (0.0000) loss 3.9162 (3.5629) grad_norm 1.2364 (1.3680) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][40/625] eta 0:03:54 lr 0.001937 wd 0.0500 time 0.3811 (0.4013) data time 0.0011 (0.0105) model time 0.0000 (0.0000) loss 3.7592 (3.5284) grad_norm 0.8867 (1.3205) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:26:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][50/625] eta 0:03:48 lr 0.001937 wd 0.0500 time 0.3724 (0.3966) data time 0.0008 (0.0087) model time 0.0000 (0.0000) loss 4.3051 (3.5580) grad_norm 1.2916 (1.2725) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:26:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][60/625] eta 0:03:42 lr 0.001936 wd 0.0500 time 0.3726 (0.3934) data time 0.0008 (0.0075) model time 0.3718 (0.3759) loss 3.5834 (3.5217) grad_norm 1.4847 (1.2810) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][70/625] eta 0:03:37 lr 0.001936 wd 0.0500 time 0.3773 (0.3911) data time 0.0011 (0.0066) model time 0.3762 (0.3760) loss 3.4256 (3.5229) grad_norm 0.9308 (1.2604) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][80/625] eta 0:03:32 lr 0.001936 wd 0.0500 time 0.3790 (0.3892) data time 0.0009 (0.0059) model time 0.3781 (0.3754) loss 3.4599 (3.5358) grad_norm 1.3688 (1.2499) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][90/625] eta 0:03:27 lr 0.001936 wd 0.0500 time 0.3761 (0.3878) data time 0.0008 (0.0054) model time 0.3753 (0.3754) loss 3.8996 (3.5586) grad_norm 1.3453 (1.2446) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][100/625] eta 0:03:23 lr 0.001936 wd 0.0500 time 0.3768 (0.3867) data time 0.0010 (0.0050) model time 0.3758 (0.3754) loss 3.5424 (3.5543) grad_norm 1.1933 (1.2345) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][110/625] eta 0:03:18 lr 0.001936 wd 0.0500 time 0.3843 (0.3858) data time 0.0008 (0.0046) model time 0.3835 (0.3754) loss 3.5644 (3.5658) grad_norm 1.0113 (1.2260) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][120/625] eta 0:03:15 lr 0.001936 wd 0.0500 time 0.3821 (0.3867) data time 0.0011 (0.0043) model time 0.3810 (0.3783) loss 3.8884 (3.5488) grad_norm 1.3188 (1.2290) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][130/625] eta 0:03:11 lr 0.001936 wd 0.0500 time 0.3772 (0.3860) data time 0.0009 (0.0041) model time 0.3764 (0.3781) loss 4.0741 (3.5418) grad_norm 1.6001 (1.2350) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][140/625] eta 0:03:06 lr 0.001936 wd 0.0500 time 0.3759 (0.3854) data time 0.0009 (0.0039) model time 0.3750 (0.3779) loss 2.7229 (3.5488) grad_norm 1.3022 (1.2279) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][150/625] eta 0:03:02 lr 0.001936 wd 0.0500 time 0.3771 (0.3850) data time 0.0008 (0.0037) model time 0.3763 (0.3780) loss 3.5789 (3.5499) grad_norm 2.1095 (1.2614) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][160/625] eta 0:02:58 lr 0.001936 wd 0.0500 time 0.3793 (0.3847) data time 0.0008 (0.0035) model time 0.3785 (0.3780) loss 3.1611 (3.5415) grad_norm 1.0387 (1.2590) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][170/625] eta 0:02:56 lr 0.001936 wd 0.0500 time 0.3846 (0.3887) data time 0.0009 (0.0034) model time 0.3837 (0.3842) loss 2.8686 (3.5579) grad_norm 1.2290 (1.2516) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][180/625] eta 0:02:52 lr 0.001936 wd 0.0500 time 0.3982 (0.3882) data time 0.0011 (0.0033) model time 0.3971 (0.3837) loss 3.4093 (3.5597) grad_norm 1.1726 (1.2469) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][190/625] eta 0:02:48 lr 0.001936 wd 0.0500 time 0.3772 (0.3876) data time 0.0010 (0.0032) model time 0.3762 (0.3831) loss 3.1276 (3.5575) grad_norm 1.4415 (1.2484) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][200/625] eta 0:02:44 lr 0.001936 wd 0.0500 time 0.3782 (0.3873) data time 0.0013 (0.0031) model time 0.3769 (0.3830) loss 3.7498 (3.5558) grad_norm 1.5759 (1.2633) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:27:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][210/625] eta 0:02:40 lr 0.001936 wd 0.0500 time 0.3766 (0.3869) data time 0.0008 (0.0030) model time 0.3758 (0.3826) loss 4.0714 (3.5676) grad_norm 0.9945 (1.2696) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:28:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][220/625] eta 0:02:36 lr 0.001935 wd 0.0500 time 0.3757 (0.3865) data time 0.0011 (0.0029) model time 0.3746 (0.3823) loss 3.6168 (3.5660) grad_norm 1.5937 (1.2723) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:28:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][230/625] eta 0:02:32 lr 0.001935 wd 0.0500 time 0.5803 (0.3870) data time 0.0012 (0.0028) model time 0.5791 (0.3830) loss 3.9174 (3.5593) grad_norm 1.0714 (inf) loss_scale 4096.0000 (8085.6104) mem 11225MB [2024-08-26 06:28:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][240/625] eta 0:02:29 lr 0.001935 wd 0.0500 time 0.3740 (0.3884) data time 0.0009 (0.0028) model time 0.3731 (0.3850) loss 3.7826 (3.5587) grad_norm 0.8878 (inf) loss_scale 4096.0000 (7920.0664) mem 11225MB [2024-08-26 06:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][250/625] eta 0:02:25 lr 0.001935 wd 0.0500 time 0.3762 (0.3880) data time 0.0008 (0.0027) model time 0.3754 (0.3846) loss 4.1279 (3.5551) grad_norm 1.4588 (inf) loss_scale 4096.0000 (7767.7131) mem 11225MB [2024-08-26 06:28:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][260/625] eta 0:02:21 lr 0.001935 wd 0.0500 time 0.3809 (0.3877) data time 0.0008 (0.0026) model time 0.3801 (0.3843) loss 3.6011 (3.5756) grad_norm 1.4215 (inf) loss_scale 4096.0000 (7627.0345) mem 11225MB [2024-08-26 06:28:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][270/625] eta 0:02:17 lr 0.001935 wd 0.0500 time 0.3822 (0.3873) data time 0.0008 (0.0026) model time 0.3814 (0.3840) loss 3.8624 (3.5802) grad_norm 1.0715 (inf) loss_scale 4096.0000 (7496.7380) mem 11225MB [2024-08-26 06:28:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][280/625] eta 0:02:13 lr 0.001935 wd 0.0500 time 0.3796 (0.3870) data time 0.0011 (0.0025) model time 0.3785 (0.3837) loss 2.8172 (3.5795) grad_norm 1.2762 (inf) loss_scale 4096.0000 (7375.7153) mem 11225MB [2024-08-26 06:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][290/625] eta 0:02:10 lr 0.001935 wd 0.0500 time 0.3811 (0.3887) data time 0.0011 (0.0025) model time 0.3801 (0.3858) loss 3.9496 (3.5838) grad_norm 2.0691 (inf) loss_scale 4096.0000 (7263.0103) mem 11225MB [2024-08-26 06:28:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][300/625] eta 0:02:06 lr 0.001935 wd 0.0500 time 0.3787 (0.3887) data time 0.0009 (0.0024) model time 0.3778 (0.3860) loss 4.5247 (3.5875) grad_norm 0.9460 (inf) loss_scale 4096.0000 (7157.7940) mem 11225MB [2024-08-26 06:28:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][310/625] eta 0:02:02 lr 0.001935 wd 0.0500 time 0.3783 (0.3884) data time 0.0011 (0.0024) model time 0.3771 (0.3856) loss 3.9328 (3.5875) grad_norm 1.2407 (inf) loss_scale 4096.0000 (7059.3441) mem 11225MB [2024-08-26 06:28:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][320/625] eta 0:01:58 lr 0.001935 wd 0.0500 time 0.3788 (0.3881) data time 0.0009 (0.0024) model time 0.3780 (0.3853) loss 2.6671 (3.5790) grad_norm 1.2963 (inf) loss_scale 4096.0000 (6967.0280) mem 11225MB [2024-08-26 06:28:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][330/625] eta 0:01:54 lr 0.001935 wd 0.0500 time 0.3782 (0.3877) data time 0.0013 (0.0023) model time 0.3769 (0.3849) loss 2.8991 (3.5711) grad_norm 1.6771 (inf) loss_scale 4096.0000 (6880.2900) mem 11225MB [2024-08-26 06:28:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][340/625] eta 0:01:50 lr 0.001935 wd 0.0500 time 0.3672 (0.3874) data time 0.0012 (0.0023) model time 0.3660 (0.3846) loss 2.5991 (3.5704) grad_norm 1.1271 (inf) loss_scale 4096.0000 (6798.6393) mem 11225MB [2024-08-26 06:28:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][350/625] eta 0:01:46 lr 0.001935 wd 0.0500 time 0.3744 (0.3871) data time 0.0011 (0.0023) model time 0.3734 (0.3843) loss 3.7483 (3.5562) grad_norm 1.4295 (inf) loss_scale 4096.0000 (6721.6410) mem 11225MB [2024-08-26 06:28:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][360/625] eta 0:01:42 lr 0.001935 wd 0.0500 time 0.3790 (0.3869) data time 0.0009 (0.0022) model time 0.3781 (0.3841) loss 3.0133 (3.5590) grad_norm 1.2285 (inf) loss_scale 4096.0000 (6648.9086) mem 11225MB [2024-08-26 06:28:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][370/625] eta 0:01:38 lr 0.001935 wd 0.0500 time 0.3816 (0.3867) data time 0.0010 (0.0022) model time 0.3806 (0.3840) loss 2.4801 (3.5499) grad_norm 1.3475 (inf) loss_scale 4096.0000 (6580.0970) mem 11225MB [2024-08-26 06:29:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][380/625] eta 0:01:34 lr 0.001934 wd 0.0500 time 0.3891 (0.3866) data time 0.0008 (0.0022) model time 0.3883 (0.3839) loss 2.7204 (3.5527) grad_norm 1.3114 (inf) loss_scale 4096.0000 (6514.8976) mem 11225MB [2024-08-26 06:29:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][390/625] eta 0:01:30 lr 0.001934 wd 0.0500 time 0.3754 (0.3863) data time 0.0011 (0.0021) model time 0.3743 (0.3836) loss 3.9678 (3.5532) grad_norm 1.4803 (inf) loss_scale 4096.0000 (6453.0332) mem 11225MB [2024-08-26 06:29:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][400/625] eta 0:01:26 lr 0.001934 wd 0.0500 time 0.3775 (0.3861) data time 0.0012 (0.0021) model time 0.3763 (0.3834) loss 3.8930 (3.5568) grad_norm 1.1367 (inf) loss_scale 4096.0000 (6394.2544) mem 11225MB [2024-08-26 06:29:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][410/625] eta 0:01:22 lr 0.001934 wd 0.0500 time 0.3824 (0.3859) data time 0.0013 (0.0021) model time 0.3811 (0.3832) loss 4.1528 (3.5598) grad_norm 1.4564 (inf) loss_scale 4096.0000 (6338.3358) mem 11225MB [2024-08-26 06:29:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][420/625] eta 0:01:19 lr 0.001934 wd 0.0500 time 0.3784 (0.3872) data time 0.0010 (0.0021) model time 0.3774 (0.3848) loss 3.7815 (3.5651) grad_norm 1.1848 (inf) loss_scale 4096.0000 (6285.0736) mem 11225MB [2024-08-26 06:29:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][430/625] eta 0:01:15 lr 0.001934 wd 0.0500 time 0.3730 (0.3870) data time 0.0008 (0.0021) model time 0.3722 (0.3845) loss 3.9329 (3.5647) grad_norm 1.1131 (inf) loss_scale 4096.0000 (6234.2831) mem 11225MB [2024-08-26 06:29:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][440/625] eta 0:01:11 lr 0.001934 wd 0.0500 time 0.3785 (0.3868) data time 0.0009 (0.0020) model time 0.3777 (0.3843) loss 2.8115 (3.5547) grad_norm 1.1798 (inf) loss_scale 4096.0000 (6185.7959) mem 11225MB [2024-08-26 06:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][450/625] eta 0:01:07 lr 0.001934 wd 0.0500 time 0.3818 (0.3866) data time 0.0008 (0.0020) model time 0.3810 (0.3841) loss 2.6106 (3.5520) grad_norm 0.8310 (inf) loss_scale 4096.0000 (6139.4590) mem 11225MB [2024-08-26 06:29:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][460/625] eta 0:01:03 lr 0.001934 wd 0.0500 time 0.3735 (0.3864) data time 0.0012 (0.0020) model time 0.3723 (0.3840) loss 3.6327 (3.5560) grad_norm 1.1142 (inf) loss_scale 4096.0000 (6095.1323) mem 11225MB [2024-08-26 06:29:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][470/625] eta 0:00:59 lr 0.001934 wd 0.0500 time 0.3857 (0.3863) data time 0.0007 (0.0020) model time 0.3850 (0.3839) loss 3.6918 (3.5602) grad_norm 1.1552 (inf) loss_scale 4096.0000 (6052.6879) mem 11225MB [2024-08-26 06:29:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][480/625] eta 0:00:55 lr 0.001934 wd 0.0500 time 0.3930 (0.3862) data time 0.0013 (0.0020) model time 0.3918 (0.3838) loss 3.5500 (3.5629) grad_norm 1.6133 (inf) loss_scale 4096.0000 (6012.0083) mem 11225MB [2024-08-26 06:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][490/625] eta 0:00:52 lr 0.001934 wd 0.0500 time 0.3818 (0.3860) data time 0.0010 (0.0019) model time 0.3808 (0.3836) loss 3.7282 (3.5610) grad_norm 1.0451 (inf) loss_scale 4096.0000 (5972.9857) mem 11225MB [2024-08-26 06:29:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][500/625] eta 0:00:48 lr 0.001934 wd 0.0500 time 0.3748 (0.3870) data time 0.0008 (0.0019) model time 0.3740 (0.3847) loss 4.2513 (3.5612) grad_norm 2.5937 (inf) loss_scale 4096.0000 (5935.5210) mem 11225MB [2024-08-26 06:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][510/625] eta 0:00:44 lr 0.001934 wd 0.0500 time 0.3757 (0.3868) data time 0.0008 (0.0019) model time 0.3749 (0.3845) loss 2.6131 (3.5667) grad_norm 1.1251 (inf) loss_scale 4096.0000 (5899.5225) mem 11225MB [2024-08-26 06:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][520/625] eta 0:00:40 lr 0.001934 wd 0.0500 time 0.3834 (0.3867) data time 0.0010 (0.0019) model time 0.3823 (0.3844) loss 3.3148 (3.5688) grad_norm 1.5804 (inf) loss_scale 4096.0000 (5864.9060) mem 11225MB [2024-08-26 06:30:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][530/625] eta 0:00:36 lr 0.001934 wd 0.0500 time 0.3754 (0.3865) data time 0.0008 (0.0019) model time 0.3746 (0.3843) loss 3.8782 (3.5747) grad_norm 0.9973 (inf) loss_scale 4096.0000 (5831.5932) mem 11225MB [2024-08-26 06:30:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][540/625] eta 0:00:32 lr 0.001933 wd 0.0500 time 0.3728 (0.3875) data time 0.0011 (0.0019) model time 0.3717 (0.3854) loss 3.8797 (3.5777) grad_norm 1.8115 (inf) loss_scale 4096.0000 (5799.5120) mem 11225MB [2024-08-26 06:30:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][550/625] eta 0:00:29 lr 0.001933 wd 0.0500 time 0.3819 (0.3873) data time 0.0008 (0.0019) model time 0.3811 (0.3852) loss 4.7597 (3.5807) grad_norm 1.1817 (inf) loss_scale 4096.0000 (5768.5953) mem 11225MB [2024-08-26 06:30:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][560/625] eta 0:00:25 lr 0.001933 wd 0.0500 time 0.3787 (0.3872) data time 0.0007 (0.0019) model time 0.3779 (0.3850) loss 4.7192 (3.5864) grad_norm 2.3673 (inf) loss_scale 4096.0000 (5738.7807) mem 11225MB [2024-08-26 06:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][570/625] eta 0:00:21 lr 0.001933 wd 0.0500 time 0.3754 (0.3870) data time 0.0009 (0.0018) model time 0.3745 (0.3848) loss 4.2856 (3.5895) grad_norm 1.6032 (inf) loss_scale 4096.0000 (5710.0105) mem 11225MB [2024-08-26 06:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][580/625] eta 0:00:17 lr 0.001933 wd 0.0500 time 0.3710 (0.3868) data time 0.0011 (0.0018) model time 0.3700 (0.3847) loss 4.0136 (3.5896) grad_norm 1.5220 (inf) loss_scale 4096.0000 (5682.2306) mem 11225MB [2024-08-26 06:30:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][590/625] eta 0:00:13 lr 0.001933 wd 0.0500 time 0.3769 (0.3867) data time 0.0011 (0.0019) model time 0.3758 (0.3846) loss 3.7465 (3.5894) grad_norm 1.0395 (inf) loss_scale 4096.0000 (5655.3909) mem 11225MB [2024-08-26 06:30:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][600/625] eta 0:00:09 lr 0.001933 wd 0.0500 time 0.3746 (0.3866) data time 0.0008 (0.0018) model time 0.3739 (0.3844) loss 2.7768 (3.5868) grad_norm 1.3053 (inf) loss_scale 4096.0000 (5629.4443) mem 11225MB [2024-08-26 06:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][610/625] eta 0:00:05 lr 0.001933 wd 0.0500 time 0.3651 (0.3864) data time 0.0008 (0.0018) model time 0.3644 (0.3842) loss 3.0305 (3.5849) grad_norm 1.0123 (inf) loss_scale 4096.0000 (5604.3470) mem 11225MB [2024-08-26 06:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [52/300][620/625] eta 0:00:01 lr 0.001933 wd 0.0500 time 0.3652 (0.3861) data time 0.0008 (0.0018) model time 0.3644 (0.3839) loss 3.7476 (3.5873) grad_norm 0.7923 (inf) loss_scale 4096.0000 (5580.0580) mem 11225MB [2024-08-26 06:30:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 52 training takes 0:04:01 [2024-08-26 06:30:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:30:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:30:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.6831 (0.6831) Acc@1 85.303 (85.303) Acc@5 97.510 (97.510) Mem 11225MB [2024-08-26 06:30:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.140) Loss 1.2051 (0.8695) Acc@1 72.119 (79.896) Acc@5 92.090 (95.632) Mem 11225MB [2024-08-26 06:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 1.3770 (1.0668) Acc@1 67.529 (75.609) Acc@5 89.404 (93.094) Mem 11225MB [2024-08-26 06:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.348 Acc@5 92.990 [2024-08-26 06:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.3% [2024-08-26 06:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 75.35% [2024-08-26 06:30:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 06:30:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 06:30:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.504 (0.504) Loss 0.5718 (0.5718) Acc@1 86.426 (86.426) Acc@5 97.559 (97.559) Mem 11225MB [2024-08-26 06:30:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.143) Loss 1.0010 (0.7330) Acc@1 75.293 (81.987) Acc@5 93.213 (96.325) Mem 11225MB [2024-08-26 06:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.2002 (0.9149) Acc@1 69.922 (77.720) Acc@5 91.064 (94.178) Mem 11225MB [2024-08-26 06:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.405 Acc@5 94.142 [2024-08-26 06:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 77.4% [2024-08-26 06:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.41% [2024-08-26 06:30:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:30:46 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:30:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][0/625] eta 0:07:56 lr 0.001933 wd 0.0500 time 0.7624 (0.7624) data time 0.4246 (0.4246) model time 0.0000 (0.0000) loss 3.7931 (3.7931) grad_norm 1.7347 (1.7347) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:30:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][10/625] eta 0:04:14 lr 0.001933 wd 0.0500 time 0.3848 (0.4144) data time 0.0009 (0.0398) model time 0.0000 (0.0000) loss 3.2035 (3.4369) grad_norm 1.2588 (1.2808) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][20/625] eta 0:03:59 lr 0.001933 wd 0.0500 time 0.3685 (0.3961) data time 0.0011 (0.0214) model time 0.0000 (0.0000) loss 3.6048 (3.5136) grad_norm 1.0582 (1.2326) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:30:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][30/625] eta 0:03:51 lr 0.001933 wd 0.0500 time 0.3807 (0.3899) data time 0.0008 (0.0148) model time 0.0000 (0.0000) loss 3.3330 (3.4778) grad_norm 1.1437 (1.1798) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][40/625] eta 0:03:46 lr 0.001933 wd 0.0500 time 0.3880 (0.3877) data time 0.0008 (0.0115) model time 0.0000 (0.0000) loss 2.7531 (3.4970) grad_norm 2.2341 (1.2789) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][50/625] eta 0:03:41 lr 0.001933 wd 0.0500 time 0.3886 (0.3861) data time 0.0007 (0.0095) model time 0.0000 (0.0000) loss 2.8269 (3.4947) grad_norm 1.6937 (1.3142) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][60/625] eta 0:03:40 lr 0.001933 wd 0.0500 time 0.3806 (0.3910) data time 0.0010 (0.0083) model time 0.3796 (0.4142) loss 3.3816 (3.5388) grad_norm 1.4841 (1.2921) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][70/625] eta 0:03:40 lr 0.001932 wd 0.0500 time 0.3808 (0.3975) data time 0.0009 (0.0073) model time 0.3799 (0.4248) loss 3.7228 (3.5197) grad_norm 1.0783 (1.2917) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][80/625] eta 0:03:36 lr 0.001932 wd 0.0500 time 0.3752 (0.3970) data time 0.0009 (0.0065) model time 0.3744 (0.4141) loss 4.3467 (3.5280) grad_norm 1.1277 (1.2860) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][90/625] eta 0:03:31 lr 0.001932 wd 0.0500 time 0.3748 (0.3949) data time 0.0007 (0.0059) model time 0.3740 (0.4049) loss 4.0580 (3.5566) grad_norm 1.3749 (1.2976) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][100/625] eta 0:03:26 lr 0.001932 wd 0.0500 time 0.3751 (0.3934) data time 0.0008 (0.0054) model time 0.3743 (0.3996) loss 3.4004 (3.5442) grad_norm 1.1707 (1.2893) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][110/625] eta 0:03:21 lr 0.001932 wd 0.0500 time 0.3681 (0.3921) data time 0.0008 (0.0050) model time 0.3673 (0.3960) loss 4.0749 (3.5246) grad_norm 1.2612 (1.2855) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][120/625] eta 0:03:17 lr 0.001932 wd 0.0500 time 0.3801 (0.3912) data time 0.0009 (0.0047) model time 0.3792 (0.3936) loss 4.0584 (3.5107) grad_norm 1.4080 (1.2921) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][130/625] eta 0:03:13 lr 0.001932 wd 0.0500 time 0.3711 (0.3917) data time 0.0010 (0.0044) model time 0.3701 (0.3940) loss 3.9569 (3.5089) grad_norm 1.3197 (1.2906) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][140/625] eta 0:03:09 lr 0.001932 wd 0.0500 time 0.3814 (0.3907) data time 0.0011 (0.0042) model time 0.3803 (0.3921) loss 3.2778 (3.5013) grad_norm 1.1469 (1.2768) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][150/625] eta 0:03:05 lr 0.001932 wd 0.0500 time 0.3860 (0.3898) data time 0.0010 (0.0040) model time 0.3850 (0.3905) loss 3.7068 (3.5092) grad_norm 0.9924 (1.2654) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][160/625] eta 0:03:00 lr 0.001932 wd 0.0500 time 0.3778 (0.3890) data time 0.0010 (0.0038) model time 0.3769 (0.3892) loss 3.4410 (3.5145) grad_norm 2.1397 (1.2860) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][170/625] eta 0:02:56 lr 0.001932 wd 0.0500 time 0.3784 (0.3885) data time 0.0008 (0.0037) model time 0.3776 (0.3883) loss 4.0569 (3.5015) grad_norm 1.0401 (1.3018) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:31:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][180/625] eta 0:02:52 lr 0.001932 wd 0.0500 time 0.3864 (0.3881) data time 0.0009 (0.0035) model time 0.3855 (0.3877) loss 4.5693 (3.5092) grad_norm 1.2403 (1.2996) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][190/625] eta 0:02:48 lr 0.001932 wd 0.0500 time 0.3754 (0.3875) data time 0.0010 (0.0034) model time 0.3744 (0.3869) loss 3.9747 (3.5250) grad_norm 0.9490 (1.2937) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][200/625] eta 0:02:44 lr 0.001932 wd 0.0500 time 0.3763 (0.3871) data time 0.0011 (0.0033) model time 0.3752 (0.3862) loss 3.8678 (3.5183) grad_norm 1.5479 (1.2929) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][210/625] eta 0:02:40 lr 0.001932 wd 0.0500 time 0.3727 (0.3866) data time 0.0008 (0.0032) model time 0.3718 (0.3856) loss 4.0632 (3.5252) grad_norm 1.5143 (1.2845) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][220/625] eta 0:02:36 lr 0.001931 wd 0.0500 time 0.3900 (0.3864) data time 0.0008 (0.0031) model time 0.3891 (0.3853) loss 2.9927 (3.5250) grad_norm 1.2675 (1.2836) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][230/625] eta 0:02:32 lr 0.001931 wd 0.0500 time 0.3765 (0.3858) data time 0.0011 (0.0030) model time 0.3755 (0.3845) loss 4.0911 (3.5333) grad_norm 2.0718 (1.3015) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][240/625] eta 0:02:28 lr 0.001931 wd 0.0500 time 0.5064 (0.3860) data time 0.0012 (0.0030) model time 0.5052 (0.3848) loss 3.3816 (3.5312) grad_norm 1.5744 (1.3021) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][250/625] eta 0:02:25 lr 0.001931 wd 0.0500 time 0.3700 (0.3873) data time 0.0011 (0.0029) model time 0.3689 (0.3864) loss 4.0667 (3.5347) grad_norm 1.2395 (1.2991) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][260/625] eta 0:02:21 lr 0.001931 wd 0.0500 time 0.3755 (0.3870) data time 0.0008 (0.0028) model time 0.3746 (0.3860) loss 3.2929 (3.5373) grad_norm 1.0445 (1.3001) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][270/625] eta 0:02:17 lr 0.001931 wd 0.0500 time 0.3750 (0.3867) data time 0.0011 (0.0028) model time 0.3739 (0.3856) loss 4.0635 (3.5362) grad_norm 1.0691 (1.2935) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][280/625] eta 0:02:13 lr 0.001931 wd 0.0500 time 0.3782 (0.3872) data time 0.0010 (0.0027) model time 0.3771 (0.3863) loss 4.3484 (3.5444) grad_norm 0.9783 (1.2986) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][290/625] eta 0:02:09 lr 0.001931 wd 0.0500 time 0.3754 (0.3876) data time 0.0008 (0.0027) model time 0.3746 (0.3867) loss 2.5070 (3.5445) grad_norm 1.3673 (1.2999) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][300/625] eta 0:02:05 lr 0.001931 wd 0.0500 time 0.3753 (0.3873) data time 0.0008 (0.0026) model time 0.3745 (0.3863) loss 2.6647 (3.5419) grad_norm 1.0144 (1.2961) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][310/625] eta 0:02:01 lr 0.001931 wd 0.0500 time 0.3737 (0.3870) data time 0.0011 (0.0026) model time 0.3726 (0.3860) loss 3.2176 (3.5452) grad_norm 1.3048 (1.2993) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][320/625] eta 0:01:57 lr 0.001931 wd 0.0500 time 0.3826 (0.3867) data time 0.0011 (0.0025) model time 0.3815 (0.3856) loss 3.8354 (3.5423) grad_norm 1.6322 (1.3033) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][330/625] eta 0:01:53 lr 0.001931 wd 0.0500 time 0.3920 (0.3864) data time 0.0009 (0.0025) model time 0.3911 (0.3853) loss 3.6996 (3.5540) grad_norm 1.0568 (1.2955) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:32:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][340/625] eta 0:01:50 lr 0.001931 wd 0.0500 time 0.3715 (0.3861) data time 0.0008 (0.0024) model time 0.3707 (0.3849) loss 2.8179 (3.5523) grad_norm 1.2634 (1.2890) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][350/625] eta 0:01:46 lr 0.001931 wd 0.0500 time 0.3803 (0.3859) data time 0.0011 (0.0024) model time 0.3792 (0.3847) loss 3.8510 (3.5506) grad_norm 1.1701 (1.2854) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][360/625] eta 0:01:42 lr 0.001931 wd 0.0500 time 0.5843 (0.3862) data time 0.0011 (0.0024) model time 0.5832 (0.3850) loss 2.7391 (3.5434) grad_norm 1.4733 (1.2837) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][370/625] eta 0:01:38 lr 0.001931 wd 0.0500 time 0.3743 (0.3870) data time 0.0008 (0.0023) model time 0.3735 (0.3859) loss 4.3717 (3.5481) grad_norm 1.0639 (1.2807) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][380/625] eta 0:01:34 lr 0.001930 wd 0.0500 time 0.3783 (0.3867) data time 0.0009 (0.0023) model time 0.3774 (0.3856) loss 4.1636 (3.5484) grad_norm 0.9566 (1.2785) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][390/625] eta 0:01:30 lr 0.001930 wd 0.0500 time 0.3738 (0.3864) data time 0.0010 (0.0023) model time 0.3728 (0.3853) loss 2.8122 (3.5450) grad_norm 1.0554 (1.2756) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][400/625] eta 0:01:26 lr 0.001930 wd 0.0500 time 0.3768 (0.3862) data time 0.0009 (0.0022) model time 0.3760 (0.3850) loss 2.2041 (3.5445) grad_norm 0.9442 (1.2711) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][410/625] eta 0:01:22 lr 0.001930 wd 0.0500 time 0.3762 (0.3859) data time 0.0011 (0.0022) model time 0.3751 (0.3847) loss 3.6192 (3.5473) grad_norm 1.6200 (1.2702) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][420/625] eta 0:01:19 lr 0.001930 wd 0.0500 time 0.3800 (0.3857) data time 0.0010 (0.0022) model time 0.3790 (0.3845) loss 3.1097 (3.5564) grad_norm 1.3436 (1.2744) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][430/625] eta 0:01:15 lr 0.001930 wd 0.0500 time 0.3755 (0.3855) data time 0.0011 (0.0022) model time 0.3743 (0.3842) loss 3.8418 (3.5589) grad_norm 1.0122 (1.2710) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][440/625] eta 0:01:11 lr 0.001930 wd 0.0500 time 0.3754 (0.3853) data time 0.0008 (0.0021) model time 0.3745 (0.3840) loss 3.6504 (3.5577) grad_norm 0.8930 (1.2662) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][450/625] eta 0:01:07 lr 0.001930 wd 0.0500 time 0.3874 (0.3852) data time 0.0010 (0.0021) model time 0.3864 (0.3839) loss 2.8940 (3.5529) grad_norm 1.0886 (1.2677) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][460/625] eta 0:01:03 lr 0.001930 wd 0.0500 time 0.3735 (0.3854) data time 0.0010 (0.0021) model time 0.3724 (0.3842) loss 3.9786 (3.5585) grad_norm 0.9379 (1.2659) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][470/625] eta 0:00:59 lr 0.001930 wd 0.0500 time 0.3793 (0.3861) data time 0.0011 (0.0021) model time 0.3782 (0.3850) loss 3.3496 (3.5595) grad_norm 1.1937 (1.2638) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][480/625] eta 0:00:55 lr 0.001930 wd 0.0500 time 0.3820 (0.3860) data time 0.0009 (0.0021) model time 0.3811 (0.3848) loss 4.2685 (3.5630) grad_norm 1.4425 (1.2659) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][490/625] eta 0:00:52 lr 0.001930 wd 0.0500 time 0.3755 (0.3858) data time 0.0008 (0.0020) model time 0.3747 (0.3846) loss 2.9351 (3.5649) grad_norm 1.3104 (1.2682) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:33:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][500/625] eta 0:00:48 lr 0.001930 wd 0.0500 time 0.3780 (0.3856) data time 0.0010 (0.0020) model time 0.3770 (0.3844) loss 3.4679 (3.5626) grad_norm 1.3197 (1.2662) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][510/625] eta 0:00:44 lr 0.001930 wd 0.0500 time 0.3792 (0.3854) data time 0.0008 (0.0020) model time 0.3784 (0.3842) loss 3.8738 (3.5639) grad_norm 1.2066 (1.2716) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][520/625] eta 0:00:40 lr 0.001930 wd 0.0500 time 0.3783 (0.3853) data time 0.0010 (0.0020) model time 0.3773 (0.3840) loss 3.6898 (3.5682) grad_norm 1.0141 (1.2702) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][530/625] eta 0:00:36 lr 0.001929 wd 0.0500 time 0.3792 (0.3852) data time 0.0011 (0.0020) model time 0.3781 (0.3839) loss 3.7421 (3.5681) grad_norm 1.1449 (1.2677) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][540/625] eta 0:00:32 lr 0.001929 wd 0.0500 time 0.3842 (0.3850) data time 0.0012 (0.0020) model time 0.3830 (0.3838) loss 4.2074 (3.5648) grad_norm 1.2804 (1.2686) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][550/625] eta 0:00:28 lr 0.001929 wd 0.0500 time 0.3693 (0.3849) data time 0.0011 (0.0019) model time 0.3681 (0.3836) loss 2.4422 (3.5638) grad_norm 1.4199 (1.2736) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][560/625] eta 0:00:25 lr 0.001929 wd 0.0500 time 0.4004 (0.3849) data time 0.0008 (0.0019) model time 0.3996 (0.3836) loss 3.5977 (3.5620) grad_norm 1.0880 (1.2705) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][570/625] eta 0:00:21 lr 0.001929 wd 0.0500 time 0.5955 (0.3855) data time 0.0011 (0.0019) model time 0.5944 (0.3843) loss 3.4308 (3.5639) grad_norm 1.1780 (1.2682) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][580/625] eta 0:00:17 lr 0.001929 wd 0.0500 time 0.5928 (0.3862) data time 0.0008 (0.0019) model time 0.5920 (0.3850) loss 4.2978 (3.5655) grad_norm 1.2716 (1.2669) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][590/625] eta 0:00:13 lr 0.001929 wd 0.0500 time 0.3764 (0.3864) data time 0.0010 (0.0019) model time 0.3754 (0.3852) loss 3.6311 (3.5682) grad_norm 1.1581 (1.2660) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][600/625] eta 0:00:09 lr 0.001929 wd 0.0500 time 0.3779 (0.3863) data time 0.0011 (0.0019) model time 0.3768 (0.3851) loss 3.9252 (3.5706) grad_norm 1.1839 (1.2644) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][610/625] eta 0:00:05 lr 0.001929 wd 0.0500 time 0.3641 (0.3861) data time 0.0006 (0.0019) model time 0.3636 (0.3849) loss 3.1400 (3.5649) grad_norm 1.0436 (1.2634) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [53/300][620/625] eta 0:00:01 lr 0.001929 wd 0.0500 time 0.3656 (0.3858) data time 0.0006 (0.0019) model time 0.3651 (0.3846) loss 3.6935 (3.5661) grad_norm 1.2850 (1.2629) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:34:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 53 training takes 0:04:01 [2024-08-26 06:34:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:34:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:34:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 0.7427 (0.7427) Acc@1 85.156 (85.156) Acc@5 96.973 (96.973) Mem 11225MB [2024-08-26 06:34:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 1.2324 (0.8988) Acc@1 71.045 (79.648) Acc@5 91.602 (95.614) Mem 11225MB [2024-08-26 06:34:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.3584 (1.0857) Acc@1 68.896 (75.295) Acc@5 90.039 (93.201) Mem 11225MB [2024-08-26 06:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.054 Acc@5 93.162 [2024-08-26 06:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.1% [2024-08-26 06:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.866 (0.866) Loss 0.5723 (0.5723) Acc@1 86.572 (86.572) Acc@5 97.510 (97.510) Mem 11225MB [2024-08-26 06:34:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.183) Loss 0.9971 (0.7311) Acc@1 75.537 (82.098) Acc@5 93.311 (96.338) Mem 11225MB [2024-08-26 06:34:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.140) Loss 1.1973 (0.9121) Acc@1 70.020 (77.820) Acc@5 91.260 (94.229) Mem 11225MB [2024-08-26 06:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.499 Acc@5 94.200 [2024-08-26 06:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 77.5% [2024-08-26 06:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.50% [2024-08-26 06:34:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:34:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:34:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][0/625] eta 0:08:47 lr 0.001929 wd 0.0500 time 0.8435 (0.8435) data time 0.5056 (0.5056) model time 0.0000 (0.0000) loss 3.8504 (3.8504) grad_norm 1.4062 (1.4062) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][10/625] eta 0:04:19 lr 0.001929 wd 0.0500 time 0.3810 (0.4212) data time 0.0013 (0.0470) model time 0.0000 (0.0000) loss 3.6161 (3.7147) grad_norm 1.2673 (1.3829) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][20/625] eta 0:04:02 lr 0.001929 wd 0.0500 time 0.3768 (0.4001) data time 0.0008 (0.0252) model time 0.0000 (0.0000) loss 3.5273 (3.6629) grad_norm 0.9473 (1.3300) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][30/625] eta 0:03:53 lr 0.001929 wd 0.0500 time 0.3808 (0.3929) data time 0.0009 (0.0174) model time 0.0000 (0.0000) loss 2.4365 (3.5585) grad_norm 1.4441 (1.3368) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][40/625] eta 0:03:47 lr 0.001929 wd 0.0500 time 0.3764 (0.3889) data time 0.0010 (0.0135) model time 0.0000 (0.0000) loss 3.3241 (3.5550) grad_norm 1.0685 (1.2957) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][50/625] eta 0:03:42 lr 0.001929 wd 0.0500 time 0.3740 (0.3863) data time 0.0008 (0.0114) model time 0.0000 (0.0000) loss 2.4127 (3.5477) grad_norm 1.1080 (1.2562) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][60/625] eta 0:03:37 lr 0.001928 wd 0.0500 time 0.3770 (0.3849) data time 0.0011 (0.0097) model time 0.3759 (0.3769) loss 3.8694 (3.5692) grad_norm 1.5743 (1.2406) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][70/625] eta 0:03:33 lr 0.001928 wd 0.0500 time 0.3759 (0.3839) data time 0.0008 (0.0085) model time 0.3751 (0.3767) loss 3.7553 (3.5576) grad_norm 1.4376 (1.2464) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][80/625] eta 0:03:28 lr 0.001928 wd 0.0500 time 0.3746 (0.3832) data time 0.0009 (0.0076) model time 0.3737 (0.3767) loss 4.1089 (3.5063) grad_norm 1.4461 (1.2666) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][90/625] eta 0:03:24 lr 0.001928 wd 0.0500 time 0.3731 (0.3825) data time 0.0009 (0.0069) model time 0.3722 (0.3766) loss 4.4760 (3.5227) grad_norm 1.4906 (1.2614) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][100/625] eta 0:03:24 lr 0.001928 wd 0.0500 time 0.6156 (0.3886) data time 0.0009 (0.0063) model time 0.6147 (0.3897) loss 4.0838 (3.5366) grad_norm 1.0740 (1.2556) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][110/625] eta 0:03:19 lr 0.001928 wd 0.0500 time 0.3755 (0.3873) data time 0.0011 (0.0059) model time 0.3745 (0.3870) loss 3.7038 (3.5500) grad_norm 1.0784 (1.2720) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][120/625] eta 0:03:15 lr 0.001928 wd 0.0500 time 0.3778 (0.3867) data time 0.0009 (0.0055) model time 0.3769 (0.3859) loss 3.3163 (3.5318) grad_norm 1.1324 (1.2634) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][130/625] eta 0:03:11 lr 0.001928 wd 0.0500 time 0.3840 (0.3863) data time 0.0011 (0.0051) model time 0.3829 (0.3852) loss 3.6563 (3.5414) grad_norm 1.0769 (1.2644) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][140/625] eta 0:03:07 lr 0.001928 wd 0.0500 time 0.3770 (0.3858) data time 0.0009 (0.0049) model time 0.3761 (0.3843) loss 4.0338 (3.5184) grad_norm 1.3671 (1.2644) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][150/625] eta 0:03:04 lr 0.001928 wd 0.0500 time 0.3795 (0.3878) data time 0.0011 (0.0046) model time 0.3785 (0.3874) loss 3.7465 (3.4868) grad_norm 0.9129 (1.2664) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:35:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][160/625] eta 0:03:00 lr 0.001928 wd 0.0500 time 0.3768 (0.3886) data time 0.0008 (0.0044) model time 0.3760 (0.3885) loss 2.7058 (3.4995) grad_norm 1.4045 (1.2701) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][170/625] eta 0:02:56 lr 0.001928 wd 0.0500 time 0.3858 (0.3881) data time 0.0011 (0.0042) model time 0.3848 (0.3877) loss 3.4006 (3.5136) grad_norm 1.1071 (1.2692) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][180/625] eta 0:02:52 lr 0.001928 wd 0.0500 time 0.3801 (0.3875) data time 0.0013 (0.0040) model time 0.3788 (0.3868) loss 3.6172 (3.5269) grad_norm 1.2801 (1.2710) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][190/625] eta 0:02:48 lr 0.001928 wd 0.0500 time 0.3939 (0.3870) data time 0.0011 (0.0039) model time 0.3928 (0.3860) loss 2.9325 (3.5305) grad_norm 1.6392 (1.2700) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][200/625] eta 0:02:44 lr 0.001928 wd 0.0500 time 0.3742 (0.3864) data time 0.0011 (0.0038) model time 0.3732 (0.3853) loss 3.5958 (3.5317) grad_norm 0.8770 (1.2659) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][210/625] eta 0:02:40 lr 0.001927 wd 0.0500 time 0.3798 (0.3860) data time 0.0008 (0.0036) model time 0.3790 (0.3847) loss 4.3323 (3.5284) grad_norm 1.0375 (1.2559) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][220/625] eta 0:02:36 lr 0.001927 wd 0.0500 time 0.3801 (0.3856) data time 0.0011 (0.0035) model time 0.3790 (0.3842) loss 4.0553 (3.5325) grad_norm 1.0442 (1.2535) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][230/625] eta 0:02:32 lr 0.001927 wd 0.0500 time 0.3713 (0.3853) data time 0.0012 (0.0034) model time 0.3701 (0.3839) loss 4.2672 (3.5448) grad_norm 1.1232 (1.2637) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][240/625] eta 0:02:28 lr 0.001927 wd 0.0500 time 0.3795 (0.3849) data time 0.0011 (0.0033) model time 0.3785 (0.3834) loss 3.8500 (3.5589) grad_norm 1.2102 (1.2621) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][250/625] eta 0:02:24 lr 0.001927 wd 0.0500 time 0.3766 (0.3845) data time 0.0012 (0.0032) model time 0.3754 (0.3829) loss 3.3328 (3.5616) grad_norm 1.2071 (1.2669) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][260/625] eta 0:02:20 lr 0.001927 wd 0.0500 time 0.3742 (0.3842) data time 0.0011 (0.0032) model time 0.3732 (0.3825) loss 4.0166 (3.5638) grad_norm 1.1222 (1.2706) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][270/625] eta 0:02:16 lr 0.001927 wd 0.0500 time 0.3701 (0.3839) data time 0.0011 (0.0031) model time 0.3690 (0.3822) loss 4.0575 (3.5671) grad_norm 0.8675 (1.2655) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][280/625] eta 0:02:12 lr 0.001927 wd 0.0500 time 0.3755 (0.3837) data time 0.0008 (0.0030) model time 0.3747 (0.3819) loss 3.9918 (3.5645) grad_norm 1.2633 (1.2622) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][290/625] eta 0:02:08 lr 0.001927 wd 0.0500 time 0.3768 (0.3835) data time 0.0011 (0.0030) model time 0.3757 (0.3817) loss 3.8458 (3.5665) grad_norm 1.2306 (1.2691) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][300/625] eta 0:02:04 lr 0.001927 wd 0.0500 time 0.3752 (0.3832) data time 0.0010 (0.0029) model time 0.3741 (0.3814) loss 3.7117 (3.5677) grad_norm 0.9508 (1.2675) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:36:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][310/625] eta 0:02:01 lr 0.001927 wd 0.0500 time 0.5699 (0.3850) data time 0.0011 (0.0028) model time 0.5688 (0.3836) loss 3.9733 (3.5611) grad_norm 1.3486 (1.2676) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][320/625] eta 0:01:57 lr 0.001927 wd 0.0500 time 0.3823 (0.3848) data time 0.0011 (0.0028) model time 0.3813 (0.3834) loss 3.5627 (3.5672) grad_norm 1.4313 (1.2685) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][330/625] eta 0:01:53 lr 0.001927 wd 0.0500 time 0.3663 (0.3846) data time 0.0008 (0.0027) model time 0.3654 (0.3831) loss 3.8181 (3.5704) grad_norm 1.2040 (1.2720) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][340/625] eta 0:01:49 lr 0.001927 wd 0.0500 time 0.3788 (0.3855) data time 0.0012 (0.0027) model time 0.3776 (0.3843) loss 3.8049 (3.5711) grad_norm 1.3329 (1.2701) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][350/625] eta 0:01:45 lr 0.001926 wd 0.0500 time 0.3759 (0.3853) data time 0.0011 (0.0026) model time 0.3748 (0.3840) loss 2.5798 (3.5743) grad_norm 1.1971 (1.2715) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][360/625] eta 0:01:42 lr 0.001926 wd 0.0500 time 0.3804 (0.3851) data time 0.0010 (0.0026) model time 0.3794 (0.3838) loss 4.1540 (3.5787) grad_norm 1.1264 (1.2709) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][370/625] eta 0:01:38 lr 0.001926 wd 0.0500 time 0.3822 (0.3849) data time 0.0011 (0.0026) model time 0.3812 (0.3835) loss 3.6831 (3.5762) grad_norm 2.2058 (1.2845) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][380/625] eta 0:01:34 lr 0.001926 wd 0.0500 time 0.3734 (0.3847) data time 0.0008 (0.0025) model time 0.3726 (0.3833) loss 3.0263 (3.5788) grad_norm 1.0059 (1.2818) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][390/625] eta 0:01:30 lr 0.001926 wd 0.0500 time 0.3817 (0.3845) data time 0.0012 (0.0025) model time 0.3805 (0.3831) loss 3.6663 (3.5745) grad_norm 1.4807 (1.2798) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][400/625] eta 0:01:26 lr 0.001926 wd 0.0500 time 0.3777 (0.3844) data time 0.0007 (0.0024) model time 0.3770 (0.3829) loss 3.9427 (3.5789) grad_norm 1.0040 (1.2771) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][410/625] eta 0:01:22 lr 0.001926 wd 0.0500 time 0.3735 (0.3842) data time 0.0011 (0.0024) model time 0.3724 (0.3827) loss 3.6005 (3.5757) grad_norm 1.0093 (1.2717) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][420/625] eta 0:01:18 lr 0.001926 wd 0.0500 time 0.3820 (0.3840) data time 0.0010 (0.0024) model time 0.3811 (0.3826) loss 3.4388 (3.5744) grad_norm 1.3845 (1.2684) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][430/625] eta 0:01:14 lr 0.001926 wd 0.0500 time 0.3754 (0.3839) data time 0.0008 (0.0024) model time 0.3746 (0.3824) loss 4.2434 (3.5793) grad_norm 1.0159 (1.2735) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][440/625] eta 0:01:10 lr 0.001926 wd 0.0500 time 0.3774 (0.3837) data time 0.0012 (0.0023) model time 0.3763 (0.3822) loss 3.1807 (3.5867) grad_norm 1.5054 (1.2757) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][450/625] eta 0:01:07 lr 0.001926 wd 0.0500 time 0.5672 (0.3840) data time 0.0010 (0.0023) model time 0.5662 (0.3825) loss 3.2857 (3.5904) grad_norm 1.0378 (1.2722) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][460/625] eta 0:01:03 lr 0.001926 wd 0.0500 time 0.3773 (0.3847) data time 0.0010 (0.0023) model time 0.3763 (0.3833) loss 3.5124 (3.5917) grad_norm 1.0634 (1.2701) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:37:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][470/625] eta 0:00:59 lr 0.001926 wd 0.0500 time 0.3755 (0.3846) data time 0.0008 (0.0023) model time 0.3747 (0.3832) loss 4.2140 (3.5924) grad_norm 1.6431 (1.2702) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][480/625] eta 0:00:55 lr 0.001926 wd 0.0500 time 0.3836 (0.3845) data time 0.0011 (0.0022) model time 0.3825 (0.3831) loss 3.2998 (3.5879) grad_norm 1.4201 (1.2696) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][490/625] eta 0:00:52 lr 0.001926 wd 0.0500 time 0.5854 (0.3853) data time 0.0008 (0.0022) model time 0.5846 (0.3840) loss 2.9805 (3.5806) grad_norm 1.3391 (1.2788) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][500/625] eta 0:00:48 lr 0.001925 wd 0.0500 time 0.3828 (0.3855) data time 0.0010 (0.0022) model time 0.3818 (0.3843) loss 3.3695 (3.5802) grad_norm 1.2292 (1.2815) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][510/625] eta 0:00:44 lr 0.001925 wd 0.0500 time 0.3810 (0.3854) data time 0.0009 (0.0022) model time 0.3802 (0.3842) loss 3.7050 (3.5802) grad_norm 1.0306 (1.2781) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][520/625] eta 0:00:40 lr 0.001925 wd 0.0500 time 0.3844 (0.3853) data time 0.0009 (0.0021) model time 0.3834 (0.3841) loss 2.7435 (3.5848) grad_norm 1.3602 (1.2753) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][530/625] eta 0:00:36 lr 0.001925 wd 0.0500 time 0.3792 (0.3851) data time 0.0011 (0.0021) model time 0.3781 (0.3839) loss 3.0262 (3.5783) grad_norm 1.2676 (1.2760) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][540/625] eta 0:00:32 lr 0.001925 wd 0.0500 time 0.3764 (0.3850) data time 0.0008 (0.0021) model time 0.3756 (0.3837) loss 3.4217 (3.5788) grad_norm 0.8755 (1.2735) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][550/625] eta 0:00:28 lr 0.001925 wd 0.0500 time 0.3776 (0.3848) data time 0.0010 (0.0021) model time 0.3766 (0.3835) loss 2.6856 (3.5795) grad_norm 1.0553 (1.2752) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][560/625] eta 0:00:25 lr 0.001925 wd 0.0500 time 0.3767 (0.3847) data time 0.0008 (0.0021) model time 0.3758 (0.3834) loss 4.4654 (3.5846) grad_norm 1.0607 (1.2762) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][570/625] eta 0:00:21 lr 0.001925 wd 0.0500 time 0.3793 (0.3845) data time 0.0011 (0.0021) model time 0.3782 (0.3832) loss 3.9626 (3.5837) grad_norm 0.9473 (1.2763) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][580/625] eta 0:00:17 lr 0.001925 wd 0.0500 time 0.3714 (0.3844) data time 0.0008 (0.0020) model time 0.3706 (0.3831) loss 4.8399 (3.5868) grad_norm 1.0643 (1.2747) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][590/625] eta 0:00:13 lr 0.001925 wd 0.0500 time 0.3735 (0.3843) data time 0.0011 (0.0020) model time 0.3723 (0.3830) loss 3.1779 (3.5828) grad_norm 1.0174 (1.2745) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][600/625] eta 0:00:09 lr 0.001925 wd 0.0500 time 0.3782 (0.3843) data time 0.0013 (0.0020) model time 0.3769 (0.3829) loss 2.6076 (3.5797) grad_norm 0.9042 (1.2747) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][610/625] eta 0:00:05 lr 0.001925 wd 0.0500 time 0.3647 (0.3848) data time 0.0008 (0.0020) model time 0.3638 (0.3835) loss 3.5376 (3.5780) grad_norm 1.3025 (1.2712) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [54/300][620/625] eta 0:00:01 lr 0.001925 wd 0.0500 time 0.3643 (0.3848) data time 0.0008 (0.0020) model time 0.3635 (0.3835) loss 3.2805 (3.5785) grad_norm 1.0541 (1.2684) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 54 training takes 0:04:00 [2024-08-26 06:38:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:38:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:38:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.6929 (0.6929) Acc@1 84.473 (84.473) Acc@5 96.680 (96.680) Mem 11225MB [2024-08-26 06:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.1982 (0.8420) Acc@1 71.973 (80.105) Acc@5 91.553 (95.597) Mem 11225MB [2024-08-26 06:39:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.3008 (1.0325) Acc@1 68.506 (75.881) Acc@5 90.479 (93.229) Mem 11225MB [2024-08-26 06:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.654 Acc@5 93.194 [2024-08-26 06:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.7% [2024-08-26 06:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 75.65% [2024-08-26 06:39:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 06:39:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 06:39:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 0.5723 (0.5723) Acc@1 86.621 (86.621) Acc@5 97.559 (97.559) Mem 11225MB [2024-08-26 06:39:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.161) Loss 0.9941 (0.7298) Acc@1 75.586 (82.089) Acc@5 93.359 (96.382) Mem 11225MB [2024-08-26 06:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.129) Loss 1.1895 (0.9095) Acc@1 70.166 (77.881) Acc@5 91.357 (94.275) Mem 11225MB [2024-08-26 06:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.559 Acc@5 94.248 [2024-08-26 06:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 77.6% [2024-08-26 06:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.56% [2024-08-26 06:39:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:39:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][0/625] eta 0:07:34 lr 0.001925 wd 0.0500 time 0.7267 (0.7267) data time 0.3868 (0.3868) model time 0.0000 (0.0000) loss 2.3699 (2.3699) grad_norm 1.0112 (1.0112) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][10/625] eta 0:04:12 lr 0.001925 wd 0.0500 time 0.3872 (0.4105) data time 0.0011 (0.0361) model time 0.0000 (0.0000) loss 4.1088 (3.4204) grad_norm 0.8659 (1.1207) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][20/625] eta 0:04:03 lr 0.001925 wd 0.0500 time 0.3695 (0.4028) data time 0.0011 (0.0195) model time 0.0000 (0.0000) loss 3.2585 (3.4727) grad_norm 0.9393 (1.1275) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][30/625] eta 0:03:58 lr 0.001924 wd 0.0500 time 0.3741 (0.4010) data time 0.0010 (0.0136) model time 0.0000 (0.0000) loss 3.8697 (3.6329) grad_norm 0.8647 (1.1274) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][40/625] eta 0:03:51 lr 0.001924 wd 0.0500 time 0.3771 (0.3957) data time 0.0011 (0.0106) model time 0.0000 (0.0000) loss 3.8495 (3.6249) grad_norm 1.4240 (1.2463) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][50/625] eta 0:03:45 lr 0.001924 wd 0.0500 time 0.3762 (0.3926) data time 0.0008 (0.0087) model time 0.0000 (0.0000) loss 3.0834 (3.7114) grad_norm 1.0408 (1.2530) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][60/625] eta 0:03:40 lr 0.001924 wd 0.0500 time 0.3727 (0.3903) data time 0.0008 (0.0075) model time 0.3719 (0.3777) loss 2.6132 (3.6405) grad_norm 1.4879 (1.2544) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][70/625] eta 0:03:35 lr 0.001924 wd 0.0500 time 0.3885 (0.3889) data time 0.0008 (0.0066) model time 0.3877 (0.3783) loss 3.7435 (3.6362) grad_norm 1.0085 (1.2576) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][80/625] eta 0:03:31 lr 0.001924 wd 0.0500 time 0.3814 (0.3876) data time 0.0011 (0.0060) model time 0.3804 (0.3780) loss 3.0955 (3.6310) grad_norm 0.9693 (1.2413) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][90/625] eta 0:03:27 lr 0.001924 wd 0.0500 time 0.3818 (0.3881) data time 0.0009 (0.0054) model time 0.3809 (0.3812) loss 3.8006 (3.6105) grad_norm 1.7846 (1.2692) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][100/625] eta 0:03:23 lr 0.001924 wd 0.0500 time 0.3741 (0.3877) data time 0.0012 (0.0053) model time 0.3729 (0.3810) loss 3.7442 (3.6208) grad_norm 0.8742 (1.2492) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][110/625] eta 0:03:19 lr 0.001924 wd 0.0500 time 0.3715 (0.3870) data time 0.0008 (0.0049) model time 0.3707 (0.3806) loss 4.5686 (3.6226) grad_norm 1.7084 (1.2371) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][120/625] eta 0:03:15 lr 0.001924 wd 0.0500 time 0.3768 (0.3862) data time 0.0011 (0.0046) model time 0.3757 (0.3799) loss 3.4608 (3.6076) grad_norm 1.0268 (1.2465) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:39:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][130/625] eta 0:03:10 lr 0.001924 wd 0.0500 time 0.3774 (0.3858) data time 0.0012 (0.0043) model time 0.3762 (0.3799) loss 4.0652 (3.5877) grad_norm 1.2106 (1.2634) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][140/625] eta 0:03:06 lr 0.001924 wd 0.0500 time 0.3740 (0.3852) data time 0.0008 (0.0041) model time 0.3732 (0.3795) loss 2.9281 (3.5820) grad_norm 1.2139 (1.2823) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][150/625] eta 0:03:02 lr 0.001924 wd 0.0500 time 0.3789 (0.3849) data time 0.0009 (0.0039) model time 0.3780 (0.3794) loss 3.9078 (3.5733) grad_norm 0.9427 (1.2783) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][160/625] eta 0:02:58 lr 0.001924 wd 0.0500 time 0.3834 (0.3845) data time 0.0011 (0.0038) model time 0.3823 (0.3793) loss 3.4984 (3.5756) grad_norm 2.2025 (1.2721) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][170/625] eta 0:02:55 lr 0.001923 wd 0.0500 time 0.3866 (0.3866) data time 0.0011 (0.0036) model time 0.3855 (0.3827) loss 3.4802 (3.5805) grad_norm 1.2386 (1.2704) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][180/625] eta 0:02:51 lr 0.001923 wd 0.0500 time 0.3745 (0.3862) data time 0.0008 (0.0035) model time 0.3737 (0.3823) loss 3.0408 (3.5699) grad_norm 1.5293 (1.2764) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][190/625] eta 0:02:47 lr 0.001923 wd 0.0500 time 0.3879 (0.3858) data time 0.0011 (0.0034) model time 0.3868 (0.3819) loss 2.4002 (3.5387) grad_norm 1.2256 (1.2780) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][200/625] eta 0:02:43 lr 0.001923 wd 0.0500 time 0.3737 (0.3854) data time 0.0012 (0.0033) model time 0.3725 (0.3815) loss 2.6419 (3.5379) grad_norm 1.4088 (1.2709) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][210/625] eta 0:02:40 lr 0.001923 wd 0.0500 time 0.3798 (0.3859) data time 0.0011 (0.0032) model time 0.3787 (0.3824) loss 3.3647 (3.5341) grad_norm 1.2303 (1.2722) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][220/625] eta 0:02:36 lr 0.001923 wd 0.0500 time 0.3780 (0.3856) data time 0.0008 (0.0031) model time 0.3772 (0.3821) loss 3.2062 (3.5307) grad_norm 1.1717 (1.2899) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][230/625] eta 0:02:32 lr 0.001923 wd 0.0500 time 0.6033 (0.3862) data time 0.0008 (0.0030) model time 0.6024 (0.3830) loss 3.9022 (3.5283) grad_norm 1.1575 (1.2806) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][240/625] eta 0:02:28 lr 0.001923 wd 0.0500 time 0.3767 (0.3858) data time 0.0011 (0.0029) model time 0.3756 (0.3826) loss 3.9621 (3.5345) grad_norm 1.4487 (1.2779) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][250/625] eta 0:02:24 lr 0.001923 wd 0.0500 time 0.3784 (0.3854) data time 0.0008 (0.0029) model time 0.3776 (0.3822) loss 2.4165 (3.5397) grad_norm 1.4052 (1.2758) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][260/625] eta 0:02:20 lr 0.001923 wd 0.0500 time 0.3709 (0.3851) data time 0.0009 (0.0028) model time 0.3700 (0.3819) loss 3.2455 (3.5328) grad_norm 1.0900 (1.2706) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][270/625] eta 0:02:16 lr 0.001923 wd 0.0500 time 0.3748 (0.3848) data time 0.0011 (0.0027) model time 0.3737 (0.3817) loss 3.8165 (3.5367) grad_norm 1.0665 (1.2676) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][280/625] eta 0:02:12 lr 0.001923 wd 0.0500 time 0.3838 (0.3847) data time 0.0012 (0.0027) model time 0.3826 (0.3816) loss 3.2612 (3.5492) grad_norm 1.2215 (1.2708) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:40:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][290/625] eta 0:02:09 lr 0.001923 wd 0.0500 time 0.3706 (0.3851) data time 0.0011 (0.0026) model time 0.3695 (0.3822) loss 3.8749 (3.5647) grad_norm 0.8968 (1.2681) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:41:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][300/625] eta 0:02:05 lr 0.001923 wd 0.0500 time 0.3740 (0.3849) data time 0.0012 (0.0026) model time 0.3728 (0.3820) loss 3.9544 (3.5768) grad_norm 0.8712 (1.2623) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:41:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][310/625] eta 0:02:01 lr 0.001923 wd 0.0500 time 0.3780 (0.3847) data time 0.0011 (0.0025) model time 0.3769 (0.3818) loss 3.0705 (3.5738) grad_norm 1.1770 (1.2596) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][320/625] eta 0:01:57 lr 0.001922 wd 0.0500 time 0.3738 (0.3845) data time 0.0011 (0.0025) model time 0.3728 (0.3817) loss 3.8563 (3.5722) grad_norm 1.1018 (1.2549) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:41:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][330/625] eta 0:01:53 lr 0.001922 wd 0.0500 time 0.3759 (0.3844) data time 0.0011 (0.0025) model time 0.3748 (0.3815) loss 3.5424 (3.5695) grad_norm 1.1213 (1.2640) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:41:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][340/625] eta 0:01:49 lr 0.001922 wd 0.0500 time 0.3752 (0.3843) data time 0.0011 (0.0024) model time 0.3741 (0.3815) loss 3.7543 (3.5656) grad_norm 1.2776 (1.2637) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 06:41:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][350/625] eta 0:01:45 lr 0.001922 wd 0.0500 time 0.3757 (0.3842) data time 0.0011 (0.0024) model time 0.3746 (0.3814) loss 2.9764 (3.5650) grad_norm 1.0077 (1.2628) loss_scale 8192.0000 (4107.6695) mem 11225MB [2024-08-26 06:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][360/625] eta 0:01:41 lr 0.001922 wd 0.0500 time 0.3790 (0.3841) data time 0.0009 (0.0024) model time 0.3781 (0.3814) loss 3.5747 (3.5692) grad_norm 1.2410 (1.2621) loss_scale 8192.0000 (4220.8089) mem 11225MB [2024-08-26 06:41:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][370/625] eta 0:01:37 lr 0.001922 wd 0.0500 time 0.3868 (0.3839) data time 0.0008 (0.0023) model time 0.3860 (0.3813) loss 2.5220 (3.5727) grad_norm 1.3546 (1.2681) loss_scale 8192.0000 (4327.8491) mem 11225MB [2024-08-26 06:41:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][380/625] eta 0:01:34 lr 0.001922 wd 0.0500 time 0.3706 (0.3837) data time 0.0009 (0.0023) model time 0.3697 (0.3811) loss 3.3633 (3.5689) grad_norm 1.2594 (1.2694) loss_scale 8192.0000 (4429.2703) mem 11225MB [2024-08-26 06:41:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][390/625] eta 0:01:30 lr 0.001922 wd 0.0500 time 0.3711 (0.3836) data time 0.0008 (0.0023) model time 0.3703 (0.3810) loss 3.5315 (3.5669) grad_norm 1.0833 (1.2707) loss_scale 8192.0000 (4525.5038) mem 11225MB [2024-08-26 06:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][400/625] eta 0:01:26 lr 0.001922 wd 0.0500 time 0.3965 (0.3835) data time 0.0010 (0.0022) model time 0.3955 (0.3809) loss 2.7253 (3.5632) grad_norm 0.9502 (1.2679) loss_scale 8192.0000 (4616.9377) mem 11225MB [2024-08-26 06:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][410/625] eta 0:01:22 lr 0.001922 wd 0.0500 time 0.3682 (0.3835) data time 0.0010 (0.0022) model time 0.3672 (0.3809) loss 3.6972 (3.5683) grad_norm 1.0585 (1.2681) loss_scale 8192.0000 (4703.9221) mem 11225MB [2024-08-26 06:41:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][420/625] eta 0:01:18 lr 0.001922 wd 0.0500 time 0.3798 (0.3848) data time 0.0008 (0.0022) model time 0.3789 (0.3824) loss 4.2339 (3.5655) grad_norm 1.1982 (1.2641) loss_scale 8192.0000 (4786.7743) mem 11225MB [2024-08-26 06:41:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][430/625] eta 0:01:15 lr 0.001922 wd 0.0500 time 0.3765 (0.3847) data time 0.0011 (0.0022) model time 0.3754 (0.3823) loss 3.9689 (3.5691) grad_norm 1.2228 (1.2609) loss_scale 8192.0000 (4865.7819) mem 11225MB [2024-08-26 06:41:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][440/625] eta 0:01:11 lr 0.001922 wd 0.0500 time 0.3823 (0.3846) data time 0.0011 (0.0022) model time 0.3812 (0.3822) loss 3.0225 (3.5682) grad_norm 0.8803 (1.2606) loss_scale 8192.0000 (4941.2063) mem 11225MB [2024-08-26 06:42:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][450/625] eta 0:01:07 lr 0.001922 wd 0.0500 time 0.3712 (0.3844) data time 0.0010 (0.0022) model time 0.3702 (0.3821) loss 2.6868 (3.5643) grad_norm 0.9001 (1.2573) loss_scale 8192.0000 (5013.2860) mem 11225MB [2024-08-26 06:42:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][460/625] eta 0:01:03 lr 0.001921 wd 0.0500 time 0.3729 (0.3843) data time 0.0011 (0.0021) model time 0.3718 (0.3819) loss 2.9907 (3.5644) grad_norm 1.0459 (1.2538) loss_scale 8192.0000 (5082.2386) mem 11225MB [2024-08-26 06:42:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][470/625] eta 0:00:59 lr 0.001921 wd 0.0500 time 0.3755 (0.3841) data time 0.0011 (0.0021) model time 0.3744 (0.3818) loss 3.2292 (3.5623) grad_norm 1.8940 (1.2534) loss_scale 8192.0000 (5148.2633) mem 11225MB [2024-08-26 06:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][480/625] eta 0:00:55 lr 0.001921 wd 0.0500 time 0.3833 (0.3840) data time 0.0008 (0.0021) model time 0.3825 (0.3817) loss 4.2857 (3.5635) grad_norm 1.3850 (1.2561) loss_scale 8192.0000 (5211.5426) mem 11225MB [2024-08-26 06:42:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][490/625] eta 0:00:51 lr 0.001921 wd 0.0500 time 0.3742 (0.3839) data time 0.0008 (0.0021) model time 0.3735 (0.3816) loss 2.7542 (3.5585) grad_norm 1.2028 (1.2557) loss_scale 8192.0000 (5272.2444) mem 11225MB [2024-08-26 06:42:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][500/625] eta 0:00:48 lr 0.001921 wd 0.0500 time 0.3822 (0.3843) data time 0.0008 (0.0020) model time 0.3814 (0.3821) loss 4.0230 (3.5569) grad_norm 1.2964 (1.2565) loss_scale 8192.0000 (5330.5230) mem 11225MB [2024-08-26 06:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][510/625] eta 0:00:44 lr 0.001921 wd 0.0500 time 0.3814 (0.3842) data time 0.0009 (0.0020) model time 0.3805 (0.3820) loss 3.6505 (3.5632) grad_norm 1.3339 (1.2550) loss_scale 8192.0000 (5386.5205) mem 11225MB [2024-08-26 06:42:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][520/625] eta 0:00:40 lr 0.001921 wd 0.0500 time 0.3750 (0.3841) data time 0.0010 (0.0020) model time 0.3739 (0.3818) loss 3.3190 (3.5685) grad_norm 1.2292 (1.2564) loss_scale 8192.0000 (5440.3685) mem 11225MB [2024-08-26 06:42:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][530/625] eta 0:00:36 lr 0.001921 wd 0.0500 time 0.3802 (0.3840) data time 0.0011 (0.0020) model time 0.3791 (0.3818) loss 2.2936 (3.5621) grad_norm 1.9733 (1.2585) loss_scale 8192.0000 (5492.1883) mem 11225MB [2024-08-26 06:42:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][540/625] eta 0:00:32 lr 0.001921 wd 0.0500 time 0.3739 (0.3846) data time 0.0009 (0.0020) model time 0.3730 (0.3825) loss 2.6765 (3.5563) grad_norm 1.3756 (1.2601) loss_scale 8192.0000 (5542.0924) mem 11225MB [2024-08-26 06:42:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][550/625] eta 0:00:28 lr 0.001921 wd 0.0500 time 0.3808 (0.3845) data time 0.0011 (0.0020) model time 0.3796 (0.3824) loss 3.6549 (3.5609) grad_norm 0.9896 (1.2586) loss_scale 8192.0000 (5590.1851) mem 11225MB [2024-08-26 06:42:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][560/625] eta 0:00:24 lr 0.001921 wd 0.0500 time 0.3854 (0.3844) data time 0.0011 (0.0020) model time 0.3843 (0.3824) loss 3.7828 (3.5609) grad_norm 1.4268 (1.2578) loss_scale 8192.0000 (5636.5633) mem 11225MB [2024-08-26 06:42:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][570/625] eta 0:00:21 lr 0.001921 wd 0.0500 time 0.3768 (0.3843) data time 0.0008 (0.0019) model time 0.3760 (0.3822) loss 4.4159 (3.5573) grad_norm 1.1958 (1.2564) loss_scale 8192.0000 (5681.3170) mem 11225MB [2024-08-26 06:42:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][580/625] eta 0:00:17 lr 0.001921 wd 0.0500 time 0.3795 (0.3842) data time 0.0008 (0.0019) model time 0.3787 (0.3822) loss 2.4521 (3.5511) grad_norm 1.3198 (1.2580) loss_scale 8192.0000 (5724.5301) mem 11225MB [2024-08-26 06:42:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][590/625] eta 0:00:13 lr 0.001921 wd 0.0500 time 0.3801 (0.3841) data time 0.0011 (0.0019) model time 0.3790 (0.3821) loss 2.5459 (3.5474) grad_norm 1.1719 (1.2566) loss_scale 8192.0000 (5766.2809) mem 11225MB [2024-08-26 06:42:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][600/625] eta 0:00:09 lr 0.001921 wd 0.0500 time 0.3750 (0.3843) data time 0.0008 (0.0019) model time 0.3742 (0.3823) loss 3.1497 (3.5400) grad_norm 1.2007 (1.2617) loss_scale 8192.0000 (5806.6423) mem 11225MB [2024-08-26 06:43:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][610/625] eta 0:00:05 lr 0.001920 wd 0.0500 time 0.3666 (0.3842) data time 0.0008 (0.0019) model time 0.3658 (0.3822) loss 3.7053 (3.5368) grad_norm 0.9321 (1.2602) loss_scale 8192.0000 (5845.6825) mem 11225MB [2024-08-26 06:43:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [55/300][620/625] eta 0:00:01 lr 0.001920 wd 0.0500 time 0.3628 (0.3840) data time 0.0006 (0.0019) model time 0.3622 (0.3819) loss 3.5442 (3.5360) grad_norm 0.9614 (1.2588) loss_scale 8192.0000 (5883.4654) mem 11225MB [2024-08-26 06:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 55 training takes 0:03:59 [2024-08-26 06:43:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:43:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:43:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 0.6831 (0.6831) Acc@1 85.205 (85.205) Acc@5 97.510 (97.510) Mem 11225MB [2024-08-26 06:43:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 1.2139 (0.8717) Acc@1 71.875 (80.398) Acc@5 91.650 (95.743) Mem 11225MB [2024-08-26 06:43:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.3057 (1.0642) Acc@1 70.117 (75.939) Acc@5 90.234 (93.308) Mem 11225MB [2024-08-26 06:43:10 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.616 Acc@5 93.242 [2024-08-26 06:43:10 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.6% [2024-08-26 06:43:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.907 (0.907) Loss 0.5723 (0.5723) Acc@1 86.670 (86.670) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 06:43:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.181) Loss 0.9917 (0.7281) Acc@1 75.879 (82.249) Acc@5 93.506 (96.373) Mem 11225MB [2024-08-26 06:43:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.139) Loss 1.1846 (0.9068) Acc@1 70.361 (78.039) Acc@5 91.406 (94.266) Mem 11225MB [2024-08-26 06:43:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.711 Acc@5 94.240 [2024-08-26 06:43:14 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 77.7% [2024-08-26 06:43:14 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.71% [2024-08-26 06:43:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:43:15 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:43:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][0/625] eta 0:08:16 lr 0.001920 wd 0.0500 time 0.7944 (0.7944) data time 0.4601 (0.4601) model time 0.0000 (0.0000) loss 3.2529 (3.2529) grad_norm 1.2644 (1.2644) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][10/625] eta 0:04:15 lr 0.001920 wd 0.0500 time 0.3812 (0.4161) data time 0.0011 (0.0432) model time 0.0000 (0.0000) loss 2.6805 (3.4667) grad_norm 1.0849 (1.3310) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][20/625] eta 0:04:00 lr 0.001920 wd 0.0500 time 0.3690 (0.3969) data time 0.0008 (0.0232) model time 0.0000 (0.0000) loss 4.3069 (3.5231) grad_norm 0.9240 (1.2207) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][30/625] eta 0:03:52 lr 0.001920 wd 0.0500 time 0.3789 (0.3911) data time 0.0011 (0.0161) model time 0.0000 (0.0000) loss 2.5914 (3.4917) grad_norm 1.3609 (1.2243) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][40/625] eta 0:03:47 lr 0.001920 wd 0.0500 time 0.3735 (0.3882) data time 0.0011 (0.0124) model time 0.0000 (0.0000) loss 3.8294 (3.5513) grad_norm 1.4083 (1.1868) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][50/625] eta 0:03:42 lr 0.001920 wd 0.0500 time 0.3793 (0.3863) data time 0.0011 (0.0102) model time 0.0000 (0.0000) loss 4.0267 (3.5186) grad_norm 1.0975 (1.1920) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][60/625] eta 0:03:37 lr 0.001920 wd 0.0500 time 0.3752 (0.3851) data time 0.0011 (0.0087) model time 0.3741 (0.3783) loss 3.8464 (3.6152) grad_norm 1.5259 (1.2139) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][70/625] eta 0:03:36 lr 0.001920 wd 0.0500 time 0.3806 (0.3899) data time 0.0008 (0.0076) model time 0.3797 (0.3982) loss 2.8754 (3.5999) grad_norm 1.4186 (1.2227) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][80/625] eta 0:03:31 lr 0.001920 wd 0.0500 time 0.3749 (0.3883) data time 0.0010 (0.0068) model time 0.3739 (0.3908) loss 2.9119 (3.5923) grad_norm 1.0682 (1.2451) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][90/625] eta 0:03:27 lr 0.001920 wd 0.0500 time 0.3833 (0.3870) data time 0.0011 (0.0062) model time 0.3822 (0.3869) loss 3.4919 (3.5721) grad_norm 1.9000 (1.2546) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][100/625] eta 0:03:23 lr 0.001920 wd 0.0500 time 0.3855 (0.3879) data time 0.0008 (0.0057) model time 0.3847 (0.3885) loss 3.9545 (3.5843) grad_norm 1.0551 (1.2448) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][110/625] eta 0:03:19 lr 0.001920 wd 0.0500 time 0.3727 (0.3868) data time 0.0012 (0.0053) model time 0.3716 (0.3860) loss 4.3521 (3.5670) grad_norm 1.1930 (1.2378) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][120/625] eta 0:03:14 lr 0.001919 wd 0.0500 time 0.3806 (0.3860) data time 0.0011 (0.0050) model time 0.3795 (0.3845) loss 3.6798 (3.5607) grad_norm 1.3074 (1.2507) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][130/625] eta 0:03:10 lr 0.001919 wd 0.0500 time 0.3874 (0.3853) data time 0.0008 (0.0047) model time 0.3866 (0.3835) loss 3.2486 (3.5386) grad_norm 0.9266 (1.2610) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][140/625] eta 0:03:06 lr 0.001919 wd 0.0500 time 0.3697 (0.3847) data time 0.0011 (0.0045) model time 0.3686 (0.3826) loss 3.7301 (3.5223) grad_norm 0.9661 (1.2620) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][150/625] eta 0:03:02 lr 0.001919 wd 0.0500 time 0.3741 (0.3842) data time 0.0008 (0.0043) model time 0.3733 (0.3819) loss 4.3119 (3.5301) grad_norm 2.1163 (1.2751) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][160/625] eta 0:02:58 lr 0.001919 wd 0.0500 time 0.3847 (0.3840) data time 0.0009 (0.0041) model time 0.3839 (0.3817) loss 3.3606 (3.5398) grad_norm 2.2920 (1.2771) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][170/625] eta 0:02:54 lr 0.001919 wd 0.0500 time 0.3742 (0.3836) data time 0.0012 (0.0039) model time 0.3729 (0.3812) loss 3.5718 (3.5501) grad_norm 1.2301 (1.2650) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][180/625] eta 0:02:50 lr 0.001919 wd 0.0500 time 0.3831 (0.3833) data time 0.0008 (0.0038) model time 0.3823 (0.3809) loss 4.2286 (3.5554) grad_norm 1.1441 (1.2662) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][190/625] eta 0:02:46 lr 0.001919 wd 0.0500 time 0.3800 (0.3831) data time 0.0011 (0.0036) model time 0.3789 (0.3808) loss 3.1157 (3.5614) grad_norm 1.2386 (1.2717) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][200/625] eta 0:02:42 lr 0.001919 wd 0.0500 time 0.3718 (0.3829) data time 0.0008 (0.0035) model time 0.3710 (0.3805) loss 3.5480 (3.5524) grad_norm 1.1694 (1.2783) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][210/625] eta 0:02:38 lr 0.001919 wd 0.0500 time 0.3768 (0.3826) data time 0.0008 (0.0034) model time 0.3760 (0.3802) loss 2.4770 (3.5521) grad_norm 1.1868 (1.2758) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][220/625] eta 0:02:35 lr 0.001919 wd 0.0500 time 0.3757 (0.3832) data time 0.0009 (0.0033) model time 0.3749 (0.3811) loss 4.1991 (3.5401) grad_norm 1.2135 (1.2711) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][230/625] eta 0:02:31 lr 0.001919 wd 0.0500 time 0.3746 (0.3830) data time 0.0011 (0.0032) model time 0.3735 (0.3808) loss 3.3572 (3.5410) grad_norm 1.5634 (1.2693) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][240/625] eta 0:02:27 lr 0.001919 wd 0.0500 time 0.3795 (0.3827) data time 0.0009 (0.0031) model time 0.3786 (0.3806) loss 2.9094 (3.5342) grad_norm 0.9408 (1.2687) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][250/625] eta 0:02:24 lr 0.001919 wd 0.0500 time 0.5568 (0.3841) data time 0.0010 (0.0031) model time 0.5558 (0.3823) loss 3.9892 (3.5427) grad_norm 1.0111 (1.2624) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][260/625] eta 0:02:20 lr 0.001919 wd 0.0500 time 0.3792 (0.3839) data time 0.0008 (0.0030) model time 0.3784 (0.3820) loss 4.1866 (3.5368) grad_norm 1.0277 (1.2601) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:44:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][270/625] eta 0:02:16 lr 0.001918 wd 0.0500 time 0.3753 (0.3837) data time 0.0010 (0.0029) model time 0.3743 (0.3819) loss 3.9052 (3.5395) grad_norm 1.1810 (1.2614) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][280/625] eta 0:02:12 lr 0.001918 wd 0.0500 time 0.3766 (0.3850) data time 0.0008 (0.0029) model time 0.3758 (0.3834) loss 2.5094 (3.5426) grad_norm 1.1442 (1.2696) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][290/625] eta 0:02:08 lr 0.001918 wd 0.0500 time 0.3777 (0.3847) data time 0.0008 (0.0028) model time 0.3768 (0.3832) loss 4.1302 (3.5473) grad_norm 1.3533 (1.2704) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][300/625] eta 0:02:04 lr 0.001918 wd 0.0500 time 0.3716 (0.3845) data time 0.0010 (0.0028) model time 0.3706 (0.3829) loss 4.3119 (3.5597) grad_norm 1.0197 (1.2707) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][310/625] eta 0:02:01 lr 0.001918 wd 0.0500 time 0.3800 (0.3843) data time 0.0011 (0.0027) model time 0.3789 (0.3827) loss 3.3364 (3.5576) grad_norm 1.3224 (1.2723) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][320/625] eta 0:01:57 lr 0.001918 wd 0.0500 time 0.3764 (0.3840) data time 0.0009 (0.0027) model time 0.3755 (0.3824) loss 4.2579 (3.5612) grad_norm 1.1422 (1.2722) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][330/625] eta 0:01:53 lr 0.001918 wd 0.0500 time 0.3793 (0.3840) data time 0.0008 (0.0026) model time 0.3785 (0.3824) loss 3.2912 (3.5432) grad_norm 1.5668 (1.2721) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][340/625] eta 0:01:49 lr 0.001918 wd 0.0500 time 0.3726 (0.3838) data time 0.0010 (0.0026) model time 0.3716 (0.3821) loss 4.2813 (3.5398) grad_norm 1.2114 (1.2703) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][350/625] eta 0:01:45 lr 0.001918 wd 0.0500 time 0.3839 (0.3836) data time 0.0011 (0.0025) model time 0.3828 (0.3820) loss 3.3203 (3.5339) grad_norm 1.1156 (1.2698) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][360/625] eta 0:01:41 lr 0.001918 wd 0.0500 time 0.3757 (0.3835) data time 0.0012 (0.0025) model time 0.3745 (0.3819) loss 3.3953 (3.5348) grad_norm 2.0769 (1.2746) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][370/625] eta 0:01:38 lr 0.001918 wd 0.0500 time 0.5844 (0.3846) data time 0.0011 (0.0024) model time 0.5833 (0.3831) loss 3.5828 (3.5293) grad_norm 1.2711 (1.2766) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][380/625] eta 0:01:34 lr 0.001918 wd 0.0500 time 0.3737 (0.3844) data time 0.0011 (0.0024) model time 0.3726 (0.3829) loss 3.0441 (3.5289) grad_norm 1.6651 (1.2835) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][390/625] eta 0:01:30 lr 0.001918 wd 0.0500 time 0.3774 (0.3842) data time 0.0011 (0.0024) model time 0.3762 (0.3827) loss 2.6435 (3.5316) grad_norm 0.9623 (1.2864) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][400/625] eta 0:01:26 lr 0.001918 wd 0.0500 time 0.3787 (0.3840) data time 0.0009 (0.0024) model time 0.3778 (0.3825) loss 3.2974 (3.5269) grad_norm 0.9755 (1.2866) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][410/625] eta 0:01:22 lr 0.001917 wd 0.0500 time 0.3763 (0.3839) data time 0.0011 (0.0023) model time 0.3752 (0.3823) loss 3.4366 (3.5275) grad_norm 1.0724 (1.2853) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:45:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][420/625] eta 0:01:18 lr 0.001917 wd 0.0500 time 0.3741 (0.3837) data time 0.0012 (0.0023) model time 0.3730 (0.3821) loss 2.8299 (3.5250) grad_norm 1.0319 (1.2848) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][430/625] eta 0:01:14 lr 0.001917 wd 0.0500 time 0.3784 (0.3840) data time 0.0011 (0.0023) model time 0.3774 (0.3825) loss 3.9254 (3.5227) grad_norm 1.2366 (1.2857) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][440/625] eta 0:01:11 lr 0.001917 wd 0.0500 time 0.3790 (0.3839) data time 0.0009 (0.0023) model time 0.3781 (0.3823) loss 3.0840 (3.5165) grad_norm 1.5627 (1.2822) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][450/625] eta 0:01:07 lr 0.001917 wd 0.0500 time 0.3777 (0.3837) data time 0.0009 (0.0022) model time 0.3768 (0.3822) loss 3.5147 (3.5151) grad_norm 1.1948 (1.2815) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][460/625] eta 0:01:03 lr 0.001917 wd 0.0500 time 0.3796 (0.3840) data time 0.0011 (0.0022) model time 0.3786 (0.3825) loss 3.8334 (3.5160) grad_norm 1.0966 (1.2761) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][470/625] eta 0:00:59 lr 0.001917 wd 0.0500 time 0.3713 (0.3844) data time 0.0011 (0.0022) model time 0.3702 (0.3829) loss 3.3837 (3.5139) grad_norm 1.1843 (1.2781) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][480/625] eta 0:00:55 lr 0.001917 wd 0.0500 time 0.3835 (0.3843) data time 0.0009 (0.0022) model time 0.3827 (0.3828) loss 3.6830 (3.5136) grad_norm 1.2412 (1.2767) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][490/625] eta 0:00:51 lr 0.001917 wd 0.0500 time 0.3788 (0.3842) data time 0.0009 (0.0021) model time 0.3779 (0.3827) loss 3.0098 (3.5127) grad_norm 0.9260 (1.2737) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][500/625] eta 0:00:48 lr 0.001917 wd 0.0500 time 0.3761 (0.3840) data time 0.0009 (0.0021) model time 0.3753 (0.3826) loss 3.2334 (3.5101) grad_norm 1.4322 (1.2770) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][510/625] eta 0:00:44 lr 0.001917 wd 0.0500 time 0.3815 (0.3840) data time 0.0011 (0.0021) model time 0.3804 (0.3825) loss 3.0574 (3.5085) grad_norm 1.0453 (1.2758) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][520/625] eta 0:00:40 lr 0.001917 wd 0.0500 time 0.3789 (0.3839) data time 0.0008 (0.0021) model time 0.3781 (0.3824) loss 3.8002 (3.5088) grad_norm 0.9320 (1.2723) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][530/625] eta 0:00:36 lr 0.001917 wd 0.0500 time 0.3832 (0.3838) data time 0.0009 (0.0021) model time 0.3824 (0.3823) loss 3.7593 (3.5089) grad_norm 1.7191 (1.2765) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][540/625] eta 0:00:32 lr 0.001917 wd 0.0500 time 0.3743 (0.3837) data time 0.0011 (0.0021) model time 0.3731 (0.3822) loss 3.9690 (3.5124) grad_norm 1.1111 (1.2765) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][550/625] eta 0:00:28 lr 0.001916 wd 0.0500 time 0.3772 (0.3836) data time 0.0009 (0.0020) model time 0.3764 (0.3821) loss 4.3036 (3.5138) grad_norm 1.1782 (1.2746) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][560/625] eta 0:00:24 lr 0.001916 wd 0.0500 time 0.3750 (0.3835) data time 0.0012 (0.0020) model time 0.3738 (0.3820) loss 2.9389 (3.5139) grad_norm 1.1759 (1.2781) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][570/625] eta 0:00:21 lr 0.001916 wd 0.0500 time 0.3753 (0.3834) data time 0.0008 (0.0020) model time 0.3744 (0.3819) loss 4.6390 (3.5135) grad_norm 1.4704 (1.2754) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][580/625] eta 0:00:17 lr 0.001916 wd 0.0500 time 0.5815 (0.3847) data time 0.0008 (0.0020) model time 0.5806 (0.3834) loss 2.4940 (3.5124) grad_norm 1.2731 (1.2793) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][590/625] eta 0:00:13 lr 0.001916 wd 0.0500 time 0.3755 (0.3846) data time 0.0008 (0.0020) model time 0.3746 (0.3833) loss 3.0169 (3.5119) grad_norm 1.5499 (1.2807) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][600/625] eta 0:00:09 lr 0.001916 wd 0.0500 time 0.3858 (0.3845) data time 0.0009 (0.0020) model time 0.3849 (0.3832) loss 2.9828 (3.5133) grad_norm 1.6611 (1.2802) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][610/625] eta 0:00:05 lr 0.001916 wd 0.0500 time 0.3661 (0.3847) data time 0.0008 (0.0020) model time 0.3653 (0.3834) loss 3.8969 (3.5112) grad_norm 1.2049 (1.2826) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [56/300][620/625] eta 0:00:01 lr 0.001916 wd 0.0500 time 0.3632 (0.3844) data time 0.0008 (0.0019) model time 0.3624 (0.3830) loss 3.5835 (3.5148) grad_norm 1.0765 (1.2796) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:15 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 56 training takes 0:04:00 [2024-08-26 06:47:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:47:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:47:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.508 (0.508) Loss 0.7373 (0.7373) Acc@1 84.619 (84.619) Acc@5 96.338 (96.338) Mem 11225MB [2024-08-26 06:47:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 1.1885 (0.8678) Acc@1 72.314 (80.278) Acc@5 91.357 (95.650) Mem 11225MB [2024-08-26 06:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.3682 (1.0595) Acc@1 68.799 (75.972) Acc@5 89.355 (93.262) Mem 11225MB [2024-08-26 06:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.816 Acc@5 93.218 [2024-08-26 06:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.8% [2024-08-26 06:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 75.82% [2024-08-26 06:47:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 06:47:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 06:47:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.508 (0.508) Loss 0.5713 (0.5713) Acc@1 86.670 (86.670) Acc@5 97.656 (97.656) Mem 11225MB [2024-08-26 06:47:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.9883 (0.7267) Acc@1 75.977 (82.355) Acc@5 93.652 (96.431) Mem 11225MB [2024-08-26 06:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.1807 (0.9044) Acc@1 70.459 (78.195) Acc@5 91.504 (94.336) Mem 11225MB [2024-08-26 06:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.883 Acc@5 94.302 [2024-08-26 06:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 77.9% [2024-08-26 06:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.88% [2024-08-26 06:47:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:47:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:47:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][0/625] eta 0:07:46 lr 0.001916 wd 0.0500 time 0.7461 (0.7461) data time 0.3943 (0.3943) model time 0.0000 (0.0000) loss 3.9537 (3.9537) grad_norm 0.9180 (0.9180) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][10/625] eta 0:04:12 lr 0.001916 wd 0.0500 time 0.3803 (0.4101) data time 0.0008 (0.0370) model time 0.0000 (0.0000) loss 2.9917 (3.3940) grad_norm 1.2116 (1.2447) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][20/625] eta 0:03:59 lr 0.001916 wd 0.0500 time 0.3814 (0.3954) data time 0.0011 (0.0199) model time 0.0000 (0.0000) loss 4.0203 (3.5077) grad_norm 3.6442 (1.4396) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][30/625] eta 0:03:51 lr 0.001916 wd 0.0500 time 0.3736 (0.3896) data time 0.0008 (0.0139) model time 0.0000 (0.0000) loss 3.1859 (3.4407) grad_norm 1.1307 (1.5059) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][40/625] eta 0:03:46 lr 0.001916 wd 0.0500 time 0.3758 (0.3865) data time 0.0008 (0.0108) model time 0.0000 (0.0000) loss 4.6650 (3.5411) grad_norm 0.9518 (1.4815) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][50/625] eta 0:03:40 lr 0.001916 wd 0.0500 time 0.3720 (0.3842) data time 0.0011 (0.0089) model time 0.0000 (0.0000) loss 3.3633 (3.5306) grad_norm 0.9989 (1.4223) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][60/625] eta 0:03:36 lr 0.001915 wd 0.0500 time 0.3809 (0.3830) data time 0.0012 (0.0076) model time 0.3797 (0.3757) loss 3.4691 (3.5306) grad_norm 1.5481 (1.3958) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][70/625] eta 0:03:33 lr 0.001915 wd 0.0500 time 0.3806 (0.3848) data time 0.0011 (0.0067) model time 0.3795 (0.3849) loss 3.7820 (3.5454) grad_norm 1.7596 (1.3754) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][80/625] eta 0:03:29 lr 0.001915 wd 0.0500 time 0.3688 (0.3837) data time 0.0008 (0.0060) model time 0.3680 (0.3818) loss 3.7874 (3.5884) grad_norm 1.1488 (1.3552) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][90/625] eta 0:03:24 lr 0.001915 wd 0.0500 time 0.3807 (0.3830) data time 0.0011 (0.0055) model time 0.3796 (0.3802) loss 3.7871 (3.5878) grad_norm 1.1051 (1.3266) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][100/625] eta 0:03:22 lr 0.001915 wd 0.0500 time 0.3720 (0.3859) data time 0.0012 (0.0051) model time 0.3707 (0.3865) loss 3.2002 (3.5945) grad_norm 1.2609 (1.3049) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][110/625] eta 0:03:18 lr 0.001915 wd 0.0500 time 0.3771 (0.3851) data time 0.0007 (0.0047) model time 0.3763 (0.3846) loss 2.5885 (3.6147) grad_norm 1.2476 (1.3109) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][120/625] eta 0:03:14 lr 0.001915 wd 0.0500 time 0.3827 (0.3845) data time 0.0009 (0.0044) model time 0.3818 (0.3836) loss 3.8057 (3.5987) grad_norm 1.2247 (1.3158) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][130/625] eta 0:03:10 lr 0.001915 wd 0.0500 time 0.3754 (0.3842) data time 0.0010 (0.0042) model time 0.3744 (0.3829) loss 2.8391 (3.5947) grad_norm 2.2554 (1.3198) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][140/625] eta 0:03:06 lr 0.001915 wd 0.0500 time 0.3581 (0.3837) data time 0.0012 (0.0040) model time 0.3569 (0.3822) loss 3.8412 (3.5793) grad_norm 1.1082 (1.3320) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][150/625] eta 0:03:02 lr 0.001915 wd 0.0500 time 0.3796 (0.3833) data time 0.0008 (0.0038) model time 0.3788 (0.3816) loss 2.5042 (3.5747) grad_norm 1.2567 (1.3367) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][160/625] eta 0:02:59 lr 0.001915 wd 0.0500 time 0.3943 (0.3857) data time 0.0008 (0.0036) model time 0.3935 (0.3853) loss 3.8816 (3.5794) grad_norm 1.1250 (1.3189) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][170/625] eta 0:02:55 lr 0.001915 wd 0.0500 time 0.3767 (0.3853) data time 0.0009 (0.0035) model time 0.3758 (0.3846) loss 3.0432 (3.5784) grad_norm 1.4019 (1.3111) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][180/625] eta 0:02:51 lr 0.001915 wd 0.0500 time 0.3843 (0.3849) data time 0.0012 (0.0033) model time 0.3832 (0.3840) loss 3.7021 (3.5752) grad_norm 1.5949 (1.3122) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][190/625] eta 0:02:47 lr 0.001915 wd 0.0500 time 0.3760 (0.3855) data time 0.0011 (0.0032) model time 0.3749 (0.3849) loss 2.9714 (3.5719) grad_norm 1.0162 (1.3072) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][200/625] eta 0:02:43 lr 0.001914 wd 0.0500 time 0.3833 (0.3852) data time 0.0011 (0.0031) model time 0.3822 (0.3844) loss 2.9942 (3.5788) grad_norm 1.3517 (1.3113) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][210/625] eta 0:02:39 lr 0.001914 wd 0.0500 time 0.3737 (0.3848) data time 0.0009 (0.0030) model time 0.3727 (0.3839) loss 4.1202 (3.5867) grad_norm 1.0557 (1.3148) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][220/625] eta 0:02:35 lr 0.001914 wd 0.0500 time 0.3840 (0.3845) data time 0.0009 (0.0029) model time 0.3831 (0.3834) loss 3.8793 (3.5829) grad_norm 0.9833 (1.3219) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][230/625] eta 0:02:31 lr 0.001914 wd 0.0500 time 0.3705 (0.3842) data time 0.0009 (0.0029) model time 0.3696 (0.3830) loss 3.6528 (3.5708) grad_norm 1.1275 (1.3242) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:48:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][240/625] eta 0:02:27 lr 0.001914 wd 0.0500 time 0.3793 (0.3839) data time 0.0012 (0.0028) model time 0.3782 (0.3827) loss 3.5906 (3.5800) grad_norm 1.1023 (1.3167) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][250/625] eta 0:02:23 lr 0.001914 wd 0.0500 time 0.3870 (0.3838) data time 0.0008 (0.0027) model time 0.3862 (0.3825) loss 4.0077 (3.5799) grad_norm 1.1394 (1.3150) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][260/625] eta 0:02:19 lr 0.001914 wd 0.0500 time 0.3754 (0.3836) data time 0.0011 (0.0027) model time 0.3743 (0.3822) loss 3.4622 (3.5807) grad_norm 1.6642 (1.3142) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][270/625] eta 0:02:16 lr 0.001914 wd 0.0500 time 0.3791 (0.3834) data time 0.0010 (0.0026) model time 0.3781 (0.3821) loss 3.8752 (3.5790) grad_norm 1.1790 (1.3098) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][280/625] eta 0:02:12 lr 0.001914 wd 0.0500 time 0.4083 (0.3833) data time 0.0012 (0.0025) model time 0.4072 (0.3820) loss 3.5882 (3.5787) grad_norm 0.7723 (1.3097) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][290/625] eta 0:02:08 lr 0.001914 wd 0.0500 time 0.3733 (0.3832) data time 0.0008 (0.0025) model time 0.3725 (0.3819) loss 2.0727 (3.5718) grad_norm 2.0117 (1.3091) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][300/625] eta 0:02:04 lr 0.001914 wd 0.0500 time 0.3796 (0.3830) data time 0.0008 (0.0025) model time 0.3787 (0.3816) loss 4.0710 (3.5642) grad_norm 0.9136 (1.3063) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][310/625] eta 0:02:01 lr 0.001914 wd 0.0500 time 0.3777 (0.3842) data time 0.0011 (0.0024) model time 0.3766 (0.3831) loss 3.7675 (3.5627) grad_norm 1.1705 (1.2999) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][320/625] eta 0:01:57 lr 0.001914 wd 0.0500 time 0.3803 (0.3841) data time 0.0009 (0.0024) model time 0.3794 (0.3829) loss 4.2633 (3.5630) grad_norm 1.0322 (1.2934) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][330/625] eta 0:01:53 lr 0.001914 wd 0.0500 time 0.3736 (0.3839) data time 0.0010 (0.0023) model time 0.3726 (0.3827) loss 3.3389 (3.5600) grad_norm 1.2503 (1.2917) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][340/625] eta 0:01:49 lr 0.001913 wd 0.0500 time 0.3748 (0.3842) data time 0.0009 (0.0023) model time 0.3738 (0.3830) loss 4.2941 (3.5615) grad_norm 1.4779 (1.2895) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][350/625] eta 0:01:45 lr 0.001913 wd 0.0500 time 0.3770 (0.3847) data time 0.0011 (0.0023) model time 0.3759 (0.3836) loss 3.4405 (3.5617) grad_norm 1.5386 (1.2948) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][360/625] eta 0:01:41 lr 0.001913 wd 0.0500 time 0.3735 (0.3844) data time 0.0011 (0.0023) model time 0.3724 (0.3833) loss 3.9086 (3.5678) grad_norm 0.9650 (1.2897) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][370/625] eta 0:01:38 lr 0.001913 wd 0.0500 time 0.3888 (0.3843) data time 0.0011 (0.0022) model time 0.3876 (0.3832) loss 2.3366 (3.5709) grad_norm 1.8237 (1.2884) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][380/625] eta 0:01:34 lr 0.001913 wd 0.0500 time 0.3845 (0.3842) data time 0.0011 (0.0022) model time 0.3834 (0.3831) loss 3.7696 (3.5685) grad_norm 1.7976 (1.2929) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][390/625] eta 0:01:30 lr 0.001913 wd 0.0500 time 0.3755 (0.3841) data time 0.0011 (0.0022) model time 0.3744 (0.3829) loss 4.3347 (3.5733) grad_norm 1.1966 (1.2908) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:49:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][400/625] eta 0:01:26 lr 0.001913 wd 0.0500 time 0.3744 (0.3843) data time 0.0011 (0.0021) model time 0.3733 (0.3832) loss 3.7346 (3.5733) grad_norm 1.2730 (1.2850) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][410/625] eta 0:01:22 lr 0.001913 wd 0.0500 time 0.3715 (0.3842) data time 0.0012 (0.0021) model time 0.3702 (0.3830) loss 3.9128 (3.5711) grad_norm 0.9453 (1.2812) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][420/625] eta 0:01:18 lr 0.001913 wd 0.0500 time 0.3929 (0.3841) data time 0.0011 (0.0021) model time 0.3918 (0.3829) loss 3.5816 (3.5682) grad_norm 1.1323 (1.2781) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][430/625] eta 0:01:14 lr 0.001913 wd 0.0500 time 0.3780 (0.3840) data time 0.0010 (0.0021) model time 0.3769 (0.3828) loss 2.9349 (3.5665) grad_norm 1.8976 (1.2826) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][440/625] eta 0:01:11 lr 0.001913 wd 0.0500 time 0.3796 (0.3839) data time 0.0008 (0.0021) model time 0.3787 (0.3827) loss 3.4831 (3.5646) grad_norm 1.6744 (1.2808) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][450/625] eta 0:01:07 lr 0.001913 wd 0.0500 time 0.3771 (0.3838) data time 0.0009 (0.0020) model time 0.3762 (0.3826) loss 3.7920 (3.5567) grad_norm 0.9254 (1.2777) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][460/625] eta 0:01:03 lr 0.001913 wd 0.0500 time 0.3745 (0.3841) data time 0.0011 (0.0020) model time 0.3734 (0.3830) loss 3.0756 (3.5517) grad_norm 1.3284 (1.2802) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][470/625] eta 0:00:59 lr 0.001913 wd 0.0500 time 0.3803 (0.3845) data time 0.0010 (0.0020) model time 0.3792 (0.3833) loss 3.6328 (3.5524) grad_norm 1.1037 (1.2796) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][480/625] eta 0:00:55 lr 0.001912 wd 0.0500 time 0.3806 (0.3844) data time 0.0008 (0.0020) model time 0.3798 (0.3832) loss 3.6366 (3.5587) grad_norm 1.0183 (1.2762) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][490/625] eta 0:00:51 lr 0.001912 wd 0.0500 time 0.3726 (0.3851) data time 0.0010 (0.0020) model time 0.3715 (0.3841) loss 3.7410 (3.5644) grad_norm 1.1112 (1.2746) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][500/625] eta 0:00:48 lr 0.001912 wd 0.0500 time 0.3812 (0.3850) data time 0.0008 (0.0019) model time 0.3804 (0.3839) loss 3.9110 (3.5674) grad_norm 1.4541 (1.2724) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][510/625] eta 0:00:44 lr 0.001912 wd 0.0500 time 0.3713 (0.3849) data time 0.0008 (0.0019) model time 0.3705 (0.3838) loss 4.2137 (3.5693) grad_norm 1.0465 (1.2788) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][520/625] eta 0:00:40 lr 0.001912 wd 0.0500 time 0.3821 (0.3847) data time 0.0008 (0.0019) model time 0.3813 (0.3836) loss 4.1278 (3.5682) grad_norm 1.0324 (1.2763) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][530/625] eta 0:00:36 lr 0.001912 wd 0.0500 time 0.3721 (0.3846) data time 0.0011 (0.0019) model time 0.3711 (0.3835) loss 3.2947 (3.5702) grad_norm 2.0280 (1.2781) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][540/625] eta 0:00:32 lr 0.001912 wd 0.0500 time 0.3774 (0.3846) data time 0.0008 (0.0019) model time 0.3766 (0.3834) loss 4.2368 (3.5719) grad_norm 0.9623 (1.2777) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:50:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][550/625] eta 0:00:28 lr 0.001912 wd 0.0500 time 0.3791 (0.3844) data time 0.0011 (0.0019) model time 0.3781 (0.3833) loss 3.1705 (3.5719) grad_norm 0.8967 (1.2745) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][560/625] eta 0:00:24 lr 0.001912 wd 0.0500 time 0.3830 (0.3843) data time 0.0008 (0.0019) model time 0.3822 (0.3832) loss 3.9665 (3.5697) grad_norm 1.0457 (1.2727) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][570/625] eta 0:00:21 lr 0.001912 wd 0.0500 time 0.3774 (0.3842) data time 0.0011 (0.0019) model time 0.3764 (0.3830) loss 2.8659 (3.5668) grad_norm 1.4708 (1.2719) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][580/625] eta 0:00:17 lr 0.001912 wd 0.0500 time 0.3806 (0.3841) data time 0.0010 (0.0019) model time 0.3795 (0.3829) loss 4.0942 (3.5632) grad_norm 0.8615 (1.2671) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][590/625] eta 0:00:13 lr 0.001912 wd 0.0500 time 0.3769 (0.3842) data time 0.0010 (0.0018) model time 0.3759 (0.3830) loss 3.2783 (3.5658) grad_norm 1.4893 (1.2654) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][600/625] eta 0:00:09 lr 0.001912 wd 0.0500 time 0.3713 (0.3841) data time 0.0011 (0.0018) model time 0.3702 (0.3829) loss 3.6513 (3.5672) grad_norm 1.0500 (1.2663) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][610/625] eta 0:00:05 lr 0.001911 wd 0.0500 time 0.3646 (0.3847) data time 0.0005 (0.0018) model time 0.3641 (0.3835) loss 4.1611 (3.5722) grad_norm 1.3241 (1.2670) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [57/300][620/625] eta 0:00:01 lr 0.001911 wd 0.0500 time 0.3671 (0.3844) data time 0.0008 (0.0019) model time 0.3664 (0.3832) loss 3.4909 (3.5733) grad_norm 1.6652 (1.2722) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:25 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 57 training takes 0:04:00 [2024-08-26 06:51:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:51:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:51:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.7002 (0.7002) Acc@1 84.033 (84.033) Acc@5 97.217 (97.217) Mem 11225MB [2024-08-26 06:51:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 1.2129 (0.8676) Acc@1 72.949 (80.083) Acc@5 91.602 (95.712) Mem 11225MB [2024-08-26 06:51:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.3340 (1.0429) Acc@1 68.994 (76.163) Acc@5 89.648 (93.424) Mem 11225MB [2024-08-26 06:51:29 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.882 Acc@5 93.388 [2024-08-26 06:51:29 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.9% [2024-08-26 06:51:29 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 75.88% [2024-08-26 06:51:29 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 06:51:30 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 06:51:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.5713 (0.5713) Acc@1 86.865 (86.865) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 06:51:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 0.9873 (0.7255) Acc@1 76.074 (82.440) Acc@5 93.750 (96.502) Mem 11225MB [2024-08-26 06:51:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.1748 (0.9020) Acc@1 70.654 (78.271) Acc@5 91.602 (94.389) Mem 11225MB [2024-08-26 06:51:33 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.975 Acc@5 94.362 [2024-08-26 06:51:33 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.0% [2024-08-26 06:51:33 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 77.98% [2024-08-26 06:51:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:51:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:51:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][0/625] eta 0:08:00 lr 0.001911 wd 0.0500 time 0.7691 (0.7691) data time 0.4377 (0.4377) model time 0.0000 (0.0000) loss 4.4242 (4.4242) grad_norm 1.6022 (1.6022) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][10/625] eta 0:04:13 lr 0.001911 wd 0.0500 time 0.3788 (0.4128) data time 0.0011 (0.0409) model time 0.0000 (0.0000) loss 2.1047 (3.4294) grad_norm 1.0670 (1.3934) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][20/625] eta 0:03:59 lr 0.001911 wd 0.0500 time 0.3797 (0.3960) data time 0.0011 (0.0220) model time 0.0000 (0.0000) loss 3.9729 (3.5572) grad_norm 1.2465 (1.3770) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][30/625] eta 0:04:03 lr 0.001911 wd 0.0500 time 0.3788 (0.4101) data time 0.0009 (0.0153) model time 0.0000 (0.0000) loss 4.2512 (3.5091) grad_norm 1.4418 (1.3330) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][40/625] eta 0:03:55 lr 0.001911 wd 0.0500 time 0.3816 (0.4024) data time 0.0012 (0.0118) model time 0.0000 (0.0000) loss 3.5533 (3.4871) grad_norm 1.0924 (1.2928) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][50/625] eta 0:03:49 lr 0.001911 wd 0.0500 time 0.3804 (0.3984) data time 0.0008 (0.0097) model time 0.0000 (0.0000) loss 4.1979 (3.5311) grad_norm 1.0803 (1.2630) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:51:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][60/625] eta 0:03:43 lr 0.001911 wd 0.0500 time 0.3797 (0.3949) data time 0.0008 (0.0083) model time 0.3789 (0.3763) loss 4.4223 (3.5701) grad_norm 1.3688 (1.2687) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][70/625] eta 0:03:38 lr 0.001911 wd 0.0500 time 0.3863 (0.3928) data time 0.0011 (0.0073) model time 0.3852 (0.3775) loss 3.8142 (3.5840) grad_norm 1.0675 (1.2671) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][80/625] eta 0:03:32 lr 0.001911 wd 0.0500 time 0.3812 (0.3908) data time 0.0011 (0.0066) model time 0.3801 (0.3767) loss 3.0992 (3.5758) grad_norm 1.2706 (1.2647) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][90/625] eta 0:03:28 lr 0.001911 wd 0.0500 time 0.3897 (0.3897) data time 0.0009 (0.0060) model time 0.3889 (0.3775) loss 3.7478 (3.5424) grad_norm 1.2794 (1.2628) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][100/625] eta 0:03:24 lr 0.001911 wd 0.0500 time 0.3870 (0.3887) data time 0.0008 (0.0055) model time 0.3861 (0.3777) loss 4.2042 (3.5770) grad_norm 1.1547 (1.2533) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][110/625] eta 0:03:19 lr 0.001911 wd 0.0500 time 0.3793 (0.3876) data time 0.0009 (0.0051) model time 0.3785 (0.3774) loss 3.8491 (3.5429) grad_norm 1.9482 (1.2722) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][120/625] eta 0:03:15 lr 0.001910 wd 0.0500 time 0.3772 (0.3870) data time 0.0011 (0.0048) model time 0.3761 (0.3775) loss 3.8550 (3.5538) grad_norm 0.8692 (1.2797) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][130/625] eta 0:03:11 lr 0.001910 wd 0.0500 time 0.3818 (0.3864) data time 0.0011 (0.0045) model time 0.3807 (0.3777) loss 3.8723 (3.5427) grad_norm 1.2853 (1.2836) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][140/625] eta 0:03:07 lr 0.001910 wd 0.0500 time 0.3767 (0.3859) data time 0.0011 (0.0043) model time 0.3756 (0.3778) loss 3.7242 (3.5522) grad_norm 1.5156 (1.2884) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][150/625] eta 0:03:03 lr 0.001910 wd 0.0500 time 0.3741 (0.3854) data time 0.0014 (0.0041) model time 0.3727 (0.3776) loss 3.6572 (3.5578) grad_norm 1.3106 (1.2731) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][160/625] eta 0:02:58 lr 0.001910 wd 0.0500 time 0.3774 (0.3849) data time 0.0010 (0.0039) model time 0.3764 (0.3775) loss 3.9338 (3.5621) grad_norm 1.1747 (1.2701) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][170/625] eta 0:02:55 lr 0.001910 wd 0.0500 time 0.3765 (0.3867) data time 0.0008 (0.0037) model time 0.3757 (0.3806) loss 2.9497 (3.5581) grad_norm 1.1739 (1.2632) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][180/625] eta 0:02:51 lr 0.001910 wd 0.0500 time 0.3785 (0.3862) data time 0.0008 (0.0036) model time 0.3777 (0.3803) loss 3.0555 (3.5465) grad_norm 0.8432 (1.2621) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][190/625] eta 0:02:47 lr 0.001910 wd 0.0500 time 0.3791 (0.3858) data time 0.0010 (0.0034) model time 0.3781 (0.3801) loss 3.8845 (3.5497) grad_norm 1.1308 (1.2691) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][200/625] eta 0:02:43 lr 0.001910 wd 0.0500 time 0.3723 (0.3854) data time 0.0011 (0.0033) model time 0.3712 (0.3799) loss 2.9876 (3.5470) grad_norm 1.3097 (1.2694) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][210/625] eta 0:02:39 lr 0.001910 wd 0.0500 time 0.3837 (0.3851) data time 0.0008 (0.0032) model time 0.3829 (0.3797) loss 3.0557 (3.5357) grad_norm 1.7101 (1.3047) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][220/625] eta 0:02:35 lr 0.001910 wd 0.0500 time 0.3774 (0.3845) data time 0.0011 (0.0031) model time 0.3763 (0.3792) loss 3.8454 (3.5432) grad_norm 1.3527 (1.3055) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][230/625] eta 0:02:31 lr 0.001910 wd 0.0500 time 0.3664 (0.3841) data time 0.0009 (0.0030) model time 0.3656 (0.3789) loss 3.5852 (3.5485) grad_norm 0.9993 (1.3000) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][240/625] eta 0:02:28 lr 0.001910 wd 0.0500 time 0.3746 (0.3864) data time 0.0008 (0.0030) model time 0.3738 (0.3821) loss 3.3556 (3.5467) grad_norm 1.0917 (1.2923) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][250/625] eta 0:02:24 lr 0.001910 wd 0.0500 time 0.3789 (0.3861) data time 0.0011 (0.0029) model time 0.3778 (0.3818) loss 3.9929 (3.5557) grad_norm 0.8594 (1.2893) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][260/625] eta 0:02:20 lr 0.001909 wd 0.0500 time 0.3785 (0.3858) data time 0.0008 (0.0028) model time 0.3777 (0.3816) loss 2.4973 (3.5463) grad_norm 1.8015 (1.2935) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][270/625] eta 0:02:16 lr 0.001909 wd 0.0500 time 0.3744 (0.3855) data time 0.0011 (0.0028) model time 0.3734 (0.3814) loss 3.7546 (3.5456) grad_norm 1.1425 (1.2928) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][280/625] eta 0:02:12 lr 0.001909 wd 0.0500 time 0.3981 (0.3854) data time 0.0011 (0.0027) model time 0.3970 (0.3813) loss 3.8783 (3.5518) grad_norm 1.4523 (1.2874) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][290/625] eta 0:02:09 lr 0.001909 wd 0.0500 time 0.3790 (0.3864) data time 0.0011 (0.0026) model time 0.3779 (0.3828) loss 3.6903 (3.5652) grad_norm 1.3405 (1.2845) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][300/625] eta 0:02:05 lr 0.001909 wd 0.0500 time 0.3761 (0.3861) data time 0.0009 (0.0026) model time 0.3752 (0.3825) loss 4.1892 (3.5611) grad_norm 1.2545 (1.2831) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][310/625] eta 0:02:01 lr 0.001909 wd 0.0500 time 0.3804 (0.3858) data time 0.0011 (0.0025) model time 0.3793 (0.3822) loss 4.1171 (3.5667) grad_norm 0.9965 (1.2803) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][320/625] eta 0:01:57 lr 0.001909 wd 0.0500 time 0.3767 (0.3856) data time 0.0008 (0.0025) model time 0.3758 (0.3821) loss 4.1510 (3.5690) grad_norm 1.4732 (1.2833) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][330/625] eta 0:01:53 lr 0.001909 wd 0.0500 time 0.3802 (0.3853) data time 0.0012 (0.0025) model time 0.3790 (0.3818) loss 3.7347 (3.5672) grad_norm 0.9912 (1.2833) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][340/625] eta 0:01:49 lr 0.001909 wd 0.0500 time 0.3758 (0.3850) data time 0.0011 (0.0024) model time 0.3747 (0.3815) loss 3.2703 (3.5609) grad_norm 0.9610 (1.2809) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][350/625] eta 0:01:45 lr 0.001909 wd 0.0500 time 0.3679 (0.3848) data time 0.0008 (0.0024) model time 0.3671 (0.3814) loss 3.6439 (3.5671) grad_norm 1.1450 (1.2820) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][360/625] eta 0:01:41 lr 0.001909 wd 0.0500 time 0.3803 (0.3846) data time 0.0008 (0.0024) model time 0.3796 (0.3812) loss 4.0777 (3.5720) grad_norm 1.2831 (1.2827) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:53:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][370/625] eta 0:01:38 lr 0.001909 wd 0.0500 time 0.3756 (0.3844) data time 0.0011 (0.0023) model time 0.3745 (0.3811) loss 3.1629 (3.5698) grad_norm 1.1001 (1.2837) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][380/625] eta 0:01:34 lr 0.001909 wd 0.0500 time 0.3775 (0.3842) data time 0.0009 (0.0023) model time 0.3766 (0.3809) loss 2.7148 (3.5652) grad_norm 1.2356 (1.2803) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][390/625] eta 0:01:30 lr 0.001908 wd 0.0500 time 0.3750 (0.3840) data time 0.0011 (0.0023) model time 0.3739 (0.3807) loss 3.8163 (3.5667) grad_norm 1.6026 (1.2834) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][400/625] eta 0:01:26 lr 0.001908 wd 0.0500 time 0.3765 (0.3838) data time 0.0011 (0.0022) model time 0.3755 (0.3806) loss 3.3525 (3.5673) grad_norm 1.2261 (1.2865) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][410/625] eta 0:01:22 lr 0.001908 wd 0.0500 time 0.3720 (0.3837) data time 0.0011 (0.0022) model time 0.3709 (0.3805) loss 3.8073 (3.5585) grad_norm 0.9519 (1.2835) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][420/625] eta 0:01:18 lr 0.001908 wd 0.0500 time 0.8361 (0.3852) data time 0.0011 (0.0022) model time 0.8350 (0.3822) loss 3.0899 (3.5534) grad_norm 1.4085 (1.2934) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][430/625] eta 0:01:15 lr 0.001908 wd 0.0500 time 0.3759 (0.3849) data time 0.0009 (0.0022) model time 0.3751 (0.3820) loss 4.4121 (3.5473) grad_norm 1.1945 (1.2971) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][440/625] eta 0:01:11 lr 0.001908 wd 0.0500 time 0.3821 (0.3848) data time 0.0010 (0.0021) model time 0.3811 (0.3819) loss 2.8248 (3.5405) grad_norm 1.2608 (1.2957) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][450/625] eta 0:01:07 lr 0.001908 wd 0.0500 time 0.3780 (0.3846) data time 0.0009 (0.0021) model time 0.3772 (0.3817) loss 3.6139 (3.5458) grad_norm 1.0206 (1.2968) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][460/625] eta 0:01:03 lr 0.001908 wd 0.0500 time 0.3944 (0.3846) data time 0.0011 (0.0021) model time 0.3933 (0.3817) loss 3.7466 (3.5491) grad_norm 1.1394 (1.2964) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][470/625] eta 0:00:59 lr 0.001908 wd 0.0500 time 0.3744 (0.3844) data time 0.0009 (0.0021) model time 0.3736 (0.3816) loss 2.6028 (3.5489) grad_norm 1.1590 (1.2958) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:54:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][480/625] eta 0:00:55 lr 0.001908 wd 0.0500 time 0.3807 (0.3843) data time 0.0011 (0.0021) model time 0.3796 (0.3815) loss 3.7874 (3.5493) grad_norm 0.9575 (1.2927) loss_scale 16384.0000 (8294.1871) mem 11225MB [2024-08-26 06:54:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][490/625] eta 0:00:51 lr 0.001908 wd 0.0500 time 0.3878 (0.3842) data time 0.0010 (0.0020) model time 0.3867 (0.3815) loss 2.1926 (3.5459) grad_norm 0.8129 (1.2870) loss_scale 16384.0000 (8458.9491) mem 11225MB [2024-08-26 06:54:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][500/625] eta 0:00:48 lr 0.001908 wd 0.0500 time 0.3750 (0.3850) data time 0.0008 (0.0020) model time 0.3743 (0.3823) loss 3.9379 (3.5439) grad_norm 1.0268 (1.2879) loss_scale 16384.0000 (8617.1337) mem 11225MB [2024-08-26 06:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][510/625] eta 0:00:44 lr 0.001908 wd 0.0500 time 0.3799 (0.3848) data time 0.0011 (0.0020) model time 0.3789 (0.3822) loss 2.7708 (3.5408) grad_norm 1.0988 (1.2855) loss_scale 16384.0000 (8769.1272) mem 11225MB [2024-08-26 06:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][520/625] eta 0:00:40 lr 0.001908 wd 0.0500 time 0.3851 (0.3847) data time 0.0009 (0.0020) model time 0.3843 (0.3820) loss 4.4759 (3.5453) grad_norm 2.2960 (1.2882) loss_scale 16384.0000 (8915.2860) mem 11225MB [2024-08-26 06:54:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][530/625] eta 0:00:36 lr 0.001907 wd 0.0500 time 0.3737 (0.3845) data time 0.0008 (0.0020) model time 0.3729 (0.3819) loss 2.9458 (3.5438) grad_norm 1.0611 (1.2900) loss_scale 16384.0000 (9055.9397) mem 11225MB [2024-08-26 06:55:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][540/625] eta 0:00:32 lr 0.001907 wd 0.0500 time 0.3360 (0.3856) data time 0.0009 (0.0020) model time 0.3351 (0.3831) loss 4.1110 (3.5405) grad_norm 1.1334 (inf) loss_scale 8192.0000 (9055.1128) mem 11225MB [2024-08-26 06:55:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][550/625] eta 0:00:28 lr 0.001907 wd 0.0500 time 0.3751 (0.3854) data time 0.0008 (0.0019) model time 0.3743 (0.3829) loss 2.5462 (3.5373) grad_norm 1.1094 (inf) loss_scale 8192.0000 (9039.4483) mem 11225MB [2024-08-26 06:55:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][560/625] eta 0:00:25 lr 0.001907 wd 0.0500 time 0.3751 (0.3852) data time 0.0008 (0.0019) model time 0.3743 (0.3828) loss 3.5531 (3.5402) grad_norm 1.2386 (inf) loss_scale 8192.0000 (9024.3422) mem 11225MB [2024-08-26 06:55:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][570/625] eta 0:00:21 lr 0.001907 wd 0.0500 time 0.3748 (0.3851) data time 0.0011 (0.0019) model time 0.3737 (0.3827) loss 3.4100 (3.5407) grad_norm 1.1491 (inf) loss_scale 8192.0000 (9009.7653) mem 11225MB [2024-08-26 06:55:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][580/625] eta 0:00:17 lr 0.001907 wd 0.0500 time 0.3810 (0.3850) data time 0.0011 (0.0019) model time 0.3798 (0.3825) loss 3.8460 (3.5414) grad_norm 0.9075 (inf) loss_scale 8192.0000 (8995.6902) mem 11225MB [2024-08-26 06:55:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][590/625] eta 0:00:13 lr 0.001907 wd 0.0500 time 0.3824 (0.3849) data time 0.0010 (0.0019) model time 0.3813 (0.3825) loss 3.1064 (3.5445) grad_norm 1.1515 (inf) loss_scale 8192.0000 (8982.0914) mem 11225MB [2024-08-26 06:55:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][600/625] eta 0:00:09 lr 0.001907 wd 0.0500 time 0.3787 (0.3848) data time 0.0008 (0.0019) model time 0.3779 (0.3824) loss 4.0553 (3.5447) grad_norm 1.1968 (inf) loss_scale 8192.0000 (8968.9451) mem 11225MB [2024-08-26 06:55:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][610/625] eta 0:00:05 lr 0.001907 wd 0.0500 time 0.3684 (0.3847) data time 0.0006 (0.0019) model time 0.3678 (0.3823) loss 3.8569 (3.5435) grad_norm 1.4106 (inf) loss_scale 8192.0000 (8956.2291) mem 11225MB [2024-08-26 06:55:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [58/300][620/625] eta 0:00:01 lr 0.001907 wd 0.0500 time 0.3679 (0.3844) data time 0.0006 (0.0019) model time 0.3674 (0.3820) loss 4.5737 (3.5432) grad_norm 1.2500 (inf) loss_scale 8192.0000 (8943.9227) mem 11225MB [2024-08-26 06:55:34 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 58 training takes 0:04:00 [2024-08-26 06:55:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:55:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:55:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.488 (0.488) Loss 0.7021 (0.7021) Acc@1 84.473 (84.473) Acc@5 96.631 (96.631) Mem 11225MB [2024-08-26 06:55:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.140) Loss 1.2715 (0.8729) Acc@1 69.922 (79.878) Acc@5 91.113 (95.663) Mem 11225MB [2024-08-26 06:55:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 1.3486 (1.0607) Acc@1 69.238 (75.958) Acc@5 89.795 (93.297) Mem 11225MB [2024-08-26 06:55:38 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.654 Acc@5 93.228 [2024-08-26 06:55:38 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.7% [2024-08-26 06:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.817 (0.817) Loss 0.5718 (0.5718) Acc@1 86.865 (86.865) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 06:55:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.177) Loss 0.9858 (0.7245) Acc@1 75.977 (82.564) Acc@5 93.799 (96.538) Mem 11225MB [2024-08-26 06:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.138) Loss 1.1699 (0.8995) Acc@1 71.094 (78.383) Acc@5 91.504 (94.436) Mem 11225MB [2024-08-26 06:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.101 Acc@5 94.410 [2024-08-26 06:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.1% [2024-08-26 06:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.10% [2024-08-26 06:55:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:55:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][0/625] eta 0:07:31 lr 0.001907 wd 0.0500 time 0.7219 (0.7219) data time 0.3790 (0.3790) model time 0.0000 (0.0000) loss 2.4766 (2.4766) grad_norm 1.4906 (1.4906) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:55:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][10/625] eta 0:04:11 lr 0.001907 wd 0.0500 time 0.3832 (0.4088) data time 0.0012 (0.0356) model time 0.0000 (0.0000) loss 3.7101 (3.4740) grad_norm 1.1971 (1.3173) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][20/625] eta 0:03:58 lr 0.001907 wd 0.0500 time 0.3686 (0.3936) data time 0.0011 (0.0192) model time 0.0000 (0.0000) loss 3.5657 (3.4673) grad_norm 0.9024 (1.2089) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][30/625] eta 0:03:50 lr 0.001906 wd 0.0500 time 0.3746 (0.3882) data time 0.0012 (0.0135) model time 0.0000 (0.0000) loss 3.7501 (3.4744) grad_norm 1.3232 (1.2361) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][40/625] eta 0:03:45 lr 0.001906 wd 0.0500 time 0.3804 (0.3856) data time 0.0017 (0.0105) model time 0.0000 (0.0000) loss 3.5999 (3.4123) grad_norm 1.2552 (1.2289) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][50/625] eta 0:03:40 lr 0.001906 wd 0.0500 time 0.3734 (0.3840) data time 0.0010 (0.0087) model time 0.0000 (0.0000) loss 2.7175 (3.4634) grad_norm 1.0070 (1.2616) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][60/625] eta 0:03:39 lr 0.001906 wd 0.0500 time 0.3829 (0.3886) data time 0.0011 (0.0074) model time 0.3819 (0.4106) loss 2.4091 (3.4850) grad_norm 1.1302 (1.2603) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][70/625] eta 0:03:37 lr 0.001906 wd 0.0500 time 0.5873 (0.3924) data time 0.0008 (0.0065) model time 0.5865 (0.4128) loss 3.1940 (3.5111) grad_norm 1.7697 (1.2691) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][80/625] eta 0:03:32 lr 0.001906 wd 0.0500 time 0.3742 (0.3903) data time 0.0008 (0.0058) model time 0.3734 (0.3999) loss 4.2842 (3.4857) grad_norm 1.0681 (1.2761) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][90/625] eta 0:03:28 lr 0.001906 wd 0.0500 time 0.3859 (0.3889) data time 0.0008 (0.0053) model time 0.3851 (0.3940) loss 3.2582 (3.4686) grad_norm 1.0655 (1.2749) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][100/625] eta 0:03:23 lr 0.001906 wd 0.0500 time 0.3733 (0.3876) data time 0.0008 (0.0049) model time 0.3724 (0.3902) loss 2.5154 (3.4811) grad_norm 1.6541 (1.2747) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][110/625] eta 0:03:19 lr 0.001906 wd 0.0500 time 0.3790 (0.3865) data time 0.0010 (0.0046) model time 0.3781 (0.3876) loss 4.1082 (3.5238) grad_norm 1.0970 (1.2646) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][120/625] eta 0:03:14 lr 0.001906 wd 0.0500 time 0.3716 (0.3856) data time 0.0008 (0.0043) model time 0.3707 (0.3857) loss 3.9371 (3.5294) grad_norm 1.0983 (1.2650) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][130/625] eta 0:03:10 lr 0.001906 wd 0.0500 time 0.3788 (0.3851) data time 0.0009 (0.0040) model time 0.3779 (0.3847) loss 4.3609 (3.5247) grad_norm 1.6466 (1.2685) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][140/625] eta 0:03:06 lr 0.001906 wd 0.0500 time 0.3751 (0.3846) data time 0.0012 (0.0038) model time 0.3739 (0.3838) loss 3.7343 (3.5048) grad_norm 2.5287 (1.2685) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][150/625] eta 0:03:02 lr 0.001906 wd 0.0500 time 0.3821 (0.3842) data time 0.0008 (0.0037) model time 0.3813 (0.3832) loss 3.5208 (3.5059) grad_norm 1.2675 (1.2771) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][160/625] eta 0:02:58 lr 0.001906 wd 0.0500 time 0.3919 (0.3838) data time 0.0008 (0.0035) model time 0.3910 (0.3826) loss 4.2254 (3.5091) grad_norm 1.2400 (1.2808) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][170/625] eta 0:02:54 lr 0.001905 wd 0.0500 time 0.3764 (0.3833) data time 0.0008 (0.0034) model time 0.3756 (0.3818) loss 2.8911 (3.4953) grad_norm 0.9686 (1.2787) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][180/625] eta 0:02:50 lr 0.001905 wd 0.0500 time 0.3818 (0.3829) data time 0.0011 (0.0032) model time 0.3806 (0.3813) loss 3.0596 (3.5059) grad_norm 1.5198 (1.2850) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][190/625] eta 0:02:46 lr 0.001905 wd 0.0500 time 0.3819 (0.3826) data time 0.0010 (0.0031) model time 0.3808 (0.3809) loss 2.9204 (3.4980) grad_norm 1.1486 (1.2719) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][200/625] eta 0:02:42 lr 0.001905 wd 0.0500 time 0.3771 (0.3823) data time 0.0011 (0.0030) model time 0.3760 (0.3806) loss 3.6942 (3.5048) grad_norm 2.2461 (1.2676) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][210/625] eta 0:02:38 lr 0.001905 wd 0.0500 time 0.3805 (0.3820) data time 0.0010 (0.0029) model time 0.3795 (0.3803) loss 3.0078 (3.5105) grad_norm 1.7955 (1.2701) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][220/625] eta 0:02:34 lr 0.001905 wd 0.0500 time 0.3723 (0.3817) data time 0.0008 (0.0028) model time 0.3714 (0.3799) loss 4.1158 (3.5070) grad_norm 1.6937 (1.2724) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][230/625] eta 0:02:30 lr 0.001905 wd 0.0500 time 0.3744 (0.3814) data time 0.0008 (0.0028) model time 0.3736 (0.3796) loss 3.4447 (3.5136) grad_norm 2.3797 (1.2822) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][240/625] eta 0:02:26 lr 0.001905 wd 0.0500 time 0.3715 (0.3812) data time 0.0010 (0.0027) model time 0.3705 (0.3793) loss 3.7573 (3.5238) grad_norm 1.4803 (1.2786) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][250/625] eta 0:02:23 lr 0.001905 wd 0.0500 time 0.5701 (0.3825) data time 0.0011 (0.0026) model time 0.5690 (0.3810) loss 3.8812 (3.5141) grad_norm 1.1055 (1.2711) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][260/625] eta 0:02:19 lr 0.001905 wd 0.0500 time 0.3719 (0.3823) data time 0.0010 (0.0026) model time 0.3710 (0.3808) loss 4.1309 (3.5158) grad_norm 1.8079 (1.2759) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][270/625] eta 0:02:15 lr 0.001905 wd 0.0500 time 0.3777 (0.3821) data time 0.0008 (0.0025) model time 0.3769 (0.3806) loss 4.1673 (3.5119) grad_norm 1.5227 (1.2738) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][280/625] eta 0:02:12 lr 0.001905 wd 0.0500 time 0.3780 (0.3827) data time 0.0011 (0.0025) model time 0.3769 (0.3813) loss 3.5251 (3.5173) grad_norm 2.8074 (1.2807) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][290/625] eta 0:02:08 lr 0.001905 wd 0.0500 time 0.3795 (0.3833) data time 0.0009 (0.0024) model time 0.3786 (0.3820) loss 3.8711 (3.5062) grad_norm 0.8330 (1.2893) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][300/625] eta 0:02:04 lr 0.001904 wd 0.0500 time 0.3865 (0.3831) data time 0.0009 (0.0024) model time 0.3856 (0.3818) loss 3.4545 (3.5095) grad_norm 1.2724 (1.2911) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][310/625] eta 0:02:00 lr 0.001904 wd 0.0500 time 0.3763 (0.3829) data time 0.0011 (0.0023) model time 0.3753 (0.3815) loss 3.5272 (3.5004) grad_norm 1.0418 (1.2832) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][320/625] eta 0:01:56 lr 0.001904 wd 0.0500 time 0.3719 (0.3826) data time 0.0011 (0.0023) model time 0.3708 (0.3813) loss 3.7473 (3.5046) grad_norm 1.2897 (1.2816) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][330/625] eta 0:01:52 lr 0.001904 wd 0.0500 time 0.3733 (0.3825) data time 0.0009 (0.0023) model time 0.3724 (0.3811) loss 3.5520 (3.4993) grad_norm 1.1465 (1.2799) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][340/625] eta 0:01:48 lr 0.001904 wd 0.0500 time 0.3783 (0.3824) data time 0.0008 (0.0022) model time 0.3775 (0.3810) loss 3.8709 (3.4969) grad_norm 1.4757 (1.2821) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:57:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][350/625] eta 0:01:45 lr 0.001904 wd 0.0500 time 0.3745 (0.3823) data time 0.0011 (0.0022) model time 0.3734 (0.3809) loss 2.9219 (3.4988) grad_norm 1.3637 (1.2795) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][360/625] eta 0:01:41 lr 0.001904 wd 0.0500 time 0.3823 (0.3823) data time 0.0011 (0.0022) model time 0.3812 (0.3809) loss 2.8219 (3.4974) grad_norm 1.5731 (1.2816) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][370/625] eta 0:01:37 lr 0.001904 wd 0.0500 time 0.5817 (0.3838) data time 0.0008 (0.0021) model time 0.5808 (0.3827) loss 4.2581 (3.5000) grad_norm 1.5095 (1.2833) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][380/625] eta 0:01:33 lr 0.001904 wd 0.0500 time 0.3755 (0.3836) data time 0.0011 (0.0021) model time 0.3744 (0.3825) loss 3.6361 (3.5036) grad_norm 1.5571 (1.2842) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][390/625] eta 0:01:30 lr 0.001904 wd 0.0500 time 0.3749 (0.3835) data time 0.0011 (0.0021) model time 0.3739 (0.3823) loss 4.2765 (3.5089) grad_norm 1.2269 (1.2818) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][400/625] eta 0:01:26 lr 0.001904 wd 0.0500 time 0.3743 (0.3834) data time 0.0009 (0.0021) model time 0.3734 (0.3822) loss 4.1642 (3.5143) grad_norm 1.1966 (1.2811) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][410/625] eta 0:01:22 lr 0.001904 wd 0.0500 time 0.3783 (0.3832) data time 0.0011 (0.0020) model time 0.3772 (0.3820) loss 3.8281 (3.5171) grad_norm 1.0281 (1.2779) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][420/625] eta 0:01:18 lr 0.001904 wd 0.0500 time 0.3762 (0.3831) data time 0.0011 (0.0020) model time 0.3751 (0.3819) loss 3.9035 (3.5224) grad_norm 0.9345 (1.2788) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][430/625] eta 0:01:14 lr 0.001903 wd 0.0500 time 0.3739 (0.3830) data time 0.0009 (0.0020) model time 0.3730 (0.3818) loss 3.6470 (3.5305) grad_norm 0.9346 (1.2776) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][440/625] eta 0:01:10 lr 0.001903 wd 0.0500 time 0.3788 (0.3829) data time 0.0008 (0.0020) model time 0.3780 (0.3817) loss 4.3694 (3.5344) grad_norm 1.0983 (1.2758) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][450/625] eta 0:01:07 lr 0.001903 wd 0.0500 time 0.3826 (0.3829) data time 0.0011 (0.0020) model time 0.3816 (0.3817) loss 3.9203 (3.5392) grad_norm 1.3230 (1.2793) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][460/625] eta 0:01:03 lr 0.001903 wd 0.0500 time 0.5624 (0.3832) data time 0.0011 (0.0019) model time 0.5613 (0.3820) loss 3.8946 (3.5383) grad_norm 1.3541 (1.2793) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][470/625] eta 0:00:59 lr 0.001903 wd 0.0500 time 0.3748 (0.3835) data time 0.0008 (0.0019) model time 0.3739 (0.3823) loss 2.6769 (3.5311) grad_norm 1.6027 (1.2800) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][480/625] eta 0:00:55 lr 0.001903 wd 0.0500 time 0.3845 (0.3834) data time 0.0008 (0.0019) model time 0.3837 (0.3822) loss 3.7582 (3.5286) grad_norm 1.4423 (1.2828) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][490/625] eta 0:00:51 lr 0.001903 wd 0.0500 time 0.3832 (0.3833) data time 0.0011 (0.0019) model time 0.3821 (0.3821) loss 3.6764 (3.5291) grad_norm 1.2292 (1.2831) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][500/625] eta 0:00:47 lr 0.001903 wd 0.0500 time 0.3744 (0.3832) data time 0.0011 (0.0019) model time 0.3733 (0.3820) loss 3.3545 (3.5298) grad_norm 1.1142 (1.2799) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:58:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][510/625] eta 0:00:44 lr 0.001903 wd 0.0500 time 0.3804 (0.3831) data time 0.0011 (0.0019) model time 0.3793 (0.3818) loss 2.2810 (3.5252) grad_norm 1.2705 (1.2796) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][520/625] eta 0:00:40 lr 0.001903 wd 0.0500 time 0.3966 (0.3830) data time 0.0008 (0.0019) model time 0.3958 (0.3818) loss 2.9007 (3.5303) grad_norm 1.6859 (1.2821) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][530/625] eta 0:00:36 lr 0.001903 wd 0.0500 time 0.3829 (0.3829) data time 0.0011 (0.0019) model time 0.3818 (0.3817) loss 3.3308 (3.5278) grad_norm 1.0082 (1.2844) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][540/625] eta 0:00:32 lr 0.001903 wd 0.0500 time 0.3756 (0.3828) data time 0.0011 (0.0018) model time 0.3745 (0.3816) loss 3.3546 (3.5293) grad_norm 1.0586 (1.2816) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][550/625] eta 0:00:28 lr 0.001903 wd 0.0500 time 0.3763 (0.3827) data time 0.0011 (0.0018) model time 0.3753 (0.3814) loss 3.3969 (3.5308) grad_norm 1.2579 (1.2811) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][560/625] eta 0:00:24 lr 0.001902 wd 0.0500 time 0.3677 (0.3826) data time 0.0010 (0.0018) model time 0.3667 (0.3814) loss 3.2873 (3.5269) grad_norm 1.3217 (1.2794) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][570/625] eta 0:00:21 lr 0.001902 wd 0.0500 time 0.5849 (0.3829) data time 0.0011 (0.0018) model time 0.5838 (0.3816) loss 3.9282 (3.5292) grad_norm 2.6986 (1.2804) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][580/625] eta 0:00:17 lr 0.001902 wd 0.0500 time 0.5855 (0.3838) data time 0.0011 (0.0018) model time 0.5845 (0.3827) loss 4.3545 (3.5306) grad_norm 1.5188 (1.2858) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][590/625] eta 0:00:13 lr 0.001902 wd 0.0500 time 0.3824 (0.3841) data time 0.0011 (0.0018) model time 0.3813 (0.3830) loss 3.4501 (3.5283) grad_norm 1.3964 (1.2875) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][600/625] eta 0:00:09 lr 0.001902 wd 0.0500 time 0.3839 (0.3840) data time 0.0010 (0.0018) model time 0.3828 (0.3828) loss 4.1222 (3.5250) grad_norm 1.5944 (1.2897) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][610/625] eta 0:00:05 lr 0.001902 wd 0.0500 time 0.3671 (0.3839) data time 0.0008 (0.0018) model time 0.3663 (0.3827) loss 3.4912 (3.5259) grad_norm 1.2242 (1.2886) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [59/300][620/625] eta 0:00:01 lr 0.001902 wd 0.0500 time 0.3649 (0.3836) data time 0.0005 (0.0018) model time 0.3644 (0.3824) loss 2.9284 (3.5280) grad_norm 1.0077 (1.2851) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:42 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 59 training takes 0:03:59 [2024-08-26 06:59:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 06:59:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 06:59:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.7104 (0.7104) Acc@1 85.254 (85.254) Acc@5 97.363 (97.363) Mem 11225MB [2024-08-26 06:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.2441 (0.8831) Acc@1 71.973 (80.402) Acc@5 91.211 (95.574) Mem 11225MB [2024-08-26 06:59:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.3770 (1.0690) Acc@1 68.506 (76.058) Acc@5 89.453 (93.315) Mem 11225MB [2024-08-26 06:59:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.856 Acc@5 93.194 [2024-08-26 06:59:46 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.9% [2024-08-26 06:59:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.886 (0.886) Loss 0.5723 (0.5723) Acc@1 86.865 (86.865) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 06:59:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.185) Loss 0.9863 (0.7235) Acc@1 75.781 (82.613) Acc@5 93.750 (96.533) Mem 11225MB [2024-08-26 06:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.141) Loss 1.1689 (0.8976) Acc@1 70.947 (78.499) Acc@5 91.699 (94.445) Mem 11225MB [2024-08-26 06:59:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.193 Acc@5 94.420 [2024-08-26 06:59:50 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.2% [2024-08-26 06:59:50 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.19% [2024-08-26 06:59:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 06:59:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 06:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][0/625] eta 0:07:42 lr 0.001902 wd 0.0500 time 0.7395 (0.7395) data time 0.3804 (0.3804) model time 0.0000 (0.0000) loss 3.4344 (3.4344) grad_norm 0.9653 (0.9653) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][10/625] eta 0:04:12 lr 0.001902 wd 0.0500 time 0.3823 (0.4110) data time 0.0010 (0.0357) model time 0.0000 (0.0000) loss 2.8485 (3.3742) grad_norm 0.9739 (1.2893) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 06:59:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][20/625] eta 0:03:59 lr 0.001902 wd 0.0500 time 0.3760 (0.3956) data time 0.0011 (0.0192) model time 0.0000 (0.0000) loss 3.9791 (3.4078) grad_norm 1.3766 (1.3962) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][30/625] eta 0:03:52 lr 0.001902 wd 0.0500 time 0.3791 (0.3912) data time 0.0012 (0.0134) model time 0.0000 (0.0000) loss 3.5906 (3.5218) grad_norm 1.3093 (1.3082) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][40/625] eta 0:03:46 lr 0.001902 wd 0.0500 time 0.3777 (0.3873) data time 0.0008 (0.0104) model time 0.0000 (0.0000) loss 2.5863 (3.5198) grad_norm 0.9797 (1.2535) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][50/625] eta 0:03:41 lr 0.001902 wd 0.0500 time 0.3808 (0.3859) data time 0.0008 (0.0087) model time 0.0000 (0.0000) loss 3.7580 (3.4686) grad_norm 1.6485 (1.2498) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][60/625] eta 0:03:37 lr 0.001901 wd 0.0500 time 0.3750 (0.3845) data time 0.0011 (0.0074) model time 0.3739 (0.3762) loss 2.1342 (3.4885) grad_norm 1.9599 (1.3115) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][70/625] eta 0:03:32 lr 0.001901 wd 0.0500 time 0.3759 (0.3837) data time 0.0008 (0.0065) model time 0.3751 (0.3769) loss 3.8645 (3.5119) grad_norm 1.2908 (1.3069) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][80/625] eta 0:03:30 lr 0.001901 wd 0.0500 time 0.3733 (0.3858) data time 0.0008 (0.0059) model time 0.3724 (0.3845) loss 4.0278 (3.4625) grad_norm 1.3357 (1.3010) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][90/625] eta 0:03:26 lr 0.001901 wd 0.0500 time 0.3874 (0.3854) data time 0.0011 (0.0054) model time 0.3863 (0.3835) loss 4.4213 (3.4719) grad_norm 1.1722 (1.2930) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][100/625] eta 0:03:24 lr 0.001901 wd 0.0500 time 0.3823 (0.3891) data time 0.0008 (0.0050) model time 0.3815 (0.3912) loss 3.3235 (3.5104) grad_norm 1.3713 (1.2959) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][110/625] eta 0:03:19 lr 0.001901 wd 0.0500 time 0.3683 (0.3882) data time 0.0012 (0.0046) model time 0.3671 (0.3889) loss 3.9102 (3.4951) grad_norm 1.2717 (1.2935) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][120/625] eta 0:03:15 lr 0.001901 wd 0.0500 time 0.3719 (0.3876) data time 0.0011 (0.0044) model time 0.3708 (0.3876) loss 3.3681 (3.5058) grad_norm 1.4165 (1.2944) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][130/625] eta 0:03:11 lr 0.001901 wd 0.0500 time 0.3722 (0.3866) data time 0.0008 (0.0041) model time 0.3715 (0.3857) loss 2.3509 (3.4937) grad_norm 1.1522 (1.2793) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][140/625] eta 0:03:07 lr 0.001901 wd 0.0500 time 0.3593 (0.3856) data time 0.0011 (0.0039) model time 0.3582 (0.3842) loss 3.7036 (3.4899) grad_norm 1.5716 (1.2819) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][150/625] eta 0:03:03 lr 0.001901 wd 0.0500 time 0.3647 (0.3858) data time 0.0011 (0.0037) model time 0.3635 (0.3846) loss 3.8945 (3.5049) grad_norm 1.4350 (1.2752) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][160/625] eta 0:02:59 lr 0.001901 wd 0.0500 time 0.3767 (0.3864) data time 0.0011 (0.0036) model time 0.3756 (0.3855) loss 3.8987 (3.4919) grad_norm 1.0140 (1.2905) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:00:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][170/625] eta 0:02:55 lr 0.001901 wd 0.0500 time 0.3666 (0.3856) data time 0.0012 (0.0034) model time 0.3654 (0.3843) loss 2.9289 (3.4975) grad_norm 1.2713 (1.2851) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][180/625] eta 0:02:51 lr 0.001901 wd 0.0500 time 0.3698 (0.3848) data time 0.0008 (0.0033) model time 0.3690 (0.3832) loss 3.0613 (3.4998) grad_norm 1.1581 (1.2816) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][190/625] eta 0:02:47 lr 0.001900 wd 0.0500 time 0.3676 (0.3842) data time 0.0011 (0.0032) model time 0.3665 (0.3824) loss 4.0348 (3.4935) grad_norm 1.0274 (1.2847) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][200/625] eta 0:02:43 lr 0.001900 wd 0.0500 time 0.3784 (0.3847) data time 0.0010 (0.0031) model time 0.3774 (0.3831) loss 3.2880 (3.5017) grad_norm 1.5351 (1.2876) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][210/625] eta 0:02:39 lr 0.001900 wd 0.0500 time 0.3762 (0.3844) data time 0.0007 (0.0030) model time 0.3754 (0.3827) loss 3.1606 (3.5196) grad_norm 1.1938 (1.3067) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][220/625] eta 0:02:35 lr 0.001900 wd 0.0500 time 0.4012 (0.3841) data time 0.0011 (0.0029) model time 0.4002 (0.3824) loss 3.7571 (3.5224) grad_norm 0.8040 (1.3113) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][230/625] eta 0:02:31 lr 0.001900 wd 0.0500 time 0.3747 (0.3839) data time 0.0008 (0.0028) model time 0.3739 (0.3821) loss 4.0312 (3.5219) grad_norm 1.1847 (1.3149) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][240/625] eta 0:02:27 lr 0.001900 wd 0.0500 time 0.3809 (0.3837) data time 0.0009 (0.0028) model time 0.3801 (0.3819) loss 2.7905 (3.5114) grad_norm 1.4760 (1.3077) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][250/625] eta 0:02:23 lr 0.001900 wd 0.0500 time 0.3685 (0.3834) data time 0.0010 (0.0027) model time 0.3675 (0.3816) loss 3.7937 (3.5207) grad_norm 1.2869 (1.3036) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][260/625] eta 0:02:19 lr 0.001900 wd 0.0500 time 0.3696 (0.3834) data time 0.0008 (0.0026) model time 0.3688 (0.3816) loss 3.1168 (3.5238) grad_norm 1.1822 (1.3023) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][270/625] eta 0:02:16 lr 0.001900 wd 0.0500 time 0.3764 (0.3832) data time 0.0009 (0.0026) model time 0.3755 (0.3814) loss 3.5842 (3.5174) grad_norm 1.2911 (1.2989) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][280/625] eta 0:02:12 lr 0.001900 wd 0.0500 time 0.3876 (0.3829) data time 0.0010 (0.0025) model time 0.3866 (0.3811) loss 3.4705 (3.5191) grad_norm 1.0481 (1.2978) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][290/625] eta 0:02:08 lr 0.001900 wd 0.0500 time 0.3734 (0.3826) data time 0.0008 (0.0025) model time 0.3726 (0.3807) loss 2.8216 (3.5129) grad_norm 1.5544 (1.2993) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][300/625] eta 0:02:04 lr 0.001900 wd 0.0500 time 0.3775 (0.3824) data time 0.0011 (0.0024) model time 0.3764 (0.3805) loss 3.9801 (3.5111) grad_norm 1.2808 (1.2983) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][310/625] eta 0:02:00 lr 0.001900 wd 0.0500 time 0.3724 (0.3836) data time 0.0008 (0.0024) model time 0.3716 (0.3819) loss 4.2339 (3.5196) grad_norm 0.9960 (1.2920) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][320/625] eta 0:01:56 lr 0.001899 wd 0.0500 time 0.3619 (0.3834) data time 0.0007 (0.0024) model time 0.3612 (0.3817) loss 2.7968 (3.5208) grad_norm 0.9321 (1.2884) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:01:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][330/625] eta 0:01:53 lr 0.001899 wd 0.0500 time 0.3690 (0.3831) data time 0.0009 (0.0023) model time 0.3681 (0.3815) loss 3.5836 (3.5176) grad_norm 1.1369 (1.2872) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][340/625] eta 0:01:49 lr 0.001899 wd 0.0500 time 0.3752 (0.3836) data time 0.0008 (0.0023) model time 0.3744 (0.3820) loss 2.4981 (3.5117) grad_norm 1.4339 (1.2851) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][350/625] eta 0:01:45 lr 0.001899 wd 0.0500 time 0.3683 (0.3838) data time 0.0008 (0.0023) model time 0.3675 (0.3823) loss 2.7090 (3.5135) grad_norm 1.1130 (1.2838) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][360/625] eta 0:01:41 lr 0.001899 wd 0.0500 time 0.3673 (0.3839) data time 0.0009 (0.0022) model time 0.3664 (0.3824) loss 3.0584 (3.5119) grad_norm 0.9548 (1.2843) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][370/625] eta 0:01:37 lr 0.001899 wd 0.0500 time 0.3797 (0.3836) data time 0.0011 (0.0022) model time 0.3786 (0.3821) loss 2.1410 (3.5049) grad_norm 1.2866 (1.2853) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][380/625] eta 0:01:33 lr 0.001899 wd 0.0500 time 0.3593 (0.3836) data time 0.0010 (0.0022) model time 0.3582 (0.3820) loss 3.7727 (3.5065) grad_norm 1.1380 (1.2849) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][390/625] eta 0:01:30 lr 0.001899 wd 0.0500 time 0.3789 (0.3834) data time 0.0009 (0.0021) model time 0.3780 (0.3818) loss 3.6167 (3.5038) grad_norm 1.2662 (1.2839) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][400/625] eta 0:01:26 lr 0.001899 wd 0.0500 time 0.3881 (0.3833) data time 0.0010 (0.0021) model time 0.3870 (0.3818) loss 3.8820 (3.5041) grad_norm 1.2690 (1.2891) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][410/625] eta 0:01:22 lr 0.001899 wd 0.0500 time 0.3749 (0.3837) data time 0.0011 (0.0021) model time 0.3738 (0.3822) loss 4.1191 (3.5055) grad_norm 1.1830 (1.2925) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][420/625] eta 0:01:18 lr 0.001899 wd 0.0500 time 0.3745 (0.3836) data time 0.0008 (0.0021) model time 0.3737 (0.3820) loss 3.0604 (3.5072) grad_norm 1.0525 (1.2921) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][430/625] eta 0:01:14 lr 0.001899 wd 0.0500 time 0.3846 (0.3835) data time 0.0010 (0.0021) model time 0.3836 (0.3820) loss 3.8230 (3.5036) grad_norm 1.2771 (1.2907) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][440/625] eta 0:01:10 lr 0.001899 wd 0.0500 time 0.3741 (0.3834) data time 0.0011 (0.0021) model time 0.3730 (0.3818) loss 3.2619 (3.5067) grad_norm 1.4192 (1.2866) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][450/625] eta 0:01:07 lr 0.001898 wd 0.0500 time 0.3686 (0.3832) data time 0.0008 (0.0020) model time 0.3678 (0.3816) loss 4.0406 (3.5104) grad_norm 1.0023 (1.2865) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][460/625] eta 0:01:03 lr 0.001898 wd 0.0500 time 0.5720 (0.3839) data time 0.0010 (0.0020) model time 0.5710 (0.3824) loss 3.0473 (3.5111) grad_norm 1.8215 (1.2916) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][470/625] eta 0:00:59 lr 0.001898 wd 0.0500 time 0.3739 (0.3838) data time 0.0008 (0.0020) model time 0.3731 (0.3823) loss 3.7037 (3.5091) grad_norm 1.1187 (1.2936) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][480/625] eta 0:00:55 lr 0.001898 wd 0.0500 time 0.3778 (0.3836) data time 0.0010 (0.0020) model time 0.3767 (0.3821) loss 3.7622 (3.5131) grad_norm 1.1227 (1.2892) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][490/625] eta 0:00:51 lr 0.001898 wd 0.0500 time 0.3714 (0.3842) data time 0.0011 (0.0020) model time 0.3703 (0.3828) loss 3.8963 (3.5172) grad_norm 1.0864 (1.2881) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:03:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][500/625] eta 0:00:48 lr 0.001898 wd 0.0500 time 0.3785 (0.3840) data time 0.0009 (0.0020) model time 0.3776 (0.3826) loss 2.9683 (3.5150) grad_norm 1.1916 (1.2874) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:03:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][510/625] eta 0:00:44 lr 0.001898 wd 0.0500 time 0.3695 (0.3839) data time 0.0009 (0.0019) model time 0.3687 (0.3825) loss 2.7500 (3.5208) grad_norm 1.2525 (1.2895) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 07:03:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][520/625] eta 0:00:40 lr 0.001898 wd 0.0500 time 0.3714 (0.3837) data time 0.0011 (0.0019) model time 0.3704 (0.3822) loss 3.0586 (3.5166) grad_norm 0.9518 (inf) loss_scale 4096.0000 (8176.2764) mem 11225MB [2024-08-26 07:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][530/625] eta 0:00:36 lr 0.001898 wd 0.0500 time 0.3715 (0.3835) data time 0.0009 (0.0019) model time 0.3706 (0.3820) loss 2.5967 (3.5187) grad_norm 1.2225 (inf) loss_scale 4096.0000 (8099.4350) mem 11225MB [2024-08-26 07:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][540/625] eta 0:00:32 lr 0.001898 wd 0.0500 time 0.3712 (0.3833) data time 0.0009 (0.0019) model time 0.3703 (0.3818) loss 4.0646 (3.5232) grad_norm 1.0813 (inf) loss_scale 4096.0000 (8025.4344) mem 11225MB [2024-08-26 07:03:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][550/625] eta 0:00:28 lr 0.001898 wd 0.0500 time 0.3742 (0.3831) data time 0.0010 (0.0019) model time 0.3732 (0.3816) loss 4.4267 (3.5246) grad_norm 1.5866 (inf) loss_scale 4096.0000 (7954.1198) mem 11225MB [2024-08-26 07:03:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][560/625] eta 0:00:24 lr 0.001898 wd 0.0500 time 0.3751 (0.3830) data time 0.0008 (0.0019) model time 0.3743 (0.3815) loss 4.0842 (3.5321) grad_norm 1.6153 (inf) loss_scale 4096.0000 (7885.3476) mem 11225MB [2024-08-26 07:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][570/625] eta 0:00:21 lr 0.001898 wd 0.0500 time 0.3699 (0.3828) data time 0.0008 (0.0019) model time 0.3691 (0.3813) loss 2.6555 (3.5338) grad_norm 1.1856 (inf) loss_scale 4096.0000 (7818.9842) mem 11225MB [2024-08-26 07:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][580/625] eta 0:00:17 lr 0.001897 wd 0.0500 time 0.3709 (0.3827) data time 0.0007 (0.0018) model time 0.3702 (0.3812) loss 4.1470 (3.5323) grad_norm 3.4775 (inf) loss_scale 4096.0000 (7754.9053) mem 11225MB [2024-08-26 07:03:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][590/625] eta 0:00:13 lr 0.001897 wd 0.0500 time 0.5690 (0.3829) data time 0.0008 (0.0018) model time 0.5683 (0.3814) loss 2.3756 (3.5323) grad_norm 1.3315 (inf) loss_scale 4096.0000 (7692.9949) mem 11225MB [2024-08-26 07:03:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][600/625] eta 0:00:09 lr 0.001897 wd 0.0500 time 0.3792 (0.3827) data time 0.0008 (0.0018) model time 0.3784 (0.3812) loss 4.3038 (3.5357) grad_norm 0.9908 (inf) loss_scale 4096.0000 (7633.1448) mem 11225MB [2024-08-26 07:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][610/625] eta 0:00:05 lr 0.001897 wd 0.0500 time 0.3657 (0.3829) data time 0.0005 (0.0018) model time 0.3652 (0.3815) loss 3.0391 (3.5367) grad_norm 0.9982 (inf) loss_scale 4096.0000 (7575.2537) mem 11225MB [2024-08-26 07:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [60/300][620/625] eta 0:00:01 lr 0.001897 wd 0.0500 time 0.3664 (0.3827) data time 0.0005 (0.0018) model time 0.3659 (0.3812) loss 3.9106 (3.5345) grad_norm 1.7125 (inf) loss_scale 4096.0000 (7519.2271) mem 11225MB [2024-08-26 07:03:50 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 60 training takes 0:03:59 [2024-08-26 07:03:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:03:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:03:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.525 (0.525) Loss 0.6343 (0.6343) Acc@1 85.400 (85.400) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 07:03:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.148) Loss 1.1230 (0.8250) Acc@1 74.854 (80.757) Acc@5 92.236 (95.827) Mem 11225MB [2024-08-26 07:03:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 1.2686 (0.9989) Acc@1 69.873 (76.602) Acc@5 90.674 (93.531) Mem 11225MB [2024-08-26 07:03:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.222 Acc@5 93.464 [2024-08-26 07:03:54 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.2% [2024-08-26 07:03:54 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 76.22% [2024-08-26 07:03:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 07:03:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 07:03:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.544 (0.544) Loss 0.5708 (0.5708) Acc@1 87.109 (87.109) Acc@5 97.656 (97.656) Mem 11225MB [2024-08-26 07:03:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.148) Loss 0.9834 (0.7219) Acc@1 75.684 (82.724) Acc@5 93.896 (96.569) Mem 11225MB [2024-08-26 07:03:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.123) Loss 1.1660 (0.8947) Acc@1 71.045 (78.623) Acc@5 91.650 (94.501) Mem 11225MB [2024-08-26 07:03:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.315 Acc@5 94.482 [2024-08-26 07:03:58 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.3% [2024-08-26 07:03:58 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.31% [2024-08-26 07:03:58 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:03:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][0/625] eta 0:08:24 lr 0.001897 wd 0.0500 time 0.8064 (0.8064) data time 0.4528 (0.4528) model time 0.0000 (0.0000) loss 4.1069 (4.1069) grad_norm 1.2119 (1.2119) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][10/625] eta 0:04:14 lr 0.001897 wd 0.0500 time 0.4054 (0.4146) data time 0.0010 (0.0422) model time 0.0000 (0.0000) loss 3.7687 (3.5112) grad_norm 1.1269 (1.1131) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][20/625] eta 0:04:03 lr 0.001897 wd 0.0500 time 0.3689 (0.4033) data time 0.0009 (0.0226) model time 0.0000 (0.0000) loss 3.9604 (3.6263) grad_norm 1.1120 (1.1453) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][30/625] eta 0:03:58 lr 0.001897 wd 0.0500 time 0.3724 (0.4004) data time 0.0008 (0.0158) model time 0.0000 (0.0000) loss 4.3778 (3.6061) grad_norm 1.0301 (1.1497) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][40/625] eta 0:03:50 lr 0.001897 wd 0.0500 time 0.3764 (0.3937) data time 0.0011 (0.0122) model time 0.0000 (0.0000) loss 3.1562 (3.5927) grad_norm 1.3511 (1.1518) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][50/625] eta 0:03:46 lr 0.001897 wd 0.0500 time 0.3629 (0.3936) data time 0.0011 (0.0101) model time 0.0000 (0.0000) loss 3.7076 (3.6649) grad_norm 1.2282 (1.1624) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][60/625] eta 0:03:40 lr 0.001897 wd 0.0500 time 0.3658 (0.3907) data time 0.0008 (0.0087) model time 0.3650 (0.3745) loss 2.5587 (3.6608) grad_norm 1.6672 (1.2228) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][70/625] eta 0:03:35 lr 0.001897 wd 0.0500 time 0.3737 (0.3885) data time 0.0009 (0.0076) model time 0.3728 (0.3744) loss 4.2544 (3.6679) grad_norm 1.2260 (1.2293) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][80/625] eta 0:03:30 lr 0.001896 wd 0.0500 time 0.3771 (0.3869) data time 0.0008 (0.0068) model time 0.3763 (0.3743) loss 2.2771 (3.6138) grad_norm 1.1067 (1.2351) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][90/625] eta 0:03:26 lr 0.001896 wd 0.0500 time 0.3702 (0.3859) data time 0.0012 (0.0062) model time 0.3691 (0.3749) loss 3.7105 (3.5814) grad_norm 0.8786 (1.2315) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][100/625] eta 0:03:22 lr 0.001896 wd 0.0500 time 0.3676 (0.3849) data time 0.0009 (0.0057) model time 0.3667 (0.3749) loss 4.2061 (3.5707) grad_norm 1.6890 (1.2490) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][110/625] eta 0:03:17 lr 0.001896 wd 0.0500 time 0.3729 (0.3841) data time 0.0010 (0.0053) model time 0.3719 (0.3748) loss 2.6012 (3.5592) grad_norm 1.0638 (1.2533) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][120/625] eta 0:03:13 lr 0.001896 wd 0.0500 time 0.3732 (0.3835) data time 0.0009 (0.0049) model time 0.3723 (0.3750) loss 3.9063 (3.5481) grad_norm 1.1115 (1.2507) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][130/625] eta 0:03:09 lr 0.001896 wd 0.0500 time 0.3796 (0.3827) data time 0.0008 (0.0046) model time 0.3787 (0.3746) loss 3.5993 (3.5502) grad_norm 1.1046 (1.2739) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][140/625] eta 0:03:05 lr 0.001896 wd 0.0500 time 0.3699 (0.3822) data time 0.0009 (0.0044) model time 0.3690 (0.3745) loss 3.9612 (3.5614) grad_norm 0.9432 (1.2616) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][150/625] eta 0:03:01 lr 0.001896 wd 0.0500 time 0.3793 (0.3816) data time 0.0010 (0.0042) model time 0.3783 (0.3744) loss 3.8516 (3.5578) grad_norm 1.0454 (1.2544) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][160/625] eta 0:02:57 lr 0.001896 wd 0.0500 time 0.3820 (0.3813) data time 0.0010 (0.0040) model time 0.3810 (0.3744) loss 3.8442 (3.5488) grad_norm 0.8725 (1.2455) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][170/625] eta 0:02:53 lr 0.001896 wd 0.0500 time 0.3696 (0.3819) data time 0.0008 (0.0038) model time 0.3688 (0.3758) loss 4.6164 (3.5528) grad_norm 1.2537 (1.2399) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][180/625] eta 0:02:49 lr 0.001896 wd 0.0500 time 0.3624 (0.3815) data time 0.0012 (0.0037) model time 0.3612 (0.3756) loss 3.8472 (3.5504) grad_norm 1.1714 (1.2432) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][190/625] eta 0:02:45 lr 0.001896 wd 0.0500 time 0.3830 (0.3812) data time 0.0008 (0.0035) model time 0.3822 (0.3755) loss 4.1433 (3.5698) grad_norm 0.9300 (1.2388) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][200/625] eta 0:02:41 lr 0.001895 wd 0.0500 time 0.3767 (0.3808) data time 0.0008 (0.0034) model time 0.3760 (0.3753) loss 2.5344 (3.5690) grad_norm 1.0715 (1.2360) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][210/625] eta 0:02:37 lr 0.001895 wd 0.0500 time 0.3774 (0.3805) data time 0.0010 (0.0033) model time 0.3764 (0.3753) loss 3.5796 (3.5597) grad_norm 1.0492 (1.2378) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][220/625] eta 0:02:33 lr 0.001895 wd 0.0500 time 0.3679 (0.3802) data time 0.0010 (0.0032) model time 0.3669 (0.3751) loss 3.7535 (3.5452) grad_norm 2.5297 (1.2595) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][230/625] eta 0:02:30 lr 0.001895 wd 0.0500 time 0.5850 (0.3808) data time 0.0008 (0.0031) model time 0.5842 (0.3761) loss 2.5849 (3.5414) grad_norm 0.8707 (1.2635) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][240/625] eta 0:02:26 lr 0.001895 wd 0.0500 time 0.3791 (0.3815) data time 0.0010 (0.0030) model time 0.3782 (0.3771) loss 3.9420 (3.5347) grad_norm 0.9768 (1.2572) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][250/625] eta 0:02:23 lr 0.001895 wd 0.0500 time 0.3717 (0.3817) data time 0.0008 (0.0029) model time 0.3709 (0.3775) loss 4.1757 (3.5239) grad_norm 1.5464 (1.2533) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][260/625] eta 0:02:19 lr 0.001895 wd 0.0500 time 0.3760 (0.3822) data time 0.0008 (0.0029) model time 0.3752 (0.3783) loss 3.0192 (3.5165) grad_norm 0.9897 (1.2471) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][270/625] eta 0:02:15 lr 0.001895 wd 0.0500 time 0.3777 (0.3820) data time 0.0008 (0.0028) model time 0.3769 (0.3782) loss 3.4073 (3.5052) grad_norm 4.1578 (1.2678) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][280/625] eta 0:02:11 lr 0.001895 wd 0.0500 time 0.3846 (0.3818) data time 0.0011 (0.0027) model time 0.3835 (0.3781) loss 4.1662 (3.5070) grad_norm 0.9590 (1.2732) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][290/625] eta 0:02:08 lr 0.001895 wd 0.0500 time 0.3639 (0.3824) data time 0.0011 (0.0027) model time 0.3628 (0.3789) loss 3.8310 (3.5093) grad_norm 1.0828 (1.2732) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][300/625] eta 0:02:04 lr 0.001895 wd 0.0500 time 0.3737 (0.3822) data time 0.0008 (0.0026) model time 0.3729 (0.3787) loss 4.1972 (3.5135) grad_norm 1.1589 (1.2681) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:05:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][310/625] eta 0:02:00 lr 0.001895 wd 0.0500 time 0.3819 (0.3820) data time 0.0011 (0.0026) model time 0.3808 (0.3786) loss 4.1034 (3.5175) grad_norm 1.1152 (1.2686) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][320/625] eta 0:01:56 lr 0.001895 wd 0.0500 time 0.3733 (0.3819) data time 0.0010 (0.0026) model time 0.3723 (0.3785) loss 3.4781 (3.5088) grad_norm 1.3357 (1.2659) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][330/625] eta 0:01:52 lr 0.001894 wd 0.0500 time 0.3762 (0.3817) data time 0.0010 (0.0026) model time 0.3752 (0.3783) loss 3.8802 (3.5111) grad_norm 1.5496 (1.2701) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][340/625] eta 0:01:48 lr 0.001894 wd 0.0500 time 0.3766 (0.3815) data time 0.0008 (0.0025) model time 0.3757 (0.3782) loss 3.3868 (3.5083) grad_norm 1.0682 (1.2683) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][350/625] eta 0:01:44 lr 0.001894 wd 0.0500 time 0.3720 (0.3812) data time 0.0011 (0.0025) model time 0.3709 (0.3779) loss 3.8816 (3.5097) grad_norm 1.1484 (1.2683) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][360/625] eta 0:01:40 lr 0.001894 wd 0.0500 time 0.3716 (0.3810) data time 0.0011 (0.0024) model time 0.3705 (0.3777) loss 2.7986 (3.5040) grad_norm 1.6000 (1.2724) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][370/625] eta 0:01:37 lr 0.001894 wd 0.0500 time 0.3912 (0.3809) data time 0.0010 (0.0024) model time 0.3901 (0.3776) loss 3.7070 (3.5070) grad_norm 0.9276 (1.2696) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][380/625] eta 0:01:33 lr 0.001894 wd 0.0500 time 0.3731 (0.3807) data time 0.0009 (0.0024) model time 0.3722 (0.3775) loss 3.3241 (3.5069) grad_norm 0.8658 (1.2684) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][390/625] eta 0:01:29 lr 0.001894 wd 0.0500 time 0.3803 (0.3807) data time 0.0011 (0.0024) model time 0.3791 (0.3775) loss 3.3555 (3.5113) grad_norm 1.2823 (1.2634) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][400/625] eta 0:01:25 lr 0.001894 wd 0.0500 time 0.3817 (0.3807) data time 0.0008 (0.0023) model time 0.3809 (0.3776) loss 4.5954 (3.5120) grad_norm 1.1906 (1.2635) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][410/625] eta 0:01:21 lr 0.001894 wd 0.0500 time 0.3642 (0.3805) data time 0.0012 (0.0023) model time 0.3630 (0.3774) loss 3.3597 (3.5137) grad_norm 2.2019 (1.2622) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][420/625] eta 0:01:18 lr 0.001894 wd 0.0500 time 0.3792 (0.3818) data time 0.0011 (0.0023) model time 0.3781 (0.3790) loss 3.6201 (3.5110) grad_norm 0.9847 (1.2585) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][430/625] eta 0:01:14 lr 0.001894 wd 0.0500 time 0.3788 (0.3816) data time 0.0008 (0.0023) model time 0.3781 (0.3788) loss 4.4371 (3.5094) grad_norm 1.2594 (1.2573) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][440/625] eta 0:01:10 lr 0.001894 wd 0.0500 time 0.3677 (0.3819) data time 0.0009 (0.0022) model time 0.3668 (0.3792) loss 3.8078 (3.5053) grad_norm 1.1583 (1.2560) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][450/625] eta 0:01:06 lr 0.001894 wd 0.0500 time 0.3705 (0.3818) data time 0.0009 (0.0022) model time 0.3696 (0.3790) loss 2.7174 (3.4976) grad_norm 1.0749 (1.2620) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][460/625] eta 0:01:02 lr 0.001893 wd 0.0500 time 0.3662 (0.3816) data time 0.0009 (0.0022) model time 0.3653 (0.3789) loss 4.5897 (3.5063) grad_norm 1.1343 (1.2598) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][470/625] eta 0:00:59 lr 0.001893 wd 0.0500 time 0.3770 (0.3815) data time 0.0008 (0.0022) model time 0.3762 (0.3788) loss 2.1226 (3.5064) grad_norm 1.7367 (1.2589) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][480/625] eta 0:00:55 lr 0.001893 wd 0.0500 time 0.3801 (0.3813) data time 0.0017 (0.0021) model time 0.3784 (0.3787) loss 3.6069 (3.5129) grad_norm 1.6817 (1.2614) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][490/625] eta 0:00:51 lr 0.001893 wd 0.0500 time 0.3860 (0.3813) data time 0.0008 (0.0021) model time 0.3852 (0.3786) loss 3.9710 (3.5117) grad_norm 1.2791 (1.2641) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][500/625] eta 0:00:47 lr 0.001893 wd 0.0500 time 0.3720 (0.3817) data time 0.0008 (0.0021) model time 0.3712 (0.3791) loss 3.4393 (3.5103) grad_norm 0.9207 (1.2638) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][510/625] eta 0:00:43 lr 0.001893 wd 0.0500 time 0.3817 (0.3816) data time 0.0011 (0.0021) model time 0.3807 (0.3791) loss 4.0309 (3.5114) grad_norm 1.3858 (1.2641) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][520/625] eta 0:00:40 lr 0.001893 wd 0.0500 time 0.3784 (0.3815) data time 0.0008 (0.0021) model time 0.3776 (0.3790) loss 3.5010 (3.5083) grad_norm 1.8117 (1.2693) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][530/625] eta 0:00:36 lr 0.001893 wd 0.0500 time 0.3709 (0.3814) data time 0.0008 (0.0020) model time 0.3701 (0.3789) loss 2.0101 (3.5067) grad_norm 1.1791 (1.2678) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][540/625] eta 0:00:32 lr 0.001893 wd 0.0500 time 0.3810 (0.3820) data time 0.0008 (0.0020) model time 0.3802 (0.3796) loss 4.6008 (3.5066) grad_norm 1.4983 (1.2662) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][550/625] eta 0:00:28 lr 0.001893 wd 0.0500 time 0.3753 (0.3819) data time 0.0009 (0.0020) model time 0.3744 (0.3795) loss 3.7565 (3.5077) grad_norm 1.5638 (1.2647) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][560/625] eta 0:00:24 lr 0.001893 wd 0.0500 time 0.3743 (0.3821) data time 0.0008 (0.0020) model time 0.3735 (0.3797) loss 3.0520 (3.5064) grad_norm 1.0784 (1.2618) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][570/625] eta 0:00:21 lr 0.001893 wd 0.0500 time 0.3746 (0.3820) data time 0.0010 (0.0020) model time 0.3736 (0.3796) loss 3.3692 (3.5033) grad_norm 1.5742 (1.2595) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][580/625] eta 0:00:17 lr 0.001892 wd 0.0500 time 0.3757 (0.3818) data time 0.0008 (0.0020) model time 0.3749 (0.3795) loss 3.0878 (3.5060) grad_norm 1.2866 (1.2610) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][590/625] eta 0:00:13 lr 0.001892 wd 0.0500 time 0.3722 (0.3817) data time 0.0008 (0.0020) model time 0.3714 (0.3794) loss 3.9311 (3.5075) grad_norm 1.3679 (1.2642) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][600/625] eta 0:00:09 lr 0.001892 wd 0.0500 time 0.3798 (0.3816) data time 0.0008 (0.0019) model time 0.3790 (0.3793) loss 4.2839 (3.5121) grad_norm 1.7747 (1.2705) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][610/625] eta 0:00:05 lr 0.001892 wd 0.0500 time 0.3714 (0.3815) data time 0.0008 (0.0019) model time 0.3706 (0.3792) loss 3.7612 (3.5199) grad_norm 0.8900 (1.2689) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [61/300][620/625] eta 0:00:01 lr 0.001892 wd 0.0500 time 0.3736 (0.3813) data time 0.0008 (0.0019) model time 0.3729 (0.3790) loss 2.8665 (3.5208) grad_norm 1.1842 (1.2678) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:07:57 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 61 training takes 0:03:58 [2024-08-26 07:07:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:07:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:07:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.566 (0.566) Loss 0.6973 (0.6973) Acc@1 84.863 (84.863) Acc@5 97.119 (97.119) Mem 11225MB [2024-08-26 07:08:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.152) Loss 1.2139 (0.8659) Acc@1 72.266 (80.278) Acc@5 92.090 (95.779) Mem 11225MB [2024-08-26 07:08:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.096 (0.125) Loss 1.3057 (1.0498) Acc@1 69.971 (76.037) Acc@5 90.137 (93.394) Mem 11225MB [2024-08-26 07:08:02 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.652 Acc@5 93.392 [2024-08-26 07:08:02 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.7% [2024-08-26 07:08:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.935 (0.935) Loss 0.5703 (0.5703) Acc@1 87.256 (87.256) Acc@5 97.900 (97.900) Mem 11225MB [2024-08-26 07:08:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.190) Loss 0.9834 (0.7211) Acc@1 75.830 (82.812) Acc@5 94.043 (96.622) Mem 11225MB [2024-08-26 07:08:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.145) Loss 1.1641 (0.8930) Acc@1 71.289 (78.764) Acc@5 91.699 (94.559) Mem 11225MB [2024-08-26 07:08:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.447 Acc@5 94.532 [2024-08-26 07:08:05 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.4% [2024-08-26 07:08:05 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.45% [2024-08-26 07:08:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:08:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:08:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][0/625] eta 0:08:41 lr 0.001892 wd 0.0500 time 0.8345 (0.8345) data time 0.5071 (0.5071) model time 0.0000 (0.0000) loss 4.1187 (4.1187) grad_norm 1.1699 (1.1699) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][10/625] eta 0:04:17 lr 0.001892 wd 0.0500 time 0.3789 (0.4195) data time 0.0008 (0.0471) model time 0.0000 (0.0000) loss 3.7231 (3.5389) grad_norm 1.0092 (1.2804) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][20/625] eta 0:04:01 lr 0.001892 wd 0.0500 time 0.3627 (0.3992) data time 0.0012 (0.0252) model time 0.0000 (0.0000) loss 4.0008 (3.4687) grad_norm 1.1406 (1.2135) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][30/625] eta 0:03:52 lr 0.001892 wd 0.0500 time 0.3698 (0.3912) data time 0.0011 (0.0175) model time 0.0000 (0.0000) loss 3.1843 (3.5429) grad_norm 1.4318 (1.2208) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][40/625] eta 0:03:46 lr 0.001892 wd 0.0500 time 0.3851 (0.3874) data time 0.0010 (0.0135) model time 0.0000 (0.0000) loss 3.5687 (3.6337) grad_norm 0.9381 (1.2445) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][50/625] eta 0:03:41 lr 0.001892 wd 0.0500 time 0.3786 (0.3852) data time 0.0011 (0.0110) model time 0.0000 (0.0000) loss 3.9768 (3.6379) grad_norm 1.2101 (1.2196) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][60/625] eta 0:03:38 lr 0.001892 wd 0.0500 time 0.6066 (0.3876) data time 0.0009 (0.0094) model time 0.6057 (0.3988) loss 3.3824 (3.5890) grad_norm 1.4529 (1.2497) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][70/625] eta 0:03:37 lr 0.001892 wd 0.0500 time 0.3855 (0.3917) data time 0.0012 (0.0083) model time 0.3843 (0.4071) loss 3.9167 (3.5980) grad_norm 1.3427 (1.2652) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][80/625] eta 0:03:33 lr 0.001891 wd 0.0500 time 0.3734 (0.3913) data time 0.0010 (0.0074) model time 0.3723 (0.4005) loss 3.7356 (3.5584) grad_norm 1.2972 (1.2454) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][90/625] eta 0:03:28 lr 0.001891 wd 0.0500 time 0.3690 (0.3896) data time 0.0010 (0.0067) model time 0.3679 (0.3940) loss 3.9252 (3.5688) grad_norm 1.2356 (1.3024) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][100/625] eta 0:03:23 lr 0.001891 wd 0.0500 time 0.3823 (0.3882) data time 0.0013 (0.0061) model time 0.3810 (0.3901) loss 2.7192 (3.5484) grad_norm 0.7905 (1.2978) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][110/625] eta 0:03:19 lr 0.001891 wd 0.0500 time 0.3580 (0.3869) data time 0.0011 (0.0057) model time 0.3570 (0.3871) loss 3.2935 (3.5466) grad_norm 1.1919 (1.2868) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][120/625] eta 0:03:14 lr 0.001891 wd 0.0500 time 0.3789 (0.3859) data time 0.0011 (0.0053) model time 0.3779 (0.3853) loss 2.7432 (3.5362) grad_norm 1.1828 (1.3047) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:08:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][130/625] eta 0:03:10 lr 0.001891 wd 0.0500 time 0.3801 (0.3857) data time 0.0008 (0.0050) model time 0.3793 (0.3848) loss 3.8591 (3.5320) grad_norm 1.2363 (1.3037) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][140/625] eta 0:03:06 lr 0.001891 wd 0.0500 time 0.3858 (0.3854) data time 0.0009 (0.0047) model time 0.3849 (0.3844) loss 3.5954 (3.5267) grad_norm 1.1322 (1.2870) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][150/625] eta 0:03:02 lr 0.001891 wd 0.0500 time 0.3722 (0.3848) data time 0.0010 (0.0045) model time 0.3712 (0.3835) loss 3.4731 (3.5162) grad_norm 1.2093 (1.2797) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][160/625] eta 0:02:58 lr 0.001891 wd 0.0500 time 0.3916 (0.3843) data time 0.0010 (0.0043) model time 0.3907 (0.3828) loss 4.1690 (3.5269) grad_norm 1.4650 (1.2990) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][170/625] eta 0:02:54 lr 0.001891 wd 0.0500 time 0.3603 (0.3838) data time 0.0008 (0.0041) model time 0.3595 (0.3820) loss 3.8049 (3.5144) grad_norm 1.2009 (1.2871) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][180/625] eta 0:02:50 lr 0.001891 wd 0.0500 time 0.3823 (0.3834) data time 0.0008 (0.0039) model time 0.3815 (0.3816) loss 2.4051 (3.5245) grad_norm 1.1800 (1.2790) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][190/625] eta 0:02:46 lr 0.001891 wd 0.0500 time 0.3755 (0.3831) data time 0.0008 (0.0038) model time 0.3747 (0.3813) loss 4.2522 (3.5424) grad_norm 1.1898 (1.2746) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][200/625] eta 0:02:42 lr 0.001890 wd 0.0500 time 0.3828 (0.3830) data time 0.0008 (0.0037) model time 0.3820 (0.3811) loss 4.4992 (3.5454) grad_norm 1.2400 (1.2760) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][210/625] eta 0:02:38 lr 0.001890 wd 0.0500 time 0.3737 (0.3828) data time 0.0011 (0.0035) model time 0.3726 (0.3808) loss 3.7296 (3.5536) grad_norm 1.1466 (1.2862) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][220/625] eta 0:02:34 lr 0.001890 wd 0.0500 time 0.3663 (0.3825) data time 0.0011 (0.0034) model time 0.3652 (0.3805) loss 3.2065 (3.5374) grad_norm 1.1159 (1.2865) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][230/625] eta 0:02:30 lr 0.001890 wd 0.0500 time 0.3725 (0.3822) data time 0.0008 (0.0033) model time 0.3717 (0.3803) loss 4.1318 (3.5309) grad_norm 1.1290 (1.2849) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][240/625] eta 0:02:27 lr 0.001890 wd 0.0500 time 0.3823 (0.3820) data time 0.0011 (0.0032) model time 0.3813 (0.3800) loss 3.5273 (3.5338) grad_norm 1.4336 (1.2860) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][250/625] eta 0:02:23 lr 0.001890 wd 0.0500 time 0.3850 (0.3826) data time 0.0009 (0.0032) model time 0.3841 (0.3808) loss 3.9807 (3.5269) grad_norm 1.2420 (1.2834) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][260/625] eta 0:02:19 lr 0.001890 wd 0.0500 time 0.3678 (0.3828) data time 0.0009 (0.0031) model time 0.3669 (0.3811) loss 3.5310 (3.5314) grad_norm 1.1356 (1.2804) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][270/625] eta 0:02:15 lr 0.001890 wd 0.0500 time 0.3797 (0.3826) data time 0.0011 (0.0030) model time 0.3786 (0.3809) loss 3.7512 (3.5356) grad_norm 1.2512 (1.2796) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][280/625] eta 0:02:12 lr 0.001890 wd 0.0500 time 0.3817 (0.3839) data time 0.0009 (0.0029) model time 0.3808 (0.3825) loss 2.4453 (3.5303) grad_norm 1.0933 (1.2759) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][290/625] eta 0:02:08 lr 0.001890 wd 0.0500 time 0.3813 (0.3836) data time 0.0008 (0.0029) model time 0.3805 (0.3822) loss 4.2670 (3.5309) grad_norm 1.4460 (1.2760) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][300/625] eta 0:02:04 lr 0.001890 wd 0.0500 time 0.3852 (0.3834) data time 0.0008 (0.0028) model time 0.3844 (0.3818) loss 3.4273 (3.5247) grad_norm 1.4119 (1.2767) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][310/625] eta 0:02:00 lr 0.001890 wd 0.0500 time 0.3683 (0.3831) data time 0.0008 (0.0028) model time 0.3675 (0.3816) loss 4.1149 (3.5177) grad_norm 1.2934 (1.2745) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][320/625] eta 0:01:56 lr 0.001889 wd 0.0500 time 0.3720 (0.3828) data time 0.0008 (0.0027) model time 0.3711 (0.3812) loss 2.8538 (3.5113) grad_norm 1.3270 (1.2760) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][330/625] eta 0:01:52 lr 0.001889 wd 0.0500 time 0.3740 (0.3827) data time 0.0008 (0.0027) model time 0.3732 (0.3811) loss 4.1475 (3.5168) grad_norm 1.3181 (1.2720) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][340/625] eta 0:01:48 lr 0.001889 wd 0.0500 time 0.3741 (0.3824) data time 0.0011 (0.0026) model time 0.3731 (0.3808) loss 3.5370 (3.5199) grad_norm 1.4515 (1.2780) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][350/625] eta 0:01:45 lr 0.001889 wd 0.0500 time 0.3660 (0.3822) data time 0.0010 (0.0026) model time 0.3650 (0.3806) loss 2.8074 (3.5202) grad_norm 1.1055 (1.2779) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][360/625] eta 0:01:41 lr 0.001889 wd 0.0500 time 0.3772 (0.3820) data time 0.0012 (0.0025) model time 0.3761 (0.3803) loss 4.1665 (3.5223) grad_norm 0.9916 (1.2783) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][370/625] eta 0:01:37 lr 0.001889 wd 0.0500 time 0.3770 (0.3824) data time 0.0010 (0.0025) model time 0.3760 (0.3808) loss 3.8299 (3.5261) grad_norm 1.6233 (1.2822) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][380/625] eta 0:01:33 lr 0.001889 wd 0.0500 time 0.3671 (0.3827) data time 0.0010 (0.0025) model time 0.3661 (0.3812) loss 4.3256 (3.5331) grad_norm 1.5435 (1.2873) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][390/625] eta 0:01:29 lr 0.001889 wd 0.0500 time 0.3801 (0.3825) data time 0.0009 (0.0024) model time 0.3793 (0.3810) loss 3.9569 (3.5361) grad_norm 0.8692 (1.2816) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][400/625] eta 0:01:26 lr 0.001889 wd 0.0500 time 0.3768 (0.3824) data time 0.0010 (0.0024) model time 0.3758 (0.3809) loss 3.1477 (3.5352) grad_norm 1.2189 (1.2789) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][410/625] eta 0:01:22 lr 0.001889 wd 0.0500 time 0.3848 (0.3823) data time 0.0011 (0.0024) model time 0.3836 (0.3807) loss 3.8923 (3.5331) grad_norm 1.4505 (1.2814) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][420/625] eta 0:01:18 lr 0.001889 wd 0.0500 time 0.3801 (0.3822) data time 0.0008 (0.0023) model time 0.3793 (0.3806) loss 3.3174 (3.5389) grad_norm 1.3400 (1.2854) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][430/625] eta 0:01:14 lr 0.001889 wd 0.0500 time 0.4125 (0.3823) data time 0.0011 (0.0023) model time 0.4114 (0.3807) loss 2.8916 (3.5322) grad_norm 1.0223 (1.2830) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][440/625] eta 0:01:10 lr 0.001889 wd 0.0500 time 0.3746 (0.3821) data time 0.0011 (0.0023) model time 0.3736 (0.3805) loss 3.5860 (3.5301) grad_norm 1.6108 (1.2837) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:10:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][450/625] eta 0:01:06 lr 0.001888 wd 0.0500 time 0.3770 (0.3820) data time 0.0009 (0.0023) model time 0.3762 (0.3805) loss 3.3743 (3.5315) grad_norm 1.1925 (1.2804) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][460/625] eta 0:01:03 lr 0.001888 wd 0.0500 time 0.5814 (0.3824) data time 0.0013 (0.0022) model time 0.5801 (0.3809) loss 3.5485 (3.5261) grad_norm 1.0276 (1.2772) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][470/625] eta 0:00:59 lr 0.001888 wd 0.0500 time 0.3757 (0.3828) data time 0.0008 (0.0022) model time 0.3749 (0.3813) loss 3.2200 (3.5240) grad_norm 1.1290 (1.2762) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][480/625] eta 0:00:55 lr 0.001888 wd 0.0500 time 0.3866 (0.3828) data time 0.0008 (0.0022) model time 0.3858 (0.3814) loss 3.9118 (3.5222) grad_norm 1.3584 (1.2735) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][490/625] eta 0:00:51 lr 0.001888 wd 0.0500 time 0.3826 (0.3827) data time 0.0010 (0.0022) model time 0.3815 (0.3813) loss 3.8003 (3.5213) grad_norm 2.9073 (1.2786) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][500/625] eta 0:00:47 lr 0.001888 wd 0.0500 time 0.3722 (0.3827) data time 0.0010 (0.0022) model time 0.3712 (0.3812) loss 3.4508 (3.5222) grad_norm 1.0147 (1.2799) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][510/625] eta 0:00:44 lr 0.001888 wd 0.0500 time 0.3801 (0.3827) data time 0.0011 (0.0021) model time 0.3791 (0.3812) loss 3.7942 (3.5259) grad_norm 0.8885 (1.2784) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][520/625] eta 0:00:40 lr 0.001888 wd 0.0500 time 0.3676 (0.3826) data time 0.0009 (0.0021) model time 0.3666 (0.3812) loss 3.0119 (3.5246) grad_norm 1.8393 (1.2813) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][530/625] eta 0:00:36 lr 0.001888 wd 0.0500 time 0.3742 (0.3825) data time 0.0011 (0.0021) model time 0.3731 (0.3810) loss 3.8600 (3.5282) grad_norm 1.5426 (1.2830) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][540/625] eta 0:00:32 lr 0.001888 wd 0.0500 time 0.3950 (0.3824) data time 0.0009 (0.0021) model time 0.3941 (0.3809) loss 4.2773 (3.5293) grad_norm 1.4685 (1.2827) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][550/625] eta 0:00:28 lr 0.001888 wd 0.0500 time 0.3859 (0.3823) data time 0.0009 (0.0021) model time 0.3850 (0.3808) loss 3.9207 (3.5291) grad_norm 1.1817 (1.2810) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][560/625] eta 0:00:24 lr 0.001888 wd 0.0500 time 0.3698 (0.3821) data time 0.0010 (0.0021) model time 0.3687 (0.3807) loss 3.9736 (3.5311) grad_norm 0.9372 (1.2814) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][570/625] eta 0:00:21 lr 0.001887 wd 0.0500 time 0.3784 (0.3820) data time 0.0009 (0.0020) model time 0.3775 (0.3806) loss 3.6320 (3.5349) grad_norm 1.3720 (1.2826) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][580/625] eta 0:00:17 lr 0.001887 wd 0.0500 time 0.6004 (0.3830) data time 0.0008 (0.0020) model time 0.5996 (0.3817) loss 3.2899 (3.5297) grad_norm 1.0462 (1.2830) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][590/625] eta 0:00:13 lr 0.001887 wd 0.0500 time 0.3674 (0.3832) data time 0.0010 (0.0020) model time 0.3664 (0.3819) loss 3.6808 (3.5267) grad_norm 1.2905 (1.2800) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:11:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][600/625] eta 0:00:09 lr 0.001887 wd 0.0500 time 0.3668 (0.3831) data time 0.0011 (0.0020) model time 0.3658 (0.3818) loss 3.5750 (3.5284) grad_norm 1.3919 (1.2799) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][610/625] eta 0:00:05 lr 0.001887 wd 0.0500 time 0.3675 (0.3829) data time 0.0008 (0.0020) model time 0.3667 (0.3815) loss 4.1689 (3.5287) grad_norm 1.0550 (1.2807) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [62/300][620/625] eta 0:00:01 lr 0.001887 wd 0.0500 time 0.3644 (0.3827) data time 0.0006 (0.0020) model time 0.3638 (0.3813) loss 3.3892 (3.5271) grad_norm 0.8796 (1.2781) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:05 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 62 training takes 0:03:59 [2024-08-26 07:12:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:12:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:12:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.549 (0.549) Loss 0.6689 (0.6689) Acc@1 85.938 (85.938) Acc@5 96.924 (96.924) Mem 11225MB [2024-08-26 07:12:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.147) Loss 1.1738 (0.8349) Acc@1 72.852 (80.611) Acc@5 91.895 (95.894) Mem 11225MB [2024-08-26 07:12:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 1.3486 (1.0247) Acc@1 67.285 (76.239) Acc@5 90.332 (93.504) Mem 11225MB [2024-08-26 07:12:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 75.918 Acc@5 93.424 [2024-08-26 07:12:09 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 75.9% [2024-08-26 07:12:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.947 (0.947) Loss 0.5703 (0.5703) Acc@1 87.256 (87.256) Acc@5 97.998 (97.998) Mem 11225MB [2024-08-26 07:12:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.185) Loss 0.9824 (0.7197) Acc@1 75.635 (82.861) Acc@5 93.945 (96.662) Mem 11225MB [2024-08-26 07:12:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.141) Loss 1.1602 (0.8908) Acc@1 71.094 (78.785) Acc@5 91.846 (94.613) Mem 11225MB [2024-08-26 07:12:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.463 Acc@5 94.588 [2024-08-26 07:12:13 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.5% [2024-08-26 07:12:13 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.46% [2024-08-26 07:12:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:12:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:12:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][0/625] eta 0:07:34 lr 0.001887 wd 0.0500 time 0.7273 (0.7273) data time 0.3777 (0.3777) model time 0.0000 (0.0000) loss 3.8835 (3.8835) grad_norm 1.2374 (1.2374) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][10/625] eta 0:04:10 lr 0.001887 wd 0.0500 time 0.3769 (0.4081) data time 0.0011 (0.0354) model time 0.0000 (0.0000) loss 3.6901 (3.6075) grad_norm 1.6013 (1.3566) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][20/625] eta 0:03:57 lr 0.001887 wd 0.0500 time 0.3731 (0.3931) data time 0.0008 (0.0191) model time 0.0000 (0.0000) loss 3.9339 (3.6188) grad_norm 0.9184 (1.3273) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][30/625] eta 0:03:50 lr 0.001887 wd 0.0500 time 0.3746 (0.3881) data time 0.0009 (0.0133) model time 0.0000 (0.0000) loss 2.7890 (3.6338) grad_norm 1.2940 (1.2675) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][40/625] eta 0:03:45 lr 0.001887 wd 0.0500 time 0.3834 (0.3858) data time 0.0008 (0.0103) model time 0.0000 (0.0000) loss 2.3300 (3.6706) grad_norm 0.9153 (1.2356) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][50/625] eta 0:03:40 lr 0.001887 wd 0.0500 time 0.3742 (0.3837) data time 0.0011 (0.0085) model time 0.0000 (0.0000) loss 4.0106 (3.6829) grad_norm 1.2730 (1.2783) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][60/625] eta 0:03:36 lr 0.001887 wd 0.0500 time 0.3790 (0.3825) data time 0.0011 (0.0073) model time 0.3779 (0.3751) loss 3.7215 (3.6428) grad_norm 1.0091 (1.2894) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][70/625] eta 0:03:31 lr 0.001886 wd 0.0500 time 0.3803 (0.3820) data time 0.0008 (0.0065) model time 0.3795 (0.3764) loss 3.3237 (3.6521) grad_norm 1.0632 (1.2833) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][80/625] eta 0:03:27 lr 0.001886 wd 0.0500 time 0.3761 (0.3814) data time 0.0011 (0.0058) model time 0.3750 (0.3764) loss 4.0530 (3.6387) grad_norm 1.2857 (1.2879) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][90/625] eta 0:03:23 lr 0.001886 wd 0.0500 time 0.3751 (0.3812) data time 0.0009 (0.0053) model time 0.3742 (0.3769) loss 2.7214 (3.6129) grad_norm 1.1481 (1.2789) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][100/625] eta 0:03:24 lr 0.001886 wd 0.0500 time 0.3807 (0.3892) data time 0.0011 (0.0049) model time 0.3796 (0.3937) loss 3.4288 (3.6026) grad_norm 0.9225 (1.2732) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][110/625] eta 0:03:19 lr 0.001886 wd 0.0500 time 0.3779 (0.3881) data time 0.0011 (0.0045) model time 0.3768 (0.3906) loss 2.6771 (3.5662) grad_norm 1.7103 (1.2906) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][120/625] eta 0:03:15 lr 0.001886 wd 0.0500 time 0.3787 (0.3872) data time 0.0008 (0.0043) model time 0.3779 (0.3887) loss 3.7929 (3.5671) grad_norm 1.6676 (1.2807) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][130/625] eta 0:03:11 lr 0.001886 wd 0.0500 time 0.3749 (0.3866) data time 0.0012 (0.0040) model time 0.3737 (0.3872) loss 3.8880 (3.5652) grad_norm 1.4425 (1.2767) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][140/625] eta 0:03:07 lr 0.001886 wd 0.0500 time 0.3760 (0.3859) data time 0.0009 (0.0038) model time 0.3750 (0.3861) loss 4.5970 (3.5693) grad_norm 1.6602 (1.2854) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][150/625] eta 0:03:03 lr 0.001886 wd 0.0500 time 0.5802 (0.3869) data time 0.0010 (0.0036) model time 0.5792 (0.3874) loss 2.8476 (3.5614) grad_norm 1.4313 (1.3102) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][160/625] eta 0:02:59 lr 0.001886 wd 0.0500 time 0.3737 (0.3864) data time 0.0012 (0.0035) model time 0.3725 (0.3865) loss 3.0963 (3.5574) grad_norm 1.2373 (1.3007) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][170/625] eta 0:02:55 lr 0.001886 wd 0.0500 time 0.3769 (0.3859) data time 0.0008 (0.0034) model time 0.3761 (0.3856) loss 2.7364 (3.5426) grad_norm 1.6100 (1.3044) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][180/625] eta 0:02:51 lr 0.001886 wd 0.0500 time 0.3837 (0.3855) data time 0.0010 (0.0032) model time 0.3828 (0.3850) loss 4.0920 (3.5532) grad_norm 1.8763 (1.3148) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][190/625] eta 0:02:47 lr 0.001885 wd 0.0500 time 0.3787 (0.3851) data time 0.0008 (0.0031) model time 0.3779 (0.3844) loss 3.7258 (3.5569) grad_norm 1.2479 (1.3087) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][200/625] eta 0:02:43 lr 0.001885 wd 0.0500 time 0.3760 (0.3847) data time 0.0010 (0.0030) model time 0.3750 (0.3840) loss 3.3678 (3.5599) grad_norm 1.1275 (1.3009) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][210/625] eta 0:02:39 lr 0.001885 wd 0.0500 time 0.3791 (0.3845) data time 0.0011 (0.0029) model time 0.3780 (0.3837) loss 3.7149 (3.5634) grad_norm 1.8013 (1.2950) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][220/625] eta 0:02:36 lr 0.001885 wd 0.0500 time 0.3888 (0.3852) data time 0.0009 (0.0029) model time 0.3879 (0.3845) loss 4.1720 (3.5542) grad_norm 1.1532 (1.2957) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][230/625] eta 0:02:32 lr 0.001885 wd 0.0500 time 0.3786 (0.3851) data time 0.0009 (0.0028) model time 0.3777 (0.3844) loss 3.6677 (3.5584) grad_norm 1.4539 (1.2952) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][240/625] eta 0:02:28 lr 0.001885 wd 0.0500 time 0.3883 (0.3849) data time 0.0010 (0.0027) model time 0.3873 (0.3840) loss 3.8937 (3.5507) grad_norm 2.2442 (1.2994) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][250/625] eta 0:02:24 lr 0.001885 wd 0.0500 time 0.3828 (0.3846) data time 0.0008 (0.0026) model time 0.3820 (0.3837) loss 2.9755 (3.5558) grad_norm 0.8854 (1.2937) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][260/625] eta 0:02:20 lr 0.001885 wd 0.0500 time 0.3798 (0.3844) data time 0.0012 (0.0026) model time 0.3786 (0.3835) loss 3.6006 (3.5588) grad_norm 1.4957 (1.2866) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:13:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][270/625] eta 0:02:16 lr 0.001885 wd 0.0500 time 0.4036 (0.3844) data time 0.0011 (0.0025) model time 0.4025 (0.3834) loss 3.4318 (3.5581) grad_norm 1.3445 (1.2878) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][280/625] eta 0:02:12 lr 0.001885 wd 0.0500 time 0.3786 (0.3841) data time 0.0009 (0.0025) model time 0.3777 (0.3831) loss 3.9527 (3.5656) grad_norm 1.2443 (1.2851) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][290/625] eta 0:02:08 lr 0.001885 wd 0.0500 time 0.3720 (0.3840) data time 0.0008 (0.0024) model time 0.3712 (0.3829) loss 3.6072 (3.5575) grad_norm 1.2313 (1.2819) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][300/625] eta 0:02:04 lr 0.001885 wd 0.0500 time 0.3792 (0.3838) data time 0.0009 (0.0024) model time 0.3783 (0.3827) loss 3.6749 (3.5391) grad_norm 1.1994 (1.2847) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][310/625] eta 0:02:01 lr 0.001884 wd 0.0500 time 0.3761 (0.3853) data time 0.0009 (0.0024) model time 0.3753 (0.3844) loss 4.6647 (3.5477) grad_norm 1.0751 (1.2822) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][320/625] eta 0:01:57 lr 0.001884 wd 0.0500 time 0.3755 (0.3851) data time 0.0009 (0.0023) model time 0.3745 (0.3842) loss 4.0096 (3.5475) grad_norm 1.0494 (1.2782) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][330/625] eta 0:01:53 lr 0.001884 wd 0.0500 time 0.3788 (0.3849) data time 0.0011 (0.0023) model time 0.3778 (0.3840) loss 3.9514 (3.5525) grad_norm 1.7254 (1.2836) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][340/625] eta 0:01:49 lr 0.001884 wd 0.0500 time 0.3799 (0.3854) data time 0.0011 (0.0023) model time 0.3789 (0.3845) loss 3.6872 (3.5497) grad_norm 1.0843 (1.2908) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][350/625] eta 0:01:45 lr 0.001884 wd 0.0500 time 0.3800 (0.3852) data time 0.0011 (0.0022) model time 0.3789 (0.3843) loss 3.2399 (3.5479) grad_norm 1.1613 (1.2863) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][360/625] eta 0:01:42 lr 0.001884 wd 0.0500 time 0.3815 (0.3850) data time 0.0011 (0.0023) model time 0.3804 (0.3840) loss 3.2548 (3.5422) grad_norm 1.6222 (1.2852) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][370/625] eta 0:01:38 lr 0.001884 wd 0.0500 time 0.3752 (0.3848) data time 0.0008 (0.0022) model time 0.3744 (0.3838) loss 3.7336 (3.5409) grad_norm 1.1133 (1.2840) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][380/625] eta 0:01:34 lr 0.001884 wd 0.0500 time 0.3743 (0.3846) data time 0.0008 (0.0022) model time 0.3735 (0.3835) loss 3.0761 (3.5380) grad_norm 1.7080 (1.2889) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][390/625] eta 0:01:30 lr 0.001884 wd 0.0500 time 0.4002 (0.3845) data time 0.0008 (0.0022) model time 0.3994 (0.3834) loss 4.5866 (3.5437) grad_norm 1.2957 (1.2873) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][400/625] eta 0:01:26 lr 0.001884 wd 0.0500 time 0.3804 (0.3844) data time 0.0011 (0.0022) model time 0.3793 (0.3833) loss 3.7252 (3.5487) grad_norm 1.2446 (1.2871) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][410/625] eta 0:01:22 lr 0.001884 wd 0.0500 time 0.3863 (0.3843) data time 0.0009 (0.0021) model time 0.3854 (0.3832) loss 4.3624 (3.5455) grad_norm 1.1486 (1.2826) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:14:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][420/625] eta 0:01:18 lr 0.001884 wd 0.0500 time 0.3803 (0.3843) data time 0.0011 (0.0021) model time 0.3793 (0.3831) loss 2.9641 (3.5446) grad_norm 1.6729 (1.2839) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][430/625] eta 0:01:14 lr 0.001883 wd 0.0500 time 0.3698 (0.3846) data time 0.0009 (0.0021) model time 0.3689 (0.3834) loss 3.4496 (3.5460) grad_norm 1.1055 (1.2813) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][440/625] eta 0:01:11 lr 0.001883 wd 0.0500 time 0.3768 (0.3845) data time 0.0011 (0.0021) model time 0.3757 (0.3833) loss 3.2350 (3.5467) grad_norm 1.8494 (1.2810) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][450/625] eta 0:01:07 lr 0.001883 wd 0.0500 time 0.3795 (0.3843) data time 0.0008 (0.0021) model time 0.3787 (0.3832) loss 3.6996 (3.5486) grad_norm 1.2319 (1.2973) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][460/625] eta 0:01:03 lr 0.001883 wd 0.0500 time 0.3763 (0.3846) data time 0.0012 (0.0020) model time 0.3751 (0.3835) loss 3.7214 (3.5457) grad_norm 1.6199 (1.2952) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][470/625] eta 0:00:59 lr 0.001883 wd 0.0500 time 0.3780 (0.3845) data time 0.0011 (0.0020) model time 0.3769 (0.3834) loss 3.8076 (3.5447) grad_norm 1.0608 (1.2968) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][480/625] eta 0:00:55 lr 0.001883 wd 0.0500 time 0.3904 (0.3844) data time 0.0011 (0.0020) model time 0.3893 (0.3832) loss 2.9113 (3.5411) grad_norm 1.5862 (1.2985) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][490/625] eta 0:00:52 lr 0.001883 wd 0.0500 time 0.3761 (0.3854) data time 0.0008 (0.0020) model time 0.3753 (0.3844) loss 2.8193 (3.5422) grad_norm 1.3106 (1.3006) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][500/625] eta 0:00:48 lr 0.001883 wd 0.0500 time 0.3790 (0.3853) data time 0.0011 (0.0020) model time 0.3779 (0.3843) loss 3.8014 (3.5438) grad_norm 1.0751 (1.3041) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][510/625] eta 0:00:44 lr 0.001883 wd 0.0500 time 0.3803 (0.3852) data time 0.0011 (0.0020) model time 0.3792 (0.3841) loss 3.7197 (3.5421) grad_norm 1.2272 (1.3019) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][520/625] eta 0:00:40 lr 0.001883 wd 0.0500 time 0.3740 (0.3850) data time 0.0012 (0.0019) model time 0.3727 (0.3839) loss 2.9549 (3.5359) grad_norm 1.2089 (1.2999) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][530/625] eta 0:00:36 lr 0.001883 wd 0.0500 time 0.3800 (0.3849) data time 0.0011 (0.0019) model time 0.3789 (0.3838) loss 3.7229 (3.5371) grad_norm 0.8414 (1.2962) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][540/625] eta 0:00:32 lr 0.001883 wd 0.0500 time 0.3765 (0.3848) data time 0.0008 (0.0019) model time 0.3757 (0.3837) loss 3.1351 (3.5356) grad_norm 1.1084 (1.2953) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][550/625] eta 0:00:28 lr 0.001882 wd 0.0500 time 0.3670 (0.3846) data time 0.0008 (0.0019) model time 0.3661 (0.3835) loss 4.0779 (3.5375) grad_norm 2.1219 (1.2985) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][560/625] eta 0:00:24 lr 0.001882 wd 0.0500 time 0.3769 (0.3846) data time 0.0009 (0.0019) model time 0.3760 (0.3834) loss 3.5024 (3.5411) grad_norm 1.2957 (1.3022) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][570/625] eta 0:00:21 lr 0.001882 wd 0.0500 time 0.3732 (0.3845) data time 0.0010 (0.0019) model time 0.3721 (0.3833) loss 3.3807 (3.5411) grad_norm 1.1160 (1.3007) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:15:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][580/625] eta 0:00:17 lr 0.001882 wd 0.0500 time 0.3830 (0.3844) data time 0.0011 (0.0019) model time 0.3819 (0.3832) loss 3.5788 (3.5373) grad_norm 1.1692 (1.2986) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:16:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][590/625] eta 0:00:13 lr 0.001882 wd 0.0500 time 0.3847 (0.3842) data time 0.0009 (0.0019) model time 0.3837 (0.3831) loss 2.7311 (3.5318) grad_norm 1.3983 (1.2991) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:16:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][600/625] eta 0:00:09 lr 0.001882 wd 0.0500 time 0.3801 (0.3841) data time 0.0009 (0.0018) model time 0.3792 (0.3829) loss 2.3713 (3.5269) grad_norm 1.0614 (1.3037) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:16:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][610/625] eta 0:00:05 lr 0.001882 wd 0.0500 time 0.3661 (0.3851) data time 0.0006 (0.0018) model time 0.3655 (0.3840) loss 4.2666 (3.5270) grad_norm 1.8463 (1.3056) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:16:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [63/300][620/625] eta 0:00:01 lr 0.001882 wd 0.0500 time 0.3685 (0.3848) data time 0.0006 (0.0018) model time 0.3680 (0.3837) loss 3.7023 (3.5263) grad_norm 0.9385 (1.3020) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:16:14 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 63 training takes 0:04:00 [2024-08-26 07:16:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:16:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:16:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.520 (0.520) Loss 0.6987 (0.6987) Acc@1 83.740 (83.740) Acc@5 97.314 (97.314) Mem 11225MB [2024-08-26 07:16:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.1172 (0.8573) Acc@1 73.877 (80.655) Acc@5 92.480 (95.836) Mem 11225MB [2024-08-26 07:16:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.3027 (1.0386) Acc@1 68.994 (76.404) Acc@5 90.088 (93.476) Mem 11225MB [2024-08-26 07:16:18 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.154 Acc@5 93.420 [2024-08-26 07:16:18 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.2% [2024-08-26 07:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.886 (0.886) Loss 0.5684 (0.5684) Acc@1 87.305 (87.305) Acc@5 97.900 (97.900) Mem 11225MB [2024-08-26 07:16:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.180) Loss 0.9810 (0.7186) Acc@1 75.928 (82.941) Acc@5 94.043 (96.697) Mem 11225MB [2024-08-26 07:16:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.138) Loss 1.1562 (0.8891) Acc@1 71.094 (78.878) Acc@5 91.895 (94.666) Mem 11225MB [2024-08-26 07:16:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.545 Acc@5 94.656 [2024-08-26 07:16:22 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.5% [2024-08-26 07:16:22 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.54% [2024-08-26 07:16:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:16:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:16:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][0/625] eta 0:08:05 lr 0.001882 wd 0.0500 time 0.7764 (0.7764) data time 0.4406 (0.4406) model time 0.0000 (0.0000) loss 3.6536 (3.6536) grad_norm 1.4263 (1.4263) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:16:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][10/625] eta 0:04:13 lr 0.001882 wd 0.0500 time 0.3817 (0.4127) data time 0.0008 (0.0411) model time 0.0000 (0.0000) loss 2.8972 (3.3535) grad_norm 1.2924 (1.2351) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:16:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][20/625] eta 0:03:59 lr 0.001882 wd 0.0500 time 0.3724 (0.3959) data time 0.0011 (0.0221) model time 0.0000 (0.0000) loss 3.7297 (3.3384) grad_norm 1.2709 (1.1834) loss_scale 8192.0000 (4486.0952) mem 11225MB [2024-08-26 07:16:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][30/625] eta 0:03:59 lr 0.001882 wd 0.0500 time 0.3714 (0.4026) data time 0.0009 (0.0153) model time 0.0000 (0.0000) loss 3.7023 (3.4065) grad_norm 1.2942 (1.2281) loss_scale 8192.0000 (5681.5484) mem 11225MB [2024-08-26 07:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][40/625] eta 0:03:52 lr 0.001881 wd 0.0500 time 0.3746 (0.3967) data time 0.0011 (0.0119) model time 0.0000 (0.0000) loss 4.0682 (3.4716) grad_norm 1.8572 (1.2446) loss_scale 8192.0000 (6293.8537) mem 11225MB [2024-08-26 07:16:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][50/625] eta 0:03:45 lr 0.001881 wd 0.0500 time 0.3595 (0.3926) data time 0.0008 (0.0098) model time 0.0000 (0.0000) loss 2.2874 (3.4619) grad_norm inf (inf) loss_scale 4096.0000 (6585.7255) mem 11225MB [2024-08-26 07:16:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][60/625] eta 0:03:40 lr 0.001881 wd 0.0500 time 0.3805 (0.3903) data time 0.0008 (0.0084) model time 0.3796 (0.3769) loss 4.3045 (3.4634) grad_norm 1.4251 (inf) loss_scale 4096.0000 (6177.5738) mem 11225MB [2024-08-26 07:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][70/625] eta 0:03:35 lr 0.001881 wd 0.0500 time 0.3905 (0.3885) data time 0.0013 (0.0074) model time 0.3891 (0.3769) loss 3.4050 (3.4805) grad_norm 1.6636 (inf) loss_scale 4096.0000 (5884.3944) mem 11225MB [2024-08-26 07:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][80/625] eta 0:03:30 lr 0.001881 wd 0.0500 time 0.3805 (0.3869) data time 0.0011 (0.0066) model time 0.3795 (0.3760) loss 2.8291 (3.4779) grad_norm 1.5666 (inf) loss_scale 4096.0000 (5663.6049) mem 11225MB [2024-08-26 07:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][90/625] eta 0:03:26 lr 0.001881 wd 0.0500 time 0.3826 (0.3860) data time 0.0011 (0.0060) model time 0.3816 (0.3763) loss 3.1411 (3.4875) grad_norm 1.4256 (inf) loss_scale 4096.0000 (5491.3407) mem 11225MB [2024-08-26 07:17:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][100/625] eta 0:03:22 lr 0.001881 wd 0.0500 time 0.3783 (0.3850) data time 0.0011 (0.0056) model time 0.3772 (0.3759) loss 3.4486 (3.5017) grad_norm 1.4135 (inf) loss_scale 4096.0000 (5353.1881) mem 11225MB [2024-08-26 07:17:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][110/625] eta 0:03:17 lr 0.001881 wd 0.0500 time 0.3691 (0.3844) data time 0.0011 (0.0052) model time 0.3680 (0.3761) loss 3.8724 (3.5315) grad_norm 1.2071 (inf) loss_scale 4096.0000 (5239.9279) mem 11225MB [2024-08-26 07:17:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][120/625] eta 0:03:14 lr 0.001881 wd 0.0500 time 0.3764 (0.3857) data time 0.0008 (0.0048) model time 0.3757 (0.3794) loss 4.2008 (3.5142) grad_norm 1.0693 (inf) loss_scale 4096.0000 (5145.3884) mem 11225MB [2024-08-26 07:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][130/625] eta 0:03:10 lr 0.001881 wd 0.0500 time 0.3836 (0.3851) data time 0.0008 (0.0046) model time 0.3828 (0.3791) loss 3.0339 (3.5041) grad_norm 1.2098 (inf) loss_scale 4096.0000 (5065.2824) mem 11225MB [2024-08-26 07:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][140/625] eta 0:03:06 lr 0.001881 wd 0.0500 time 0.3749 (0.3845) data time 0.0009 (0.0043) model time 0.3740 (0.3787) loss 2.8951 (3.4843) grad_norm 1.4597 (inf) loss_scale 4096.0000 (4996.5390) mem 11225MB [2024-08-26 07:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][150/625] eta 0:03:02 lr 0.001881 wd 0.0500 time 0.3749 (0.3841) data time 0.0012 (0.0041) model time 0.3736 (0.3785) loss 3.2041 (3.4898) grad_norm 1.6880 (inf) loss_scale 4096.0000 (4936.9007) mem 11225MB [2024-08-26 07:17:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][160/625] eta 0:02:58 lr 0.001880 wd 0.0500 time 0.3873 (0.3840) data time 0.0012 (0.0039) model time 0.3862 (0.3788) loss 3.5755 (3.4947) grad_norm 1.0035 (inf) loss_scale 4096.0000 (4884.6708) mem 11225MB [2024-08-26 07:17:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][170/625] eta 0:02:55 lr 0.001880 wd 0.0500 time 0.3696 (0.3860) data time 0.0008 (0.0038) model time 0.3688 (0.3820) loss 4.1166 (3.5004) grad_norm 1.3211 (inf) loss_scale 4096.0000 (4838.5497) mem 11225MB [2024-08-26 07:17:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][180/625] eta 0:02:51 lr 0.001880 wd 0.0500 time 0.3794 (0.3856) data time 0.0011 (0.0036) model time 0.3783 (0.3816) loss 3.0252 (3.5111) grad_norm 1.0510 (inf) loss_scale 4096.0000 (4797.5249) mem 11225MB [2024-08-26 07:17:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][190/625] eta 0:02:47 lr 0.001880 wd 0.0500 time 0.3737 (0.3853) data time 0.0011 (0.0035) model time 0.3726 (0.3814) loss 3.5612 (3.5015) grad_norm 0.9231 (inf) loss_scale 4096.0000 (4760.7958) mem 11225MB [2024-08-26 07:17:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][200/625] eta 0:02:43 lr 0.001880 wd 0.0500 time 0.3748 (0.3848) data time 0.0008 (0.0034) model time 0.3741 (0.3810) loss 2.8874 (3.5121) grad_norm 1.2946 (inf) loss_scale 4096.0000 (4727.7214) mem 11225MB [2024-08-26 07:17:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][210/625] eta 0:02:39 lr 0.001880 wd 0.0500 time 0.3739 (0.3844) data time 0.0010 (0.0033) model time 0.3729 (0.3806) loss 2.7871 (3.5164) grad_norm 1.0652 (inf) loss_scale 4096.0000 (4697.7820) mem 11225MB [2024-08-26 07:17:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][220/625] eta 0:02:35 lr 0.001880 wd 0.0500 time 0.3763 (0.3840) data time 0.0011 (0.0032) model time 0.3752 (0.3802) loss 3.5551 (3.5255) grad_norm 2.0063 (inf) loss_scale 4096.0000 (4670.5520) mem 11225MB [2024-08-26 07:17:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][230/625] eta 0:02:31 lr 0.001880 wd 0.0500 time 0.3752 (0.3836) data time 0.0012 (0.0031) model time 0.3740 (0.3798) loss 3.9229 (3.5231) grad_norm 1.2113 (inf) loss_scale 4096.0000 (4645.6797) mem 11225MB [2024-08-26 07:17:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][240/625] eta 0:02:28 lr 0.001880 wd 0.0500 time 0.3741 (0.3851) data time 0.0011 (0.0030) model time 0.3729 (0.3819) loss 4.0848 (3.5332) grad_norm 1.2092 (inf) loss_scale 4096.0000 (4622.8714) mem 11225MB [2024-08-26 07:18:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][250/625] eta 0:02:24 lr 0.001880 wd 0.0500 time 0.3885 (0.3848) data time 0.0008 (0.0029) model time 0.3877 (0.3816) loss 2.6309 (3.5347) grad_norm 1.8423 (inf) loss_scale 4096.0000 (4601.8805) mem 11225MB [2024-08-26 07:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][260/625] eta 0:02:20 lr 0.001880 wd 0.0500 time 0.3804 (0.3844) data time 0.0011 (0.0029) model time 0.3794 (0.3813) loss 3.9192 (3.5514) grad_norm 1.6047 (inf) loss_scale 4096.0000 (4582.4981) mem 11225MB [2024-08-26 07:18:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][270/625] eta 0:02:16 lr 0.001880 wd 0.0500 time 0.3828 (0.3842) data time 0.0011 (0.0028) model time 0.3817 (0.3811) loss 3.8228 (3.5575) grad_norm 1.5371 (inf) loss_scale 4096.0000 (4564.5461) mem 11225MB [2024-08-26 07:18:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][280/625] eta 0:02:12 lr 0.001879 wd 0.0500 time 0.3729 (0.3840) data time 0.0011 (0.0028) model time 0.3718 (0.3809) loss 3.0158 (3.5599) grad_norm 2.1083 (inf) loss_scale 4096.0000 (4547.8719) mem 11225MB [2024-08-26 07:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][290/625] eta 0:02:09 lr 0.001879 wd 0.0500 time 0.3773 (0.3852) data time 0.0011 (0.0027) model time 0.3762 (0.3824) loss 3.3419 (3.5586) grad_norm 1.4609 (inf) loss_scale 4096.0000 (4532.3436) mem 11225MB [2024-08-26 07:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][300/625] eta 0:02:05 lr 0.001879 wd 0.0500 time 0.3348 (0.3856) data time 0.0009 (0.0026) model time 0.3338 (0.3829) loss 2.9867 (3.5586) grad_norm 1.1972 (inf) loss_scale 4096.0000 (4517.8472) mem 11225MB [2024-08-26 07:18:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][310/625] eta 0:02:01 lr 0.001879 wd 0.0500 time 0.3804 (0.3854) data time 0.0009 (0.0026) model time 0.3795 (0.3828) loss 3.9706 (3.5684) grad_norm 1.0138 (inf) loss_scale 4096.0000 (4504.2830) mem 11225MB [2024-08-26 07:18:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][320/625] eta 0:01:57 lr 0.001879 wd 0.0500 time 0.3739 (0.3851) data time 0.0013 (0.0026) model time 0.3726 (0.3825) loss 3.5388 (3.5725) grad_norm 1.3084 (inf) loss_scale 4096.0000 (4491.5639) mem 11225MB [2024-08-26 07:18:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][330/625] eta 0:01:53 lr 0.001879 wd 0.0500 time 0.3801 (0.3849) data time 0.0012 (0.0025) model time 0.3789 (0.3823) loss 3.7489 (3.5727) grad_norm 0.9469 (inf) loss_scale 4096.0000 (4479.6133) mem 11225MB [2024-08-26 07:18:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][340/625] eta 0:01:49 lr 0.001879 wd 0.0500 time 0.3762 (0.3846) data time 0.0011 (0.0025) model time 0.3751 (0.3820) loss 3.0336 (3.5689) grad_norm 1.1827 (inf) loss_scale 4096.0000 (4468.3636) mem 11225MB [2024-08-26 07:18:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][350/625] eta 0:01:45 lr 0.001879 wd 0.0500 time 0.3808 (0.3844) data time 0.0010 (0.0024) model time 0.3798 (0.3819) loss 3.5461 (3.5664) grad_norm 1.2680 (inf) loss_scale 4096.0000 (4457.7550) mem 11225MB [2024-08-26 07:18:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][360/625] eta 0:01:41 lr 0.001879 wd 0.0500 time 0.3752 (0.3842) data time 0.0011 (0.0024) model time 0.3741 (0.3817) loss 3.3515 (3.5702) grad_norm 1.1464 (inf) loss_scale 4096.0000 (4447.7341) mem 11225MB [2024-08-26 07:18:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][370/625] eta 0:01:37 lr 0.001879 wd 0.0500 time 0.3763 (0.3840) data time 0.0009 (0.0024) model time 0.3754 (0.3815) loss 2.5524 (3.5669) grad_norm 0.8895 (inf) loss_scale 4096.0000 (4438.2534) mem 11225MB [2024-08-26 07:18:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][380/625] eta 0:01:34 lr 0.001879 wd 0.0500 time 0.3667 (0.3839) data time 0.0011 (0.0023) model time 0.3656 (0.3813) loss 2.6101 (3.5722) grad_norm 1.2848 (inf) loss_scale 4096.0000 (4429.2703) mem 11225MB [2024-08-26 07:18:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][390/625] eta 0:01:30 lr 0.001878 wd 0.0500 time 0.3757 (0.3838) data time 0.0009 (0.0023) model time 0.3748 (0.3813) loss 2.9785 (3.5635) grad_norm 1.1891 (inf) loss_scale 4096.0000 (4420.7468) mem 11225MB [2024-08-26 07:18:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][400/625] eta 0:01:26 lr 0.001878 wd 0.0500 time 0.3782 (0.3837) data time 0.0011 (0.0023) model time 0.3771 (0.3812) loss 2.1622 (3.5546) grad_norm 1.2395 (inf) loss_scale 4096.0000 (4412.6484) mem 11225MB [2024-08-26 07:19:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][410/625] eta 0:01:22 lr 0.001878 wd 0.0500 time 0.3752 (0.3835) data time 0.0009 (0.0023) model time 0.3742 (0.3811) loss 4.5804 (3.5557) grad_norm 1.2939 (inf) loss_scale 4096.0000 (4404.9440) mem 11225MB [2024-08-26 07:19:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][420/625] eta 0:01:18 lr 0.001878 wd 0.0500 time 0.3783 (0.3848) data time 0.0010 (0.0022) model time 0.3774 (0.3826) loss 3.7500 (3.5558) grad_norm 0.9860 (inf) loss_scale 4096.0000 (4397.6057) mem 11225MB [2024-08-26 07:19:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][430/625] eta 0:01:15 lr 0.001878 wd 0.0500 time 0.3730 (0.3847) data time 0.0011 (0.0022) model time 0.3719 (0.3824) loss 3.6137 (3.5538) grad_norm 1.9182 (inf) loss_scale 4096.0000 (4390.6079) mem 11225MB [2024-08-26 07:19:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][440/625] eta 0:01:11 lr 0.001878 wd 0.0500 time 0.3725 (0.3845) data time 0.0009 (0.0022) model time 0.3716 (0.3823) loss 3.1874 (3.5512) grad_norm 1.0906 (inf) loss_scale 4096.0000 (4383.9274) mem 11225MB [2024-08-26 07:19:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][450/625] eta 0:01:07 lr 0.001878 wd 0.0500 time 0.3771 (0.3844) data time 0.0008 (0.0022) model time 0.3763 (0.3822) loss 2.2910 (3.5498) grad_norm 0.8371 (inf) loss_scale 4096.0000 (4377.5432) mem 11225MB [2024-08-26 07:19:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][460/625] eta 0:01:03 lr 0.001878 wd 0.0500 time 0.3788 (0.3843) data time 0.0008 (0.0021) model time 0.3780 (0.3821) loss 3.6535 (3.5525) grad_norm 0.9225 (inf) loss_scale 4096.0000 (4371.4360) mem 11225MB [2024-08-26 07:19:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][470/625] eta 0:00:59 lr 0.001878 wd 0.0500 time 0.3767 (0.3842) data time 0.0008 (0.0021) model time 0.3760 (0.3820) loss 4.1561 (3.5509) grad_norm 0.8945 (inf) loss_scale 4096.0000 (4365.5881) mem 11225MB [2024-08-26 07:19:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][480/625] eta 0:00:55 lr 0.001878 wd 0.0500 time 0.3739 (0.3841) data time 0.0010 (0.0021) model time 0.3729 (0.3819) loss 3.7807 (3.5507) grad_norm 1.8349 (inf) loss_scale 4096.0000 (4359.9834) mem 11225MB [2024-08-26 07:19:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][490/625] eta 0:00:51 lr 0.001878 wd 0.0500 time 0.3780 (0.3840) data time 0.0011 (0.0021) model time 0.3769 (0.3818) loss 3.6745 (3.5522) grad_norm 1.1415 (inf) loss_scale 4096.0000 (4354.6069) mem 11225MB [2024-08-26 07:19:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][500/625] eta 0:00:48 lr 0.001878 wd 0.0500 time 0.3739 (0.3851) data time 0.0008 (0.0021) model time 0.3730 (0.3830) loss 2.4967 (3.5545) grad_norm 0.8995 (inf) loss_scale 4096.0000 (4349.4451) mem 11225MB [2024-08-26 07:19:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][510/625] eta 0:00:44 lr 0.001877 wd 0.0500 time 0.3760 (0.3849) data time 0.0010 (0.0020) model time 0.3750 (0.3829) loss 3.1962 (3.5586) grad_norm 1.7018 (inf) loss_scale 4096.0000 (4344.4853) mem 11225MB [2024-08-26 07:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][520/625] eta 0:00:40 lr 0.001877 wd 0.0500 time 0.3855 (0.3848) data time 0.0011 (0.0020) model time 0.3844 (0.3827) loss 2.5969 (3.5558) grad_norm 1.3441 (inf) loss_scale 4096.0000 (4339.7159) mem 11225MB [2024-08-26 07:19:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][530/625] eta 0:00:36 lr 0.001877 wd 0.0500 time 0.3748 (0.3846) data time 0.0008 (0.0020) model time 0.3740 (0.3825) loss 3.3723 (3.5588) grad_norm 0.9075 (inf) loss_scale 4096.0000 (4335.1262) mem 11225MB [2024-08-26 07:19:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][540/625] eta 0:00:32 lr 0.001877 wd 0.0500 time 0.3740 (0.3852) data time 0.0008 (0.0020) model time 0.3731 (0.3833) loss 4.0097 (3.5615) grad_norm 1.1764 (inf) loss_scale 4096.0000 (4330.7061) mem 11225MB [2024-08-26 07:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][550/625] eta 0:00:28 lr 0.001877 wd 0.0500 time 0.3709 (0.3851) data time 0.0012 (0.0020) model time 0.3697 (0.3831) loss 4.2491 (3.5634) grad_norm 0.9284 (inf) loss_scale 4096.0000 (4326.4465) mem 11225MB [2024-08-26 07:19:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][560/625] eta 0:00:25 lr 0.001877 wd 0.0500 time 0.3746 (0.3850) data time 0.0010 (0.0020) model time 0.3736 (0.3830) loss 3.1462 (3.5646) grad_norm 1.2422 (inf) loss_scale 4096.0000 (4322.3387) mem 11225MB [2024-08-26 07:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][570/625] eta 0:00:21 lr 0.001877 wd 0.0500 time 0.3797 (0.3848) data time 0.0011 (0.0019) model time 0.3786 (0.3829) loss 3.7277 (3.5641) grad_norm 1.1217 (inf) loss_scale 4096.0000 (4318.3748) mem 11225MB [2024-08-26 07:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][580/625] eta 0:00:17 lr 0.001877 wd 0.0500 time 0.3759 (0.3847) data time 0.0008 (0.0019) model time 0.3750 (0.3828) loss 3.6090 (3.5682) grad_norm 1.0548 (inf) loss_scale 4096.0000 (4314.5473) mem 11225MB [2024-08-26 07:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][590/625] eta 0:00:13 lr 0.001877 wd 0.0500 time 0.3750 (0.3846) data time 0.0011 (0.0019) model time 0.3738 (0.3827) loss 3.7003 (3.5676) grad_norm 0.8874 (inf) loss_scale 4096.0000 (4310.8494) mem 11225MB [2024-08-26 07:20:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][600/625] eta 0:00:09 lr 0.001877 wd 0.0500 time 0.3762 (0.3845) data time 0.0009 (0.0019) model time 0.3753 (0.3825) loss 4.3137 (3.5684) grad_norm 1.2129 (inf) loss_scale 4096.0000 (4307.2745) mem 11225MB [2024-08-26 07:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][610/625] eta 0:00:05 lr 0.001877 wd 0.0500 time 0.3684 (0.3843) data time 0.0008 (0.0019) model time 0.3676 (0.3824) loss 3.6983 (3.5669) grad_norm 1.2748 (inf) loss_scale 4096.0000 (4303.8167) mem 11225MB [2024-08-26 07:20:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [64/300][620/625] eta 0:00:01 lr 0.001877 wd 0.0500 time 0.3638 (0.3840) data time 0.0006 (0.0019) model time 0.3633 (0.3821) loss 4.2666 (3.5680) grad_norm 1.4481 (inf) loss_scale 4096.0000 (4300.4702) mem 11225MB [2024-08-26 07:20:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 64 training takes 0:03:59 [2024-08-26 07:20:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:20:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.517 (0.517) Loss 0.7100 (0.7100) Acc@1 84.277 (84.277) Acc@5 97.021 (97.021) Mem 11225MB [2024-08-26 07:20:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.1357 (0.8479) Acc@1 72.314 (80.362) Acc@5 92.285 (95.858) Mem 11225MB [2024-08-26 07:20:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.2734 (1.0273) Acc@1 70.068 (76.258) Acc@5 90.869 (93.587) Mem 11225MB [2024-08-26 07:20:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.124 Acc@5 93.536 [2024-08-26 07:20:27 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.1% [2024-08-26 07:20:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.878 (0.878) Loss 0.5684 (0.5684) Acc@1 87.305 (87.305) Acc@5 97.900 (97.900) Mem 11225MB [2024-08-26 07:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.180) Loss 0.9800 (0.7183) Acc@1 76.465 (83.083) Acc@5 93.994 (96.715) Mem 11225MB [2024-08-26 07:20:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.139) Loss 1.1523 (0.8880) Acc@1 71.045 (78.948) Acc@5 91.846 (94.722) Mem 11225MB [2024-08-26 07:20:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.613 Acc@5 94.708 [2024-08-26 07:20:31 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.6% [2024-08-26 07:20:31 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.61% [2024-08-26 07:20:31 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:20:32 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][0/625] eta 0:07:58 lr 0.001876 wd 0.0500 time 0.7660 (0.7660) data time 0.4241 (0.4241) model time 0.0000 (0.0000) loss 4.0885 (4.0885) grad_norm 1.8169 (1.8169) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][10/625] eta 0:04:14 lr 0.001876 wd 0.0500 time 0.3802 (0.4140) data time 0.0009 (0.0396) model time 0.0000 (0.0000) loss 2.9702 (3.3523) grad_norm 1.6189 (1.5392) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][20/625] eta 0:04:00 lr 0.001876 wd 0.0500 time 0.3792 (0.3981) data time 0.0011 (0.0213) model time 0.0000 (0.0000) loss 3.9154 (3.4167) grad_norm 1.1343 (1.3943) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][30/625] eta 0:03:53 lr 0.001876 wd 0.0500 time 0.3788 (0.3921) data time 0.0011 (0.0148) model time 0.0000 (0.0000) loss 3.8733 (3.3826) grad_norm 1.1082 (1.3124) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][40/625] eta 0:03:47 lr 0.001876 wd 0.0500 time 0.3769 (0.3888) data time 0.0008 (0.0115) model time 0.0000 (0.0000) loss 4.2530 (3.4211) grad_norm 1.7908 (1.2710) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:20:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][50/625] eta 0:03:42 lr 0.001876 wd 0.0500 time 0.3748 (0.3865) data time 0.0009 (0.0095) model time 0.0000 (0.0000) loss 4.0030 (3.4397) grad_norm 2.3732 (1.3379) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:20:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][60/625] eta 0:03:39 lr 0.001876 wd 0.0500 time 0.5773 (0.3886) data time 0.0008 (0.0081) model time 0.5765 (0.3976) loss 3.7570 (3.4749) grad_norm 1.0775 (1.3925) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][70/625] eta 0:03:40 lr 0.001876 wd 0.0500 time 0.3841 (0.3977) data time 0.0011 (0.0071) model time 0.3830 (0.4251) loss 4.0741 (3.4766) grad_norm 1.1794 (1.3759) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][80/625] eta 0:03:35 lr 0.001876 wd 0.0500 time 0.3765 (0.3950) data time 0.0011 (0.0064) model time 0.3754 (0.4081) loss 4.1354 (3.4898) grad_norm 1.8123 (1.3589) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][90/625] eta 0:03:30 lr 0.001876 wd 0.0500 time 0.3816 (0.3929) data time 0.0009 (0.0058) model time 0.3807 (0.3997) loss 4.5950 (3.4813) grad_norm 1.0886 (1.3614) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][100/625] eta 0:03:25 lr 0.001876 wd 0.0500 time 0.3710 (0.3912) data time 0.0009 (0.0054) model time 0.3701 (0.3947) loss 4.5338 (3.4977) grad_norm 1.0802 (1.3497) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][110/625] eta 0:03:20 lr 0.001876 wd 0.0500 time 0.3711 (0.3898) data time 0.0012 (0.0050) model time 0.3699 (0.3914) loss 3.5551 (3.5007) grad_norm 0.9061 (1.3415) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][120/625] eta 0:03:16 lr 0.001875 wd 0.0500 time 0.3788 (0.3889) data time 0.0008 (0.0047) model time 0.3779 (0.3895) loss 2.3490 (3.4715) grad_norm 1.1422 (1.3312) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][130/625] eta 0:03:12 lr 0.001875 wd 0.0500 time 0.3820 (0.3880) data time 0.0011 (0.0044) model time 0.3808 (0.3878) loss 3.4458 (3.4631) grad_norm 1.5035 (1.3202) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][140/625] eta 0:03:07 lr 0.001875 wd 0.0500 time 0.3793 (0.3873) data time 0.0009 (0.0042) model time 0.3785 (0.3866) loss 2.4271 (3.4510) grad_norm 1.1224 (1.3136) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][150/625] eta 0:03:04 lr 0.001875 wd 0.0500 time 0.3820 (0.3881) data time 0.0012 (0.0040) model time 0.3808 (0.3878) loss 4.3002 (3.4635) grad_norm 1.0365 (1.3056) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][160/625] eta 0:03:00 lr 0.001875 wd 0.0500 time 0.3755 (0.3876) data time 0.0008 (0.0038) model time 0.3747 (0.3869) loss 3.7900 (3.4911) grad_norm 1.1021 (1.3054) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][170/625] eta 0:02:56 lr 0.001875 wd 0.0500 time 0.3796 (0.3872) data time 0.0011 (0.0036) model time 0.3785 (0.3862) loss 3.7182 (3.4834) grad_norm 1.6193 (1.3033) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][180/625] eta 0:02:52 lr 0.001875 wd 0.0500 time 0.3773 (0.3868) data time 0.0009 (0.0035) model time 0.3764 (0.3857) loss 3.9464 (3.4822) grad_norm 1.2750 (1.3028) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][190/625] eta 0:02:48 lr 0.001875 wd 0.0500 time 0.3860 (0.3863) data time 0.0011 (0.0034) model time 0.3849 (0.3850) loss 3.5718 (3.4867) grad_norm 1.2830 (1.3001) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][200/625] eta 0:02:43 lr 0.001875 wd 0.0500 time 0.3801 (0.3858) data time 0.0009 (0.0033) model time 0.3793 (0.3844) loss 3.1450 (3.4837) grad_norm 1.0169 (1.2917) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][210/625] eta 0:02:39 lr 0.001875 wd 0.0500 time 0.3814 (0.3855) data time 0.0011 (0.0032) model time 0.3803 (0.3840) loss 3.7130 (3.5009) grad_norm 1.5188 (1.2937) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:21:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][220/625] eta 0:02:36 lr 0.001875 wd 0.0500 time 0.3749 (0.3853) data time 0.0009 (0.0031) model time 0.3740 (0.3838) loss 3.2375 (3.4958) grad_norm 1.1631 (1.2952) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][230/625] eta 0:02:32 lr 0.001874 wd 0.0500 time 0.3705 (0.3849) data time 0.0008 (0.0030) model time 0.3696 (0.3833) loss 3.4693 (3.4958) grad_norm 1.0942 (1.2897) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][240/625] eta 0:02:28 lr 0.001874 wd 0.0500 time 0.3714 (0.3846) data time 0.0010 (0.0029) model time 0.3703 (0.3829) loss 3.6434 (3.4926) grad_norm 1.5610 (1.2862) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][250/625] eta 0:02:24 lr 0.001874 wd 0.0500 time 0.3742 (0.3866) data time 0.0009 (0.0029) model time 0.3733 (0.3854) loss 4.0632 (3.5101) grad_norm 1.1492 (1.2799) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][260/625] eta 0:02:20 lr 0.001874 wd 0.0500 time 0.3782 (0.3862) data time 0.0012 (0.0028) model time 0.3771 (0.3850) loss 2.7228 (3.5134) grad_norm 1.2166 (1.2785) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][270/625] eta 0:02:17 lr 0.001874 wd 0.0500 time 0.3788 (0.3866) data time 0.0011 (0.0027) model time 0.3777 (0.3854) loss 4.0461 (3.5126) grad_norm 1.7331 (1.2873) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][280/625] eta 0:02:13 lr 0.001874 wd 0.0500 time 0.3759 (0.3878) data time 0.0008 (0.0027) model time 0.3751 (0.3870) loss 4.0591 (3.5126) grad_norm 1.0834 (1.2877) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][290/625] eta 0:02:09 lr 0.001874 wd 0.0500 time 0.3827 (0.3876) data time 0.0012 (0.0026) model time 0.3815 (0.3866) loss 4.2418 (3.5109) grad_norm 1.3096 (1.2823) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][300/625] eta 0:02:05 lr 0.001874 wd 0.0500 time 0.3886 (0.3873) data time 0.0010 (0.0026) model time 0.3876 (0.3863) loss 3.4025 (3.5151) grad_norm 1.2654 (1.2778) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][310/625] eta 0:02:01 lr 0.001874 wd 0.0500 time 0.3779 (0.3870) data time 0.0008 (0.0025) model time 0.3771 (0.3860) loss 3.5246 (3.5111) grad_norm 1.2117 (1.2788) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][320/625] eta 0:01:57 lr 0.001874 wd 0.0500 time 0.3837 (0.3868) data time 0.0008 (0.0025) model time 0.3829 (0.3857) loss 4.4526 (3.5150) grad_norm 1.1425 (1.2775) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][330/625] eta 0:01:54 lr 0.001874 wd 0.0500 time 0.3816 (0.3865) data time 0.0008 (0.0024) model time 0.3808 (0.3853) loss 3.4193 (3.5121) grad_norm 1.0964 (1.2767) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][340/625] eta 0:01:50 lr 0.001874 wd 0.0500 time 0.3745 (0.3863) data time 0.0011 (0.0024) model time 0.3734 (0.3850) loss 3.4571 (3.5154) grad_norm 1.4015 (1.2793) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][350/625] eta 0:01:46 lr 0.001873 wd 0.0500 time 0.3783 (0.3860) data time 0.0009 (0.0024) model time 0.3774 (0.3848) loss 3.6852 (3.5250) grad_norm 1.0510 (1.2821) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][360/625] eta 0:01:42 lr 0.001873 wd 0.0500 time 0.3831 (0.3858) data time 0.0011 (0.0023) model time 0.3820 (0.3845) loss 3.7272 (3.5230) grad_norm 1.1890 (1.2853) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][370/625] eta 0:01:38 lr 0.001873 wd 0.0500 time 0.3773 (0.3867) data time 0.0009 (0.0023) model time 0.3765 (0.3855) loss 3.9642 (3.5117) grad_norm 1.3136 (1.3001) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:22:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][380/625] eta 0:01:34 lr 0.001873 wd 0.0500 time 0.3776 (0.3865) data time 0.0010 (0.0023) model time 0.3766 (0.3853) loss 3.3482 (3.5041) grad_norm 1.1217 (1.2992) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][390/625] eta 0:01:30 lr 0.001873 wd 0.0500 time 0.3850 (0.3863) data time 0.0011 (0.0022) model time 0.3840 (0.3851) loss 2.6667 (3.5075) grad_norm 1.2892 (1.2984) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][400/625] eta 0:01:26 lr 0.001873 wd 0.0500 time 0.3747 (0.3863) data time 0.0008 (0.0022) model time 0.3738 (0.3850) loss 4.1768 (3.5207) grad_norm 1.3861 (1.3000) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][410/625] eta 0:01:23 lr 0.001873 wd 0.0500 time 0.3772 (0.3861) data time 0.0010 (0.0022) model time 0.3762 (0.3849) loss 4.3486 (3.5195) grad_norm 1.2926 (1.2995) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][420/625] eta 0:01:19 lr 0.001873 wd 0.0500 time 0.3836 (0.3860) data time 0.0010 (0.0022) model time 0.3826 (0.3847) loss 3.2125 (3.5218) grad_norm 1.0122 (1.2973) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][430/625] eta 0:01:15 lr 0.001873 wd 0.0500 time 0.3807 (0.3859) data time 0.0011 (0.0022) model time 0.3796 (0.3846) loss 3.8493 (3.5185) grad_norm 0.8902 (1.2965) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][440/625] eta 0:01:11 lr 0.001873 wd 0.0500 time 0.3749 (0.3857) data time 0.0012 (0.0021) model time 0.3737 (0.3844) loss 3.8888 (3.5211) grad_norm 0.9612 (1.2948) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][450/625] eta 0:01:07 lr 0.001873 wd 0.0500 time 0.3865 (0.3856) data time 0.0009 (0.0021) model time 0.3856 (0.3843) loss 3.4066 (3.5168) grad_norm 1.1997 (1.2938) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][460/625] eta 0:01:03 lr 0.001872 wd 0.0500 time 0.3810 (0.3856) data time 0.0008 (0.0021) model time 0.3801 (0.3843) loss 3.9534 (3.5182) grad_norm 1.4902 (1.2932) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][470/625] eta 0:00:59 lr 0.001872 wd 0.0500 time 0.3809 (0.3862) data time 0.0010 (0.0021) model time 0.3799 (0.3850) loss 3.8836 (3.5174) grad_norm 1.4220 (1.2954) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][480/625] eta 0:00:56 lr 0.001872 wd 0.0500 time 0.3998 (0.3865) data time 0.0009 (0.0021) model time 0.3989 (0.3853) loss 3.0342 (3.5169) grad_norm 1.0374 (1.2929) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][490/625] eta 0:00:52 lr 0.001872 wd 0.0500 time 0.3731 (0.3864) data time 0.0010 (0.0020) model time 0.3721 (0.3852) loss 3.6213 (3.5226) grad_norm 1.3404 (1.2894) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][500/625] eta 0:00:48 lr 0.001872 wd 0.0500 time 0.3743 (0.3863) data time 0.0014 (0.0021) model time 0.3729 (0.3850) loss 3.8033 (3.5241) grad_norm 0.8984 (1.2858) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][510/625] eta 0:00:44 lr 0.001872 wd 0.0500 time 0.3795 (0.3862) data time 0.0009 (0.0020) model time 0.3786 (0.3850) loss 2.6254 (3.5245) grad_norm 1.6597 (1.2863) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][520/625] eta 0:00:40 lr 0.001872 wd 0.0500 time 0.3707 (0.3861) data time 0.0009 (0.0020) model time 0.3699 (0.3849) loss 4.1990 (3.5301) grad_norm 1.1664 (1.2911) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:23:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][530/625] eta 0:00:36 lr 0.001872 wd 0.0500 time 0.3715 (0.3860) data time 0.0012 (0.0020) model time 0.3703 (0.3847) loss 2.5907 (3.5294) grad_norm 1.0625 (1.2881) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][540/625] eta 0:00:32 lr 0.001872 wd 0.0500 time 0.3832 (0.3858) data time 0.0009 (0.0020) model time 0.3823 (0.3845) loss 2.4518 (3.5284) grad_norm 1.7869 (1.2881) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][550/625] eta 0:00:28 lr 0.001872 wd 0.0500 time 0.3747 (0.3857) data time 0.0008 (0.0020) model time 0.3739 (0.3844) loss 3.4709 (3.5311) grad_norm 1.3969 (1.2906) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][560/625] eta 0:00:25 lr 0.001872 wd 0.0500 time 0.3734 (0.3855) data time 0.0011 (0.0020) model time 0.3724 (0.3842) loss 3.8601 (3.5295) grad_norm 1.2256 (1.2876) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][570/625] eta 0:00:21 lr 0.001872 wd 0.0500 time 0.3764 (0.3854) data time 0.0008 (0.0019) model time 0.3757 (0.3841) loss 3.6501 (3.5303) grad_norm 1.0658 (1.2856) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][580/625] eta 0:00:17 lr 0.001871 wd 0.0500 time 0.3792 (0.3860) data time 0.0011 (0.0019) model time 0.3781 (0.3847) loss 4.1483 (3.5306) grad_norm 0.9650 (1.2837) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][590/625] eta 0:00:13 lr 0.001871 wd 0.0500 time 0.3815 (0.3865) data time 0.0012 (0.0019) model time 0.3803 (0.3853) loss 3.7694 (3.5344) grad_norm 1.8089 (1.2859) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][600/625] eta 0:00:09 lr 0.001871 wd 0.0500 time 0.3770 (0.3864) data time 0.0011 (0.0019) model time 0.3759 (0.3852) loss 3.5020 (3.5343) grad_norm 1.0904 (1.2879) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][610/625] eta 0:00:05 lr 0.001871 wd 0.0500 time 0.3669 (0.3863) data time 0.0006 (0.0019) model time 0.3663 (0.3851) loss 4.3329 (3.5383) grad_norm 1.1580 (1.2900) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [65/300][620/625] eta 0:00:01 lr 0.001871 wd 0.0500 time 0.3658 (0.3860) data time 0.0006 (0.0019) model time 0.3652 (0.3848) loss 1.9427 (3.5406) grad_norm 1.2049 (1.2921) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 65 training takes 0:04:01 [2024-08-26 07:24:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:24:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:24:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.517 (0.517) Loss 0.7515 (0.7515) Acc@1 83.154 (83.154) Acc@5 96.631 (96.631) Mem 11225MB [2024-08-26 07:24:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 1.1670 (0.8702) Acc@1 72.559 (80.220) Acc@5 92.334 (95.796) Mem 11225MB [2024-08-26 07:24:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.2998 (1.0378) Acc@1 69.824 (76.411) Acc@5 90.283 (93.550) Mem 11225MB [2024-08-26 07:24:37 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.150 Acc@5 93.468 [2024-08-26 07:24:37 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.1% [2024-08-26 07:24:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.872 (0.872) Loss 0.5693 (0.5693) Acc@1 87.402 (87.402) Acc@5 97.998 (97.998) Mem 11225MB [2024-08-26 07:24:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.179) Loss 0.9790 (0.7178) Acc@1 76.660 (83.163) Acc@5 93.994 (96.751) Mem 11225MB [2024-08-26 07:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.138) Loss 1.1514 (0.8869) Acc@1 71.240 (79.053) Acc@5 91.895 (94.754) Mem 11225MB [2024-08-26 07:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.711 Acc@5 94.740 [2024-08-26 07:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.7% [2024-08-26 07:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.71% [2024-08-26 07:24:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:24:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:24:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][0/625] eta 0:08:08 lr 0.001871 wd 0.0500 time 0.7822 (0.7822) data time 0.4443 (0.4443) model time 0.0000 (0.0000) loss 3.2359 (3.2359) grad_norm 1.0868 (1.0868) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][10/625] eta 0:04:16 lr 0.001871 wd 0.0500 time 0.3804 (0.4163) data time 0.0010 (0.0415) model time 0.0000 (0.0000) loss 3.9048 (3.5546) grad_norm 1.7047 (1.2093) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][20/625] eta 0:04:01 lr 0.001871 wd 0.0500 time 0.3779 (0.3994) data time 0.0008 (0.0223) model time 0.0000 (0.0000) loss 4.1349 (3.5155) grad_norm 1.1947 (1.2520) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][30/625] eta 0:03:53 lr 0.001871 wd 0.0500 time 0.3691 (0.3918) data time 0.0011 (0.0155) model time 0.0000 (0.0000) loss 4.1928 (3.5648) grad_norm 1.4461 (1.2198) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:24:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][40/625] eta 0:03:47 lr 0.001871 wd 0.0500 time 0.4036 (0.3892) data time 0.0012 (0.0120) model time 0.0000 (0.0000) loss 3.8226 (3.5201) grad_norm 1.1229 (1.2475) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][50/625] eta 0:03:42 lr 0.001871 wd 0.0500 time 0.3770 (0.3874) data time 0.0008 (0.0100) model time 0.0000 (0.0000) loss 1.9454 (3.4486) grad_norm 0.9716 (1.2655) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][60/625] eta 0:03:37 lr 0.001871 wd 0.0500 time 0.3758 (0.3857) data time 0.0011 (0.0085) model time 0.3746 (0.3756) loss 3.4236 (3.4612) grad_norm 1.3033 (1.2636) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][70/625] eta 0:03:33 lr 0.001870 wd 0.0500 time 0.3766 (0.3848) data time 0.0011 (0.0075) model time 0.3754 (0.3769) loss 3.3337 (3.4773) grad_norm 1.2673 (1.2782) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][80/625] eta 0:03:29 lr 0.001870 wd 0.0500 time 0.3715 (0.3840) data time 0.0014 (0.0067) model time 0.3701 (0.3770) loss 3.9859 (3.4755) grad_norm 1.3113 (1.3103) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][90/625] eta 0:03:25 lr 0.001870 wd 0.0500 time 0.3827 (0.3836) data time 0.0008 (0.0061) model time 0.3819 (0.3775) loss 2.8309 (3.4604) grad_norm 0.9657 (1.2992) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][100/625] eta 0:03:22 lr 0.001870 wd 0.0500 time 0.3748 (0.3867) data time 0.0009 (0.0056) model time 0.3739 (0.3847) loss 2.6973 (3.4761) grad_norm 1.7687 (1.2912) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][110/625] eta 0:03:18 lr 0.001870 wd 0.0500 time 0.3835 (0.3860) data time 0.0011 (0.0052) model time 0.3824 (0.3836) loss 3.3227 (3.4764) grad_norm 2.1448 (1.3115) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][120/625] eta 0:03:14 lr 0.001870 wd 0.0500 time 0.3785 (0.3853) data time 0.0008 (0.0049) model time 0.3777 (0.3826) loss 4.2583 (3.4957) grad_norm 0.9865 (1.2945) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][130/625] eta 0:03:10 lr 0.001870 wd 0.0500 time 0.3798 (0.3849) data time 0.0008 (0.0046) model time 0.3790 (0.3821) loss 2.6363 (3.4994) grad_norm 1.1380 (1.2896) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][140/625] eta 0:03:06 lr 0.001870 wd 0.0500 time 0.3718 (0.3844) data time 0.0011 (0.0044) model time 0.3707 (0.3815) loss 4.0289 (3.5114) grad_norm 1.2069 (1.2842) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][150/625] eta 0:03:03 lr 0.001870 wd 0.0500 time 0.5870 (0.3854) data time 0.0008 (0.0041) model time 0.5861 (0.3833) loss 3.9910 (3.5274) grad_norm 1.2979 (1.2946) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][160/625] eta 0:02:59 lr 0.001870 wd 0.0500 time 0.3723 (0.3871) data time 0.0011 (0.0040) model time 0.3712 (0.3858) loss 3.9544 (3.5103) grad_norm 1.0257 (1.3081) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][170/625] eta 0:02:55 lr 0.001870 wd 0.0500 time 0.3773 (0.3864) data time 0.0008 (0.0038) model time 0.3765 (0.3849) loss 3.6085 (3.5085) grad_norm 1.1943 (1.2995) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][180/625] eta 0:02:51 lr 0.001869 wd 0.0500 time 0.3746 (0.3859) data time 0.0008 (0.0036) model time 0.3738 (0.3842) loss 2.8686 (3.4857) grad_norm 1.5829 (1.3011) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][190/625] eta 0:02:47 lr 0.001869 wd 0.0500 time 0.3660 (0.3855) data time 0.0011 (0.0035) model time 0.3650 (0.3836) loss 2.5598 (3.4936) grad_norm 1.2069 (1.3073) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][200/625] eta 0:02:43 lr 0.001869 wd 0.0500 time 0.3797 (0.3851) data time 0.0009 (0.0034) model time 0.3788 (0.3832) loss 4.0970 (3.4951) grad_norm 0.9986 (1.3138) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][210/625] eta 0:02:40 lr 0.001869 wd 0.0500 time 0.3871 (0.3857) data time 0.0009 (0.0033) model time 0.3862 (0.3840) loss 2.7043 (3.4817) grad_norm 1.0304 (1.3064) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][220/625] eta 0:02:36 lr 0.001869 wd 0.0500 time 0.3724 (0.3853) data time 0.0009 (0.0032) model time 0.3715 (0.3835) loss 4.1782 (3.4891) grad_norm 1.1875 (1.2996) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][230/625] eta 0:02:32 lr 0.001869 wd 0.0500 time 0.3738 (0.3849) data time 0.0008 (0.0031) model time 0.3730 (0.3830) loss 2.5240 (3.4875) grad_norm 1.1968 (1.2930) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][240/625] eta 0:02:28 lr 0.001869 wd 0.0500 time 0.3811 (0.3846) data time 0.0008 (0.0030) model time 0.3803 (0.3826) loss 3.1367 (3.4784) grad_norm 1.2655 (1.2953) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][250/625] eta 0:02:24 lr 0.001869 wd 0.0500 time 0.3750 (0.3843) data time 0.0011 (0.0030) model time 0.3739 (0.3823) loss 3.1153 (3.4752) grad_norm 1.1855 (1.2965) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][260/625] eta 0:02:20 lr 0.001869 wd 0.0500 time 0.3796 (0.3841) data time 0.0011 (0.0029) model time 0.3785 (0.3821) loss 3.4768 (3.4786) grad_norm 1.6655 (1.3052) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][270/625] eta 0:02:16 lr 0.001869 wd 0.0500 time 0.3831 (0.3838) data time 0.0011 (0.0028) model time 0.3820 (0.3818) loss 3.9522 (3.4839) grad_norm 1.3097 (1.3069) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][280/625] eta 0:02:12 lr 0.001869 wd 0.0500 time 0.3761 (0.3836) data time 0.0013 (0.0028) model time 0.3748 (0.3815) loss 2.7184 (3.4888) grad_norm 1.3030 (1.3041) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][290/625] eta 0:02:08 lr 0.001868 wd 0.0500 time 0.3821 (0.3834) data time 0.0011 (0.0027) model time 0.3810 (0.3813) loss 3.3437 (3.4900) grad_norm 0.8622 (1.2983) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][300/625] eta 0:02:04 lr 0.001868 wd 0.0500 time 0.3827 (0.3832) data time 0.0008 (0.0027) model time 0.3819 (0.3811) loss 3.7904 (3.4940) grad_norm 1.1846 (1.2996) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][310/625] eta 0:02:00 lr 0.001868 wd 0.0500 time 0.3807 (0.3836) data time 0.0011 (0.0026) model time 0.3796 (0.3817) loss 4.1017 (3.4992) grad_norm 0.8354 (1.2946) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][320/625] eta 0:01:56 lr 0.001868 wd 0.0500 time 0.3687 (0.3834) data time 0.0010 (0.0026) model time 0.3676 (0.3814) loss 4.1088 (3.5124) grad_norm 0.8903 (1.2919) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][330/625] eta 0:01:53 lr 0.001868 wd 0.0500 time 0.3745 (0.3832) data time 0.0011 (0.0025) model time 0.3735 (0.3812) loss 3.9409 (3.5159) grad_norm 1.1736 (1.2933) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][340/625] eta 0:01:49 lr 0.001868 wd 0.0500 time 0.3739 (0.3841) data time 0.0008 (0.0025) model time 0.3731 (0.3824) loss 4.3617 (3.5198) grad_norm 1.1930 (1.3017) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:26:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][350/625] eta 0:01:45 lr 0.001868 wd 0.0500 time 0.3807 (0.3840) data time 0.0009 (0.0025) model time 0.3798 (0.3823) loss 2.2977 (3.5221) grad_norm 1.4391 (1.3004) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][360/625] eta 0:01:41 lr 0.001868 wd 0.0500 time 0.3830 (0.3839) data time 0.0010 (0.0024) model time 0.3820 (0.3822) loss 4.1715 (3.5265) grad_norm 1.1727 (1.2967) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][370/625] eta 0:01:37 lr 0.001868 wd 0.0500 time 0.3732 (0.3837) data time 0.0009 (0.0024) model time 0.3723 (0.3820) loss 3.0869 (3.5312) grad_norm 1.3788 (1.2915) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][380/625] eta 0:01:33 lr 0.001868 wd 0.0500 time 0.3821 (0.3836) data time 0.0009 (0.0024) model time 0.3813 (0.3819) loss 4.0242 (3.5351) grad_norm 1.0130 (1.2939) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][390/625] eta 0:01:30 lr 0.001868 wd 0.0500 time 0.3840 (0.3840) data time 0.0011 (0.0023) model time 0.3830 (0.3823) loss 2.7444 (3.5387) grad_norm 1.3643 (1.2976) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][400/625] eta 0:01:26 lr 0.001868 wd 0.0500 time 0.3855 (0.3839) data time 0.0008 (0.0023) model time 0.3847 (0.3822) loss 3.6931 (3.5393) grad_norm 0.9139 (1.2954) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][410/625] eta 0:01:22 lr 0.001867 wd 0.0500 time 0.3753 (0.3838) data time 0.0008 (0.0023) model time 0.3745 (0.3821) loss 4.3063 (3.5421) grad_norm 1.1366 (1.2944) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][420/625] eta 0:01:18 lr 0.001867 wd 0.0500 time 0.3822 (0.3837) data time 0.0011 (0.0023) model time 0.3812 (0.3820) loss 3.4653 (3.5413) grad_norm 1.0939 (1.2974) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][430/625] eta 0:01:14 lr 0.001867 wd 0.0500 time 0.3848 (0.3837) data time 0.0009 (0.0022) model time 0.3839 (0.3820) loss 3.0414 (3.5360) grad_norm 1.7769 (1.2958) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][440/625] eta 0:01:10 lr 0.001867 wd 0.0500 time 0.3914 (0.3836) data time 0.0010 (0.0022) model time 0.3904 (0.3819) loss 3.6111 (3.5407) grad_norm 0.9133 (1.2902) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][450/625] eta 0:01:07 lr 0.001867 wd 0.0500 time 0.3773 (0.3835) data time 0.0011 (0.0022) model time 0.3762 (0.3818) loss 3.9700 (3.5402) grad_norm 1.2413 (1.2904) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][460/625] eta 0:01:03 lr 0.001867 wd 0.0500 time 0.3786 (0.3843) data time 0.0012 (0.0022) model time 0.3774 (0.3827) loss 3.5880 (3.5412) grad_norm 1.1515 (1.2869) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][470/625] eta 0:00:59 lr 0.001867 wd 0.0500 time 0.3784 (0.3842) data time 0.0011 (0.0021) model time 0.3773 (0.3826) loss 3.0421 (3.5416) grad_norm 1.5688 (1.2878) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][480/625] eta 0:00:55 lr 0.001867 wd 0.0500 time 0.3763 (0.3841) data time 0.0010 (0.0021) model time 0.3753 (0.3825) loss 3.8216 (3.5420) grad_norm 1.1733 (1.2883) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][490/625] eta 0:00:51 lr 0.001867 wd 0.0500 time 0.3783 (0.3848) data time 0.0008 (0.0021) model time 0.3774 (0.3833) loss 3.6045 (3.5385) grad_norm 1.2996 (1.2848) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][500/625] eta 0:00:48 lr 0.001867 wd 0.0500 time 0.3799 (0.3847) data time 0.0011 (0.0021) model time 0.3788 (0.3832) loss 3.1744 (3.5417) grad_norm 1.9215 (1.2900) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:27:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][510/625] eta 0:00:44 lr 0.001867 wd 0.0500 time 0.3784 (0.3849) data time 0.0011 (0.0021) model time 0.3773 (0.3835) loss 3.6553 (3.5467) grad_norm 1.0645 (1.2906) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][520/625] eta 0:00:40 lr 0.001866 wd 0.0500 time 0.3791 (0.3848) data time 0.0011 (0.0021) model time 0.3780 (0.3833) loss 3.6155 (3.5481) grad_norm 1.3242 (1.2941) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][530/625] eta 0:00:36 lr 0.001866 wd 0.0500 time 0.3813 (0.3847) data time 0.0011 (0.0020) model time 0.3802 (0.3832) loss 3.6956 (3.5501) grad_norm 1.1687 (1.2924) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][540/625] eta 0:00:32 lr 0.001866 wd 0.0500 time 0.3777 (0.3846) data time 0.0008 (0.0020) model time 0.3768 (0.3831) loss 4.1510 (3.5569) grad_norm 1.1296 (1.2921) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][550/625] eta 0:00:28 lr 0.001866 wd 0.0500 time 0.3778 (0.3845) data time 0.0009 (0.0020) model time 0.3769 (0.3830) loss 3.9819 (3.5550) grad_norm 1.9308 (1.2935) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][560/625] eta 0:00:24 lr 0.001866 wd 0.0500 time 0.3770 (0.3844) data time 0.0008 (0.0020) model time 0.3762 (0.3829) loss 2.1763 (3.5519) grad_norm 1.2882 (1.2922) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][570/625] eta 0:00:21 lr 0.001866 wd 0.0500 time 0.3757 (0.3843) data time 0.0008 (0.0020) model time 0.3749 (0.3827) loss 3.6441 (3.5500) grad_norm 1.0739 (1.2899) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][580/625] eta 0:00:17 lr 0.001866 wd 0.0500 time 0.3690 (0.3841) data time 0.0008 (0.0020) model time 0.3681 (0.3826) loss 3.8297 (3.5515) grad_norm 0.9912 (1.2921) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][590/625] eta 0:00:13 lr 0.001866 wd 0.0500 time 0.3752 (0.3840) data time 0.0011 (0.0020) model time 0.3741 (0.3825) loss 3.3577 (3.5503) grad_norm 1.2070 (1.2899) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][600/625] eta 0:00:09 lr 0.001866 wd 0.0500 time 0.3733 (0.3839) data time 0.0008 (0.0019) model time 0.3725 (0.3824) loss 4.4056 (3.5501) grad_norm 1.4111 (1.2918) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][610/625] eta 0:00:05 lr 0.001866 wd 0.0500 time 0.3649 (0.3845) data time 0.0008 (0.0019) model time 0.3641 (0.3830) loss 3.2835 (3.5478) grad_norm 1.2780 (1.2920) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [66/300][620/625] eta 0:00:01 lr 0.001866 wd 0.0500 time 0.3653 (0.3842) data time 0.0008 (0.0019) model time 0.3645 (0.3827) loss 3.7276 (3.5491) grad_norm 1.0436 (1.2914) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 66 training takes 0:04:00 [2024-08-26 07:28:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:28:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:28:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 0.6729 (0.6729) Acc@1 83.594 (83.594) Acc@5 97.363 (97.363) Mem 11225MB [2024-08-26 07:28:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 1.1377 (0.8423) Acc@1 73.047 (80.362) Acc@5 92.383 (95.761) Mem 11225MB [2024-08-26 07:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.3242 (1.0183) Acc@1 69.238 (76.302) Acc@5 89.893 (93.624) Mem 11225MB [2024-08-26 07:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.096 Acc@5 93.578 [2024-08-26 07:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.1% [2024-08-26 07:28:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.831 (0.831) Loss 0.5688 (0.5688) Acc@1 87.646 (87.646) Acc@5 97.900 (97.900) Mem 11225MB [2024-08-26 07:28:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.180) Loss 0.9795 (0.7173) Acc@1 76.709 (83.243) Acc@5 93.945 (96.746) Mem 11225MB [2024-08-26 07:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.138) Loss 1.1475 (0.8852) Acc@1 71.680 (79.136) Acc@5 91.895 (94.766) Mem 11225MB [2024-08-26 07:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.791 Acc@5 94.748 [2024-08-26 07:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.8% [2024-08-26 07:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.79% [2024-08-26 07:28:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:28:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:28:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][0/625] eta 0:07:43 lr 0.001866 wd 0.0500 time 0.7411 (0.7411) data time 0.3948 (0.3948) model time 0.0000 (0.0000) loss 4.4266 (4.4266) grad_norm 1.0851 (1.0851) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][10/625] eta 0:04:11 lr 0.001865 wd 0.0500 time 0.3817 (0.4088) data time 0.0008 (0.0370) model time 0.0000 (0.0000) loss 3.9846 (3.5624) grad_norm 1.5595 (1.3424) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:28:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][20/625] eta 0:03:58 lr 0.001865 wd 0.0500 time 0.3773 (0.3935) data time 0.0011 (0.0199) model time 0.0000 (0.0000) loss 3.7883 (3.5387) grad_norm 1.1622 (1.2790) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][30/625] eta 0:03:57 lr 0.001865 wd 0.0500 time 0.3725 (0.3991) data time 0.0009 (0.0138) model time 0.0000 (0.0000) loss 4.1045 (3.5228) grad_norm 0.9814 (1.1994) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][40/625] eta 0:03:50 lr 0.001865 wd 0.0500 time 0.3782 (0.3943) data time 0.0008 (0.0107) model time 0.0000 (0.0000) loss 3.5209 (3.4913) grad_norm 1.3654 (1.3024) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][50/625] eta 0:03:45 lr 0.001865 wd 0.0500 time 0.3845 (0.3914) data time 0.0011 (0.0088) model time 0.0000 (0.0000) loss 3.0705 (3.4822) grad_norm 1.1350 (1.3118) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][60/625] eta 0:03:39 lr 0.001865 wd 0.0500 time 0.3748 (0.3890) data time 0.0008 (0.0076) model time 0.3740 (0.3758) loss 2.9956 (3.4616) grad_norm 1.1298 (1.3075) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][70/625] eta 0:03:34 lr 0.001865 wd 0.0500 time 0.3861 (0.3874) data time 0.0011 (0.0067) model time 0.3850 (0.3761) loss 3.6995 (3.4731) grad_norm 1.3319 (1.2976) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][80/625] eta 0:03:30 lr 0.001865 wd 0.0500 time 0.3766 (0.3862) data time 0.0010 (0.0060) model time 0.3756 (0.3761) loss 3.4797 (3.4745) grad_norm 1.2826 (1.3145) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][90/625] eta 0:03:26 lr 0.001865 wd 0.0500 time 0.3717 (0.3852) data time 0.0011 (0.0055) model time 0.3707 (0.3761) loss 3.6271 (3.4633) grad_norm 1.1833 (1.3208) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][100/625] eta 0:03:21 lr 0.001865 wd 0.0500 time 0.3788 (0.3844) data time 0.0011 (0.0051) model time 0.3777 (0.3760) loss 4.0493 (3.4570) grad_norm 1.7265 (1.3151) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][110/625] eta 0:03:17 lr 0.001865 wd 0.0500 time 0.3773 (0.3837) data time 0.0010 (0.0047) model time 0.3763 (0.3760) loss 3.0989 (3.4416) grad_norm 1.2952 (1.3148) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][120/625] eta 0:03:13 lr 0.001864 wd 0.0500 time 0.3843 (0.3831) data time 0.0011 (0.0044) model time 0.3832 (0.3760) loss 3.3377 (3.4337) grad_norm 0.9194 (1.3231) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 07:29:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][130/625] eta 0:03:09 lr 0.001864 wd 0.0500 time 0.3789 (0.3826) data time 0.0009 (0.0042) model time 0.3780 (0.3758) loss 4.1672 (3.4248) grad_norm 1.0291 (inf) loss_scale 2048.0000 (4033.4656) mem 11225MB [2024-08-26 07:29:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][140/625] eta 0:03:05 lr 0.001864 wd 0.0500 time 0.3680 (0.3823) data time 0.0009 (0.0039) model time 0.3671 (0.3760) loss 3.3010 (3.4452) grad_norm 0.9085 (inf) loss_scale 2048.0000 (3892.6525) mem 11225MB [2024-08-26 07:29:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][150/625] eta 0:03:01 lr 0.001864 wd 0.0500 time 0.3821 (0.3820) data time 0.0011 (0.0038) model time 0.3810 (0.3760) loss 3.2528 (3.4673) grad_norm 1.0532 (inf) loss_scale 2048.0000 (3770.4901) mem 11225MB [2024-08-26 07:29:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][160/625] eta 0:02:57 lr 0.001864 wd 0.0500 time 0.3839 (0.3817) data time 0.0011 (0.0036) model time 0.3829 (0.3761) loss 2.8524 (3.4645) grad_norm 1.1729 (inf) loss_scale 2048.0000 (3663.5031) mem 11225MB [2024-08-26 07:29:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][170/625] eta 0:02:54 lr 0.001864 wd 0.0500 time 0.5662 (0.3838) data time 0.0008 (0.0035) model time 0.5654 (0.3794) loss 2.8222 (3.4554) grad_norm 1.4654 (inf) loss_scale 2048.0000 (3569.0292) mem 11225MB [2024-08-26 07:30:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][180/625] eta 0:02:50 lr 0.001864 wd 0.0500 time 0.3732 (0.3834) data time 0.0008 (0.0033) model time 0.3724 (0.3791) loss 4.1872 (3.4582) grad_norm 1.8520 (inf) loss_scale 2048.0000 (3484.9945) mem 11225MB [2024-08-26 07:30:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][190/625] eta 0:02:46 lr 0.001864 wd 0.0500 time 0.3785 (0.3831) data time 0.0011 (0.0032) model time 0.3774 (0.3789) loss 3.5237 (3.4438) grad_norm 1.5233 (inf) loss_scale 2048.0000 (3409.7592) mem 11225MB [2024-08-26 07:30:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][200/625] eta 0:02:43 lr 0.001864 wd 0.0500 time 0.4267 (0.3838) data time 0.0010 (0.0031) model time 0.4256 (0.3801) loss 3.1922 (3.4437) grad_norm 1.0141 (inf) loss_scale 2048.0000 (3342.0100) mem 11225MB [2024-08-26 07:30:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][210/625] eta 0:02:39 lr 0.001864 wd 0.0500 time 0.3322 (0.3840) data time 0.0008 (0.0030) model time 0.3314 (0.3804) loss 4.2100 (3.4665) grad_norm 1.0000 (inf) loss_scale 2048.0000 (3280.6825) mem 11225MB [2024-08-26 07:30:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][220/625] eta 0:02:35 lr 0.001864 wd 0.0500 time 0.3731 (0.3834) data time 0.0009 (0.0030) model time 0.3722 (0.3799) loss 4.1661 (3.4713) grad_norm 1.3699 (inf) loss_scale 2048.0000 (3224.9050) mem 11225MB [2024-08-26 07:30:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][230/625] eta 0:02:31 lr 0.001863 wd 0.0500 time 0.3698 (0.3833) data time 0.0012 (0.0029) model time 0.3687 (0.3799) loss 2.5959 (3.4615) grad_norm 1.0540 (inf) loss_scale 2048.0000 (3173.9567) mem 11225MB [2024-08-26 07:30:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][240/625] eta 0:02:28 lr 0.001863 wd 0.0500 time 0.3671 (0.3846) data time 0.0011 (0.0028) model time 0.3660 (0.3816) loss 3.3319 (3.4649) grad_norm 1.6495 (inf) loss_scale 2048.0000 (3127.2365) mem 11225MB [2024-08-26 07:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][250/625] eta 0:02:23 lr 0.001863 wd 0.0500 time 0.3663 (0.3840) data time 0.0009 (0.0027) model time 0.3653 (0.3809) loss 4.2666 (3.4719) grad_norm 1.1512 (inf) loss_scale 2048.0000 (3084.2390) mem 11225MB [2024-08-26 07:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][260/625] eta 0:02:20 lr 0.001863 wd 0.0500 time 0.3734 (0.3836) data time 0.0009 (0.0027) model time 0.3724 (0.3805) loss 3.0365 (3.4669) grad_norm 1.0166 (inf) loss_scale 2048.0000 (3044.5364) mem 11225MB [2024-08-26 07:30:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][270/625] eta 0:02:15 lr 0.001863 wd 0.0500 time 0.3642 (0.3831) data time 0.0008 (0.0026) model time 0.3634 (0.3800) loss 3.0660 (3.4610) grad_norm 1.2398 (inf) loss_scale 2048.0000 (3007.7638) mem 11225MB [2024-08-26 07:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][280/625] eta 0:02:12 lr 0.001863 wd 0.0500 time 0.3887 (0.3828) data time 0.0009 (0.0026) model time 0.3878 (0.3797) loss 3.3585 (3.4638) grad_norm 1.5799 (inf) loss_scale 2048.0000 (2973.6085) mem 11225MB [2024-08-26 07:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][290/625] eta 0:02:08 lr 0.001863 wd 0.0500 time 0.5892 (0.3839) data time 0.0011 (0.0025) model time 0.5880 (0.3811) loss 3.8639 (3.4695) grad_norm 1.3683 (inf) loss_scale 2048.0000 (2941.8007) mem 11225MB [2024-08-26 07:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][300/625] eta 0:02:04 lr 0.001863 wd 0.0500 time 0.3726 (0.3836) data time 0.0008 (0.0025) model time 0.3718 (0.3808) loss 4.3475 (3.4766) grad_norm 1.3113 (inf) loss_scale 2048.0000 (2912.1063) mem 11225MB [2024-08-26 07:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][310/625] eta 0:02:00 lr 0.001863 wd 0.0500 time 0.3757 (0.3833) data time 0.0014 (0.0024) model time 0.3743 (0.3805) loss 3.9572 (3.4823) grad_norm 1.4395 (inf) loss_scale 2048.0000 (2884.3215) mem 11225MB [2024-08-26 07:30:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][320/625] eta 0:01:56 lr 0.001863 wd 0.0500 time 0.3654 (0.3830) data time 0.0008 (0.0024) model time 0.3645 (0.3802) loss 2.9522 (3.4781) grad_norm 2.3602 (inf) loss_scale 2048.0000 (2858.2679) mem 11225MB [2024-08-26 07:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][330/625] eta 0:01:53 lr 0.001863 wd 0.0500 time 0.3395 (0.3832) data time 0.0009 (0.0024) model time 0.3386 (0.3806) loss 4.4134 (3.4800) grad_norm 1.2142 (inf) loss_scale 2048.0000 (2833.7885) mem 11225MB [2024-08-26 07:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][340/625] eta 0:01:49 lr 0.001862 wd 0.0500 time 0.3745 (0.3829) data time 0.0008 (0.0023) model time 0.3737 (0.3802) loss 4.1379 (3.4761) grad_norm 1.0353 (inf) loss_scale 2048.0000 (2810.7449) mem 11225MB [2024-08-26 07:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][350/625] eta 0:01:45 lr 0.001862 wd 0.0500 time 0.3841 (0.3826) data time 0.0011 (0.0023) model time 0.3830 (0.3799) loss 3.4418 (3.4802) grad_norm 1.1696 (inf) loss_scale 2048.0000 (2789.0142) mem 11225MB [2024-08-26 07:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][360/625] eta 0:01:41 lr 0.001862 wd 0.0500 time 0.3697 (0.3823) data time 0.0010 (0.0023) model time 0.3687 (0.3797) loss 3.4304 (3.4858) grad_norm 1.2775 (inf) loss_scale 2048.0000 (2768.4875) mem 11225MB [2024-08-26 07:31:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][370/625] eta 0:01:37 lr 0.001862 wd 0.0500 time 0.3764 (0.3820) data time 0.0008 (0.0022) model time 0.3756 (0.3793) loss 2.2231 (3.4893) grad_norm 1.3538 (inf) loss_scale 2048.0000 (2749.0674) mem 11225MB [2024-08-26 07:31:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][380/625] eta 0:01:33 lr 0.001862 wd 0.0500 time 0.3603 (0.3816) data time 0.0008 (0.0022) model time 0.3595 (0.3789) loss 4.1853 (3.4892) grad_norm 2.1204 (inf) loss_scale 2048.0000 (2730.6667) mem 11225MB [2024-08-26 07:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][390/625] eta 0:01:29 lr 0.001862 wd 0.0500 time 0.3696 (0.3813) data time 0.0009 (0.0022) model time 0.3688 (0.3786) loss 3.4353 (3.4944) grad_norm 1.6805 (inf) loss_scale 2048.0000 (2713.2072) mem 11225MB [2024-08-26 07:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][400/625] eta 0:01:25 lr 0.001862 wd 0.0500 time 0.3663 (0.3810) data time 0.0010 (0.0022) model time 0.3653 (0.3783) loss 3.9483 (3.5057) grad_norm 1.0248 (inf) loss_scale 2048.0000 (2696.6185) mem 11225MB [2024-08-26 07:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][410/625] eta 0:01:21 lr 0.001862 wd 0.0500 time 0.3741 (0.3807) data time 0.0018 (0.0021) model time 0.3723 (0.3780) loss 3.6579 (3.5035) grad_norm 1.3151 (inf) loss_scale 2048.0000 (2680.8370) mem 11225MB [2024-08-26 07:31:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][420/625] eta 0:01:18 lr 0.001862 wd 0.0500 time 0.3675 (0.3815) data time 0.0008 (0.0021) model time 0.3666 (0.3790) loss 4.5058 (3.5107) grad_norm 1.3584 (inf) loss_scale 2048.0000 (2665.8052) mem 11225MB [2024-08-26 07:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][430/625] eta 0:01:14 lr 0.001862 wd 0.0500 time 0.3647 (0.3812) data time 0.0011 (0.0021) model time 0.3636 (0.3787) loss 3.6529 (3.5156) grad_norm 1.4103 (inf) loss_scale 2048.0000 (2651.4710) mem 11225MB [2024-08-26 07:31:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][440/625] eta 0:01:10 lr 0.001862 wd 0.0500 time 0.3763 (0.3809) data time 0.0008 (0.0021) model time 0.3754 (0.3784) loss 2.9265 (3.5156) grad_norm 0.8835 (inf) loss_scale 2048.0000 (2637.7868) mem 11225MB [2024-08-26 07:31:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][450/625] eta 0:01:06 lr 0.001861 wd 0.0500 time 0.3800 (0.3807) data time 0.0011 (0.0020) model time 0.3789 (0.3782) loss 3.1535 (3.5137) grad_norm 1.6120 (inf) loss_scale 2048.0000 (2624.7095) mem 11225MB [2024-08-26 07:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][460/625] eta 0:01:02 lr 0.001861 wd 0.0500 time 0.3777 (0.3805) data time 0.0008 (0.0020) model time 0.3768 (0.3780) loss 2.5305 (3.5065) grad_norm 1.0544 (inf) loss_scale 2048.0000 (2612.1996) mem 11225MB [2024-08-26 07:31:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][470/625] eta 0:00:58 lr 0.001861 wd 0.0500 time 0.3714 (0.3804) data time 0.0011 (0.0020) model time 0.3704 (0.3779) loss 3.8439 (3.5090) grad_norm 1.3100 (inf) loss_scale 2048.0000 (2600.2208) mem 11225MB [2024-08-26 07:31:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][480/625] eta 0:00:55 lr 0.001861 wd 0.0500 time 0.3736 (0.3803) data time 0.0008 (0.0020) model time 0.3728 (0.3778) loss 3.5992 (3.5044) grad_norm 1.5768 (inf) loss_scale 2048.0000 (2588.7401) mem 11225MB [2024-08-26 07:31:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][490/625] eta 0:00:51 lr 0.001861 wd 0.0500 time 0.3672 (0.3801) data time 0.0009 (0.0020) model time 0.3663 (0.3777) loss 3.6425 (3.5029) grad_norm 1.2637 (inf) loss_scale 2048.0000 (2577.7271) mem 11225MB [2024-08-26 07:32:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][500/625] eta 0:00:47 lr 0.001861 wd 0.0500 time 0.3768 (0.3810) data time 0.0010 (0.0019) model time 0.3758 (0.3787) loss 3.2723 (3.4971) grad_norm 0.9028 (inf) loss_scale 2048.0000 (2567.1537) mem 11225MB [2024-08-26 07:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][510/625] eta 0:00:43 lr 0.001861 wd 0.0500 time 0.3814 (0.3810) data time 0.0010 (0.0019) model time 0.3804 (0.3786) loss 4.1088 (3.5077) grad_norm 1.3088 (inf) loss_scale 2048.0000 (2556.9941) mem 11225MB [2024-08-26 07:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][520/625] eta 0:00:39 lr 0.001861 wd 0.0500 time 0.3840 (0.3809) data time 0.0012 (0.0019) model time 0.3828 (0.3786) loss 2.9316 (3.5079) grad_norm 1.1797 (inf) loss_scale 2048.0000 (2547.2246) mem 11225MB [2024-08-26 07:32:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][530/625] eta 0:00:36 lr 0.001861 wd 0.0500 time 0.3616 (0.3808) data time 0.0009 (0.0019) model time 0.3607 (0.3785) loss 2.9092 (3.5009) grad_norm 1.2640 (inf) loss_scale 2048.0000 (2537.8230) mem 11225MB [2024-08-26 07:32:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][540/625] eta 0:00:32 lr 0.001861 wd 0.0500 time 0.3850 (0.3818) data time 0.0008 (0.0019) model time 0.3842 (0.3796) loss 3.5357 (3.4989) grad_norm 1.6284 (inf) loss_scale 2048.0000 (2528.7689) mem 11225MB [2024-08-26 07:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][550/625] eta 0:00:28 lr 0.001861 wd 0.0500 time 0.3745 (0.3816) data time 0.0011 (0.0019) model time 0.3734 (0.3794) loss 3.1587 (3.4960) grad_norm 1.7101 (inf) loss_scale 2048.0000 (2520.0436) mem 11225MB [2024-08-26 07:32:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][560/625] eta 0:00:24 lr 0.001860 wd 0.0500 time 0.3737 (0.3814) data time 0.0009 (0.0019) model time 0.3729 (0.3792) loss 3.1973 (3.4926) grad_norm 1.3554 (inf) loss_scale 2048.0000 (2511.6292) mem 11225MB [2024-08-26 07:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][570/625] eta 0:00:20 lr 0.001860 wd 0.0500 time 0.3698 (0.3813) data time 0.0008 (0.0019) model time 0.3690 (0.3791) loss 2.5630 (3.4892) grad_norm 1.8595 (inf) loss_scale 2048.0000 (2503.5096) mem 11225MB [2024-08-26 07:32:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][580/625] eta 0:00:17 lr 0.001860 wd 0.0500 time 0.3680 (0.3811) data time 0.0008 (0.0018) model time 0.3672 (0.3790) loss 3.6575 (3.4939) grad_norm 1.9519 (inf) loss_scale 2048.0000 (2495.6695) mem 11225MB [2024-08-26 07:32:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][590/625] eta 0:00:13 lr 0.001860 wd 0.0500 time 0.3661 (0.3810) data time 0.0011 (0.0018) model time 0.3649 (0.3788) loss 3.9032 (3.4907) grad_norm 2.2059 (inf) loss_scale 2048.0000 (2488.0948) mem 11225MB [2024-08-26 07:32:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][600/625] eta 0:00:09 lr 0.001860 wd 0.0500 time 0.3734 (0.3808) data time 0.0009 (0.0018) model time 0.3725 (0.3787) loss 2.8041 (3.4971) grad_norm 1.5256 (inf) loss_scale 2048.0000 (2480.7720) mem 11225MB [2024-08-26 07:32:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][610/625] eta 0:00:05 lr 0.001860 wd 0.0500 time 0.3606 (0.3807) data time 0.0008 (0.0018) model time 0.3598 (0.3785) loss 3.3879 (3.4999) grad_norm 1.4334 (inf) loss_scale 2048.0000 (2473.6890) mem 11225MB [2024-08-26 07:32:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [67/300][620/625] eta 0:00:01 lr 0.001860 wd 0.0500 time 0.3560 (0.3803) data time 0.0008 (0.0018) model time 0.3552 (0.3782) loss 3.6568 (3.5070) grad_norm 1.4407 (inf) loss_scale 2048.0000 (2466.8341) mem 11225MB [2024-08-26 07:32:48 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 67 training takes 0:03:57 [2024-08-26 07:32:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:32:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.530 (0.530) Loss 0.6978 (0.6978) Acc@1 84.814 (84.814) Acc@5 97.217 (97.217) Mem 11225MB [2024-08-26 07:32:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.147) Loss 1.1250 (0.8358) Acc@1 73.779 (80.740) Acc@5 92.920 (95.934) Mem 11225MB [2024-08-26 07:32:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.2842 (1.0238) Acc@1 70.361 (76.507) Acc@5 90.186 (93.643) Mem 11225MB [2024-08-26 07:32:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.220 Acc@5 93.598 [2024-08-26 07:32:52 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.2% [2024-08-26 07:32:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.929 (0.929) Loss 0.5688 (0.5688) Acc@1 87.646 (87.646) Acc@5 97.998 (97.998) Mem 11225MB [2024-08-26 07:32:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.191) Loss 0.9771 (0.7168) Acc@1 76.514 (83.265) Acc@5 94.141 (96.773) Mem 11225MB [2024-08-26 07:32:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.145) Loss 1.1445 (0.8839) Acc@1 71.680 (79.215) Acc@5 91.895 (94.810) Mem 11225MB [2024-08-26 07:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.867 Acc@5 94.784 [2024-08-26 07:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.9% [2024-08-26 07:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.87% [2024-08-26 07:32:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:32:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:32:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][0/625] eta 0:08:23 lr 0.001860 wd 0.0500 time 0.8049 (0.8049) data time 0.4671 (0.4671) model time 0.0000 (0.0000) loss 3.2806 (3.2806) grad_norm 1.0646 (1.0646) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][10/625] eta 0:04:12 lr 0.001860 wd 0.0500 time 0.3681 (0.4109) data time 0.0008 (0.0435) model time 0.0000 (0.0000) loss 2.5271 (3.5731) grad_norm 1.0316 (1.4029) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][20/625] eta 0:03:57 lr 0.001860 wd 0.0500 time 0.3652 (0.3922) data time 0.0010 (0.0233) model time 0.0000 (0.0000) loss 4.0561 (3.5147) grad_norm 1.2809 (1.4641) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][30/625] eta 0:03:49 lr 0.001860 wd 0.0500 time 0.3680 (0.3858) data time 0.0009 (0.0162) model time 0.0000 (0.0000) loss 4.2697 (3.5824) grad_norm 1.8474 (1.4201) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][40/625] eta 0:03:43 lr 0.001859 wd 0.0500 time 0.3685 (0.3821) data time 0.0008 (0.0125) model time 0.0000 (0.0000) loss 3.4194 (3.5887) grad_norm 1.5376 (1.4398) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][50/625] eta 0:03:40 lr 0.001859 wd 0.0500 time 0.3697 (0.3838) data time 0.0010 (0.0103) model time 0.0000 (0.0000) loss 3.7982 (3.5565) grad_norm 0.9014 (1.3811) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][60/625] eta 0:03:35 lr 0.001859 wd 0.0500 time 0.3788 (0.3817) data time 0.0008 (0.0088) model time 0.3780 (0.3700) loss 2.8488 (3.6031) grad_norm 1.0005 (1.3542) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][70/625] eta 0:03:36 lr 0.001859 wd 0.0500 time 0.3719 (0.3893) data time 0.0011 (0.0077) model time 0.3708 (0.4021) loss 2.3512 (3.5917) grad_norm 2.0712 (1.3655) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][80/625] eta 0:03:31 lr 0.001859 wd 0.0500 time 0.3689 (0.3874) data time 0.0011 (0.0069) model time 0.3678 (0.3924) loss 3.9265 (3.5949) grad_norm 0.9243 (1.3410) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][90/625] eta 0:03:26 lr 0.001859 wd 0.0500 time 0.3943 (0.3866) data time 0.0015 (0.0063) model time 0.3928 (0.3891) loss 3.3868 (3.5390) grad_norm 1.1036 (1.3237) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][100/625] eta 0:03:22 lr 0.001859 wd 0.0500 time 0.3843 (0.3854) data time 0.0009 (0.0058) model time 0.3835 (0.3858) loss 4.0106 (3.5211) grad_norm 1.1124 (1.3097) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][110/625] eta 0:03:17 lr 0.001859 wd 0.0500 time 0.3700 (0.3842) data time 0.0008 (0.0053) model time 0.3692 (0.3835) loss 3.5936 (3.5338) grad_norm 1.0956 (1.3029) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][120/625] eta 0:03:13 lr 0.001859 wd 0.0500 time 0.3749 (0.3834) data time 0.0008 (0.0050) model time 0.3741 (0.3820) loss 3.5534 (3.5285) grad_norm 0.9706 (1.2906) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][130/625] eta 0:03:09 lr 0.001859 wd 0.0500 time 0.3782 (0.3826) data time 0.0008 (0.0047) model time 0.3774 (0.3808) loss 4.0852 (3.5171) grad_norm 1.3239 (1.2876) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][140/625] eta 0:03:05 lr 0.001859 wd 0.0500 time 0.3671 (0.3821) data time 0.0009 (0.0044) model time 0.3662 (0.3801) loss 2.5621 (3.5203) grad_norm 1.0162 (1.2880) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:33:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][150/625] eta 0:03:01 lr 0.001858 wd 0.0500 time 0.3755 (0.3816) data time 0.0008 (0.0042) model time 0.3746 (0.3794) loss 4.1033 (3.5324) grad_norm 1.2825 (inf) loss_scale 1024.0000 (1986.9669) mem 11225MB [2024-08-26 07:33:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][160/625] eta 0:02:57 lr 0.001858 wd 0.0500 time 0.3844 (0.3812) data time 0.0011 (0.0040) model time 0.3833 (0.3788) loss 3.5063 (3.5220) grad_norm 0.9406 (inf) loss_scale 1024.0000 (1927.1553) mem 11225MB [2024-08-26 07:34:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][170/625] eta 0:02:53 lr 0.001858 wd 0.0500 time 0.3590 (0.3817) data time 0.0008 (0.0039) model time 0.3582 (0.3796) loss 4.0337 (3.4931) grad_norm 1.2560 (inf) loss_scale 1024.0000 (1874.3392) mem 11225MB [2024-08-26 07:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][180/625] eta 0:02:49 lr 0.001858 wd 0.0500 time 0.3616 (0.3811) data time 0.0012 (0.0037) model time 0.3604 (0.3789) loss 3.6358 (3.4958) grad_norm 1.0489 (inf) loss_scale 1024.0000 (1827.3591) mem 11225MB [2024-08-26 07:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][190/625] eta 0:02:45 lr 0.001858 wd 0.0500 time 0.3681 (0.3805) data time 0.0011 (0.0036) model time 0.3670 (0.3781) loss 3.9010 (3.5147) grad_norm 1.2723 (inf) loss_scale 1024.0000 (1785.2984) mem 11225MB [2024-08-26 07:34:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][200/625] eta 0:02:41 lr 0.001858 wd 0.0500 time 0.3713 (0.3801) data time 0.0010 (0.0035) model time 0.3703 (0.3777) loss 3.6470 (3.5063) grad_norm 0.8967 (inf) loss_scale 1024.0000 (1747.4229) mem 11225MB [2024-08-26 07:34:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][210/625] eta 0:02:37 lr 0.001858 wd 0.0500 time 0.3736 (0.3799) data time 0.0012 (0.0033) model time 0.3724 (0.3775) loss 3.5929 (3.5091) grad_norm 1.2801 (inf) loss_scale 1024.0000 (1713.1374) mem 11225MB [2024-08-26 07:34:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][220/625] eta 0:02:33 lr 0.001858 wd 0.0500 time 0.3660 (0.3794) data time 0.0011 (0.0032) model time 0.3649 (0.3770) loss 4.1788 (3.5120) grad_norm 0.9576 (inf) loss_scale 1024.0000 (1681.9548) mem 11225MB [2024-08-26 07:34:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][230/625] eta 0:02:29 lr 0.001858 wd 0.0500 time 0.3815 (0.3792) data time 0.0008 (0.0031) model time 0.3807 (0.3768) loss 4.1634 (3.5224) grad_norm 0.9015 (inf) loss_scale 1024.0000 (1653.4719) mem 11225MB [2024-08-26 07:34:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][240/625] eta 0:02:25 lr 0.001858 wd 0.0500 time 0.3625 (0.3789) data time 0.0010 (0.0031) model time 0.3615 (0.3765) loss 3.7682 (3.5255) grad_norm 1.3610 (inf) loss_scale 1024.0000 (1627.3527) mem 11225MB [2024-08-26 07:34:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][250/625] eta 0:02:22 lr 0.001858 wd 0.0500 time 0.3676 (0.3802) data time 0.0009 (0.0030) model time 0.3667 (0.3782) loss 3.6533 (3.5287) grad_norm 1.3717 (inf) loss_scale 1024.0000 (1603.3147) mem 11225MB [2024-08-26 07:34:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][260/625] eta 0:02:18 lr 0.001857 wd 0.0500 time 0.3693 (0.3800) data time 0.0011 (0.0029) model time 0.3682 (0.3779) loss 3.8810 (3.5298) grad_norm 1.4928 (inf) loss_scale 1024.0000 (1581.1188) mem 11225MB [2024-08-26 07:34:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][270/625] eta 0:02:14 lr 0.001857 wd 0.0500 time 0.3634 (0.3797) data time 0.0011 (0.0029) model time 0.3623 (0.3776) loss 3.7573 (3.5379) grad_norm 1.6783 (inf) loss_scale 1024.0000 (1560.5609) mem 11225MB [2024-08-26 07:34:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][280/625] eta 0:02:11 lr 0.001857 wd 0.0500 time 0.3751 (0.3810) data time 0.0011 (0.0028) model time 0.3740 (0.3793) loss 2.7994 (3.5400) grad_norm 0.9406 (inf) loss_scale 1024.0000 (1541.4662) mem 11225MB [2024-08-26 07:34:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][290/625] eta 0:02:07 lr 0.001857 wd 0.0500 time 0.3692 (0.3808) data time 0.0009 (0.0027) model time 0.3683 (0.3790) loss 4.1208 (3.5401) grad_norm 1.4102 (inf) loss_scale 1024.0000 (1523.6838) mem 11225MB [2024-08-26 07:34:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][300/625] eta 0:02:03 lr 0.001857 wd 0.0500 time 0.3680 (0.3805) data time 0.0009 (0.0027) model time 0.3671 (0.3787) loss 4.1070 (3.5445) grad_norm 0.9155 (inf) loss_scale 1024.0000 (1507.0831) mem 11225MB [2024-08-26 07:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][310/625] eta 0:01:59 lr 0.001857 wd 0.0500 time 0.3713 (0.3805) data time 0.0011 (0.0026) model time 0.3703 (0.3787) loss 2.3366 (3.5298) grad_norm 1.2030 (inf) loss_scale 1024.0000 (1491.5498) mem 11225MB [2024-08-26 07:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][320/625] eta 0:01:55 lr 0.001857 wd 0.0500 time 0.3604 (0.3803) data time 0.0008 (0.0026) model time 0.3596 (0.3785) loss 4.1632 (3.5269) grad_norm 1.3616 (inf) loss_scale 1024.0000 (1476.9844) mem 11225MB [2024-08-26 07:35:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][330/625] eta 0:01:52 lr 0.001857 wd 0.0500 time 0.3690 (0.3801) data time 0.0013 (0.0026) model time 0.3677 (0.3783) loss 3.7188 (3.5274) grad_norm 1.3514 (inf) loss_scale 1024.0000 (1463.2991) mem 11225MB [2024-08-26 07:35:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][340/625] eta 0:01:48 lr 0.001857 wd 0.0500 time 0.3779 (0.3799) data time 0.0009 (0.0025) model time 0.3770 (0.3780) loss 2.7049 (3.5239) grad_norm 1.0010 (inf) loss_scale 1024.0000 (1450.4164) mem 11225MB [2024-08-26 07:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][350/625] eta 0:01:44 lr 0.001857 wd 0.0500 time 0.3738 (0.3797) data time 0.0010 (0.0025) model time 0.3728 (0.3778) loss 2.8381 (3.5203) grad_norm 1.1709 (inf) loss_scale 1024.0000 (1438.2678) mem 11225MB [2024-08-26 07:35:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][360/625] eta 0:01:40 lr 0.001857 wd 0.0500 time 0.3754 (0.3794) data time 0.0009 (0.0024) model time 0.3744 (0.3776) loss 2.5468 (3.5074) grad_norm 1.2846 (inf) loss_scale 1024.0000 (1426.7922) mem 11225MB [2024-08-26 07:35:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][370/625] eta 0:01:37 lr 0.001856 wd 0.0500 time 0.3968 (0.3804) data time 0.0008 (0.0024) model time 0.3960 (0.3788) loss 3.0691 (3.5058) grad_norm 1.2067 (inf) loss_scale 1024.0000 (1415.9353) mem 11225MB [2024-08-26 07:35:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][380/625] eta 0:01:33 lr 0.001856 wd 0.0500 time 0.3756 (0.3807) data time 0.0009 (0.0024) model time 0.3747 (0.3790) loss 3.4509 (3.5026) grad_norm 1.0948 (inf) loss_scale 1024.0000 (1405.6483) mem 11225MB [2024-08-26 07:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][390/625] eta 0:01:29 lr 0.001856 wd 0.0500 time 0.3684 (0.3804) data time 0.0011 (0.0023) model time 0.3673 (0.3787) loss 3.5088 (3.4994) grad_norm 1.2529 (inf) loss_scale 1024.0000 (1395.8875) mem 11225MB [2024-08-26 07:35:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][400/625] eta 0:01:25 lr 0.001856 wd 0.0500 time 0.3648 (0.3801) data time 0.0008 (0.0023) model time 0.3640 (0.3784) loss 3.9727 (3.4954) grad_norm 1.2861 (inf) loss_scale 1024.0000 (1386.6135) mem 11225MB [2024-08-26 07:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][410/625] eta 0:01:21 lr 0.001856 wd 0.0500 time 0.3706 (0.3798) data time 0.0009 (0.0023) model time 0.3697 (0.3781) loss 4.1414 (3.4970) grad_norm 1.0800 (inf) loss_scale 1024.0000 (1377.7908) mem 11225MB [2024-08-26 07:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][420/625] eta 0:01:17 lr 0.001856 wd 0.0500 time 0.3803 (0.3797) data time 0.0009 (0.0023) model time 0.3795 (0.3780) loss 4.2501 (3.5064) grad_norm 1.0646 (inf) loss_scale 1024.0000 (1369.3872) mem 11225MB [2024-08-26 07:35:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][430/625] eta 0:01:13 lr 0.001856 wd 0.0500 time 0.3798 (0.3795) data time 0.0011 (0.0022) model time 0.3787 (0.3777) loss 3.4971 (3.5138) grad_norm 1.3644 (inf) loss_scale 1024.0000 (1361.3735) mem 11225MB [2024-08-26 07:35:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][440/625] eta 0:01:10 lr 0.001856 wd 0.0500 time 0.3640 (0.3793) data time 0.0009 (0.0022) model time 0.3632 (0.3775) loss 3.8322 (3.5083) grad_norm 1.5353 (inf) loss_scale 1024.0000 (1353.7234) mem 11225MB [2024-08-26 07:35:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][450/625] eta 0:01:06 lr 0.001856 wd 0.0500 time 0.3690 (0.3791) data time 0.0011 (0.0022) model time 0.3679 (0.3774) loss 4.2241 (3.5167) grad_norm 0.9124 (inf) loss_scale 1024.0000 (1346.4124) mem 11225MB [2024-08-26 07:35:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][460/625] eta 0:01:02 lr 0.001856 wd 0.0500 time 0.5507 (0.3793) data time 0.0011 (0.0022) model time 0.5496 (0.3776) loss 3.6176 (3.5194) grad_norm 1.1499 (inf) loss_scale 1024.0000 (1339.4187) mem 11225MB [2024-08-26 07:35:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][470/625] eta 0:00:58 lr 0.001856 wd 0.0500 time 0.3794 (0.3797) data time 0.0011 (0.0021) model time 0.3784 (0.3780) loss 3.4070 (3.5204) grad_norm 1.4007 (inf) loss_scale 1024.0000 (1332.7219) mem 11225MB [2024-08-26 07:35:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][480/625] eta 0:00:55 lr 0.001855 wd 0.0500 time 0.3693 (0.3796) data time 0.0008 (0.0021) model time 0.3685 (0.3779) loss 3.3044 (3.5233) grad_norm 0.9218 (inf) loss_scale 1024.0000 (1326.3035) mem 11225MB [2024-08-26 07:36:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][490/625] eta 0:00:51 lr 0.001855 wd 0.0500 time 0.3690 (0.3794) data time 0.0012 (0.0021) model time 0.3678 (0.3778) loss 3.8652 (3.5229) grad_norm 0.8955 (inf) loss_scale 1024.0000 (1320.1466) mem 11225MB [2024-08-26 07:36:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][500/625] eta 0:00:47 lr 0.001855 wd 0.0500 time 0.3726 (0.3792) data time 0.0009 (0.0021) model time 0.3718 (0.3776) loss 3.9216 (3.5220) grad_norm 1.4079 (inf) loss_scale 1024.0000 (1314.2355) mem 11225MB [2024-08-26 07:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][510/625] eta 0:00:43 lr 0.001855 wd 0.0500 time 0.3741 (0.3791) data time 0.0009 (0.0021) model time 0.3732 (0.3774) loss 3.2902 (3.5246) grad_norm 1.8649 (inf) loss_scale 1024.0000 (1308.5558) mem 11225MB [2024-08-26 07:36:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][520/625] eta 0:00:39 lr 0.001855 wd 0.0500 time 0.3729 (0.3790) data time 0.0011 (0.0020) model time 0.3718 (0.3773) loss 4.2612 (3.5284) grad_norm 1.3616 (inf) loss_scale 1024.0000 (1303.0940) mem 11225MB [2024-08-26 07:36:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][530/625] eta 0:00:35 lr 0.001855 wd 0.0500 time 0.3753 (0.3789) data time 0.0010 (0.0020) model time 0.3743 (0.3773) loss 3.3027 (3.5269) grad_norm 2.1066 (inf) loss_scale 1024.0000 (1297.8380) mem 11225MB [2024-08-26 07:36:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][540/625] eta 0:00:32 lr 0.001855 wd 0.0500 time 0.3693 (0.3788) data time 0.0011 (0.0020) model time 0.3683 (0.3772) loss 3.6452 (3.5283) grad_norm 2.3815 (inf) loss_scale 1024.0000 (1292.7763) mem 11225MB [2024-08-26 07:36:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][550/625] eta 0:00:28 lr 0.001855 wd 0.0500 time 0.3875 (0.3788) data time 0.0008 (0.0020) model time 0.3867 (0.3771) loss 2.2833 (3.5283) grad_norm 0.9883 (inf) loss_scale 1024.0000 (1287.8984) mem 11225MB [2024-08-26 07:36:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][560/625] eta 0:00:24 lr 0.001855 wd 0.0500 time 0.3619 (0.3789) data time 0.0011 (0.0020) model time 0.3608 (0.3772) loss 3.6221 (3.5362) grad_norm 1.3675 (inf) loss_scale 1024.0000 (1283.1943) mem 11225MB [2024-08-26 07:36:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][570/625] eta 0:00:20 lr 0.001855 wd 0.0500 time 0.3681 (0.3787) data time 0.0011 (0.0020) model time 0.3670 (0.3771) loss 3.6754 (3.5349) grad_norm 1.0414 (inf) loss_scale 1024.0000 (1278.6550) mem 11225MB [2024-08-26 07:36:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][580/625] eta 0:00:17 lr 0.001855 wd 0.0500 time 0.5786 (0.3798) data time 0.0009 (0.0020) model time 0.5777 (0.3783) loss 2.9026 (3.5365) grad_norm 1.1339 (inf) loss_scale 1024.0000 (1274.2719) mem 11225MB [2024-08-26 07:36:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][590/625] eta 0:00:13 lr 0.001854 wd 0.0500 time 0.3731 (0.3801) data time 0.0008 (0.0019) model time 0.3722 (0.3786) loss 2.6657 (3.5301) grad_norm 1.2202 (inf) loss_scale 1024.0000 (1270.0372) mem 11225MB [2024-08-26 07:36:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][600/625] eta 0:00:09 lr 0.001854 wd 0.0500 time 0.3639 (0.3800) data time 0.0011 (0.0019) model time 0.3628 (0.3785) loss 2.8655 (3.5276) grad_norm 2.4693 (inf) loss_scale 1024.0000 (1265.9434) mem 11225MB [2024-08-26 07:36:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][610/625] eta 0:00:05 lr 0.001854 wd 0.0500 time 0.3636 (0.3798) data time 0.0008 (0.0019) model time 0.3627 (0.3783) loss 3.3929 (3.5270) grad_norm 1.5553 (inf) loss_scale 1024.0000 (1261.9836) mem 11225MB [2024-08-26 07:36:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [68/300][620/625] eta 0:00:01 lr 0.001854 wd 0.0500 time 0.3623 (0.3795) data time 0.0008 (0.0019) model time 0.3614 (0.3780) loss 3.4432 (3.5265) grad_norm 1.3392 (inf) loss_scale 1024.0000 (1258.1514) mem 11225MB [2024-08-26 07:36:54 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 68 training takes 0:03:57 [2024-08-26 07:36:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:36:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:36:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.533 (0.533) Loss 0.6416 (0.6416) Acc@1 85.352 (85.352) Acc@5 97.314 (97.314) Mem 11225MB [2024-08-26 07:36:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.147) Loss 1.1797 (0.8244) Acc@1 70.410 (80.473) Acc@5 92.627 (95.894) Mem 11225MB [2024-08-26 07:36:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 1.2305 (0.9973) Acc@1 69.727 (76.507) Acc@5 90.381 (93.659) Mem 11225MB [2024-08-26 07:36:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.360 Acc@5 93.600 [2024-08-26 07:36:58 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.4% [2024-08-26 07:36:58 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 76.36% [2024-08-26 07:36:58 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 07:36:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 07:37:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.539 (0.539) Loss 0.5688 (0.5688) Acc@1 87.744 (87.744) Acc@5 97.998 (97.998) Mem 11225MB [2024-08-26 07:37:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.147) Loss 0.9736 (0.7155) Acc@1 76.611 (83.354) Acc@5 94.141 (96.808) Mem 11225MB [2024-08-26 07:37:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.1416 (0.8815) Acc@1 71.680 (79.260) Acc@5 91.992 (94.829) Mem 11225MB [2024-08-26 07:37:02 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.937 Acc@5 94.796 [2024-08-26 07:37:02 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 78.9% [2024-08-26 07:37:02 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 78.94% [2024-08-26 07:37:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:37:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:37:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][0/625] eta 0:07:40 lr 0.001854 wd 0.0500 time 0.7369 (0.7369) data time 0.3841 (0.3841) model time 0.0000 (0.0000) loss 3.5343 (3.5343) grad_norm 1.2007 (1.2007) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][10/625] eta 0:04:18 lr 0.001854 wd 0.0500 time 0.3311 (0.4203) data time 0.0011 (0.0360) model time 0.0000 (0.0000) loss 3.7821 (3.3768) grad_norm 1.6707 (1.2923) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][20/625] eta 0:04:00 lr 0.001854 wd 0.0500 time 0.3751 (0.3975) data time 0.0009 (0.0194) model time 0.0000 (0.0000) loss 4.1256 (3.4540) grad_norm 1.8750 (1.2771) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][30/625] eta 0:03:52 lr 0.001854 wd 0.0500 time 0.4014 (0.3906) data time 0.0011 (0.0135) model time 0.0000 (0.0000) loss 3.7336 (3.4793) grad_norm 1.6049 (1.2852) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][40/625] eta 0:03:46 lr 0.001854 wd 0.0500 time 0.3742 (0.3869) data time 0.0009 (0.0105) model time 0.0000 (0.0000) loss 2.8870 (3.3984) grad_norm 1.1751 (1.2703) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][50/625] eta 0:03:41 lr 0.001854 wd 0.0500 time 0.3745 (0.3850) data time 0.0008 (0.0087) model time 0.0000 (0.0000) loss 3.6139 (3.3648) grad_norm 1.1535 (1.3407) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][60/625] eta 0:03:36 lr 0.001854 wd 0.0500 time 0.3822 (0.3837) data time 0.0011 (0.0074) model time 0.3811 (0.3758) loss 3.9842 (3.4212) grad_norm 1.0093 (1.3592) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][70/625] eta 0:03:32 lr 0.001853 wd 0.0500 time 0.3756 (0.3822) data time 0.0012 (0.0065) model time 0.3744 (0.3738) loss 3.2949 (3.4440) grad_norm 1.2687 (1.3609) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][80/625] eta 0:03:27 lr 0.001853 wd 0.0500 time 0.3673 (0.3805) data time 0.0009 (0.0059) model time 0.3664 (0.3717) loss 2.9410 (3.4228) grad_norm 1.1289 (1.3513) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][90/625] eta 0:03:22 lr 0.001853 wd 0.0500 time 0.3690 (0.3792) data time 0.0011 (0.0054) model time 0.3679 (0.3705) loss 2.4928 (3.4225) grad_norm 1.1421 (1.3275) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][100/625] eta 0:03:20 lr 0.001853 wd 0.0500 time 0.3719 (0.3825) data time 0.0008 (0.0049) model time 0.3710 (0.3787) loss 2.2413 (3.4226) grad_norm 1.3562 (1.3251) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][110/625] eta 0:03:16 lr 0.001853 wd 0.0500 time 0.3747 (0.3816) data time 0.0009 (0.0046) model time 0.3738 (0.3775) loss 2.3758 (3.4000) grad_norm 1.2954 (1.3559) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][120/625] eta 0:03:12 lr 0.001853 wd 0.0500 time 0.3757 (0.3808) data time 0.0008 (0.0043) model time 0.3749 (0.3765) loss 4.1589 (3.4299) grad_norm 0.9041 (1.3523) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][130/625] eta 0:03:08 lr 0.001853 wd 0.0500 time 0.4064 (0.3803) data time 0.0011 (0.0041) model time 0.4053 (0.3761) loss 3.8699 (3.4567) grad_norm 1.9108 (1.3415) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:37:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][140/625] eta 0:03:04 lr 0.001853 wd 0.0500 time 0.3549 (0.3799) data time 0.0009 (0.0039) model time 0.3540 (0.3759) loss 2.7311 (3.4679) grad_norm 1.0594 (1.3275) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][150/625] eta 0:03:00 lr 0.001853 wd 0.0500 time 0.3743 (0.3795) data time 0.0009 (0.0037) model time 0.3734 (0.3755) loss 4.2132 (3.4766) grad_norm 1.0535 (1.3218) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][160/625] eta 0:02:57 lr 0.001853 wd 0.0500 time 0.3761 (0.3819) data time 0.0011 (0.0036) model time 0.3750 (0.3793) loss 3.1525 (3.4728) grad_norm 1.1321 (1.3154) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][170/625] eta 0:02:53 lr 0.001853 wd 0.0500 time 0.3698 (0.3814) data time 0.0008 (0.0034) model time 0.3690 (0.3787) loss 4.4692 (3.4837) grad_norm 1.2681 (1.3081) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][180/625] eta 0:02:49 lr 0.001852 wd 0.0500 time 0.3678 (0.3808) data time 0.0009 (0.0033) model time 0.3669 (0.3779) loss 3.7447 (3.4986) grad_norm 1.1352 (1.3023) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][190/625] eta 0:02:45 lr 0.001852 wd 0.0500 time 0.3866 (0.3803) data time 0.0008 (0.0032) model time 0.3858 (0.3774) loss 4.5907 (3.5179) grad_norm 1.1982 (1.2933) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][200/625] eta 0:02:41 lr 0.001852 wd 0.0500 time 0.3657 (0.3800) data time 0.0011 (0.0031) model time 0.3646 (0.3771) loss 3.1123 (3.5199) grad_norm 1.0369 (1.2938) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][210/625] eta 0:02:37 lr 0.001852 wd 0.0500 time 0.3802 (0.3797) data time 0.0020 (0.0030) model time 0.3782 (0.3768) loss 4.1212 (3.5282) grad_norm 1.3476 (1.2955) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][220/625] eta 0:02:34 lr 0.001852 wd 0.0500 time 0.3771 (0.3804) data time 0.0009 (0.0029) model time 0.3762 (0.3779) loss 2.4850 (3.5201) grad_norm 1.2228 (1.3011) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][230/625] eta 0:02:30 lr 0.001852 wd 0.0500 time 0.3705 (0.3801) data time 0.0011 (0.0029) model time 0.3693 (0.3774) loss 3.8399 (3.5070) grad_norm 1.0430 (1.3009) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][240/625] eta 0:02:26 lr 0.001852 wd 0.0500 time 0.3692 (0.3798) data time 0.0009 (0.0028) model time 0.3682 (0.3771) loss 2.7557 (3.4972) grad_norm 0.8952 (1.2947) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][250/625] eta 0:02:22 lr 0.001852 wd 0.0500 time 0.3803 (0.3794) data time 0.0011 (0.0027) model time 0.3792 (0.3768) loss 3.9466 (3.5013) grad_norm 0.9503 (1.2927) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][260/625] eta 0:02:18 lr 0.001852 wd 0.0500 time 0.3691 (0.3793) data time 0.0009 (0.0027) model time 0.3682 (0.3766) loss 2.4597 (3.5052) grad_norm 1.5272 (1.2912) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][270/625] eta 0:02:14 lr 0.001852 wd 0.0500 time 0.3655 (0.3789) data time 0.0008 (0.0026) model time 0.3647 (0.3762) loss 2.2821 (3.4962) grad_norm 1.2720 (1.2897) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][280/625] eta 0:02:10 lr 0.001852 wd 0.0500 time 0.3852 (0.3788) data time 0.0010 (0.0026) model time 0.3842 (0.3762) loss 2.8965 (3.4847) grad_norm 1.7435 (1.2891) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][290/625] eta 0:02:06 lr 0.001851 wd 0.0500 time 0.3831 (0.3787) data time 0.0008 (0.0025) model time 0.3823 (0.3761) loss 2.4180 (3.4826) grad_norm 0.9229 (1.2918) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][300/625] eta 0:02:02 lr 0.001851 wd 0.0500 time 0.3631 (0.3784) data time 0.0010 (0.0025) model time 0.3621 (0.3758) loss 4.1141 (3.4945) grad_norm 1.1885 (1.2902) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][310/625] eta 0:01:59 lr 0.001851 wd 0.0500 time 0.3818 (0.3796) data time 0.0008 (0.0024) model time 0.3809 (0.3773) loss 3.9863 (3.4850) grad_norm 0.9178 (1.2836) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][320/625] eta 0:01:55 lr 0.001851 wd 0.0500 time 0.3687 (0.3793) data time 0.0010 (0.0024) model time 0.3677 (0.3770) loss 3.7310 (3.4842) grad_norm 1.0204 (1.2828) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][330/625] eta 0:01:51 lr 0.001851 wd 0.0500 time 0.3687 (0.3790) data time 0.0011 (0.0024) model time 0.3676 (0.3766) loss 2.3137 (3.4820) grad_norm 1.1602 (1.2876) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][340/625] eta 0:01:48 lr 0.001851 wd 0.0500 time 0.3690 (0.3802) data time 0.0009 (0.0023) model time 0.3682 (0.3782) loss 3.5210 (3.4834) grad_norm 1.1742 (1.2837) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][350/625] eta 0:01:44 lr 0.001851 wd 0.0500 time 0.3659 (0.3800) data time 0.0010 (0.0023) model time 0.3649 (0.3780) loss 3.7911 (3.4833) grad_norm 1.5643 (1.2799) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][360/625] eta 0:01:40 lr 0.001851 wd 0.0500 time 0.3743 (0.3799) data time 0.0010 (0.0023) model time 0.3732 (0.3778) loss 3.9607 (3.4874) grad_norm 0.8484 (1.2767) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][370/625] eta 0:01:36 lr 0.001851 wd 0.0500 time 0.3691 (0.3796) data time 0.0010 (0.0022) model time 0.3680 (0.3775) loss 3.6133 (3.4877) grad_norm 1.4045 (1.2789) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][380/625] eta 0:01:32 lr 0.001851 wd 0.0500 time 0.3744 (0.3794) data time 0.0010 (0.0022) model time 0.3733 (0.3773) loss 3.8128 (3.4861) grad_norm 1.8547 (1.2788) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][390/625] eta 0:01:29 lr 0.001850 wd 0.0500 time 0.3807 (0.3792) data time 0.0011 (0.0022) model time 0.3796 (0.3771) loss 3.0094 (3.4898) grad_norm 1.6036 (1.2766) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][400/625] eta 0:01:25 lr 0.001850 wd 0.0500 time 0.3784 (0.3789) data time 0.0008 (0.0022) model time 0.3776 (0.3768) loss 3.1465 (3.4925) grad_norm 1.2775 (1.2811) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][410/625] eta 0:01:21 lr 0.001850 wd 0.0500 time 0.3643 (0.3792) data time 0.0009 (0.0021) model time 0.3634 (0.3772) loss 4.3755 (3.4899) grad_norm 1.5170 (1.2785) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][420/625] eta 0:01:17 lr 0.001850 wd 0.0500 time 0.3730 (0.3790) data time 0.0008 (0.0021) model time 0.3722 (0.3770) loss 2.3274 (3.4902) grad_norm 1.7925 (1.2772) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][430/625] eta 0:01:13 lr 0.001850 wd 0.0500 time 0.3727 (0.3789) data time 0.0008 (0.0021) model time 0.3719 (0.3768) loss 3.1722 (3.4854) grad_norm 1.0386 (1.2770) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][440/625] eta 0:01:10 lr 0.001850 wd 0.0500 time 0.3649 (0.3787) data time 0.0009 (0.0021) model time 0.3640 (0.3766) loss 3.3980 (3.4884) grad_norm 1.2082 (1.2753) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][450/625] eta 0:01:06 lr 0.001850 wd 0.0500 time 0.3685 (0.3785) data time 0.0008 (0.0020) model time 0.3677 (0.3765) loss 3.6514 (3.4869) grad_norm 1.1693 (1.2747) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:39:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][460/625] eta 0:01:02 lr 0.001850 wd 0.0500 time 0.3735 (0.3794) data time 0.0009 (0.0020) model time 0.3726 (0.3775) loss 4.1006 (3.4922) grad_norm 1.5169 (1.2768) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][470/625] eta 0:00:58 lr 0.001850 wd 0.0500 time 0.3764 (0.3793) data time 0.0008 (0.0020) model time 0.3756 (0.3774) loss 2.5392 (3.4832) grad_norm 1.8060 (1.2806) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][480/625] eta 0:00:54 lr 0.001850 wd 0.0500 time 0.3716 (0.3792) data time 0.0008 (0.0020) model time 0.3708 (0.3773) loss 2.8308 (3.4868) grad_norm 1.3734 (1.2794) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][490/625] eta 0:00:51 lr 0.001850 wd 0.0500 time 0.3818 (0.3799) data time 0.0008 (0.0020) model time 0.3810 (0.3782) loss 2.2338 (3.4832) grad_norm 1.3400 (1.2780) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][500/625] eta 0:00:47 lr 0.001849 wd 0.0500 time 0.4544 (0.3801) data time 0.0015 (0.0020) model time 0.4529 (0.3783) loss 3.6669 (3.4813) grad_norm 1.2498 (1.2771) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][510/625] eta 0:00:43 lr 0.001849 wd 0.0500 time 0.3803 (0.3800) data time 0.0009 (0.0019) model time 0.3795 (0.3782) loss 4.4234 (3.4881) grad_norm 1.0227 (1.2718) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][520/625] eta 0:00:39 lr 0.001849 wd 0.0500 time 0.6118 (0.3803) data time 0.0012 (0.0019) model time 0.6106 (0.3786) loss 2.1965 (3.4748) grad_norm 2.6281 (1.2741) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][530/625] eta 0:00:36 lr 0.001849 wd 0.0500 time 0.3758 (0.3801) data time 0.0011 (0.0019) model time 0.3747 (0.3784) loss 3.3095 (3.4755) grad_norm 1.1587 (1.2748) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][540/625] eta 0:00:32 lr 0.001849 wd 0.0500 time 0.3777 (0.3800) data time 0.0009 (0.0019) model time 0.3767 (0.3783) loss 2.7711 (3.4709) grad_norm 1.2474 (1.2759) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][550/625] eta 0:00:28 lr 0.001849 wd 0.0500 time 0.3777 (0.3799) data time 0.0012 (0.0019) model time 0.3765 (0.3782) loss 3.8338 (3.4654) grad_norm 1.3028 (1.2753) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][560/625] eta 0:00:24 lr 0.001849 wd 0.0500 time 0.3766 (0.3798) data time 0.0011 (0.0019) model time 0.3755 (0.3781) loss 4.0997 (3.4701) grad_norm 1.0515 (1.2764) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][570/625] eta 0:00:20 lr 0.001849 wd 0.0500 time 0.3766 (0.3797) data time 0.0008 (0.0019) model time 0.3758 (0.3780) loss 2.2977 (3.4692) grad_norm 1.2619 (1.2778) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][580/625] eta 0:00:17 lr 0.001849 wd 0.0500 time 0.3790 (0.3796) data time 0.0009 (0.0018) model time 0.3781 (0.3779) loss 4.3280 (3.4706) grad_norm 1.0433 (1.2871) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][590/625] eta 0:00:13 lr 0.001849 wd 0.0500 time 0.3675 (0.3796) data time 0.0009 (0.0018) model time 0.3666 (0.3779) loss 3.3291 (3.4682) grad_norm 1.1819 (1.2866) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][600/625] eta 0:00:09 lr 0.001848 wd 0.0500 time 0.3796 (0.3795) data time 0.0012 (0.0018) model time 0.3784 (0.3778) loss 3.5387 (3.4689) grad_norm 1.4307 (1.2871) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][610/625] eta 0:00:05 lr 0.001848 wd 0.0500 time 0.3584 (0.3801) data time 0.0008 (0.0018) model time 0.3575 (0.3784) loss 3.7084 (3.4719) grad_norm 1.3392 (1.2870) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:40:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [69/300][620/625] eta 0:00:01 lr 0.001848 wd 0.0500 time 0.3658 (0.3799) data time 0.0008 (0.0018) model time 0.3649 (0.3782) loss 3.7482 (3.4705) grad_norm 1.7458 (1.2895) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:01 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 69 training takes 0:03:57 [2024-08-26 07:41:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:41:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:41:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.550 (0.550) Loss 0.6704 (0.6704) Acc@1 85.107 (85.107) Acc@5 97.461 (97.461) Mem 11225MB [2024-08-26 07:41:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.154) Loss 1.1846 (0.8385) Acc@1 73.486 (80.779) Acc@5 92.480 (95.952) Mem 11225MB [2024-08-26 07:41:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.125) Loss 1.2998 (1.0173) Acc@1 69.873 (76.844) Acc@5 90.381 (93.713) Mem 11225MB [2024-08-26 07:41:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.478 Acc@5 93.654 [2024-08-26 07:41:05 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.5% [2024-08-26 07:41:05 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 76.48% [2024-08-26 07:41:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 07:41:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 07:41:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.525 (0.525) Loss 0.5674 (0.5674) Acc@1 87.744 (87.744) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 07:41:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.143) Loss 0.9707 (0.7144) Acc@1 76.611 (83.403) Acc@5 94.092 (96.835) Mem 11225MB [2024-08-26 07:41:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.1377 (0.8795) Acc@1 71.777 (79.332) Acc@5 92.139 (94.885) Mem 11225MB [2024-08-26 07:41:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.011 Acc@5 94.832 [2024-08-26 07:41:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.0% [2024-08-26 07:41:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.01% [2024-08-26 07:41:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:41:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:41:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][0/625] eta 0:07:56 lr 0.001848 wd 0.0500 time 0.7619 (0.7619) data time 0.4180 (0.4180) model time 0.0000 (0.0000) loss 4.3859 (4.3859) grad_norm 1.4287 (1.4287) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][10/625] eta 0:04:11 lr 0.001848 wd 0.0500 time 0.3744 (0.4088) data time 0.0010 (0.0390) model time 0.0000 (0.0000) loss 4.0925 (3.5220) grad_norm 1.1794 (1.2629) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][20/625] eta 0:04:03 lr 0.001848 wd 0.0500 time 0.3720 (0.4017) data time 0.0011 (0.0210) model time 0.0000 (0.0000) loss 3.6053 (3.6458) grad_norm 0.7968 (1.2509) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][30/625] eta 0:04:01 lr 0.001848 wd 0.0500 time 0.3711 (0.4066) data time 0.0009 (0.0146) model time 0.0000 (0.0000) loss 3.7416 (3.5114) grad_norm 1.1618 (1.2393) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][40/625] eta 0:03:53 lr 0.001848 wd 0.0500 time 0.3739 (0.3993) data time 0.0008 (0.0113) model time 0.0000 (0.0000) loss 2.6669 (3.4075) grad_norm 1.3286 (1.2557) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][50/625] eta 0:03:47 lr 0.001848 wd 0.0500 time 0.3712 (0.3949) data time 0.0009 (0.0093) model time 0.0000 (0.0000) loss 4.1625 (3.4438) grad_norm 1.0786 (1.3020) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][60/625] eta 0:03:41 lr 0.001848 wd 0.0500 time 0.3758 (0.3918) data time 0.0008 (0.0079) model time 0.3750 (0.3747) loss 4.1081 (3.5033) grad_norm 1.3620 (1.3120) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][70/625] eta 0:03:36 lr 0.001848 wd 0.0500 time 0.3765 (0.3894) data time 0.0008 (0.0070) model time 0.3757 (0.3745) loss 3.9919 (3.5200) grad_norm 1.1346 (1.3046) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][80/625] eta 0:03:31 lr 0.001847 wd 0.0500 time 0.3878 (0.3878) data time 0.0009 (0.0062) model time 0.3869 (0.3747) loss 3.6469 (3.5208) grad_norm 1.2800 (1.2957) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][90/625] eta 0:03:26 lr 0.001847 wd 0.0500 time 0.3843 (0.3866) data time 0.0008 (0.0057) model time 0.3834 (0.3749) loss 4.1120 (3.5363) grad_norm 1.0037 (1.2834) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][100/625] eta 0:03:22 lr 0.001847 wd 0.0500 time 0.3750 (0.3855) data time 0.0012 (0.0052) model time 0.3738 (0.3748) loss 3.3748 (3.5342) grad_norm 1.0120 (1.2740) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][110/625] eta 0:03:18 lr 0.001847 wd 0.0500 time 0.3714 (0.3846) data time 0.0011 (0.0049) model time 0.3703 (0.3749) loss 2.9639 (3.5362) grad_norm 1.4702 (1.2640) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:41:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][120/625] eta 0:03:13 lr 0.001847 wd 0.0500 time 0.3732 (0.3836) data time 0.0009 (0.0045) model time 0.3723 (0.3742) loss 4.1463 (3.5078) grad_norm 1.5258 (1.2732) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][130/625] eta 0:03:09 lr 0.001847 wd 0.0500 time 0.3710 (0.3829) data time 0.0008 (0.0043) model time 0.3702 (0.3741) loss 2.4437 (3.4985) grad_norm 1.2273 (1.2889) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][140/625] eta 0:03:05 lr 0.001847 wd 0.0500 time 0.3739 (0.3824) data time 0.0011 (0.0041) model time 0.3728 (0.3741) loss 2.6630 (3.5098) grad_norm 1.5711 (1.2991) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][150/625] eta 0:03:01 lr 0.001847 wd 0.0500 time 0.3829 (0.3818) data time 0.0010 (0.0039) model time 0.3818 (0.3740) loss 2.9424 (3.5130) grad_norm 0.9644 (1.2934) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][160/625] eta 0:02:57 lr 0.001847 wd 0.0500 time 0.3725 (0.3813) data time 0.0008 (0.0037) model time 0.3717 (0.3738) loss 3.3445 (3.5074) grad_norm 1.1438 (1.2819) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][170/625] eta 0:02:54 lr 0.001847 wd 0.0500 time 0.3778 (0.3832) data time 0.0010 (0.0036) model time 0.3768 (0.3771) loss 4.0249 (3.5013) grad_norm 1.8362 (1.2920) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][180/625] eta 0:02:50 lr 0.001847 wd 0.0500 time 0.3670 (0.3826) data time 0.0008 (0.0034) model time 0.3662 (0.3766) loss 3.3199 (3.5145) grad_norm 1.1222 (1.2844) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][190/625] eta 0:02:46 lr 0.001846 wd 0.0500 time 0.3692 (0.3821) data time 0.0008 (0.0033) model time 0.3684 (0.3764) loss 3.5019 (3.5218) grad_norm 1.2425 (1.2851) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][200/625] eta 0:02:42 lr 0.001846 wd 0.0500 time 0.3742 (0.3827) data time 0.0012 (0.0032) model time 0.3729 (0.3774) loss 3.7423 (3.5226) grad_norm 1.0112 (1.2828) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][210/625] eta 0:02:38 lr 0.001846 wd 0.0500 time 0.3694 (0.3822) data time 0.0009 (0.0031) model time 0.3685 (0.3770) loss 3.7671 (3.5112) grad_norm 1.0891 (1.2864) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][220/625] eta 0:02:34 lr 0.001846 wd 0.0500 time 0.3755 (0.3819) data time 0.0010 (0.0030) model time 0.3745 (0.3769) loss 3.9920 (3.5101) grad_norm 1.0539 (1.2827) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][230/625] eta 0:02:31 lr 0.001846 wd 0.0500 time 0.5847 (0.3826) data time 0.0008 (0.0029) model time 0.5839 (0.3780) loss 3.8855 (3.5181) grad_norm 1.1852 (1.2790) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][240/625] eta 0:02:27 lr 0.001846 wd 0.0500 time 0.3707 (0.3831) data time 0.0018 (0.0029) model time 0.3689 (0.3788) loss 2.4634 (3.5105) grad_norm 0.9601 (1.2746) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][250/625] eta 0:02:23 lr 0.001846 wd 0.0500 time 0.3696 (0.3826) data time 0.0010 (0.0028) model time 0.3687 (0.3784) loss 3.4600 (3.5195) grad_norm 1.5760 (1.2715) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][260/625] eta 0:02:19 lr 0.001846 wd 0.0500 time 0.3725 (0.3823) data time 0.0009 (0.0027) model time 0.3715 (0.3782) loss 3.5434 (3.5147) grad_norm 1.6760 (1.2908) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][270/625] eta 0:02:15 lr 0.001846 wd 0.0500 time 0.3721 (0.3820) data time 0.0011 (0.0027) model time 0.3709 (0.3779) loss 3.7103 (3.5159) grad_norm 0.9890 (1.2843) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:42:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][280/625] eta 0:02:11 lr 0.001846 wd 0.0500 time 0.3768 (0.3818) data time 0.0011 (0.0026) model time 0.3757 (0.3778) loss 3.4970 (3.5134) grad_norm 1.7860 (1.2855) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][290/625] eta 0:02:08 lr 0.001846 wd 0.0500 time 0.3706 (0.3830) data time 0.0009 (0.0026) model time 0.3697 (0.3793) loss 3.1326 (3.5035) grad_norm 1.9451 (1.2837) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][300/625] eta 0:02:04 lr 0.001845 wd 0.0500 time 0.3743 (0.3827) data time 0.0010 (0.0025) model time 0.3733 (0.3791) loss 3.0352 (3.5048) grad_norm 1.3342 (1.2827) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][310/625] eta 0:02:00 lr 0.001845 wd 0.0500 time 0.3683 (0.3825) data time 0.0008 (0.0025) model time 0.3675 (0.3789) loss 2.1476 (3.4929) grad_norm 1.1108 (1.2798) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][320/625] eta 0:01:56 lr 0.001845 wd 0.0500 time 0.3769 (0.3828) data time 0.0009 (0.0024) model time 0.3759 (0.3794) loss 3.3979 (3.4862) grad_norm 1.1618 (1.2860) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][330/625] eta 0:01:52 lr 0.001845 wd 0.0500 time 0.3799 (0.3826) data time 0.0008 (0.0024) model time 0.3791 (0.3792) loss 3.2170 (3.4845) grad_norm 1.3286 (1.2845) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][340/625] eta 0:01:48 lr 0.001845 wd 0.0500 time 0.3767 (0.3824) data time 0.0010 (0.0024) model time 0.3757 (0.3790) loss 3.7395 (3.4737) grad_norm 1.4013 (1.2887) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][350/625] eta 0:01:45 lr 0.001845 wd 0.0500 time 0.3713 (0.3821) data time 0.0008 (0.0023) model time 0.3705 (0.3788) loss 3.4431 (3.4708) grad_norm 2.2121 (1.2974) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][360/625] eta 0:01:41 lr 0.001845 wd 0.0500 time 0.3845 (0.3819) data time 0.0011 (0.0023) model time 0.3834 (0.3787) loss 4.0352 (3.4692) grad_norm 0.9819 (1.3056) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][370/625] eta 0:01:37 lr 0.001845 wd 0.0500 time 0.3830 (0.3818) data time 0.0008 (0.0023) model time 0.3822 (0.3786) loss 2.7706 (3.4692) grad_norm 1.8326 (1.3031) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][380/625] eta 0:01:33 lr 0.001845 wd 0.0500 time 0.3732 (0.3816) data time 0.0010 (0.0022) model time 0.3721 (0.3784) loss 3.6867 (3.4681) grad_norm 2.4099 (1.3056) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][390/625] eta 0:01:29 lr 0.001845 wd 0.0500 time 0.3678 (0.3815) data time 0.0008 (0.0022) model time 0.3669 (0.3783) loss 4.2887 (3.4790) grad_norm 1.1979 (1.3086) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][400/625] eta 0:01:25 lr 0.001844 wd 0.0500 time 0.3853 (0.3813) data time 0.0009 (0.0022) model time 0.3844 (0.3782) loss 4.7435 (3.4901) grad_norm 1.2127 (1.3049) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][410/625] eta 0:01:21 lr 0.001844 wd 0.0500 time 0.3774 (0.3811) data time 0.0011 (0.0022) model time 0.3763 (0.3781) loss 3.5409 (3.4960) grad_norm 1.5384 (1.3055) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][420/625] eta 0:01:18 lr 0.001844 wd 0.0500 time 0.3705 (0.3825) data time 0.0010 (0.0021) model time 0.3694 (0.3796) loss 3.6735 (3.4965) grad_norm 2.0157 (1.3079) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][430/625] eta 0:01:14 lr 0.001844 wd 0.0500 time 0.3695 (0.3823) data time 0.0011 (0.0021) model time 0.3684 (0.3794) loss 2.3829 (3.4958) grad_norm 1.0227 (1.3117) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:43:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][440/625] eta 0:01:10 lr 0.001844 wd 0.0500 time 0.3694 (0.3821) data time 0.0009 (0.0021) model time 0.3685 (0.3793) loss 3.1602 (3.4932) grad_norm 1.1723 (1.3160) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][450/625] eta 0:01:06 lr 0.001844 wd 0.0500 time 0.3620 (0.3819) data time 0.0009 (0.0021) model time 0.3611 (0.3791) loss 4.0496 (3.4986) grad_norm 1.8864 (1.3135) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][460/625] eta 0:01:02 lr 0.001844 wd 0.0500 time 0.3730 (0.3817) data time 0.0011 (0.0020) model time 0.3719 (0.3789) loss 3.4801 (3.4989) grad_norm 1.2250 (1.3124) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][470/625] eta 0:00:59 lr 0.001844 wd 0.0500 time 0.3710 (0.3815) data time 0.0010 (0.0020) model time 0.3700 (0.3787) loss 3.2461 (3.5039) grad_norm 1.8502 (1.3114) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][480/625] eta 0:00:55 lr 0.001844 wd 0.0500 time 0.3710 (0.3814) data time 0.0008 (0.0020) model time 0.3702 (0.3786) loss 2.9041 (3.5030) grad_norm 1.2026 (1.3106) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][490/625] eta 0:00:51 lr 0.001844 wd 0.0500 time 0.3813 (0.3813) data time 0.0009 (0.0020) model time 0.3803 (0.3786) loss 3.6188 (3.5008) grad_norm 1.2285 (1.3068) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][500/625] eta 0:00:47 lr 0.001843 wd 0.0500 time 0.3690 (0.3820) data time 0.0012 (0.0020) model time 0.3679 (0.3794) loss 3.6633 (3.4963) grad_norm 1.3980 (1.3036) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][510/625] eta 0:00:43 lr 0.001843 wd 0.0500 time 0.3681 (0.3819) data time 0.0012 (0.0020) model time 0.3669 (0.3793) loss 2.7611 (3.4952) grad_norm 1.2924 (1.3025) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][520/625] eta 0:00:40 lr 0.001843 wd 0.0500 time 0.3806 (0.3817) data time 0.0011 (0.0019) model time 0.3796 (0.3792) loss 3.7992 (3.4935) grad_norm 1.6211 (1.3084) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][530/625] eta 0:00:36 lr 0.001843 wd 0.0500 time 0.3732 (0.3820) data time 0.0009 (0.0019) model time 0.3723 (0.3795) loss 4.1855 (3.4966) grad_norm 1.4447 (1.3119) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][540/625] eta 0:00:32 lr 0.001843 wd 0.0500 time 0.3765 (0.3831) data time 0.0010 (0.0019) model time 0.3755 (0.3807) loss 3.5444 (3.4984) grad_norm 1.5085 (1.3184) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][550/625] eta 0:00:28 lr 0.001843 wd 0.0500 time 0.3717 (0.3829) data time 0.0008 (0.0019) model time 0.3709 (0.3806) loss 3.5862 (3.4974) grad_norm 1.3282 (1.3159) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][560/625] eta 0:00:24 lr 0.001843 wd 0.0500 time 0.3677 (0.3828) data time 0.0008 (0.0019) model time 0.3668 (0.3804) loss 2.5553 (3.4970) grad_norm 1.2919 (1.3138) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][570/625] eta 0:00:21 lr 0.001843 wd 0.0500 time 0.3612 (0.3826) data time 0.0009 (0.0019) model time 0.3603 (0.3803) loss 3.2430 (3.5004) grad_norm 1.3392 (1.3141) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][580/625] eta 0:00:17 lr 0.001843 wd 0.0500 time 0.3731 (0.3825) data time 0.0011 (0.0019) model time 0.3720 (0.3802) loss 3.8279 (3.4981) grad_norm 1.2408 (1.3121) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:44:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][590/625] eta 0:00:13 lr 0.001843 wd 0.0500 time 0.3714 (0.3824) data time 0.0011 (0.0018) model time 0.3703 (0.3800) loss 2.9960 (3.4917) grad_norm 1.1259 (1.3124) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][600/625] eta 0:00:09 lr 0.001843 wd 0.0500 time 0.3766 (0.3822) data time 0.0011 (0.0018) model time 0.3755 (0.3799) loss 3.2123 (3.4947) grad_norm 1.1445 (1.3127) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][610/625] eta 0:00:05 lr 0.001842 wd 0.0500 time 0.3679 (0.3821) data time 0.0006 (0.0018) model time 0.3674 (0.3798) loss 4.0032 (3.4962) grad_norm 1.4572 (1.3140) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [70/300][620/625] eta 0:00:01 lr 0.001842 wd 0.0500 time 0.3573 (0.3818) data time 0.0008 (0.0018) model time 0.3565 (0.3795) loss 3.2778 (3.4965) grad_norm 1.7874 (1.3138) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:09 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 70 training takes 0:03:58 [2024-08-26 07:45:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:45:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:45:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.571 (0.571) Loss 0.6851 (0.6851) Acc@1 84.863 (84.863) Acc@5 96.875 (96.875) Mem 11225MB [2024-08-26 07:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.149) Loss 1.1699 (0.8423) Acc@1 71.631 (80.646) Acc@5 92.090 (95.841) Mem 11225MB [2024-08-26 07:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.123) Loss 1.3281 (1.0241) Acc@1 67.773 (76.442) Acc@5 90.479 (93.673) Mem 11225MB [2024-08-26 07:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.246 Acc@5 93.648 [2024-08-26 07:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.2% [2024-08-26 07:45:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 1.090 (1.090) Loss 0.5669 (0.5669) Acc@1 87.646 (87.646) Acc@5 97.949 (97.949) Mem 11225MB [2024-08-26 07:45:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.196) Loss 0.9692 (0.7136) Acc@1 76.758 (83.416) Acc@5 94.141 (96.826) Mem 11225MB [2024-08-26 07:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.147) Loss 1.1338 (0.8777) Acc@1 71.631 (79.292) Acc@5 92.041 (94.906) Mem 11225MB [2024-08-26 07:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.985 Acc@5 94.860 [2024-08-26 07:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.0% [2024-08-26 07:45:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][0/625] eta 0:14:39 lr 0.001842 wd 0.0500 time 1.4067 (1.4067) data time 0.8865 (0.8865) model time 0.0000 (0.0000) loss 4.3814 (4.3814) grad_norm 1.5452 (1.5452) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][10/625] eta 0:04:50 lr 0.001842 wd 0.0500 time 0.3754 (0.4721) data time 0.0011 (0.0817) model time 0.0000 (0.0000) loss 3.3910 (3.4796) grad_norm 1.0785 (1.3449) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][20/625] eta 0:04:17 lr 0.001842 wd 0.0500 time 0.3746 (0.4261) data time 0.0011 (0.0433) model time 0.0000 (0.0000) loss 3.8020 (3.5827) grad_norm 1.6731 (1.2750) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][30/625] eta 0:04:04 lr 0.001842 wd 0.0500 time 0.3750 (0.4107) data time 0.0008 (0.0297) model time 0.0000 (0.0000) loss 3.0323 (3.6046) grad_norm 1.1751 (1.2390) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][40/625] eta 0:03:55 lr 0.001842 wd 0.0500 time 0.3800 (0.4022) data time 0.0011 (0.0228) model time 0.0000 (0.0000) loss 3.5077 (3.6116) grad_norm 0.9600 (1.2217) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][50/625] eta 0:03:48 lr 0.001842 wd 0.0500 time 0.3755 (0.3970) data time 0.0011 (0.0185) model time 0.0000 (0.0000) loss 3.4427 (3.5381) grad_norm 2.2540 (1.2332) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][60/625] eta 0:03:46 lr 0.001842 wd 0.0500 time 0.5824 (0.4003) data time 0.0011 (0.0157) model time 0.5813 (0.4157) loss 3.8753 (3.5038) grad_norm 1.1729 (1.2018) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][70/625] eta 0:03:46 lr 0.001842 wd 0.0500 time 0.5506 (0.4079) data time 0.0010 (0.0136) model time 0.5496 (0.4345) loss 3.7174 (3.5020) grad_norm 1.6291 (1.1941) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][80/625] eta 0:03:40 lr 0.001842 wd 0.0500 time 0.3718 (0.4042) data time 0.0011 (0.0121) model time 0.3708 (0.4152) loss 3.2628 (3.4651) grad_norm 1.2138 (1.1963) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][90/625] eta 0:03:34 lr 0.001841 wd 0.0500 time 0.3864 (0.4013) data time 0.0007 (0.0109) model time 0.3857 (0.4056) loss 3.4743 (3.4514) grad_norm 1.8947 (1.1959) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:45:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][100/625] eta 0:03:29 lr 0.001841 wd 0.0500 time 0.3807 (0.3991) data time 0.0010 (0.0100) model time 0.3797 (0.4000) loss 3.6361 (3.4444) grad_norm 1.0773 (1.2022) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][110/625] eta 0:03:24 lr 0.001841 wd 0.0500 time 0.3842 (0.3973) data time 0.0009 (0.0092) model time 0.3834 (0.3963) loss 3.1664 (3.4505) grad_norm 1.6070 (1.2188) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][120/625] eta 0:03:19 lr 0.001841 wd 0.0500 time 0.3660 (0.3957) data time 0.0009 (0.0085) model time 0.3652 (0.3935) loss 4.1820 (3.4366) grad_norm 0.9613 (1.2180) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][130/625] eta 0:03:15 lr 0.001841 wd 0.0500 time 0.3833 (0.3943) data time 0.0010 (0.0079) model time 0.3823 (0.3914) loss 3.7332 (3.4402) grad_norm 0.9872 (1.2231) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][140/625] eta 0:03:10 lr 0.001841 wd 0.0500 time 0.3741 (0.3931) data time 0.0011 (0.0074) model time 0.3730 (0.3897) loss 3.5729 (3.4447) grad_norm 1.5085 (1.2402) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][150/625] eta 0:03:06 lr 0.001841 wd 0.0500 time 0.3796 (0.3919) data time 0.0010 (0.0070) model time 0.3786 (0.3881) loss 3.0167 (3.4216) grad_norm 1.0089 (1.2333) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][160/625] eta 0:03:01 lr 0.001841 wd 0.0500 time 0.3779 (0.3910) data time 0.0011 (0.0067) model time 0.3768 (0.3870) loss 2.4661 (3.4263) grad_norm 1.7048 (1.2392) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][170/625] eta 0:02:57 lr 0.001841 wd 0.0500 time 0.3737 (0.3899) data time 0.0008 (0.0063) model time 0.3728 (0.3857) loss 2.6422 (3.4248) grad_norm 1.3108 (1.2566) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][180/625] eta 0:02:53 lr 0.001841 wd 0.0500 time 0.3707 (0.3901) data time 0.0011 (0.0060) model time 0.3697 (0.3863) loss 4.2348 (3.4230) grad_norm 1.1764 (1.2657) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][190/625] eta 0:02:49 lr 0.001840 wd 0.0500 time 0.3797 (0.3893) data time 0.0008 (0.0058) model time 0.3788 (0.3853) loss 4.5396 (3.4495) grad_norm 1.1026 (1.2646) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][200/625] eta 0:02:45 lr 0.001840 wd 0.0500 time 0.3744 (0.3886) data time 0.0009 (0.0056) model time 0.3735 (0.3846) loss 4.3083 (3.4612) grad_norm 1.2316 (1.2667) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][210/625] eta 0:02:41 lr 0.001840 wd 0.0500 time 0.3745 (0.3880) data time 0.0011 (0.0054) model time 0.3734 (0.3839) loss 3.7701 (3.4756) grad_norm 0.9978 (1.2665) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][220/625] eta 0:02:36 lr 0.001840 wd 0.0500 time 0.3702 (0.3874) data time 0.0009 (0.0052) model time 0.3692 (0.3834) loss 3.2800 (3.4696) grad_norm 1.4275 (1.2656) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][230/625] eta 0:02:32 lr 0.001840 wd 0.0500 time 0.3772 (0.3869) data time 0.0011 (0.0050) model time 0.3761 (0.3829) loss 2.6750 (3.4698) grad_norm 1.2011 (1.2750) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][240/625] eta 0:02:28 lr 0.001840 wd 0.0500 time 0.3965 (0.3865) data time 0.0011 (0.0048) model time 0.3954 (0.3825) loss 2.9288 (3.4775) grad_norm 1.1881 (1.2703) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][250/625] eta 0:02:25 lr 0.001840 wd 0.0500 time 0.3730 (0.3883) data time 0.0009 (0.0047) model time 0.3721 (0.3849) loss 4.3088 (3.4858) grad_norm 1.2858 (1.2665) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][260/625] eta 0:02:21 lr 0.001840 wd 0.0500 time 0.3676 (0.3876) data time 0.0008 (0.0046) model time 0.3668 (0.3842) loss 3.4984 (3.4924) grad_norm 1.2621 (1.2635) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 07:47:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][270/625] eta 0:02:17 lr 0.001840 wd 0.0500 time 0.3695 (0.3871) data time 0.0008 (0.0044) model time 0.3687 (0.3837) loss 3.8588 (3.4938) grad_norm 2.2934 (1.2758) loss_scale 2048.0000 (1039.1144) mem 11225MB [2024-08-26 07:47:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][280/625] eta 0:02:14 lr 0.001840 wd 0.0500 time 0.5869 (0.3889) data time 0.0008 (0.0043) model time 0.5861 (0.3860) loss 3.1026 (3.4958) grad_norm 0.8739 (1.2715) loss_scale 2048.0000 (1075.0178) mem 11225MB [2024-08-26 07:47:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][290/625] eta 0:02:10 lr 0.001839 wd 0.0500 time 0.3707 (0.3884) data time 0.0012 (0.0042) model time 0.3695 (0.3854) loss 3.4892 (3.4938) grad_norm 1.8135 (1.2778) loss_scale 2048.0000 (1108.4536) mem 11225MB [2024-08-26 07:47:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][300/625] eta 0:02:06 lr 0.001839 wd 0.0500 time 0.3819 (0.3881) data time 0.0008 (0.0041) model time 0.3811 (0.3851) loss 3.8365 (3.4962) grad_norm 0.9478 (1.2705) loss_scale 2048.0000 (1139.6678) mem 11225MB [2024-08-26 07:47:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][310/625] eta 0:02:02 lr 0.001839 wd 0.0500 time 0.3819 (0.3877) data time 0.0009 (0.0040) model time 0.3810 (0.3847) loss 3.7923 (3.4923) grad_norm 1.0700 (1.2630) loss_scale 2048.0000 (1168.8746) mem 11225MB [2024-08-26 07:47:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][320/625] eta 0:01:58 lr 0.001839 wd 0.0500 time 0.3759 (0.3874) data time 0.0011 (0.0039) model time 0.3748 (0.3844) loss 3.4757 (3.5020) grad_norm 1.2180 (1.2679) loss_scale 2048.0000 (1196.2617) mem 11225MB [2024-08-26 07:47:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][330/625] eta 0:01:54 lr 0.001839 wd 0.0500 time 0.3709 (0.3871) data time 0.0008 (0.0038) model time 0.3701 (0.3841) loss 4.1375 (3.5079) grad_norm 1.0498 (1.2723) loss_scale 2048.0000 (1221.9940) mem 11225MB [2024-08-26 07:47:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][340/625] eta 0:01:50 lr 0.001839 wd 0.0500 time 0.3813 (0.3868) data time 0.0011 (0.0038) model time 0.3802 (0.3839) loss 1.9791 (3.4981) grad_norm 1.1976 (1.2678) loss_scale 2048.0000 (1246.2170) mem 11225MB [2024-08-26 07:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][350/625] eta 0:01:46 lr 0.001839 wd 0.0500 time 0.3753 (0.3865) data time 0.0008 (0.0037) model time 0.3745 (0.3836) loss 3.8833 (3.4939) grad_norm 1.4630 (1.2676) loss_scale 2048.0000 (1269.0598) mem 11225MB [2024-08-26 07:47:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][360/625] eta 0:01:42 lr 0.001839 wd 0.0500 time 0.3797 (0.3862) data time 0.0008 (0.0036) model time 0.3789 (0.3833) loss 2.6177 (3.4810) grad_norm 1.0989 (1.2632) loss_scale 2048.0000 (1290.6371) mem 11225MB [2024-08-26 07:47:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][370/625] eta 0:01:38 lr 0.001839 wd 0.0500 time 0.3804 (0.3871) data time 0.0010 (0.0035) model time 0.3794 (0.3843) loss 3.3209 (3.4824) grad_norm 1.0540 (1.2630) loss_scale 2048.0000 (1311.0512) mem 11225MB [2024-08-26 07:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][380/625] eta 0:01:34 lr 0.001839 wd 0.0500 time 0.3769 (0.3868) data time 0.0008 (0.0035) model time 0.3761 (0.3841) loss 2.9703 (3.4826) grad_norm 1.3672 (1.2655) loss_scale 2048.0000 (1330.3937) mem 11225MB [2024-08-26 07:47:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][390/625] eta 0:01:30 lr 0.001839 wd 0.0500 time 0.3757 (0.3871) data time 0.0008 (0.0034) model time 0.3749 (0.3844) loss 3.8615 (3.4804) grad_norm 0.9968 (1.2648) loss_scale 2048.0000 (1348.7468) mem 11225MB [2024-08-26 07:47:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][400/625] eta 0:01:27 lr 0.001838 wd 0.0500 time 0.3746 (0.3868) data time 0.0010 (0.0034) model time 0.3736 (0.3842) loss 3.8385 (3.4771) grad_norm 1.1029 (1.2629) loss_scale 2048.0000 (1366.1845) mem 11225MB [2024-08-26 07:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][410/625] eta 0:01:23 lr 0.001838 wd 0.0500 time 0.3740 (0.3866) data time 0.0011 (0.0033) model time 0.3730 (0.3840) loss 3.3911 (3.4746) grad_norm 0.9858 (1.2628) loss_scale 2048.0000 (1382.7737) mem 11225MB [2024-08-26 07:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][420/625] eta 0:01:19 lr 0.001838 wd 0.0500 time 0.3790 (0.3863) data time 0.0010 (0.0033) model time 0.3780 (0.3837) loss 3.2487 (3.4718) grad_norm 0.8851 (1.2650) loss_scale 2048.0000 (1398.5748) mem 11225MB [2024-08-26 07:48:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][430/625] eta 0:01:15 lr 0.001838 wd 0.0500 time 0.3674 (0.3861) data time 0.0008 (0.0032) model time 0.3666 (0.3834) loss 3.6755 (3.4664) grad_norm 1.2428 (1.2657) loss_scale 2048.0000 (1413.6427) mem 11225MB [2024-08-26 07:48:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][440/625] eta 0:01:11 lr 0.001838 wd 0.0500 time 0.3767 (0.3858) data time 0.0010 (0.0032) model time 0.3758 (0.3832) loss 3.0684 (3.4697) grad_norm 1.2894 (1.2676) loss_scale 2048.0000 (1428.0272) mem 11225MB [2024-08-26 07:48:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][450/625] eta 0:01:07 lr 0.001838 wd 0.0500 time 0.3855 (0.3856) data time 0.0008 (0.0031) model time 0.3847 (0.3830) loss 3.7706 (3.4624) grad_norm 1.1987 (1.2708) loss_scale 2048.0000 (1441.7738) mem 11225MB [2024-08-26 07:48:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][460/625] eta 0:01:03 lr 0.001838 wd 0.0500 time 0.3758 (0.3859) data time 0.0009 (0.0031) model time 0.3748 (0.3834) loss 3.1028 (3.4629) grad_norm 1.3930 (1.2758) loss_scale 2048.0000 (1454.9241) mem 11225MB [2024-08-26 07:48:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][470/625] eta 0:00:59 lr 0.001838 wd 0.0500 time 0.3732 (0.3865) data time 0.0010 (0.0030) model time 0.3722 (0.3841) loss 3.5677 (3.4535) grad_norm 1.1236 (1.2749) loss_scale 2048.0000 (1467.5159) mem 11225MB [2024-08-26 07:48:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][480/625] eta 0:00:56 lr 0.001838 wd 0.0500 time 0.3742 (0.3863) data time 0.0009 (0.0030) model time 0.3733 (0.3838) loss 4.3745 (3.4509) grad_norm 1.1451 (1.2756) loss_scale 2048.0000 (1479.5842) mem 11225MB [2024-08-26 07:48:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][490/625] eta 0:00:52 lr 0.001838 wd 0.0500 time 0.3680 (0.3860) data time 0.0008 (0.0030) model time 0.3672 (0.3836) loss 4.0518 (3.4532) grad_norm 0.9972 (1.2709) loss_scale 2048.0000 (1491.1609) mem 11225MB [2024-08-26 07:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][500/625] eta 0:00:48 lr 0.001837 wd 0.0500 time 0.3726 (0.3858) data time 0.0011 (0.0029) model time 0.3715 (0.3834) loss 3.8173 (3.4535) grad_norm 1.4051 (1.2703) loss_scale 2048.0000 (1502.2754) mem 11225MB [2024-08-26 07:48:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][510/625] eta 0:00:44 lr 0.001837 wd 0.0500 time 0.3733 (0.3856) data time 0.0010 (0.0029) model time 0.3724 (0.3832) loss 3.5700 (3.4556) grad_norm 1.2007 (1.2691) loss_scale 2048.0000 (1512.9550) mem 11225MB [2024-08-26 07:48:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][520/625] eta 0:00:40 lr 0.001837 wd 0.0500 time 0.3789 (0.3855) data time 0.0009 (0.0028) model time 0.3780 (0.3831) loss 2.9723 (3.4534) grad_norm 1.1800 (1.2671) loss_scale 2048.0000 (1523.2246) mem 11225MB [2024-08-26 07:48:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][530/625] eta 0:00:36 lr 0.001837 wd 0.0500 time 0.3684 (0.3853) data time 0.0009 (0.0028) model time 0.3675 (0.3829) loss 3.8580 (3.4513) grad_norm 1.4286 (1.2660) loss_scale 2048.0000 (1533.1073) mem 11225MB [2024-08-26 07:48:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][540/625] eta 0:00:32 lr 0.001837 wd 0.0500 time 0.3865 (0.3852) data time 0.0008 (0.0028) model time 0.3856 (0.3828) loss 3.7105 (3.4556) grad_norm 1.2662 (1.2673) loss_scale 2048.0000 (1542.6248) mem 11225MB [2024-08-26 07:48:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][550/625] eta 0:00:28 lr 0.001837 wd 0.0500 time 0.3789 (0.3850) data time 0.0011 (0.0028) model time 0.3778 (0.3827) loss 3.7369 (3.4581) grad_norm 1.1449 (1.2638) loss_scale 2048.0000 (1551.7967) mem 11225MB [2024-08-26 07:48:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][560/625] eta 0:00:25 lr 0.001837 wd 0.0500 time 0.3720 (0.3849) data time 0.0011 (0.0027) model time 0.3708 (0.3825) loss 4.3925 (3.4596) grad_norm 1.4066 (1.2642) loss_scale 2048.0000 (1560.6417) mem 11225MB [2024-08-26 07:48:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][570/625] eta 0:00:21 lr 0.001837 wd 0.0500 time 0.3743 (0.3850) data time 0.0011 (0.0027) model time 0.3732 (0.3827) loss 3.3814 (3.4592) grad_norm 1.1916 (1.2655) loss_scale 2048.0000 (1569.1769) mem 11225MB [2024-08-26 07:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][580/625] eta 0:00:17 lr 0.001837 wd 0.0500 time 0.3775 (0.3859) data time 0.0011 (0.0027) model time 0.3764 (0.3837) loss 3.7190 (3.4592) grad_norm 1.0506 (1.2711) loss_scale 2048.0000 (1577.4182) mem 11225MB [2024-08-26 07:49:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][590/625] eta 0:00:13 lr 0.001837 wd 0.0500 time 0.3674 (0.3861) data time 0.0008 (0.0026) model time 0.3666 (0.3840) loss 2.2684 (3.4599) grad_norm 1.0183 (1.2721) loss_scale 2048.0000 (1585.3807) mem 11225MB [2024-08-26 07:49:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][600/625] eta 0:00:09 lr 0.001836 wd 0.0500 time 0.3867 (0.3860) data time 0.0008 (0.0026) model time 0.3859 (0.3838) loss 2.3890 (3.4586) grad_norm 1.1196 (1.2695) loss_scale 2048.0000 (1593.0782) mem 11225MB [2024-08-26 07:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][610/625] eta 0:00:05 lr 0.001836 wd 0.0500 time 0.3687 (0.3859) data time 0.0005 (0.0026) model time 0.3681 (0.3837) loss 2.7100 (3.4534) grad_norm 1.4506 (1.2676) loss_scale 2048.0000 (1600.5237) mem 11225MB [2024-08-26 07:49:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [71/300][620/625] eta 0:00:01 lr 0.001836 wd 0.0500 time 0.3596 (0.3856) data time 0.0006 (0.0026) model time 0.3589 (0.3834) loss 3.5825 (3.4562) grad_norm 0.7842 (1.2658) loss_scale 2048.0000 (1607.7295) mem 11225MB [2024-08-26 07:49:17 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 71 training takes 0:04:00 [2024-08-26 07:49:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:49:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:49:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.529 (0.529) Loss 0.6846 (0.6846) Acc@1 85.547 (85.547) Acc@5 97.119 (97.119) Mem 11225MB [2024-08-26 07:49:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.146) Loss 1.1309 (0.8549) Acc@1 72.852 (80.948) Acc@5 92.529 (96.027) Mem 11225MB [2024-08-26 07:49:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.2881 (1.0321) Acc@1 70.068 (76.744) Acc@5 90.527 (93.738) Mem 11225MB [2024-08-26 07:49:21 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.581 Acc@5 93.732 [2024-08-26 07:49:21 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.6% [2024-08-26 07:49:21 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 76.58% [2024-08-26 07:49:21 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 07:49:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 07:49:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.523 (0.523) Loss 0.5674 (0.5674) Acc@1 87.646 (87.646) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 07:49:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 0.9668 (0.7124) Acc@1 76.660 (83.461) Acc@5 94.238 (96.862) Mem 11225MB [2024-08-26 07:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.1318 (0.8760) Acc@1 71.338 (79.329) Acc@5 92.139 (94.945) Mem 11225MB [2024-08-26 07:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.035 Acc@5 94.888 [2024-08-26 07:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.0% [2024-08-26 07:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.03% [2024-08-26 07:49:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:49:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:49:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][0/625] eta 0:10:07 lr 0.001836 wd 0.0500 time 0.9723 (0.9723) data time 0.6398 (0.6398) model time 0.0000 (0.0000) loss 3.4501 (3.4501) grad_norm 1.2457 (1.2457) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:49:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][10/625] eta 0:04:36 lr 0.001836 wd 0.0500 time 0.3793 (0.4493) data time 0.0010 (0.0593) model time 0.0000 (0.0000) loss 4.0759 (3.5494) grad_norm 2.0496 (1.6312) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:49:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][20/625] eta 0:04:10 lr 0.001836 wd 0.0500 time 0.3727 (0.4137) data time 0.0011 (0.0315) model time 0.0000 (0.0000) loss 4.1836 (3.6432) grad_norm 1.6154 (1.7258) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:49:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][30/625] eta 0:03:58 lr 0.001836 wd 0.0500 time 0.3765 (0.4016) data time 0.0010 (0.0217) model time 0.0000 (0.0000) loss 4.0818 (3.6816) grad_norm 0.9068 (1.5643) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:49:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][40/625] eta 0:03:51 lr 0.001836 wd 0.0500 time 0.3831 (0.3959) data time 0.0011 (0.0167) model time 0.0000 (0.0000) loss 3.9003 (3.6184) grad_norm 0.7871 (1.4608) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][50/625] eta 0:03:45 lr 0.001836 wd 0.0500 time 0.3790 (0.3919) data time 0.0011 (0.0137) model time 0.0000 (0.0000) loss 3.3031 (3.6543) grad_norm 1.5839 (1.4394) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][60/625] eta 0:03:40 lr 0.001836 wd 0.0500 time 0.3706 (0.3897) data time 0.0011 (0.0116) model time 0.3695 (0.3776) loss 2.6254 (3.6258) grad_norm 0.9562 (1.4033) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][70/625] eta 0:03:35 lr 0.001836 wd 0.0500 time 0.3690 (0.3875) data time 0.0008 (0.0101) model time 0.3682 (0.3751) loss 4.0889 (3.6659) grad_norm 0.9194 (1.3631) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:49:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][80/625] eta 0:03:30 lr 0.001835 wd 0.0500 time 0.3672 (0.3861) data time 0.0008 (0.0090) model time 0.3664 (0.3751) loss 3.4963 (3.6331) grad_norm 1.6190 (1.3863) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][90/625] eta 0:03:25 lr 0.001835 wd 0.0500 time 0.3697 (0.3846) data time 0.0009 (0.0082) model time 0.3688 (0.3742) loss 2.6432 (3.5804) grad_norm 1.0838 (1.3966) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][100/625] eta 0:03:23 lr 0.001835 wd 0.0500 time 0.3852 (0.3874) data time 0.0011 (0.0075) model time 0.3841 (0.3817) loss 3.1441 (3.5589) grad_norm 1.1915 (1.3761) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][110/625] eta 0:03:18 lr 0.001835 wd 0.0500 time 0.3779 (0.3862) data time 0.0010 (0.0069) model time 0.3768 (0.3801) loss 2.6697 (3.5409) grad_norm 1.0086 (1.3573) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][120/625] eta 0:03:14 lr 0.001835 wd 0.0500 time 0.3832 (0.3855) data time 0.0011 (0.0064) model time 0.3821 (0.3797) loss 3.7292 (3.5133) grad_norm 1.1172 (1.3622) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][130/625] eta 0:03:10 lr 0.001835 wd 0.0500 time 0.3844 (0.3850) data time 0.0012 (0.0060) model time 0.3832 (0.3794) loss 3.6292 (3.5010) grad_norm 1.0234 (1.3805) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][140/625] eta 0:03:06 lr 0.001835 wd 0.0500 time 0.3814 (0.3846) data time 0.0011 (0.0057) model time 0.3803 (0.3793) loss 3.5275 (3.4888) grad_norm 1.2987 (1.3600) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][150/625] eta 0:03:03 lr 0.001835 wd 0.0500 time 0.5779 (0.3856) data time 0.0011 (0.0054) model time 0.5768 (0.3813) loss 3.8129 (3.4887) grad_norm 0.9782 (1.3552) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][160/625] eta 0:02:59 lr 0.001835 wd 0.0500 time 0.3833 (0.3866) data time 0.0008 (0.0051) model time 0.3824 (0.3830) loss 4.1417 (3.4722) grad_norm 1.0504 (1.3511) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][170/625] eta 0:02:55 lr 0.001835 wd 0.0500 time 0.3783 (0.3863) data time 0.0011 (0.0049) model time 0.3772 (0.3827) loss 3.0145 (3.4578) grad_norm 1.1676 (1.3547) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][180/625] eta 0:02:51 lr 0.001834 wd 0.0500 time 0.3826 (0.3860) data time 0.0008 (0.0047) model time 0.3818 (0.3825) loss 2.2129 (3.4545) grad_norm 1.2454 (1.3429) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][190/625] eta 0:02:47 lr 0.001834 wd 0.0500 time 0.3814 (0.3858) data time 0.0011 (0.0045) model time 0.3803 (0.3823) loss 3.5217 (3.4469) grad_norm 1.2974 (1.3408) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][200/625] eta 0:02:43 lr 0.001834 wd 0.0500 time 0.3859 (0.3853) data time 0.0009 (0.0043) model time 0.3850 (0.3819) loss 4.3825 (3.4677) grad_norm 1.2144 (1.3470) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][210/625] eta 0:02:39 lr 0.001834 wd 0.0500 time 0.3749 (0.3849) data time 0.0008 (0.0042) model time 0.3741 (0.3815) loss 2.4385 (3.4556) grad_norm 0.9962 (1.3440) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][220/625] eta 0:02:36 lr 0.001834 wd 0.0500 time 0.3737 (0.3855) data time 0.0009 (0.0040) model time 0.3728 (0.3824) loss 2.5507 (3.4525) grad_norm 1.3824 (1.3410) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][230/625] eta 0:02:32 lr 0.001834 wd 0.0500 time 0.3860 (0.3853) data time 0.0011 (0.0039) model time 0.3849 (0.3822) loss 3.4258 (3.4551) grad_norm 1.6588 (1.3372) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:50:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][240/625] eta 0:02:28 lr 0.001834 wd 0.0500 time 0.3892 (0.3851) data time 0.0011 (0.0038) model time 0.3881 (0.3821) loss 4.0388 (3.4626) grad_norm 1.5199 (1.3378) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][250/625] eta 0:02:24 lr 0.001834 wd 0.0500 time 0.3830 (0.3848) data time 0.0010 (0.0037) model time 0.3820 (0.3819) loss 3.5403 (3.4722) grad_norm 1.2432 (1.3320) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][260/625] eta 0:02:20 lr 0.001834 wd 0.0500 time 0.3725 (0.3845) data time 0.0012 (0.0036) model time 0.3713 (0.3816) loss 3.5565 (3.4657) grad_norm 0.9857 (1.3387) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][270/625] eta 0:02:16 lr 0.001834 wd 0.0500 time 0.3900 (0.3844) data time 0.0010 (0.0035) model time 0.3889 (0.3815) loss 3.7088 (3.4652) grad_norm 1.4205 (1.3469) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][280/625] eta 0:02:12 lr 0.001833 wd 0.0500 time 0.3782 (0.3841) data time 0.0010 (0.0034) model time 0.3772 (0.3813) loss 3.6782 (3.4656) grad_norm 1.4939 (1.3393) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][290/625] eta 0:02:08 lr 0.001833 wd 0.0500 time 0.3752 (0.3838) data time 0.0010 (0.0033) model time 0.3742 (0.3810) loss 3.8725 (3.4709) grad_norm 0.9953 (1.3391) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][300/625] eta 0:02:04 lr 0.001833 wd 0.0500 time 0.3748 (0.3836) data time 0.0011 (0.0033) model time 0.3737 (0.3808) loss 4.0532 (3.4640) grad_norm 0.8283 (1.3429) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][310/625] eta 0:02:01 lr 0.001833 wd 0.0500 time 0.3704 (0.3842) data time 0.0011 (0.0032) model time 0.3693 (0.3815) loss 3.6815 (3.4614) grad_norm 1.2222 (1.3384) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][320/625] eta 0:01:57 lr 0.001833 wd 0.0500 time 0.3779 (0.3840) data time 0.0012 (0.0031) model time 0.3767 (0.3813) loss 3.7044 (3.4567) grad_norm 1.1462 (1.3346) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][330/625] eta 0:01:53 lr 0.001833 wd 0.0500 time 0.3850 (0.3839) data time 0.0009 (0.0031) model time 0.3840 (0.3813) loss 2.6577 (3.4473) grad_norm 1.2509 (1.3298) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][340/625] eta 0:01:49 lr 0.001833 wd 0.0500 time 0.3740 (0.3853) data time 0.0011 (0.0030) model time 0.3730 (0.3830) loss 3.9585 (3.4480) grad_norm 1.2729 (1.3291) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][350/625] eta 0:01:45 lr 0.001833 wd 0.0500 time 0.3781 (0.3852) data time 0.0011 (0.0030) model time 0.3770 (0.3829) loss 3.6014 (3.4491) grad_norm 1.5569 (1.3245) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][360/625] eta 0:01:42 lr 0.001833 wd 0.0500 time 0.3765 (0.3850) data time 0.0008 (0.0029) model time 0.3757 (0.3827) loss 3.3571 (3.4526) grad_norm 1.0333 (1.3231) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][370/625] eta 0:01:38 lr 0.001833 wd 0.0500 time 0.3684 (0.3848) data time 0.0009 (0.0029) model time 0.3675 (0.3825) loss 3.9857 (3.4547) grad_norm 1.1292 (1.3189) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][380/625] eta 0:01:34 lr 0.001832 wd 0.0500 time 0.3779 (0.3846) data time 0.0011 (0.0028) model time 0.3769 (0.3823) loss 3.3482 (3.4486) grad_norm 1.2097 (1.3199) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:51:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][390/625] eta 0:01:30 lr 0.001832 wd 0.0500 time 0.3764 (0.3845) data time 0.0011 (0.0028) model time 0.3753 (0.3822) loss 3.3987 (3.4557) grad_norm 1.6538 (1.3184) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][400/625] eta 0:01:26 lr 0.001832 wd 0.0500 time 0.3805 (0.3843) data time 0.0009 (0.0027) model time 0.3796 (0.3821) loss 3.7089 (3.4616) grad_norm 1.2393 (1.3202) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][410/625] eta 0:01:22 lr 0.001832 wd 0.0500 time 0.3891 (0.3845) data time 0.0012 (0.0027) model time 0.3879 (0.3823) loss 3.1212 (3.4671) grad_norm 1.9865 (1.3256) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][420/625] eta 0:01:18 lr 0.001832 wd 0.0500 time 0.3751 (0.3843) data time 0.0010 (0.0027) model time 0.3741 (0.3822) loss 3.0878 (3.4608) grad_norm 1.5335 (1.3198) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][430/625] eta 0:01:14 lr 0.001832 wd 0.0500 time 0.3707 (0.3842) data time 0.0010 (0.0026) model time 0.3697 (0.3820) loss 3.8687 (3.4553) grad_norm 1.0206 (1.3230) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][440/625] eta 0:01:11 lr 0.001832 wd 0.0500 time 0.4008 (0.3841) data time 0.0011 (0.0026) model time 0.3998 (0.3819) loss 2.3800 (3.4506) grad_norm 1.6382 (1.3229) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][450/625] eta 0:01:07 lr 0.001832 wd 0.0500 time 0.3804 (0.3840) data time 0.0010 (0.0026) model time 0.3793 (0.3819) loss 3.5181 (3.4541) grad_norm 0.9542 (1.3223) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][460/625] eta 0:01:03 lr 0.001832 wd 0.0500 time 0.3778 (0.3848) data time 0.0010 (0.0025) model time 0.3767 (0.3828) loss 3.3383 (3.4540) grad_norm 1.7080 (1.3208) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][470/625] eta 0:00:59 lr 0.001832 wd 0.0500 time 0.3716 (0.3846) data time 0.0012 (0.0025) model time 0.3705 (0.3826) loss 3.6846 (3.4536) grad_norm 1.2595 (1.3186) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][480/625] eta 0:00:55 lr 0.001831 wd 0.0500 time 0.3839 (0.3846) data time 0.0012 (0.0025) model time 0.3828 (0.3825) loss 3.5855 (3.4583) grad_norm 1.3633 (1.3207) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][490/625] eta 0:00:52 lr 0.001831 wd 0.0500 time 0.3779 (0.3853) data time 0.0010 (0.0025) model time 0.3769 (0.3833) loss 3.7777 (3.4584) grad_norm 1.3394 (1.3250) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][500/625] eta 0:00:48 lr 0.001831 wd 0.0500 time 0.3763 (0.3851) data time 0.0010 (0.0024) model time 0.3753 (0.3832) loss 2.7659 (3.4551) grad_norm 1.2378 (1.3230) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][510/625] eta 0:00:44 lr 0.001831 wd 0.0500 time 0.3712 (0.3850) data time 0.0013 (0.0024) model time 0.3699 (0.3831) loss 3.8018 (3.4623) grad_norm 1.1023 (1.3186) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][520/625] eta 0:00:40 lr 0.001831 wd 0.0500 time 0.3813 (0.3849) data time 0.0010 (0.0024) model time 0.3802 (0.3830) loss 2.9978 (3.4625) grad_norm 1.1471 (1.3159) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][530/625] eta 0:00:36 lr 0.001831 wd 0.0500 time 0.3778 (0.3852) data time 0.0009 (0.0024) model time 0.3770 (0.3833) loss 4.1658 (3.4589) grad_norm 1.0585 (1.3139) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][540/625] eta 0:00:32 lr 0.001831 wd 0.0500 time 0.3743 (0.3851) data time 0.0010 (0.0023) model time 0.3732 (0.3832) loss 3.7457 (3.4610) grad_norm 1.3465 (1.3191) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][550/625] eta 0:00:28 lr 0.001831 wd 0.0500 time 0.3850 (0.3850) data time 0.0008 (0.0023) model time 0.3841 (0.3832) loss 2.5941 (3.4576) grad_norm 2.0506 (1.3195) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][560/625] eta 0:00:25 lr 0.001831 wd 0.0500 time 0.3712 (0.3849) data time 0.0009 (0.0023) model time 0.3703 (0.3830) loss 3.2289 (3.4617) grad_norm 0.9465 (1.3238) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][570/625] eta 0:00:21 lr 0.001831 wd 0.0500 time 0.3744 (0.3847) data time 0.0011 (0.0023) model time 0.3733 (0.3829) loss 3.7460 (3.4589) grad_norm 1.1952 (1.3260) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][580/625] eta 0:00:17 lr 0.001831 wd 0.0500 time 0.3763 (0.3846) data time 0.0009 (0.0023) model time 0.3753 (0.3827) loss 3.3486 (3.4555) grad_norm 1.6876 (1.3246) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][590/625] eta 0:00:13 lr 0.001830 wd 0.0500 time 0.3744 (0.3845) data time 0.0011 (0.0022) model time 0.3733 (0.3827) loss 3.1320 (3.4542) grad_norm 1.7449 (1.3238) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][600/625] eta 0:00:09 lr 0.001830 wd 0.0500 time 0.3881 (0.3844) data time 0.0008 (0.0022) model time 0.3873 (0.3826) loss 3.9773 (3.4557) grad_norm 1.2224 (1.3266) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][610/625] eta 0:00:05 lr 0.001830 wd 0.0500 time 0.3671 (0.3850) data time 0.0005 (0.0022) model time 0.3665 (0.3832) loss 3.0292 (3.4599) grad_norm 1.6119 (1.3233) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [72/300][620/625] eta 0:00:01 lr 0.001830 wd 0.0500 time 0.3661 (0.3847) data time 0.0008 (0.0022) model time 0.3653 (0.3829) loss 3.2748 (3.4580) grad_norm 1.3941 (1.3222) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 72 training takes 0:04:00 [2024-08-26 07:53:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:53:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:53:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.6313 (0.6313) Acc@1 86.279 (86.279) Acc@5 97.510 (97.510) Mem 11225MB [2024-08-26 07:53:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.1729 (0.8114) Acc@1 70.801 (81.064) Acc@5 91.846 (95.792) Mem 11225MB [2024-08-26 07:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.2754 (0.9919) Acc@1 69.727 (76.793) Acc@5 89.941 (93.622) Mem 11225MB [2024-08-26 07:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.504 Acc@5 93.604 [2024-08-26 07:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.5% [2024-08-26 07:53:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.870 (0.870) Loss 0.5679 (0.5679) Acc@1 87.646 (87.646) Acc@5 97.998 (97.998) Mem 11225MB [2024-08-26 07:53:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.179) Loss 0.9629 (0.7115) Acc@1 76.465 (83.478) Acc@5 94.336 (96.875) Mem 11225MB [2024-08-26 07:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.138) Loss 1.1270 (0.8745) Acc@1 71.631 (79.418) Acc@5 92.236 (94.996) Mem 11225MB [2024-08-26 07:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.129 Acc@5 94.942 [2024-08-26 07:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.1% [2024-08-26 07:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.13% [2024-08-26 07:53:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:53:36 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][0/625] eta 0:07:51 lr 0.001830 wd 0.0500 time 0.7551 (0.7551) data time 0.4086 (0.4086) model time 0.0000 (0.0000) loss 3.4414 (3.4414) grad_norm 0.8883 (0.8883) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][10/625] eta 0:04:12 lr 0.001830 wd 0.0500 time 0.3853 (0.4109) data time 0.0008 (0.0388) model time 0.0000 (0.0000) loss 3.5722 (3.5339) grad_norm 1.9377 (1.4980) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][20/625] eta 0:03:58 lr 0.001830 wd 0.0500 time 0.3776 (0.3949) data time 0.0013 (0.0209) model time 0.0000 (0.0000) loss 3.1445 (3.4415) grad_norm 1.1988 (1.3586) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][30/625] eta 0:03:56 lr 0.001830 wd 0.0500 time 0.3782 (0.3974) data time 0.0009 (0.0145) model time 0.0000 (0.0000) loss 3.4417 (3.4511) grad_norm 1.0566 (1.3312) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][40/625] eta 0:03:49 lr 0.001830 wd 0.0500 time 0.3881 (0.3928) data time 0.0011 (0.0113) model time 0.0000 (0.0000) loss 3.6986 (3.4340) grad_norm 0.9567 (1.3549) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][50/625] eta 0:03:45 lr 0.001830 wd 0.0500 time 0.3829 (0.3925) data time 0.0011 (0.0093) model time 0.0000 (0.0000) loss 3.1202 (3.4209) grad_norm 1.2847 (1.3287) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][60/625] eta 0:03:40 lr 0.001829 wd 0.0500 time 0.3762 (0.3903) data time 0.0011 (0.0080) model time 0.3751 (0.3780) loss 3.8566 (3.4342) grad_norm 1.2859 (1.3627) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][70/625] eta 0:03:35 lr 0.001829 wd 0.0500 time 0.3807 (0.3888) data time 0.0011 (0.0071) model time 0.3796 (0.3783) loss 3.0771 (3.4516) grad_norm 1.6721 (1.3455) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][80/625] eta 0:03:31 lr 0.001829 wd 0.0500 time 0.3815 (0.3876) data time 0.0010 (0.0063) model time 0.3804 (0.3779) loss 3.3456 (3.4530) grad_norm 1.3421 (1.3418) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][90/625] eta 0:03:26 lr 0.001829 wd 0.0500 time 0.3823 (0.3867) data time 0.0010 (0.0058) model time 0.3813 (0.3781) loss 3.6384 (3.4701) grad_norm 1.4427 (1.3364) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][100/625] eta 0:03:22 lr 0.001829 wd 0.0500 time 0.3805 (0.3858) data time 0.0011 (0.0053) model time 0.3794 (0.3777) loss 3.6100 (3.4861) grad_norm 1.2371 (1.3284) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][110/625] eta 0:03:18 lr 0.001829 wd 0.0500 time 0.3808 (0.3851) data time 0.0008 (0.0049) model time 0.3799 (0.3776) loss 3.5480 (3.4942) grad_norm 1.2024 (1.3226) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][120/625] eta 0:03:14 lr 0.001829 wd 0.0500 time 0.3728 (0.3847) data time 0.0014 (0.0046) model time 0.3714 (0.3778) loss 3.4439 (3.4651) grad_norm 1.1867 (1.3019) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][130/625] eta 0:03:10 lr 0.001829 wd 0.0500 time 0.3918 (0.3841) data time 0.0009 (0.0043) model time 0.3909 (0.3776) loss 3.2650 (3.4619) grad_norm 1.2050 (1.2957) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][140/625] eta 0:03:06 lr 0.001829 wd 0.0500 time 0.3756 (0.3836) data time 0.0008 (0.0041) model time 0.3748 (0.3774) loss 4.1156 (3.4542) grad_norm 1.0850 (1.2881) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][150/625] eta 0:03:01 lr 0.001829 wd 0.0500 time 0.3736 (0.3831) data time 0.0008 (0.0039) model time 0.3728 (0.3772) loss 2.3470 (3.4457) grad_norm 0.9899 (1.2764) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][160/625] eta 0:02:58 lr 0.001828 wd 0.0500 time 0.3709 (0.3828) data time 0.0010 (0.0037) model time 0.3699 (0.3771) loss 3.6978 (3.4239) grad_norm 1.0668 (1.2837) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][170/625] eta 0:02:55 lr 0.001828 wd 0.0500 time 0.3751 (0.3847) data time 0.0008 (0.0036) model time 0.3742 (0.3803) loss 2.9329 (3.4222) grad_norm 1.1244 (1.2973) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][180/625] eta 0:02:51 lr 0.001828 wd 0.0500 time 0.3796 (0.3844) data time 0.0010 (0.0035) model time 0.3786 (0.3800) loss 3.8317 (3.4335) grad_norm 1.4924 (1.2978) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][190/625] eta 0:02:47 lr 0.001828 wd 0.0500 time 0.3791 (0.3840) data time 0.0008 (0.0033) model time 0.3783 (0.3798) loss 3.5939 (3.4353) grad_norm 0.9776 (1.2989) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][200/625] eta 0:02:43 lr 0.001828 wd 0.0500 time 0.3814 (0.3837) data time 0.0008 (0.0032) model time 0.3806 (0.3796) loss 4.3332 (3.4345) grad_norm 0.8368 (1.3059) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][210/625] eta 0:02:39 lr 0.001828 wd 0.0500 time 0.3759 (0.3837) data time 0.0011 (0.0032) model time 0.3749 (0.3797) loss 3.0572 (3.4272) grad_norm 1.2702 (1.3165) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][220/625] eta 0:02:35 lr 0.001828 wd 0.0500 time 0.3799 (0.3835) data time 0.0008 (0.0031) model time 0.3791 (0.3796) loss 4.0140 (3.4460) grad_norm 1.3635 (1.3159) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][230/625] eta 0:02:31 lr 0.001828 wd 0.0500 time 0.5849 (0.3841) data time 0.0008 (0.0030) model time 0.5841 (0.3806) loss 4.2816 (3.4578) grad_norm 0.9997 (1.3104) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][240/625] eta 0:02:28 lr 0.001828 wd 0.0500 time 0.3722 (0.3847) data time 0.0008 (0.0029) model time 0.3715 (0.3814) loss 4.2332 (3.4486) grad_norm 0.9827 (1.3121) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][250/625] eta 0:02:24 lr 0.001828 wd 0.0500 time 0.3784 (0.3845) data time 0.0011 (0.0028) model time 0.3774 (0.3813) loss 3.0695 (3.4417) grad_norm 1.1732 (1.3065) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][260/625] eta 0:02:20 lr 0.001827 wd 0.0500 time 0.3896 (0.3844) data time 0.0010 (0.0028) model time 0.3886 (0.3812) loss 3.4649 (3.4563) grad_norm 1.3422 (1.3063) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][270/625] eta 0:02:16 lr 0.001827 wd 0.0500 time 0.3759 (0.3841) data time 0.0008 (0.0027) model time 0.3751 (0.3810) loss 2.5265 (3.4559) grad_norm 1.7551 (1.3065) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][280/625] eta 0:02:12 lr 0.001827 wd 0.0500 time 0.3763 (0.3839) data time 0.0008 (0.0026) model time 0.3755 (0.3808) loss 2.8468 (3.4573) grad_norm 1.1349 (1.3056) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][290/625] eta 0:02:08 lr 0.001827 wd 0.0500 time 0.3796 (0.3851) data time 0.0011 (0.0026) model time 0.3785 (0.3823) loss 4.0441 (3.4578) grad_norm 1.3302 (1.3044) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][300/625] eta 0:02:05 lr 0.001827 wd 0.0500 time 0.3814 (0.3848) data time 0.0008 (0.0025) model time 0.3806 (0.3821) loss 2.5835 (3.4495) grad_norm 0.8111 (1.2976) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][310/625] eta 0:02:01 lr 0.001827 wd 0.0500 time 0.3748 (0.3846) data time 0.0010 (0.0025) model time 0.3738 (0.3819) loss 2.7553 (3.4517) grad_norm 2.6709 (1.3004) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][320/625] eta 0:01:57 lr 0.001827 wd 0.0500 time 0.3708 (0.3844) data time 0.0009 (0.0025) model time 0.3699 (0.3817) loss 3.0865 (3.4575) grad_norm 1.2529 (1.3070) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][330/625] eta 0:01:53 lr 0.001827 wd 0.0500 time 0.3770 (0.3842) data time 0.0011 (0.0024) model time 0.3760 (0.3816) loss 3.4811 (3.4616) grad_norm 1.5303 (1.3082) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][340/625] eta 0:01:49 lr 0.001827 wd 0.0500 time 0.3773 (0.3840) data time 0.0011 (0.0024) model time 0.3761 (0.3814) loss 4.0499 (3.4647) grad_norm 0.9121 (1.3040) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][350/625] eta 0:01:45 lr 0.001827 wd 0.0500 time 0.3748 (0.3839) data time 0.0008 (0.0024) model time 0.3740 (0.3813) loss 3.1180 (3.4654) grad_norm 1.3787 (1.3023) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][360/625] eta 0:01:41 lr 0.001826 wd 0.0500 time 0.3782 (0.3844) data time 0.0009 (0.0023) model time 0.3773 (0.3818) loss 3.8698 (3.4649) grad_norm 1.1132 (1.3001) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][370/625] eta 0:01:37 lr 0.001826 wd 0.0500 time 0.3753 (0.3842) data time 0.0010 (0.0023) model time 0.3743 (0.3817) loss 3.2675 (3.4681) grad_norm 1.1211 (1.2981) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][380/625] eta 0:01:34 lr 0.001826 wd 0.0500 time 0.3724 (0.3839) data time 0.0012 (0.0023) model time 0.3712 (0.3815) loss 2.4033 (3.4620) grad_norm 1.2925 (1.3010) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][390/625] eta 0:01:30 lr 0.001826 wd 0.0500 time 0.3801 (0.3838) data time 0.0008 (0.0022) model time 0.3793 (0.3814) loss 4.4797 (3.4621) grad_norm 1.1518 (1.2984) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][400/625] eta 0:01:26 lr 0.001826 wd 0.0500 time 0.3740 (0.3837) data time 0.0011 (0.0022) model time 0.3729 (0.3812) loss 3.5169 (3.4633) grad_norm 0.9291 (1.3013) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][410/625] eta 0:01:22 lr 0.001826 wd 0.0500 time 0.3783 (0.3836) data time 0.0010 (0.0022) model time 0.3773 (0.3812) loss 3.5834 (3.4684) grad_norm 1.2451 (1.3008) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][420/625] eta 0:01:18 lr 0.001826 wd 0.0500 time 0.3716 (0.3848) data time 0.0008 (0.0021) model time 0.3707 (0.3825) loss 4.4383 (3.4732) grad_norm 1.1064 (1.2971) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][430/625] eta 0:01:15 lr 0.001826 wd 0.0500 time 0.3827 (0.3846) data time 0.0010 (0.0021) model time 0.3816 (0.3824) loss 3.7583 (3.4737) grad_norm 1.0844 (1.2921) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][440/625] eta 0:01:11 lr 0.001826 wd 0.0500 time 0.3749 (0.3845) data time 0.0008 (0.0021) model time 0.3741 (0.3822) loss 3.0290 (3.4717) grad_norm 1.3063 (1.2920) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][450/625] eta 0:01:07 lr 0.001826 wd 0.0500 time 0.3756 (0.3843) data time 0.0011 (0.0021) model time 0.3745 (0.3821) loss 3.7735 (3.4695) grad_norm 1.2802 (1.2884) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][460/625] eta 0:01:03 lr 0.001825 wd 0.0500 time 0.3731 (0.3842) data time 0.0008 (0.0021) model time 0.3723 (0.3820) loss 4.4700 (3.4700) grad_norm 1.0303 (1.2880) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][470/625] eta 0:00:59 lr 0.001825 wd 0.0500 time 0.3780 (0.3840) data time 0.0008 (0.0020) model time 0.3772 (0.3818) loss 4.4176 (3.4686) grad_norm 2.2089 (1.2917) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][480/625] eta 0:00:55 lr 0.001825 wd 0.0500 time 0.3834 (0.3839) data time 0.0011 (0.0021) model time 0.3823 (0.3817) loss 3.7143 (3.4738) grad_norm 1.0661 (1.2913) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][490/625] eta 0:00:51 lr 0.001825 wd 0.0500 time 0.3771 (0.3838) data time 0.0008 (0.0020) model time 0.3762 (0.3816) loss 3.7231 (3.4772) grad_norm 0.9236 (1.2889) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][500/625] eta 0:00:48 lr 0.001825 wd 0.0500 time 0.3715 (0.3845) data time 0.0009 (0.0020) model time 0.3707 (0.3824) loss 3.4467 (3.4733) grad_norm 1.0256 (1.2888) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][510/625] eta 0:00:44 lr 0.001825 wd 0.0500 time 0.3743 (0.3844) data time 0.0009 (0.0020) model time 0.3734 (0.3823) loss 3.9978 (3.4733) grad_norm 1.9852 (1.2946) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][520/625] eta 0:00:40 lr 0.001825 wd 0.0500 time 0.3760 (0.3844) data time 0.0011 (0.0020) model time 0.3749 (0.3823) loss 4.0657 (3.4682) grad_norm 0.8740 (1.2930) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][530/625] eta 0:00:36 lr 0.001825 wd 0.0500 time 0.3697 (0.3842) data time 0.0011 (0.0020) model time 0.3686 (0.3821) loss 3.5909 (3.4710) grad_norm 1.0456 (1.2886) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][540/625] eta 0:00:32 lr 0.001825 wd 0.0500 time 0.3735 (0.3848) data time 0.0011 (0.0020) model time 0.3724 (0.3828) loss 2.7327 (3.4681) grad_norm 0.8781 (1.2880) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][550/625] eta 0:00:28 lr 0.001825 wd 0.0500 time 0.3741 (0.3847) data time 0.0008 (0.0019) model time 0.3733 (0.3827) loss 2.2273 (3.4698) grad_norm 0.8183 (1.2876) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][560/625] eta 0:00:25 lr 0.001824 wd 0.0500 time 0.3806 (0.3846) data time 0.0008 (0.0019) model time 0.3797 (0.3827) loss 3.0831 (3.4687) grad_norm 1.3509 (1.2874) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][570/625] eta 0:00:21 lr 0.001824 wd 0.0500 time 0.3769 (0.3849) data time 0.0008 (0.0019) model time 0.3761 (0.3829) loss 4.1021 (3.4682) grad_norm 1.4392 (1.2890) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][580/625] eta 0:00:17 lr 0.001824 wd 0.0500 time 0.3773 (0.3848) data time 0.0011 (0.0019) model time 0.3763 (0.3829) loss 3.8923 (3.4753) grad_norm 1.2893 (1.2881) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][590/625] eta 0:00:13 lr 0.001824 wd 0.0500 time 0.3749 (0.3847) data time 0.0008 (0.0019) model time 0.3741 (0.3828) loss 3.3033 (3.4752) grad_norm 1.1900 (1.2859) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][600/625] eta 0:00:09 lr 0.001824 wd 0.0500 time 0.3705 (0.3846) data time 0.0011 (0.0019) model time 0.3695 (0.3827) loss 2.9357 (3.4730) grad_norm 1.0716 (1.2849) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][610/625] eta 0:00:05 lr 0.001824 wd 0.0500 time 0.3705 (0.3844) data time 0.0008 (0.0019) model time 0.3697 (0.3825) loss 3.6961 (3.4711) grad_norm 1.0784 (1.2857) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [73/300][620/625] eta 0:00:01 lr 0.001824 wd 0.0500 time 0.3671 (0.3842) data time 0.0006 (0.0019) model time 0.3665 (0.3823) loss 3.8513 (3.4677) grad_norm 0.9799 (1.2844) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:36 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 73 training takes 0:04:00 [2024-08-26 07:57:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 07:57:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 07:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.526 (0.526) Loss 0.7065 (0.7065) Acc@1 84.863 (84.863) Acc@5 97.021 (97.021) Mem 11225MB [2024-08-26 07:57:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 1.1641 (0.8405) Acc@1 72.461 (80.740) Acc@5 92.822 (95.992) Mem 11225MB [2024-08-26 07:57:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.2812 (1.0216) Acc@1 69.873 (76.772) Acc@5 90.723 (93.745) Mem 11225MB [2024-08-26 07:57:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.556 Acc@5 93.754 [2024-08-26 07:57:40 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.6% [2024-08-26 07:57:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.811 (0.811) Loss 0.5669 (0.5669) Acc@1 87.549 (87.549) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 07:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.173) Loss 0.9595 (0.7105) Acc@1 76.611 (83.540) Acc@5 94.482 (96.884) Mem 11225MB [2024-08-26 07:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.135) Loss 1.1279 (0.8731) Acc@1 71.777 (79.513) Acc@5 92.285 (94.996) Mem 11225MB [2024-08-26 07:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.207 Acc@5 94.942 [2024-08-26 07:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.2% [2024-08-26 07:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.21% [2024-08-26 07:57:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 07:57:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 07:57:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][0/625] eta 0:08:09 lr 0.001824 wd 0.0500 time 0.7828 (0.7828) data time 0.4460 (0.4460) model time 0.0000 (0.0000) loss 3.0975 (3.0975) grad_norm 1.1783 (1.1783) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][10/625] eta 0:04:15 lr 0.001824 wd 0.0500 time 0.3867 (0.4156) data time 0.0011 (0.0416) model time 0.0000 (0.0000) loss 3.5393 (3.2165) grad_norm 4.0219 (1.7142) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][20/625] eta 0:04:00 lr 0.001824 wd 0.0500 time 0.3754 (0.3974) data time 0.0011 (0.0223) model time 0.0000 (0.0000) loss 3.3251 (3.3942) grad_norm 1.0327 (1.5148) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:57:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][30/625] eta 0:03:52 lr 0.001823 wd 0.0500 time 0.3804 (0.3909) data time 0.0012 (0.0155) model time 0.0000 (0.0000) loss 2.9998 (3.3985) grad_norm 1.3846 (1.3851) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][40/625] eta 0:03:46 lr 0.001823 wd 0.0500 time 0.3816 (0.3875) data time 0.0011 (0.0120) model time 0.0000 (0.0000) loss 4.1204 (3.4657) grad_norm 1.0907 (1.3510) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][50/625] eta 0:03:41 lr 0.001823 wd 0.0500 time 0.3849 (0.3857) data time 0.0009 (0.0099) model time 0.0000 (0.0000) loss 4.3271 (3.4337) grad_norm 1.1812 (1.3613) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][60/625] eta 0:03:39 lr 0.001823 wd 0.0500 time 0.3772 (0.3878) data time 0.0011 (0.0085) model time 0.3761 (0.3974) loss 3.6154 (3.4419) grad_norm 1.3185 (1.3772) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][70/625] eta 0:03:39 lr 0.001823 wd 0.0500 time 0.3871 (0.3954) data time 0.0011 (0.0075) model time 0.3860 (0.4187) loss 2.8333 (3.4560) grad_norm 0.7152 (1.3560) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][80/625] eta 0:03:35 lr 0.001823 wd 0.0500 time 0.3747 (0.3955) data time 0.0011 (0.0067) model time 0.3736 (0.4108) loss 3.3592 (3.4623) grad_norm 1.1391 (1.3315) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][90/625] eta 0:03:30 lr 0.001823 wd 0.0500 time 0.3756 (0.3935) data time 0.0008 (0.0061) model time 0.3748 (0.4021) loss 3.7214 (3.4681) grad_norm 1.5735 (1.3313) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][100/625] eta 0:03:25 lr 0.001823 wd 0.0500 time 0.3737 (0.3919) data time 0.0011 (0.0059) model time 0.3726 (0.3964) loss 3.7564 (3.4720) grad_norm 1.4820 (1.3426) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][110/625] eta 0:03:21 lr 0.001823 wd 0.0500 time 0.3746 (0.3905) data time 0.0008 (0.0055) model time 0.3738 (0.3929) loss 4.4783 (3.4695) grad_norm 1.2611 (1.3387) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][120/625] eta 0:03:16 lr 0.001823 wd 0.0500 time 0.3735 (0.3894) data time 0.0011 (0.0051) model time 0.3724 (0.3904) loss 3.6507 (3.4619) grad_norm 1.3941 (1.3400) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][130/625] eta 0:03:12 lr 0.001822 wd 0.0500 time 0.3810 (0.3885) data time 0.0010 (0.0048) model time 0.3800 (0.3888) loss 3.8037 (3.4790) grad_norm 1.2873 (1.3437) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][140/625] eta 0:03:08 lr 0.001822 wd 0.0500 time 0.3864 (0.3879) data time 0.0008 (0.0045) model time 0.3856 (0.3876) loss 2.4031 (3.4714) grad_norm 1.3899 (1.3380) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][150/625] eta 0:03:03 lr 0.001822 wd 0.0500 time 0.3795 (0.3872) data time 0.0012 (0.0043) model time 0.3783 (0.3865) loss 3.4219 (3.4772) grad_norm 0.9621 (1.3210) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][160/625] eta 0:02:59 lr 0.001822 wd 0.0500 time 0.3734 (0.3867) data time 0.0009 (0.0041) model time 0.3725 (0.3858) loss 3.7555 (3.4855) grad_norm 1.0751 (1.3181) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][170/625] eta 0:02:55 lr 0.001822 wd 0.0500 time 0.3763 (0.3862) data time 0.0012 (0.0039) model time 0.3751 (0.3850) loss 3.5805 (3.4881) grad_norm 1.8408 (1.3265) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][180/625] eta 0:02:51 lr 0.001822 wd 0.0500 time 0.3895 (0.3858) data time 0.0008 (0.0038) model time 0.3887 (0.3844) loss 4.5514 (3.4984) grad_norm 1.1973 (1.3202) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:58:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][190/625] eta 0:02:47 lr 0.001822 wd 0.0500 time 0.3874 (0.3854) data time 0.0010 (0.0037) model time 0.3864 (0.3840) loss 3.4087 (3.5089) grad_norm 1.0022 (1.3143) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][200/625] eta 0:02:44 lr 0.001822 wd 0.0500 time 0.3684 (0.3860) data time 0.0012 (0.0035) model time 0.3672 (0.3848) loss 3.9589 (3.5206) grad_norm 1.9504 (1.3140) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][210/625] eta 0:02:40 lr 0.001822 wd 0.0500 time 0.3790 (0.3857) data time 0.0010 (0.0034) model time 0.3780 (0.3843) loss 2.9406 (3.5067) grad_norm 1.2657 (1.3112) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][220/625] eta 0:02:36 lr 0.001822 wd 0.0500 time 0.3798 (0.3853) data time 0.0011 (0.0033) model time 0.3788 (0.3838) loss 3.2891 (3.4972) grad_norm 1.4822 (1.3181) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][230/625] eta 0:02:32 lr 0.001821 wd 0.0500 time 0.3781 (0.3850) data time 0.0009 (0.0032) model time 0.3772 (0.3835) loss 3.1851 (3.4897) grad_norm 1.6109 (1.3208) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][240/625] eta 0:02:28 lr 0.001821 wd 0.0500 time 0.3833 (0.3848) data time 0.0008 (0.0031) model time 0.3825 (0.3832) loss 3.4187 (3.4868) grad_norm 1.2467 (1.3094) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][250/625] eta 0:02:25 lr 0.001821 wd 0.0500 time 0.3806 (0.3871) data time 0.0009 (0.0031) model time 0.3798 (0.3862) loss 3.8419 (3.4794) grad_norm 1.1756 (1.3144) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][260/625] eta 0:02:21 lr 0.001821 wd 0.0500 time 0.3778 (0.3868) data time 0.0011 (0.0030) model time 0.3768 (0.3858) loss 3.1579 (3.4759) grad_norm 1.1214 (1.3103) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][270/625] eta 0:02:17 lr 0.001821 wd 0.0500 time 0.3793 (0.3865) data time 0.0011 (0.0029) model time 0.3782 (0.3854) loss 3.3578 (3.4833) grad_norm 0.9828 (1.3050) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][280/625] eta 0:02:14 lr 0.001821 wd 0.0500 time 0.3850 (0.3884) data time 0.0009 (0.0029) model time 0.3841 (0.3877) loss 4.0508 (3.4850) grad_norm 1.2064 (1.3019) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][290/625] eta 0:02:10 lr 0.001821 wd 0.0500 time 0.3738 (0.3881) data time 0.0008 (0.0028) model time 0.3730 (0.3873) loss 3.1765 (3.4725) grad_norm 1.3387 (1.3027) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][300/625] eta 0:02:06 lr 0.001821 wd 0.0500 time 0.3811 (0.3878) data time 0.0008 (0.0027) model time 0.3803 (0.3870) loss 2.7586 (3.4749) grad_norm 1.3017 (1.3060) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][310/625] eta 0:02:02 lr 0.001821 wd 0.0500 time 0.3754 (0.3875) data time 0.0011 (0.0027) model time 0.3743 (0.3866) loss 2.8966 (3.4837) grad_norm 1.2918 (1.3097) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][320/625] eta 0:01:58 lr 0.001821 wd 0.0500 time 0.3751 (0.3872) data time 0.0012 (0.0026) model time 0.3739 (0.3862) loss 3.6092 (3.4837) grad_norm 1.6438 (1.3172) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][330/625] eta 0:01:54 lr 0.001820 wd 0.0500 time 0.3797 (0.3869) data time 0.0008 (0.0026) model time 0.3789 (0.3859) loss 4.1969 (3.4913) grad_norm 1.0110 (1.3180) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 07:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][340/625] eta 0:01:50 lr 0.001820 wd 0.0500 time 0.3765 (0.3867) data time 0.0008 (0.0026) model time 0.3757 (0.3856) loss 3.3792 (3.4843) grad_norm 0.9705 (1.3163) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][350/625] eta 0:01:46 lr 0.001820 wd 0.0500 time 0.3763 (0.3865) data time 0.0009 (0.0025) model time 0.3754 (0.3854) loss 4.2816 (3.4852) grad_norm 1.2075 (1.3146) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][360/625] eta 0:01:42 lr 0.001820 wd 0.0500 time 0.3838 (0.3862) data time 0.0009 (0.0025) model time 0.3828 (0.3851) loss 3.7122 (3.4836) grad_norm 1.1424 (1.3133) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:00:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][370/625] eta 0:01:38 lr 0.001820 wd 0.0500 time 0.3834 (0.3877) data time 0.0011 (0.0024) model time 0.3823 (0.3868) loss 3.2937 (3.4878) grad_norm 2.4904 (1.3156) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:00:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][380/625] eta 0:01:34 lr 0.001820 wd 0.0500 time 0.3823 (0.3875) data time 0.0010 (0.0024) model time 0.3813 (0.3865) loss 3.9520 (3.4929) grad_norm 1.3790 (1.3191) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:00:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][390/625] eta 0:01:31 lr 0.001820 wd 0.0500 time 0.3723 (0.3873) data time 0.0009 (0.0024) model time 0.3715 (0.3863) loss 2.7799 (3.4826) grad_norm 1.5945 (1.3174) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:00:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][400/625] eta 0:01:27 lr 0.001820 wd 0.0500 time 0.3869 (0.3871) data time 0.0008 (0.0024) model time 0.3861 (0.3861) loss 3.4268 (3.4794) grad_norm 0.9625 (1.3168) loss_scale 4096.0000 (2093.9651) mem 11225MB [2024-08-26 08:00:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][410/625] eta 0:01:23 lr 0.001820 wd 0.0500 time 0.3753 (0.3874) data time 0.0008 (0.0023) model time 0.3744 (0.3864) loss 3.4359 (3.4706) grad_norm 1.3747 (1.3136) loss_scale 4096.0000 (2142.6764) mem 11225MB [2024-08-26 08:00:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][420/625] eta 0:01:19 lr 0.001820 wd 0.0500 time 0.3834 (0.3872) data time 0.0008 (0.0023) model time 0.3826 (0.3862) loss 4.2702 (3.4714) grad_norm 1.4793 (1.3152) loss_scale 4096.0000 (2189.0736) mem 11225MB [2024-08-26 08:00:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][430/625] eta 0:01:15 lr 0.001819 wd 0.0500 time 0.3853 (0.3871) data time 0.0011 (0.0023) model time 0.3842 (0.3860) loss 3.4051 (3.4705) grad_norm 1.1156 (1.3157) loss_scale 4096.0000 (2233.3179) mem 11225MB [2024-08-26 08:00:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][440/625] eta 0:01:11 lr 0.001819 wd 0.0500 time 0.3825 (0.3869) data time 0.0011 (0.0023) model time 0.3815 (0.3858) loss 3.7863 (3.4711) grad_norm 1.4473 (1.3146) loss_scale 4096.0000 (2275.5556) mem 11225MB [2024-08-26 08:00:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][450/625] eta 0:01:07 lr 0.001819 wd 0.0500 time 0.3849 (0.3867) data time 0.0010 (0.0022) model time 0.3838 (0.3855) loss 3.7971 (3.4705) grad_norm 1.0183 (1.3145) loss_scale 4096.0000 (2315.9202) mem 11225MB [2024-08-26 08:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][460/625] eta 0:01:03 lr 0.001819 wd 0.0500 time 0.3718 (0.3873) data time 0.0008 (0.0022) model time 0.3710 (0.3862) loss 4.0070 (3.4749) grad_norm 1.7270 (1.3217) loss_scale 4096.0000 (2354.5336) mem 11225MB [2024-08-26 08:00:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][470/625] eta 0:01:00 lr 0.001819 wd 0.0500 time 0.3724 (0.3874) data time 0.0010 (0.0022) model time 0.3714 (0.3864) loss 3.6923 (3.4748) grad_norm 1.8146 (1.3223) loss_scale 4096.0000 (2391.5074) mem 11225MB [2024-08-26 08:00:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][480/625] eta 0:00:56 lr 0.001819 wd 0.0500 time 0.3757 (0.3873) data time 0.0009 (0.0022) model time 0.3748 (0.3862) loss 4.0159 (3.4799) grad_norm 1.6050 (1.3198) loss_scale 4096.0000 (2426.9439) mem 11225MB [2024-08-26 08:00:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][490/625] eta 0:00:52 lr 0.001819 wd 0.0500 time 0.3813 (0.3871) data time 0.0011 (0.0021) model time 0.3803 (0.3860) loss 3.7812 (3.4782) grad_norm 1.1811 (1.3204) loss_scale 4096.0000 (2460.9369) mem 11225MB [2024-08-26 08:00:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][500/625] eta 0:00:48 lr 0.001819 wd 0.0500 time 0.3767 (0.3869) data time 0.0008 (0.0021) model time 0.3758 (0.3858) loss 4.0518 (3.4797) grad_norm 1.0099 (1.3184) loss_scale 4096.0000 (2493.5729) mem 11225MB [2024-08-26 08:01:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][510/625] eta 0:00:44 lr 0.001819 wd 0.0500 time 0.3719 (0.3867) data time 0.0011 (0.0021) model time 0.3707 (0.3856) loss 3.2789 (3.4816) grad_norm 1.7935 (1.3231) loss_scale 4096.0000 (2524.9315) mem 11225MB [2024-08-26 08:01:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][520/625] eta 0:00:40 lr 0.001818 wd 0.0500 time 0.3763 (0.3865) data time 0.0008 (0.0021) model time 0.3755 (0.3854) loss 4.3320 (3.4813) grad_norm 1.0998 (1.3207) loss_scale 4096.0000 (2555.0864) mem 11225MB [2024-08-26 08:01:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][530/625] eta 0:00:36 lr 0.001818 wd 0.0500 time 0.3841 (0.3864) data time 0.0008 (0.0021) model time 0.3833 (0.3852) loss 3.0313 (3.4829) grad_norm 1.5009 (1.3188) loss_scale 4096.0000 (2584.1055) mem 11225MB [2024-08-26 08:01:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][540/625] eta 0:00:32 lr 0.001818 wd 0.0500 time 0.3683 (0.3862) data time 0.0011 (0.0021) model time 0.3671 (0.3851) loss 3.9932 (3.4815) grad_norm 1.3498 (1.3169) loss_scale 4096.0000 (2612.0518) mem 11225MB [2024-08-26 08:01:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][550/625] eta 0:00:28 lr 0.001818 wd 0.0500 time 0.3720 (0.3861) data time 0.0012 (0.0020) model time 0.3709 (0.3849) loss 3.5191 (3.4829) grad_norm 1.1937 (1.3196) loss_scale 4096.0000 (2638.9837) mem 11225MB [2024-08-26 08:01:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][560/625] eta 0:00:25 lr 0.001818 wd 0.0500 time 0.3820 (0.3860) data time 0.0008 (0.0020) model time 0.3812 (0.3848) loss 3.9425 (3.4845) grad_norm 1.0583 (1.3162) loss_scale 4096.0000 (2664.9554) mem 11225MB [2024-08-26 08:01:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][570/625] eta 0:00:21 lr 0.001818 wd 0.0500 time 0.5822 (0.3862) data time 0.0008 (0.0020) model time 0.5814 (0.3850) loss 3.3146 (3.4819) grad_norm 1.1282 (1.3157) loss_scale 4096.0000 (2690.0175) mem 11225MB [2024-08-26 08:01:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][580/625] eta 0:00:17 lr 0.001818 wd 0.0500 time 0.3811 (0.3872) data time 0.0009 (0.0020) model time 0.3802 (0.3861) loss 3.7580 (3.4815) grad_norm 1.1845 (1.3162) loss_scale 4096.0000 (2714.2169) mem 11225MB [2024-08-26 08:01:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][590/625] eta 0:00:13 lr 0.001818 wd 0.0500 time 0.3834 (0.3877) data time 0.0008 (0.0020) model time 0.3826 (0.3867) loss 3.9354 (3.4848) grad_norm 1.1424 (1.3133) loss_scale 4096.0000 (2737.5973) mem 11225MB [2024-08-26 08:01:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][600/625] eta 0:00:09 lr 0.001818 wd 0.0500 time 0.3740 (0.3876) data time 0.0011 (0.0020) model time 0.3729 (0.3866) loss 2.9380 (3.4835) grad_norm 1.4288 (1.3123) loss_scale 4096.0000 (2760.1997) mem 11225MB [2024-08-26 08:01:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][610/625] eta 0:00:05 lr 0.001818 wd 0.0500 time 0.3662 (0.3876) data time 0.0006 (0.0020) model time 0.3656 (0.3865) loss 3.3042 (3.4839) grad_norm 0.8846 (1.3107) loss_scale 4096.0000 (2782.0622) mem 11225MB [2024-08-26 08:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [74/300][620/625] eta 0:00:01 lr 0.001817 wd 0.0500 time 0.3650 (0.3872) data time 0.0005 (0.0020) model time 0.3645 (0.3861) loss 2.2183 (3.4851) grad_norm 1.2420 (1.3080) loss_scale 4096.0000 (2803.2206) mem 11225MB [2024-08-26 08:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 74 training takes 0:04:01 [2024-08-26 08:01:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:01:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:01:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.501 (0.501) Loss 0.7417 (0.7417) Acc@1 84.668 (84.668) Acc@5 97.070 (97.070) Mem 11225MB [2024-08-26 08:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.1621 (0.8672) Acc@1 72.803 (80.944) Acc@5 93.018 (96.085) Mem 11225MB [2024-08-26 08:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.3027 (1.0523) Acc@1 69.775 (76.730) Acc@5 90.771 (93.752) Mem 11225MB [2024-08-26 08:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.486 Acc@5 93.708 [2024-08-26 08:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.5% [2024-08-26 08:01:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.824 (0.824) Loss 0.5669 (0.5669) Acc@1 87.793 (87.793) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 08:01:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.176) Loss 0.9565 (0.7095) Acc@1 76.855 (83.674) Acc@5 94.727 (96.924) Mem 11225MB [2024-08-26 08:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.1240 (0.8713) Acc@1 71.826 (79.622) Acc@5 92.285 (95.040) Mem 11225MB [2024-08-26 08:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.315 Acc@5 94.984 [2024-08-26 08:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.3% [2024-08-26 08:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.31% [2024-08-26 08:01:53 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:01:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:01:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][0/625] eta 0:08:17 lr 0.001817 wd 0.0500 time 0.7952 (0.7952) data time 0.4606 (0.4606) model time 0.0000 (0.0000) loss 2.7236 (2.7236) grad_norm 1.3544 (1.3544) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:01:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][10/625] eta 0:04:16 lr 0.001817 wd 0.0500 time 0.3786 (0.4168) data time 0.0008 (0.0441) model time 0.0000 (0.0000) loss 4.4128 (3.3902) grad_norm 1.2956 (1.3703) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][20/625] eta 0:04:01 lr 0.001817 wd 0.0500 time 0.3704 (0.3997) data time 0.0009 (0.0236) model time 0.0000 (0.0000) loss 2.5039 (3.4359) grad_norm 1.3837 (1.4911) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][30/625] eta 0:03:53 lr 0.001817 wd 0.0500 time 0.3756 (0.3932) data time 0.0011 (0.0164) model time 0.0000 (0.0000) loss 3.6378 (3.4270) grad_norm 0.8577 (1.3744) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][40/625] eta 0:03:48 lr 0.001817 wd 0.0500 time 0.4084 (0.3912) data time 0.0008 (0.0126) model time 0.0000 (0.0000) loss 2.7537 (3.4196) grad_norm 1.3506 (1.3174) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][50/625] eta 0:03:45 lr 0.001817 wd 0.0500 time 0.3858 (0.3928) data time 0.0008 (0.0104) model time 0.0000 (0.0000) loss 4.1333 (3.3999) grad_norm 1.0014 (1.2906) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][60/625] eta 0:03:40 lr 0.001817 wd 0.0500 time 0.3807 (0.3905) data time 0.0008 (0.0089) model time 0.3799 (0.3767) loss 2.3470 (3.3882) grad_norm 0.9980 (1.2883) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][70/625] eta 0:03:35 lr 0.001817 wd 0.0500 time 0.3805 (0.3886) data time 0.0008 (0.0078) model time 0.3796 (0.3766) loss 3.0943 (3.3661) grad_norm 1.0651 (1.2925) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][80/625] eta 0:03:31 lr 0.001817 wd 0.0500 time 0.3744 (0.3872) data time 0.0008 (0.0070) model time 0.3736 (0.3762) loss 4.3613 (3.3886) grad_norm 1.1879 (1.2997) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][90/625] eta 0:03:26 lr 0.001816 wd 0.0500 time 0.3734 (0.3859) data time 0.0009 (0.0064) model time 0.3726 (0.3757) loss 4.4137 (3.3890) grad_norm 2.7123 (1.3176) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][100/625] eta 0:03:24 lr 0.001816 wd 0.0500 time 0.3824 (0.3887) data time 0.0011 (0.0059) model time 0.3813 (0.3832) loss 3.4840 (3.3804) grad_norm 0.9014 (1.3086) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][110/625] eta 0:03:19 lr 0.001816 wd 0.0500 time 0.3786 (0.3876) data time 0.0007 (0.0055) model time 0.3779 (0.3819) loss 4.4640 (3.3833) grad_norm 1.1763 (1.3068) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][120/625] eta 0:03:15 lr 0.001816 wd 0.0500 time 0.3693 (0.3869) data time 0.0012 (0.0051) model time 0.3681 (0.3812) loss 3.7452 (3.3923) grad_norm 1.2632 (1.3080) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][130/625] eta 0:03:11 lr 0.001816 wd 0.0500 time 0.3760 (0.3862) data time 0.0011 (0.0048) model time 0.3749 (0.3808) loss 3.5203 (3.4069) grad_norm 2.1966 (1.3301) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][140/625] eta 0:03:07 lr 0.001816 wd 0.0500 time 0.3805 (0.3858) data time 0.0011 (0.0045) model time 0.3794 (0.3805) loss 3.4018 (3.4128) grad_norm 1.1998 (1.3326) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][150/625] eta 0:03:04 lr 0.001816 wd 0.0500 time 0.3821 (0.3877) data time 0.0008 (0.0043) model time 0.3813 (0.3839) loss 3.4661 (3.4223) grad_norm 1.5911 (1.3292) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:02:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][160/625] eta 0:03:00 lr 0.001816 wd 0.0500 time 0.3901 (0.3884) data time 0.0009 (0.0041) model time 0.3893 (0.3852) loss 3.7306 (3.4310) grad_norm 1.3826 (1.3453) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][170/625] eta 0:02:56 lr 0.001816 wd 0.0500 time 0.3769 (0.3878) data time 0.0011 (0.0040) model time 0.3758 (0.3844) loss 2.3549 (3.4237) grad_norm 1.5483 (1.3462) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][180/625] eta 0:02:52 lr 0.001816 wd 0.0500 time 0.3741 (0.3872) data time 0.0011 (0.0038) model time 0.3730 (0.3838) loss 3.2567 (3.4295) grad_norm 1.3355 (1.3409) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][190/625] eta 0:02:48 lr 0.001815 wd 0.0500 time 0.3790 (0.3868) data time 0.0008 (0.0037) model time 0.3782 (0.3834) loss 2.6307 (3.4257) grad_norm 1.0044 (1.3325) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][200/625] eta 0:02:44 lr 0.001815 wd 0.0500 time 0.3806 (0.3863) data time 0.0012 (0.0036) model time 0.3794 (0.3829) loss 3.0428 (3.4229) grad_norm 1.1280 (1.3287) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][210/625] eta 0:02:40 lr 0.001815 wd 0.0500 time 0.3748 (0.3859) data time 0.0008 (0.0034) model time 0.3740 (0.3825) loss 4.1298 (3.4426) grad_norm 2.2230 (1.3244) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][220/625] eta 0:02:36 lr 0.001815 wd 0.0500 time 0.3758 (0.3855) data time 0.0008 (0.0033) model time 0.3750 (0.3821) loss 4.7057 (3.4656) grad_norm 1.1605 (1.3282) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][230/625] eta 0:02:32 lr 0.001815 wd 0.0500 time 0.3631 (0.3851) data time 0.0009 (0.0032) model time 0.3622 (0.3817) loss 3.3789 (3.4655) grad_norm 1.0707 (1.3226) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][240/625] eta 0:02:28 lr 0.001815 wd 0.0500 time 0.3770 (0.3848) data time 0.0008 (0.0032) model time 0.3762 (0.3814) loss 4.5845 (3.4704) grad_norm 1.8507 (1.3167) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][250/625] eta 0:02:24 lr 0.001815 wd 0.0500 time 0.3781 (0.3846) data time 0.0011 (0.0031) model time 0.3770 (0.3813) loss 2.8923 (3.4658) grad_norm 1.4621 (1.3244) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][260/625] eta 0:02:20 lr 0.001815 wd 0.0500 time 0.3811 (0.3849) data time 0.0008 (0.0030) model time 0.3802 (0.3818) loss 4.2845 (3.4728) grad_norm 1.0659 (1.3233) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][270/625] eta 0:02:16 lr 0.001815 wd 0.0500 time 0.3770 (0.3847) data time 0.0011 (0.0030) model time 0.3760 (0.3815) loss 3.9344 (3.4782) grad_norm 2.0899 (1.3231) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][280/625] eta 0:02:12 lr 0.001815 wd 0.0500 time 0.3709 (0.3844) data time 0.0009 (0.0029) model time 0.3700 (0.3813) loss 2.9693 (3.4800) grad_norm 1.1612 (1.3267) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][290/625] eta 0:02:08 lr 0.001814 wd 0.0500 time 0.3794 (0.3843) data time 0.0009 (0.0028) model time 0.3785 (0.3812) loss 2.8487 (3.4855) grad_norm 1.1539 (1.3218) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][300/625] eta 0:02:04 lr 0.001814 wd 0.0500 time 0.3794 (0.3840) data time 0.0008 (0.0028) model time 0.3786 (0.3810) loss 3.3479 (3.4847) grad_norm 1.5519 (1.3251) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][310/625] eta 0:02:01 lr 0.001814 wd 0.0500 time 0.3774 (0.3859) data time 0.0009 (0.0027) model time 0.3765 (0.3833) loss 3.7886 (3.4834) grad_norm 1.4716 (1.3309) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:03:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][320/625] eta 0:01:57 lr 0.001814 wd 0.0500 time 0.3749 (0.3856) data time 0.0011 (0.0027) model time 0.3738 (0.3830) loss 3.6988 (3.4662) grad_norm 1.2423 (1.3306) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][330/625] eta 0:01:53 lr 0.001814 wd 0.0500 time 0.3779 (0.3854) data time 0.0009 (0.0026) model time 0.3771 (0.3828) loss 4.0814 (3.4654) grad_norm 1.3345 (1.3363) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][340/625] eta 0:01:50 lr 0.001814 wd 0.0500 time 0.4021 (0.3870) data time 0.0011 (0.0026) model time 0.4010 (0.3847) loss 3.3346 (3.4642) grad_norm 0.9561 (1.3311) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][350/625] eta 0:01:46 lr 0.001814 wd 0.0500 time 0.3822 (0.3868) data time 0.0010 (0.0026) model time 0.3811 (0.3845) loss 3.9461 (3.4632) grad_norm 0.8567 (1.3252) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][360/625] eta 0:01:42 lr 0.001814 wd 0.0500 time 0.3827 (0.3865) data time 0.0011 (0.0025) model time 0.3816 (0.3843) loss 3.5172 (3.4528) grad_norm 1.1098 (1.3252) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][370/625] eta 0:01:38 lr 0.001814 wd 0.0500 time 0.3850 (0.3864) data time 0.0009 (0.0025) model time 0.3842 (0.3841) loss 3.5724 (3.4574) grad_norm 1.4548 (1.3232) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][380/625] eta 0:01:34 lr 0.001813 wd 0.0500 time 0.3786 (0.3862) data time 0.0008 (0.0024) model time 0.3778 (0.3839) loss 3.0402 (3.4514) grad_norm 1.2371 (1.3224) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][390/625] eta 0:01:30 lr 0.001813 wd 0.0500 time 0.3766 (0.3860) data time 0.0009 (0.0024) model time 0.3758 (0.3838) loss 3.6278 (3.4492) grad_norm 1.0474 (1.3247) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][400/625] eta 0:01:26 lr 0.001813 wd 0.0500 time 0.3754 (0.3858) data time 0.0011 (0.0024) model time 0.3743 (0.3835) loss 3.6975 (3.4487) grad_norm 1.4674 (1.3220) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][410/625] eta 0:01:22 lr 0.001813 wd 0.0500 time 0.3777 (0.3856) data time 0.0012 (0.0023) model time 0.3765 (0.3833) loss 3.9805 (3.4469) grad_norm 1.6179 (1.3242) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][420/625] eta 0:01:18 lr 0.001813 wd 0.0500 time 0.3756 (0.3854) data time 0.0008 (0.0023) model time 0.3748 (0.3831) loss 3.0314 (3.4456) grad_norm 1.3015 (1.3219) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][430/625] eta 0:01:15 lr 0.001813 wd 0.0500 time 0.3781 (0.3852) data time 0.0008 (0.0023) model time 0.3773 (0.3829) loss 4.2862 (3.4536) grad_norm 1.6228 (1.3226) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][440/625] eta 0:01:11 lr 0.001813 wd 0.0500 time 0.3613 (0.3854) data time 0.0008 (0.0023) model time 0.3605 (0.3832) loss 2.9959 (3.4485) grad_norm 1.2097 (1.3201) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][450/625] eta 0:01:07 lr 0.001813 wd 0.0500 time 0.3771 (0.3852) data time 0.0008 (0.0022) model time 0.3763 (0.3830) loss 3.6271 (3.4536) grad_norm 1.4116 (1.3193) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][460/625] eta 0:01:03 lr 0.001813 wd 0.0500 time 0.3753 (0.3864) data time 0.0012 (0.0022) model time 0.3741 (0.3843) loss 3.8713 (3.4555) grad_norm 1.4050 (1.3195) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][470/625] eta 0:00:59 lr 0.001813 wd 0.0500 time 0.3765 (0.3862) data time 0.0011 (0.0022) model time 0.3754 (0.3842) loss 3.0447 (3.4538) grad_norm 1.0317 (1.3205) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][480/625] eta 0:00:55 lr 0.001812 wd 0.0500 time 0.3853 (0.3860) data time 0.0010 (0.0022) model time 0.3843 (0.3840) loss 3.5214 (3.4571) grad_norm 1.9913 (1.3181) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][490/625] eta 0:00:52 lr 0.001812 wd 0.0500 time 0.3746 (0.3871) data time 0.0009 (0.0021) model time 0.3737 (0.3852) loss 3.7062 (3.4541) grad_norm 1.1872 (1.3165) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][500/625] eta 0:00:48 lr 0.001812 wd 0.0500 time 0.3754 (0.3869) data time 0.0014 (0.0021) model time 0.3740 (0.3850) loss 3.6016 (3.4565) grad_norm 1.1246 (1.3149) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][510/625] eta 0:00:44 lr 0.001812 wd 0.0500 time 0.3840 (0.3868) data time 0.0008 (0.0021) model time 0.3832 (0.3849) loss 4.0190 (3.4529) grad_norm 0.8761 (1.3173) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][520/625] eta 0:00:40 lr 0.001812 wd 0.0500 time 0.3749 (0.3866) data time 0.0009 (0.0021) model time 0.3740 (0.3848) loss 2.3674 (3.4461) grad_norm 1.2332 (1.3193) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][530/625] eta 0:00:36 lr 0.001812 wd 0.0500 time 0.3817 (0.3865) data time 0.0011 (0.0021) model time 0.3807 (0.3846) loss 3.9693 (3.4487) grad_norm 0.9538 (1.3191) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][540/625] eta 0:00:32 lr 0.001812 wd 0.0500 time 0.3756 (0.3863) data time 0.0012 (0.0021) model time 0.3745 (0.3844) loss 3.6705 (3.4489) grad_norm 1.4329 (1.3163) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][550/625] eta 0:00:28 lr 0.001812 wd 0.0500 time 0.3722 (0.3861) data time 0.0011 (0.0020) model time 0.3711 (0.3842) loss 3.1060 (3.4488) grad_norm 2.5962 (1.3169) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][560/625] eta 0:00:25 lr 0.001812 wd 0.0500 time 0.3714 (0.3863) data time 0.0009 (0.0020) model time 0.3705 (0.3845) loss 4.0668 (3.4451) grad_norm 1.2744 (1.3172) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][570/625] eta 0:00:21 lr 0.001812 wd 0.0500 time 0.3801 (0.3861) data time 0.0010 (0.0020) model time 0.3790 (0.3843) loss 3.0347 (3.4475) grad_norm 1.1747 (1.3166) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][580/625] eta 0:00:17 lr 0.001811 wd 0.0500 time 0.3756 (0.3860) data time 0.0008 (0.0020) model time 0.3747 (0.3841) loss 3.9921 (3.4436) grad_norm 1.2311 (1.3168) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][590/625] eta 0:00:13 lr 0.001811 wd 0.0500 time 0.3770 (0.3858) data time 0.0008 (0.0020) model time 0.3762 (0.3840) loss 3.1347 (3.4408) grad_norm 1.1504 (1.3170) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][600/625] eta 0:00:09 lr 0.001811 wd 0.0500 time 0.3778 (0.3857) data time 0.0008 (0.0020) model time 0.3770 (0.3839) loss 3.9738 (3.4421) grad_norm 1.6718 (1.3182) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][610/625] eta 0:00:05 lr 0.001811 wd 0.0500 time 0.3838 (0.3866) data time 0.0005 (0.0020) model time 0.3833 (0.3848) loss 3.9187 (3.4411) grad_norm 1.3478 (1.3178) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [75/300][620/625] eta 0:00:01 lr 0.001811 wd 0.0500 time 0.3667 (0.3863) data time 0.0006 (0.0019) model time 0.3661 (0.3846) loss 3.9760 (3.4420) grad_norm 1.3954 (1.3180) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 75 training takes 0:04:01 [2024-08-26 08:05:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:05:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:05:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/75_ckpt.pth saving...... [2024-08-26 08:05:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/75_ckpt.pth saved !!! [2024-08-26 08:05:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.6499 (0.6499) Acc@1 85.156 (85.156) Acc@5 97.314 (97.314) Mem 11225MB [2024-08-26 08:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 1.1025 (0.8207) Acc@1 73.877 (81.272) Acc@5 92.334 (95.987) Mem 11225MB [2024-08-26 08:06:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 1.2646 (0.9978) Acc@1 70.996 (76.972) Acc@5 90.381 (93.794) Mem 11225MB [2024-08-26 08:06:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.651 Acc@5 93.714 [2024-08-26 08:06:01 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.7% [2024-08-26 08:06:01 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 76.65% [2024-08-26 08:06:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 08:06:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 08:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.507 (0.507) Loss 0.5664 (0.5664) Acc@1 87.744 (87.744) Acc@5 98.096 (98.096) Mem 11225MB [2024-08-26 08:06:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.151) Loss 0.9546 (0.7088) Acc@1 76.904 (83.674) Acc@5 94.971 (96.964) Mem 11225MB [2024-08-26 08:06:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.099 (0.124) Loss 1.1221 (0.8698) Acc@1 71.729 (79.653) Acc@5 92.383 (95.094) Mem 11225MB [2024-08-26 08:06:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.359 Acc@5 95.030 [2024-08-26 08:06:05 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.4% [2024-08-26 08:06:05 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.36% [2024-08-26 08:06:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:06:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][0/625] eta 0:07:47 lr 0.001811 wd 0.0500 time 0.7485 (0.7485) data time 0.4010 (0.4010) model time 0.0000 (0.0000) loss 3.4883 (3.4883) grad_norm 1.2353 (1.2353) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][10/625] eta 0:04:14 lr 0.001811 wd 0.0500 time 0.3802 (0.4142) data time 0.0011 (0.0376) model time 0.0000 (0.0000) loss 3.5759 (3.4553) grad_norm 1.1991 (1.2235) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][20/625] eta 0:04:06 lr 0.001811 wd 0.0500 time 0.3752 (0.4075) data time 0.0012 (0.0203) model time 0.0000 (0.0000) loss 3.5214 (3.3760) grad_norm 1.8426 (1.2392) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][30/625] eta 0:04:02 lr 0.001811 wd 0.0500 time 0.3759 (0.4079) data time 0.0011 (0.0141) model time 0.0000 (0.0000) loss 3.8021 (3.3240) grad_norm 1.1128 (1.2320) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][40/625] eta 0:03:57 lr 0.001811 wd 0.0500 time 0.3739 (0.4052) data time 0.0011 (0.0110) model time 0.0000 (0.0000) loss 4.0404 (3.3650) grad_norm 1.0699 (1.2005) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][50/625] eta 0:03:49 lr 0.001810 wd 0.0500 time 0.3817 (0.3998) data time 0.0008 (0.0091) model time 0.0000 (0.0000) loss 3.5838 (3.4391) grad_norm 1.5754 (1.1898) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][60/625] eta 0:03:43 lr 0.001810 wd 0.0500 time 0.3786 (0.3962) data time 0.0012 (0.0078) model time 0.3774 (0.3765) loss 3.7408 (3.4401) grad_norm 1.4096 (1.1920) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][70/625] eta 0:03:38 lr 0.001810 wd 0.0500 time 0.3743 (0.3937) data time 0.0009 (0.0069) model time 0.3734 (0.3767) loss 3.8206 (3.4665) grad_norm 1.1850 (1.2167) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][80/625] eta 0:03:33 lr 0.001810 wd 0.0500 time 0.3815 (0.3919) data time 0.0011 (0.0062) model time 0.3804 (0.3770) loss 2.6760 (3.4701) grad_norm 1.1396 (1.2038) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][90/625] eta 0:03:28 lr 0.001810 wd 0.0500 time 0.3662 (0.3902) data time 0.0011 (0.0056) model time 0.3651 (0.3766) loss 3.9899 (3.4943) grad_norm 1.9135 (1.2320) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][100/625] eta 0:03:24 lr 0.001810 wd 0.0500 time 0.3731 (0.3893) data time 0.0010 (0.0052) model time 0.3721 (0.3773) loss 3.9585 (3.5228) grad_norm 1.1948 (1.2270) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][110/625] eta 0:03:20 lr 0.001810 wd 0.0500 time 0.3834 (0.3884) data time 0.0011 (0.0048) model time 0.3824 (0.3774) loss 3.6597 (3.5098) grad_norm 1.2132 (1.2351) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][120/625] eta 0:03:15 lr 0.001810 wd 0.0500 time 0.3812 (0.3876) data time 0.0011 (0.0045) model time 0.3801 (0.3775) loss 3.4810 (3.5057) grad_norm 1.1323 (1.2405) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][130/625] eta 0:03:11 lr 0.001810 wd 0.0500 time 0.3801 (0.3869) data time 0.0009 (0.0043) model time 0.3792 (0.3775) loss 3.5380 (3.4842) grad_norm 1.4903 (1.2373) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][140/625] eta 0:03:07 lr 0.001809 wd 0.0500 time 0.3726 (0.3861) data time 0.0009 (0.0041) model time 0.3718 (0.3770) loss 3.1422 (3.4861) grad_norm 0.8003 (1.2431) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][150/625] eta 0:03:03 lr 0.001809 wd 0.0500 time 0.3758 (0.3855) data time 0.0009 (0.0039) model time 0.3748 (0.3769) loss 4.4714 (3.4869) grad_norm 1.3318 (1.2423) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][160/625] eta 0:02:59 lr 0.001809 wd 0.0500 time 0.3849 (0.3851) data time 0.0009 (0.0037) model time 0.3840 (0.3770) loss 3.7931 (3.4930) grad_norm 1.4060 (1.2641) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][170/625] eta 0:02:56 lr 0.001809 wd 0.0500 time 0.3705 (0.3876) data time 0.0008 (0.0036) model time 0.3696 (0.3811) loss 2.4463 (3.4785) grad_norm 1.0406 (1.2696) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][180/625] eta 0:02:52 lr 0.001809 wd 0.0500 time 0.3776 (0.3870) data time 0.0009 (0.0035) model time 0.3767 (0.3808) loss 4.3835 (3.4892) grad_norm 1.8140 (1.2806) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][190/625] eta 0:02:48 lr 0.001809 wd 0.0500 time 0.3744 (0.3866) data time 0.0012 (0.0033) model time 0.3732 (0.3805) loss 3.9180 (3.4909) grad_norm 1.5210 (1.2810) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][200/625] eta 0:02:44 lr 0.001809 wd 0.0500 time 0.3748 (0.3862) data time 0.0009 (0.0032) model time 0.3740 (0.3803) loss 4.1189 (3.4905) grad_norm 1.2889 (1.2808) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][210/625] eta 0:02:40 lr 0.001809 wd 0.0500 time 0.3741 (0.3858) data time 0.0008 (0.0031) model time 0.3733 (0.3801) loss 3.1256 (3.4929) grad_norm 0.8968 (1.2899) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][220/625] eta 0:02:36 lr 0.001809 wd 0.0500 time 0.3782 (0.3862) data time 0.0013 (0.0030) model time 0.3769 (0.3809) loss 3.6728 (3.5029) grad_norm 1.6097 (1.2941) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][230/625] eta 0:02:32 lr 0.001809 wd 0.0500 time 0.5836 (0.3868) data time 0.0008 (0.0030) model time 0.5828 (0.3819) loss 4.7912 (3.5099) grad_norm 1.8972 (1.2918) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][240/625] eta 0:02:29 lr 0.001808 wd 0.0500 time 0.3780 (0.3882) data time 0.0008 (0.0029) model time 0.3772 (0.3838) loss 4.1525 (3.5079) grad_norm 1.4809 (1.2925) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][250/625] eta 0:02:25 lr 0.001808 wd 0.0500 time 0.3779 (0.3877) data time 0.0011 (0.0028) model time 0.3768 (0.3834) loss 2.5070 (3.5034) grad_norm 1.1703 (1.2934) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][260/625] eta 0:02:21 lr 0.001808 wd 0.0500 time 0.3728 (0.3872) data time 0.0010 (0.0028) model time 0.3718 (0.3830) loss 3.7212 (3.4975) grad_norm 1.2281 (1.2927) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][270/625] eta 0:02:17 lr 0.001808 wd 0.0500 time 0.3911 (0.3868) data time 0.0008 (0.0027) model time 0.3903 (0.3826) loss 3.4764 (3.4879) grad_norm 1.2092 (1.3011) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][280/625] eta 0:02:13 lr 0.001808 wd 0.0500 time 0.3749 (0.3865) data time 0.0011 (0.0026) model time 0.3739 (0.3823) loss 2.3042 (3.4848) grad_norm 1.4227 (1.2975) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:07:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][290/625] eta 0:02:10 lr 0.001808 wd 0.0500 time 0.3777 (0.3883) data time 0.0009 (0.0026) model time 0.3769 (0.3846) loss 2.6975 (3.4829) grad_norm 1.3237 (1.2912) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][300/625] eta 0:02:06 lr 0.001808 wd 0.0500 time 0.3697 (0.3878) data time 0.0009 (0.0025) model time 0.3689 (0.3842) loss 4.3129 (3.4828) grad_norm 1.4685 (1.2988) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][310/625] eta 0:02:02 lr 0.001808 wd 0.0500 time 0.3775 (0.3875) data time 0.0011 (0.0025) model time 0.3764 (0.3839) loss 3.2566 (3.4826) grad_norm 1.0035 (1.3013) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][320/625] eta 0:01:58 lr 0.001808 wd 0.0500 time 0.3750 (0.3872) data time 0.0009 (0.0025) model time 0.3741 (0.3836) loss 3.9304 (3.4779) grad_norm 1.8732 (1.3004) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][330/625] eta 0:01:54 lr 0.001807 wd 0.0500 time 0.3747 (0.3869) data time 0.0011 (0.0024) model time 0.3736 (0.3833) loss 3.8113 (3.4722) grad_norm 1.1583 (1.3027) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][340/625] eta 0:01:50 lr 0.001807 wd 0.0500 time 0.3741 (0.3872) data time 0.0010 (0.0024) model time 0.3732 (0.3838) loss 4.3405 (3.4738) grad_norm 1.1345 (1.2967) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][350/625] eta 0:01:46 lr 0.001807 wd 0.0500 time 0.3719 (0.3869) data time 0.0008 (0.0023) model time 0.3710 (0.3835) loss 4.3873 (3.4696) grad_norm 1.0901 (1.2970) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][360/625] eta 0:01:42 lr 0.001807 wd 0.0500 time 0.3757 (0.3866) data time 0.0010 (0.0023) model time 0.3747 (0.3833) loss 3.3236 (3.4659) grad_norm 1.7271 (1.2961) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][370/625] eta 0:01:38 lr 0.001807 wd 0.0500 time 0.3724 (0.3864) data time 0.0008 (0.0023) model time 0.3716 (0.3830) loss 3.6555 (3.4662) grad_norm 1.4285 (1.2946) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][380/625] eta 0:01:34 lr 0.001807 wd 0.0500 time 0.3801 (0.3862) data time 0.0009 (0.0023) model time 0.3792 (0.3828) loss 3.1907 (3.4690) grad_norm 1.4483 (1.2964) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][390/625] eta 0:01:30 lr 0.001807 wd 0.0500 time 0.3814 (0.3859) data time 0.0013 (0.0022) model time 0.3801 (0.3826) loss 3.6602 (3.4696) grad_norm 1.6221 (1.2987) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][400/625] eta 0:01:26 lr 0.001807 wd 0.0500 time 0.3749 (0.3857) data time 0.0013 (0.0022) model time 0.3736 (0.3824) loss 3.3164 (3.4626) grad_norm 1.5148 (1.2988) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][410/625] eta 0:01:22 lr 0.001807 wd 0.0500 time 0.3795 (0.3855) data time 0.0011 (0.0022) model time 0.3784 (0.3822) loss 3.8947 (3.4625) grad_norm 1.1351 (1.2972) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][420/625] eta 0:01:19 lr 0.001807 wd 0.0500 time 0.3743 (0.3869) data time 0.0011 (0.0022) model time 0.3732 (0.3839) loss 3.7982 (3.4661) grad_norm 1.3931 (1.2962) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][430/625] eta 0:01:15 lr 0.001806 wd 0.0500 time 0.3834 (0.3867) data time 0.0012 (0.0021) model time 0.3822 (0.3837) loss 3.7162 (3.4644) grad_norm 1.1704 (1.2958) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:08:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][440/625] eta 0:01:11 lr 0.001806 wd 0.0500 time 0.3746 (0.3865) data time 0.0011 (0.0021) model time 0.3735 (0.3835) loss 3.3079 (3.4652) grad_norm 1.3527 (1.2968) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][450/625] eta 0:01:07 lr 0.001806 wd 0.0500 time 0.3744 (0.3863) data time 0.0009 (0.0021) model time 0.3735 (0.3834) loss 3.8339 (3.4667) grad_norm 0.9255 (1.2985) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][460/625] eta 0:01:03 lr 0.001806 wd 0.0500 time 0.3701 (0.3861) data time 0.0011 (0.0021) model time 0.3690 (0.3832) loss 3.7715 (3.4692) grad_norm 1.3536 (1.2982) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][470/625] eta 0:00:59 lr 0.001806 wd 0.0500 time 0.3750 (0.3860) data time 0.0008 (0.0021) model time 0.3742 (0.3831) loss 4.2395 (3.4712) grad_norm 1.4734 (1.2988) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][480/625] eta 0:00:55 lr 0.001806 wd 0.0500 time 0.3778 (0.3858) data time 0.0008 (0.0020) model time 0.3770 (0.3829) loss 4.2948 (3.4722) grad_norm 1.3748 (1.2981) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][490/625] eta 0:00:52 lr 0.001806 wd 0.0500 time 0.3738 (0.3856) data time 0.0011 (0.0020) model time 0.3727 (0.3827) loss 3.0844 (3.4664) grad_norm 1.7551 (1.2994) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][500/625] eta 0:00:48 lr 0.001806 wd 0.0500 time 0.3739 (0.3867) data time 0.0011 (0.0020) model time 0.3727 (0.3840) loss 3.6703 (3.4602) grad_norm 1.3661 (1.2986) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][510/625] eta 0:00:44 lr 0.001806 wd 0.0500 time 0.3757 (0.3865) data time 0.0008 (0.0020) model time 0.3749 (0.3838) loss 3.1219 (3.4586) grad_norm 2.3358 (1.2995) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][520/625] eta 0:00:40 lr 0.001805 wd 0.0500 time 0.3782 (0.3864) data time 0.0008 (0.0020) model time 0.3774 (0.3837) loss 3.0443 (3.4582) grad_norm 1.1580 (1.2989) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][530/625] eta 0:00:36 lr 0.001805 wd 0.0500 time 0.3731 (0.3862) data time 0.0008 (0.0019) model time 0.3722 (0.3836) loss 3.7682 (3.4607) grad_norm 1.0941 (1.2999) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][540/625] eta 0:00:32 lr 0.001805 wd 0.0500 time 0.3717 (0.3872) data time 0.0011 (0.0019) model time 0.3706 (0.3847) loss 3.1519 (3.4572) grad_norm 1.0817 (1.2952) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][550/625] eta 0:00:29 lr 0.001805 wd 0.0500 time 0.3804 (0.3874) data time 0.0009 (0.0019) model time 0.3796 (0.3849) loss 2.1792 (3.4599) grad_norm 1.3033 (1.2954) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][560/625] eta 0:00:25 lr 0.001805 wd 0.0500 time 0.3734 (0.3873) data time 0.0009 (0.0019) model time 0.3726 (0.3848) loss 2.8347 (3.4611) grad_norm 1.2750 (1.2938) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][570/625] eta 0:00:21 lr 0.001805 wd 0.0500 time 0.4226 (0.3872) data time 0.0012 (0.0019) model time 0.4213 (0.3848) loss 2.8741 (3.4637) grad_norm 1.1623 (1.2922) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][580/625] eta 0:00:17 lr 0.001805 wd 0.0500 time 0.3807 (0.3871) data time 0.0008 (0.0019) model time 0.3799 (0.3846) loss 4.0013 (3.4623) grad_norm 1.3920 (1.2904) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][590/625] eta 0:00:13 lr 0.001805 wd 0.0500 time 0.3795 (0.3870) data time 0.0011 (0.0019) model time 0.3784 (0.3846) loss 3.3672 (3.4616) grad_norm 1.2418 (1.2917) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:09:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][600/625] eta 0:00:09 lr 0.001805 wd 0.0500 time 0.3741 (0.3868) data time 0.0009 (0.0019) model time 0.3732 (0.3845) loss 2.8063 (3.4587) grad_norm 1.1825 (1.2940) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][610/625] eta 0:00:05 lr 0.001805 wd 0.0500 time 0.3705 (0.3867) data time 0.0005 (0.0018) model time 0.3700 (0.3843) loss 2.6670 (3.4592) grad_norm 1.0446 (1.2932) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [76/300][620/625] eta 0:00:01 lr 0.001804 wd 0.0500 time 0.3682 (0.3864) data time 0.0005 (0.0018) model time 0.3677 (0.3840) loss 2.1824 (3.4575) grad_norm 0.9231 (1.3001) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 76 training takes 0:04:01 [2024-08-26 08:10:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:10:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:10:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.522 (0.522) Loss 0.6616 (0.6616) Acc@1 86.426 (86.426) Acc@5 96.729 (96.729) Mem 11225MB [2024-08-26 08:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.145) Loss 1.1309 (0.8319) Acc@1 74.854 (81.614) Acc@5 92.188 (95.916) Mem 11225MB [2024-08-26 08:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.2998 (1.0151) Acc@1 69.824 (77.297) Acc@5 90.625 (93.799) Mem 11225MB [2024-08-26 08:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.945 Acc@5 93.698 [2024-08-26 08:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.9% [2024-08-26 08:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 76.95% [2024-08-26 08:10:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 08:10:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 08:10:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 0.5654 (0.5654) Acc@1 87.891 (87.891) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 08:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.9517 (0.7078) Acc@1 76.807 (83.700) Acc@5 94.824 (96.933) Mem 11225MB [2024-08-26 08:10:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.1191 (0.8679) Acc@1 71.826 (79.667) Acc@5 92.285 (95.078) Mem 11225MB [2024-08-26 08:10:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.387 Acc@5 95.030 [2024-08-26 08:10:16 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.4% [2024-08-26 08:10:16 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.39% [2024-08-26 08:10:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:10:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][0/625] eta 0:07:50 lr 0.001804 wd 0.0500 time 0.7526 (0.7526) data time 0.4191 (0.4191) model time 0.0000 (0.0000) loss 3.3452 (3.3452) grad_norm 1.2444 (1.2444) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][10/625] eta 0:04:13 lr 0.001804 wd 0.0500 time 0.3878 (0.4119) data time 0.0011 (0.0392) model time 0.0000 (0.0000) loss 3.4411 (3.3382) grad_norm 1.0245 (1.1099) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][20/625] eta 0:03:59 lr 0.001804 wd 0.0500 time 0.3703 (0.3963) data time 0.0012 (0.0211) model time 0.0000 (0.0000) loss 4.1548 (3.4540) grad_norm 1.2590 (1.1078) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][30/625] eta 0:03:56 lr 0.001804 wd 0.0500 time 0.3704 (0.3976) data time 0.0009 (0.0146) model time 0.0000 (0.0000) loss 4.4006 (3.4631) grad_norm 1.5589 (1.1615) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][40/625] eta 0:03:49 lr 0.001804 wd 0.0500 time 0.3785 (0.3926) data time 0.0014 (0.0114) model time 0.0000 (0.0000) loss 3.9040 (3.4591) grad_norm 1.5821 (1.2127) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][50/625] eta 0:03:43 lr 0.001804 wd 0.0500 time 0.3711 (0.3893) data time 0.0011 (0.0094) model time 0.0000 (0.0000) loss 3.5431 (3.4622) grad_norm 1.2555 (1.2244) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][60/625] eta 0:03:40 lr 0.001804 wd 0.0500 time 0.3781 (0.3896) data time 0.0009 (0.0080) model time 0.3772 (0.3898) loss 3.2027 (3.5148) grad_norm 1.6645 (1.2733) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][70/625] eta 0:03:41 lr 0.001804 wd 0.0500 time 0.3782 (0.3995) data time 0.0008 (0.0070) model time 0.3774 (0.4243) loss 4.1218 (3.5407) grad_norm 1.4122 (1.2709) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][80/625] eta 0:03:37 lr 0.001803 wd 0.0500 time 0.3729 (0.3990) data time 0.0009 (0.0063) model time 0.3721 (0.4143) loss 2.7707 (3.5107) grad_norm 1.2728 (1.2886) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][90/625] eta 0:03:32 lr 0.001803 wd 0.0500 time 0.3786 (0.3966) data time 0.0011 (0.0057) model time 0.3775 (0.4047) loss 3.3468 (3.4789) grad_norm 1.2271 (1.2952) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:10:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][100/625] eta 0:03:27 lr 0.001803 wd 0.0500 time 0.4079 (0.3951) data time 0.0011 (0.0053) model time 0.4069 (0.3999) loss 3.9354 (3.4799) grad_norm 1.2506 (1.2897) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][110/625] eta 0:03:22 lr 0.001803 wd 0.0500 time 0.3745 (0.3934) data time 0.0009 (0.0049) model time 0.3736 (0.3957) loss 4.0838 (3.4780) grad_norm 0.9765 (1.2730) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][120/625] eta 0:03:17 lr 0.001803 wd 0.0500 time 0.3727 (0.3920) data time 0.0011 (0.0046) model time 0.3717 (0.3928) loss 3.2280 (3.4818) grad_norm 1.2836 (1.2757) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][130/625] eta 0:03:13 lr 0.001803 wd 0.0500 time 0.3755 (0.3910) data time 0.0011 (0.0043) model time 0.3744 (0.3909) loss 2.7740 (3.4632) grad_norm 1.2504 (1.2840) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][140/625] eta 0:03:09 lr 0.001803 wd 0.0500 time 0.3322 (0.3913) data time 0.0009 (0.0041) model time 0.3313 (0.3914) loss 3.9160 (3.4376) grad_norm 1.1518 (1.2969) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][150/625] eta 0:03:05 lr 0.001803 wd 0.0500 time 0.3753 (0.3906) data time 0.0009 (0.0039) model time 0.3744 (0.3901) loss 4.4104 (3.4075) grad_norm 1.3140 (1.3136) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][160/625] eta 0:03:01 lr 0.001803 wd 0.0500 time 0.3791 (0.3899) data time 0.0011 (0.0037) model time 0.3780 (0.3891) loss 3.7235 (3.3955) grad_norm 0.9403 (1.3113) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][170/625] eta 0:02:57 lr 0.001803 wd 0.0500 time 0.3961 (0.3894) data time 0.0011 (0.0036) model time 0.3950 (0.3882) loss 3.6678 (3.3912) grad_norm 1.6141 (1.3136) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][180/625] eta 0:02:52 lr 0.001802 wd 0.0500 time 0.3721 (0.3887) data time 0.0011 (0.0035) model time 0.3710 (0.3874) loss 3.5648 (3.3819) grad_norm 0.8167 (1.3012) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][190/625] eta 0:02:48 lr 0.001802 wd 0.0500 time 0.3704 (0.3883) data time 0.0013 (0.0033) model time 0.3691 (0.3868) loss 2.5139 (3.3937) grad_norm 1.9743 (1.3163) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][200/625] eta 0:02:44 lr 0.001802 wd 0.0500 time 0.3774 (0.3880) data time 0.0009 (0.0032) model time 0.3765 (0.3864) loss 3.8215 (3.4128) grad_norm 1.1033 (1.3193) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][210/625] eta 0:02:40 lr 0.001802 wd 0.0500 time 0.3785 (0.3873) data time 0.0010 (0.0031) model time 0.3775 (0.3856) loss 3.6880 (3.4243) grad_norm 1.2481 (1.3165) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][220/625] eta 0:02:36 lr 0.001802 wd 0.0500 time 0.3756 (0.3869) data time 0.0011 (0.0030) model time 0.3745 (0.3850) loss 3.7715 (3.4316) grad_norm 0.9695 (1.3057) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][230/625] eta 0:02:32 lr 0.001802 wd 0.0500 time 0.3805 (0.3866) data time 0.0010 (0.0030) model time 0.3794 (0.3847) loss 3.1389 (3.4422) grad_norm 1.3483 (1.3025) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][240/625] eta 0:02:28 lr 0.001802 wd 0.0500 time 0.5228 (0.3869) data time 0.0011 (0.0029) model time 0.5217 (0.3851) loss 3.3692 (3.4403) grad_norm 1.1005 (1.2971) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][250/625] eta 0:02:25 lr 0.001802 wd 0.0500 time 0.3756 (0.3880) data time 0.0011 (0.0028) model time 0.3746 (0.3865) loss 3.3117 (3.4283) grad_norm 1.3038 (1.3021) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:11:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][260/625] eta 0:02:21 lr 0.001802 wd 0.0500 time 0.3782 (0.3876) data time 0.0011 (0.0028) model time 0.3771 (0.3861) loss 2.7930 (3.4206) grad_norm 1.0905 (1.3025) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][270/625] eta 0:02:17 lr 0.001801 wd 0.0500 time 0.3827 (0.3874) data time 0.0011 (0.0027) model time 0.3816 (0.3858) loss 2.6460 (3.4177) grad_norm 1.5121 (1.3005) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][280/625] eta 0:02:13 lr 0.001801 wd 0.0500 time 0.3713 (0.3879) data time 0.0011 (0.0026) model time 0.3702 (0.3865) loss 3.7074 (3.4337) grad_norm 0.9271 (1.2969) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][290/625] eta 0:02:10 lr 0.001801 wd 0.0500 time 0.3622 (0.3884) data time 0.0008 (0.0026) model time 0.3614 (0.3870) loss 4.1455 (3.4278) grad_norm 1.5901 (1.2911) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][300/625] eta 0:02:06 lr 0.001801 wd 0.0500 time 0.3764 (0.3880) data time 0.0009 (0.0025) model time 0.3755 (0.3866) loss 4.1923 (3.4280) grad_norm 2.0495 (1.2924) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][310/625] eta 0:02:02 lr 0.001801 wd 0.0500 time 0.3828 (0.3877) data time 0.0008 (0.0025) model time 0.3820 (0.3863) loss 3.1807 (3.4330) grad_norm 1.0743 (1.2929) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][320/625] eta 0:01:58 lr 0.001801 wd 0.0500 time 0.3769 (0.3874) data time 0.0013 (0.0025) model time 0.3756 (0.3859) loss 2.9493 (3.4340) grad_norm 1.1069 (1.2947) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][330/625] eta 0:01:54 lr 0.001801 wd 0.0500 time 0.3992 (0.3872) data time 0.0008 (0.0024) model time 0.3984 (0.3857) loss 3.3143 (3.4316) grad_norm 1.2287 (1.2921) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][340/625] eta 0:01:50 lr 0.001801 wd 0.0500 time 0.3743 (0.3869) data time 0.0011 (0.0024) model time 0.3732 (0.3854) loss 3.0924 (3.4242) grad_norm 2.0999 (1.2884) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][350/625] eta 0:01:46 lr 0.001801 wd 0.0500 time 0.5987 (0.3873) data time 0.0013 (0.0023) model time 0.5973 (0.3858) loss 3.4505 (3.4239) grad_norm 1.2639 (1.2917) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:12:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][360/625] eta 0:01:42 lr 0.001801 wd 0.0500 time 0.5587 (0.3874) data time 0.0013 (0.0023) model time 0.5574 (0.3859) loss 3.3419 (3.4315) grad_norm 1.5565 (inf) loss_scale 2048.0000 (4061.9612) mem 11225MB [2024-08-26 08:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][370/625] eta 0:01:38 lr 0.001800 wd 0.0500 time 0.3754 (0.3882) data time 0.0008 (0.0023) model time 0.3746 (0.3869) loss 3.7589 (3.4422) grad_norm 1.4297 (inf) loss_scale 2048.0000 (4007.6765) mem 11225MB [2024-08-26 08:12:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][380/625] eta 0:01:35 lr 0.001800 wd 0.0500 time 0.3672 (0.3879) data time 0.0011 (0.0023) model time 0.3661 (0.3865) loss 3.9561 (3.4339) grad_norm 1.0760 (inf) loss_scale 2048.0000 (3956.2415) mem 11225MB [2024-08-26 08:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][390/625] eta 0:01:31 lr 0.001800 wd 0.0500 time 0.3791 (0.3876) data time 0.0011 (0.0022) model time 0.3781 (0.3862) loss 4.0230 (3.4413) grad_norm 0.8601 (inf) loss_scale 2048.0000 (3907.4373) mem 11225MB [2024-08-26 08:12:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][400/625] eta 0:01:27 lr 0.001800 wd 0.0500 time 0.3747 (0.3874) data time 0.0009 (0.0022) model time 0.3738 (0.3859) loss 3.2992 (3.4438) grad_norm 1.2611 (inf) loss_scale 2048.0000 (3861.0673) mem 11225MB [2024-08-26 08:12:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][410/625] eta 0:01:23 lr 0.001800 wd 0.0500 time 0.3830 (0.3871) data time 0.0011 (0.0022) model time 0.3818 (0.3857) loss 3.6267 (3.4438) grad_norm 1.4131 (inf) loss_scale 2048.0000 (3816.9538) mem 11225MB [2024-08-26 08:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][420/625] eta 0:01:19 lr 0.001800 wd 0.0500 time 0.3737 (0.3869) data time 0.0011 (0.0021) model time 0.3726 (0.3854) loss 3.6939 (3.4402) grad_norm 1.5364 (inf) loss_scale 2048.0000 (3774.9359) mem 11225MB [2024-08-26 08:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][430/625] eta 0:01:15 lr 0.001800 wd 0.0500 time 0.3856 (0.3866) data time 0.0010 (0.0021) model time 0.3846 (0.3851) loss 2.7915 (3.4421) grad_norm 1.3682 (inf) loss_scale 2048.0000 (3734.8677) mem 11225MB [2024-08-26 08:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][440/625] eta 0:01:11 lr 0.001800 wd 0.0500 time 0.3750 (0.3864) data time 0.0010 (0.0021) model time 0.3740 (0.3849) loss 3.5497 (3.4494) grad_norm 1.0866 (inf) loss_scale 2048.0000 (3696.6168) mem 11225MB [2024-08-26 08:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][450/625] eta 0:01:07 lr 0.001800 wd 0.0500 time 0.3781 (0.3862) data time 0.0010 (0.0021) model time 0.3772 (0.3847) loss 3.0158 (3.4483) grad_norm 1.1252 (inf) loss_scale 2048.0000 (3660.0621) mem 11225MB [2024-08-26 08:13:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][460/625] eta 0:01:03 lr 0.001799 wd 0.0500 time 0.3842 (0.3860) data time 0.0012 (0.0021) model time 0.3830 (0.3844) loss 3.7153 (3.4430) grad_norm 0.9263 (inf) loss_scale 2048.0000 (3625.0933) mem 11225MB [2024-08-26 08:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][470/625] eta 0:00:59 lr 0.001799 wd 0.0500 time 0.3794 (0.3867) data time 0.0011 (0.0020) model time 0.3783 (0.3853) loss 2.5475 (3.4379) grad_norm 1.0947 (inf) loss_scale 2048.0000 (3591.6093) mem 11225MB [2024-08-26 08:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][480/625] eta 0:00:56 lr 0.001799 wd 0.0500 time 0.3760 (0.3866) data time 0.0011 (0.0020) model time 0.3749 (0.3851) loss 4.2159 (3.4355) grad_norm 1.5064 (inf) loss_scale 2048.0000 (3559.5177) mem 11225MB [2024-08-26 08:13:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][490/625] eta 0:00:52 lr 0.001799 wd 0.0500 time 0.3846 (0.3864) data time 0.0008 (0.0020) model time 0.3838 (0.3849) loss 3.9202 (3.4355) grad_norm 1.0436 (inf) loss_scale 2048.0000 (3528.7332) mem 11225MB [2024-08-26 08:13:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][500/625] eta 0:00:48 lr 0.001799 wd 0.0500 time 0.3795 (0.3862) data time 0.0010 (0.0020) model time 0.3784 (0.3847) loss 2.8291 (3.4403) grad_norm 1.4185 (inf) loss_scale 2048.0000 (3499.1776) mem 11225MB [2024-08-26 08:13:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][510/625] eta 0:00:44 lr 0.001799 wd 0.0500 time 0.3801 (0.3861) data time 0.0008 (0.0020) model time 0.3793 (0.3846) loss 3.7467 (3.4422) grad_norm 1.2364 (inf) loss_scale 2048.0000 (3470.7789) mem 11225MB [2024-08-26 08:13:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][520/625] eta 0:00:40 lr 0.001799 wd 0.0500 time 0.3777 (0.3859) data time 0.0011 (0.0020) model time 0.3765 (0.3844) loss 3.6691 (3.4440) grad_norm 1.4633 (inf) loss_scale 2048.0000 (3443.4702) mem 11225MB [2024-08-26 08:13:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][530/625] eta 0:00:36 lr 0.001799 wd 0.0500 time 0.3731 (0.3857) data time 0.0010 (0.0019) model time 0.3721 (0.3842) loss 4.0553 (3.4434) grad_norm 1.3336 (inf) loss_scale 2048.0000 (3417.1902) mem 11225MB [2024-08-26 08:13:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][540/625] eta 0:00:32 lr 0.001799 wd 0.0500 time 0.3761 (0.3858) data time 0.0011 (0.0019) model time 0.3750 (0.3843) loss 3.7317 (3.4457) grad_norm 1.0530 (inf) loss_scale 2048.0000 (3391.8817) mem 11225MB [2024-08-26 08:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][550/625] eta 0:00:28 lr 0.001798 wd 0.0500 time 0.3751 (0.3856) data time 0.0009 (0.0019) model time 0.3742 (0.3841) loss 3.9740 (3.4449) grad_norm 1.1903 (inf) loss_scale 2048.0000 (3367.4918) mem 11225MB [2024-08-26 08:13:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][560/625] eta 0:00:25 lr 0.001798 wd 0.0500 time 0.3801 (0.3855) data time 0.0012 (0.0019) model time 0.3789 (0.3839) loss 2.4961 (3.4429) grad_norm 1.4367 (inf) loss_scale 2048.0000 (3343.9715) mem 11225MB [2024-08-26 08:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][570/625] eta 0:00:21 lr 0.001798 wd 0.0500 time 0.3753 (0.3857) data time 0.0008 (0.0019) model time 0.3745 (0.3842) loss 3.4652 (3.4412) grad_norm 1.3856 (inf) loss_scale 2048.0000 (3321.2750) mem 11225MB [2024-08-26 08:14:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][580/625] eta 0:00:17 lr 0.001798 wd 0.0500 time 0.5914 (0.3866) data time 0.0011 (0.0019) model time 0.5903 (0.3852) loss 3.6880 (3.4454) grad_norm 1.1427 (inf) loss_scale 2048.0000 (3299.3597) mem 11225MB [2024-08-26 08:14:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][590/625] eta 0:00:13 lr 0.001798 wd 0.0500 time 0.3749 (0.3868) data time 0.0008 (0.0019) model time 0.3740 (0.3854) loss 2.8750 (3.4450) grad_norm 1.1002 (inf) loss_scale 2048.0000 (3278.1861) mem 11225MB [2024-08-26 08:14:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][600/625] eta 0:00:09 lr 0.001798 wd 0.0500 time 0.3793 (0.3866) data time 0.0009 (0.0018) model time 0.3785 (0.3852) loss 4.4046 (3.4460) grad_norm 1.5134 (inf) loss_scale 2048.0000 (3257.7171) mem 11225MB [2024-08-26 08:14:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][610/625] eta 0:00:05 lr 0.001798 wd 0.0500 time 0.3679 (0.3864) data time 0.0008 (0.0018) model time 0.3671 (0.3850) loss 3.5235 (3.4440) grad_norm 1.3566 (inf) loss_scale 2048.0000 (3237.9182) mem 11225MB [2024-08-26 08:14:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [77/300][620/625] eta 0:00:01 lr 0.001798 wd 0.0500 time 0.3622 (0.3861) data time 0.0008 (0.0018) model time 0.3614 (0.3847) loss 3.8154 (3.4410) grad_norm 1.4817 (inf) loss_scale 2048.0000 (3218.7568) mem 11225MB [2024-08-26 08:14:18 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 77 training takes 0:04:01 [2024-08-26 08:14:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:14:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:14:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.7021 (0.7021) Acc@1 84.863 (84.863) Acc@5 96.924 (96.924) Mem 11225MB [2024-08-26 08:14:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 1.0889 (0.8317) Acc@1 74.707 (81.157) Acc@5 93.066 (95.925) Mem 11225MB [2024-08-26 08:14:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.2764 (0.9974) Acc@1 68.604 (77.051) Acc@5 91.064 (93.762) Mem 11225MB [2024-08-26 08:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.801 Acc@5 93.714 [2024-08-26 08:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.8% [2024-08-26 08:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.862 (0.862) Loss 0.5654 (0.5654) Acc@1 87.891 (87.891) Acc@5 98.145 (98.145) Mem 11225MB [2024-08-26 08:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.178) Loss 0.9502 (0.7072) Acc@1 76.709 (83.727) Acc@5 94.824 (96.950) Mem 11225MB [2024-08-26 08:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.138) Loss 1.1172 (0.8667) Acc@1 71.875 (79.711) Acc@5 92.383 (95.096) Mem 11225MB [2024-08-26 08:14:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.429 Acc@5 95.048 [2024-08-26 08:14:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.4% [2024-08-26 08:14:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.43% [2024-08-26 08:14:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:14:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:14:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][0/625] eta 0:08:17 lr 0.001798 wd 0.0500 time 0.7966 (0.7966) data time 0.4646 (0.4646) model time 0.0000 (0.0000) loss 3.4183 (3.4183) grad_norm 1.9048 (1.9048) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][10/625] eta 0:04:15 lr 0.001798 wd 0.0500 time 0.3734 (0.4154) data time 0.0008 (0.0433) model time 0.0000 (0.0000) loss 4.1081 (3.4679) grad_norm 1.6310 (1.4300) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:14:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][20/625] eta 0:04:00 lr 0.001797 wd 0.0500 time 0.3738 (0.3969) data time 0.0008 (0.0232) model time 0.0000 (0.0000) loss 3.9700 (3.5204) grad_norm 1.3818 (1.3368) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:14:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][30/625] eta 0:03:52 lr 0.001797 wd 0.0500 time 0.3754 (0.3907) data time 0.0009 (0.0160) model time 0.0000 (0.0000) loss 3.3362 (3.4055) grad_norm 1.3669 (1.2575) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:14:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][40/625] eta 0:03:46 lr 0.001797 wd 0.0500 time 0.3904 (0.3871) data time 0.0011 (0.0124) model time 0.0000 (0.0000) loss 3.5278 (3.3626) grad_norm 1.4420 (1.2961) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:14:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][50/625] eta 0:03:41 lr 0.001797 wd 0.0500 time 0.3771 (0.3849) data time 0.0008 (0.0102) model time 0.0000 (0.0000) loss 2.8699 (3.3763) grad_norm 1.0828 (1.3348) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:14:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][60/625] eta 0:03:36 lr 0.001797 wd 0.0500 time 0.3711 (0.3834) data time 0.0011 (0.0087) model time 0.3700 (0.3744) loss 3.3482 (3.4144) grad_norm 1.0866 (1.3014) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:14:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][70/625] eta 0:03:32 lr 0.001797 wd 0.0500 time 0.3772 (0.3822) data time 0.0008 (0.0076) model time 0.3764 (0.3742) loss 3.9820 (3.4291) grad_norm 1.4871 (1.3379) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:14:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][80/625] eta 0:03:27 lr 0.001797 wd 0.0500 time 0.3805 (0.3815) data time 0.0008 (0.0068) model time 0.3797 (0.3745) loss 2.6837 (3.4105) grad_norm 1.0054 (1.3306) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][90/625] eta 0:03:23 lr 0.001797 wd 0.0500 time 0.3756 (0.3811) data time 0.0009 (0.0062) model time 0.3747 (0.3751) loss 4.6877 (3.4450) grad_norm 1.3055 (1.3142) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][100/625] eta 0:03:21 lr 0.001797 wd 0.0500 time 0.3833 (0.3830) data time 0.0011 (0.0057) model time 0.3821 (0.3799) loss 3.3602 (3.4677) grad_norm 1.4471 (1.3267) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][110/625] eta 0:03:16 lr 0.001796 wd 0.0500 time 0.3779 (0.3825) data time 0.0011 (0.0053) model time 0.3768 (0.3793) loss 3.5391 (3.4605) grad_norm 1.0509 (1.3131) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][120/625] eta 0:03:13 lr 0.001796 wd 0.0500 time 0.6104 (0.3840) data time 0.0010 (0.0049) model time 0.6094 (0.3822) loss 3.3207 (3.4665) grad_norm 1.0684 (1.3086) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][130/625] eta 0:03:09 lr 0.001796 wd 0.0500 time 0.3751 (0.3833) data time 0.0009 (0.0046) model time 0.3742 (0.3812) loss 3.8869 (3.4657) grad_norm 1.0330 (1.2913) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][140/625] eta 0:03:05 lr 0.001796 wd 0.0500 time 0.3756 (0.3831) data time 0.0011 (0.0044) model time 0.3745 (0.3809) loss 3.3529 (3.4599) grad_norm 1.7781 (1.3016) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][150/625] eta 0:03:02 lr 0.001796 wd 0.0500 time 0.3792 (0.3840) data time 0.0011 (0.0042) model time 0.3781 (0.3824) loss 2.6845 (3.4579) grad_norm 0.9468 (1.3015) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][160/625] eta 0:02:59 lr 0.001796 wd 0.0500 time 0.3815 (0.3861) data time 0.0008 (0.0040) model time 0.3807 (0.3854) loss 2.9884 (3.4542) grad_norm 2.0575 (1.2996) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][170/625] eta 0:02:55 lr 0.001796 wd 0.0500 time 0.3852 (0.3857) data time 0.0010 (0.0038) model time 0.3841 (0.3849) loss 3.3904 (3.4571) grad_norm 1.5629 (1.3141) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][180/625] eta 0:02:51 lr 0.001796 wd 0.0500 time 0.3813 (0.3853) data time 0.0011 (0.0037) model time 0.3802 (0.3842) loss 3.0789 (3.4743) grad_norm 0.8979 (1.3094) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][190/625] eta 0:02:47 lr 0.001796 wd 0.0500 time 0.3824 (0.3849) data time 0.0008 (0.0036) model time 0.3816 (0.3837) loss 3.1319 (3.4618) grad_norm 1.1579 (1.2996) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][200/625] eta 0:02:43 lr 0.001796 wd 0.0500 time 0.3727 (0.3845) data time 0.0009 (0.0034) model time 0.3718 (0.3833) loss 3.6039 (3.4713) grad_norm 1.1010 (1.2913) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][210/625] eta 0:02:39 lr 0.001795 wd 0.0500 time 0.3741 (0.3843) data time 0.0011 (0.0033) model time 0.3730 (0.3829) loss 3.9297 (3.4733) grad_norm 1.3794 (1.2950) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][220/625] eta 0:02:35 lr 0.001795 wd 0.0500 time 0.3769 (0.3841) data time 0.0011 (0.0032) model time 0.3758 (0.3827) loss 3.4451 (3.4928) grad_norm 1.0231 (1.2923) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][230/625] eta 0:02:31 lr 0.001795 wd 0.0500 time 0.3740 (0.3838) data time 0.0010 (0.0031) model time 0.3729 (0.3824) loss 3.6849 (3.4876) grad_norm 1.6445 (1.2907) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][240/625] eta 0:02:27 lr 0.001795 wd 0.0500 time 0.3850 (0.3836) data time 0.0008 (0.0031) model time 0.3842 (0.3821) loss 3.6945 (3.4780) grad_norm 0.9453 (1.2927) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][250/625] eta 0:02:23 lr 0.001795 wd 0.0500 time 0.3823 (0.3834) data time 0.0008 (0.0030) model time 0.3815 (0.3819) loss 4.1235 (3.4666) grad_norm 1.7942 (1.2905) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][260/625] eta 0:02:19 lr 0.001795 wd 0.0500 time 0.3744 (0.3833) data time 0.0008 (0.0029) model time 0.3736 (0.3817) loss 2.2936 (3.4618) grad_norm 1.5100 (1.3005) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][270/625] eta 0:02:16 lr 0.001795 wd 0.0500 time 0.3809 (0.3831) data time 0.0011 (0.0028) model time 0.3798 (0.3816) loss 3.6366 (3.4680) grad_norm 1.0696 (1.3017) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][280/625] eta 0:02:12 lr 0.001795 wd 0.0500 time 0.3840 (0.3830) data time 0.0010 (0.0028) model time 0.3830 (0.3814) loss 3.1809 (3.4732) grad_norm 1.6746 (1.2972) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][290/625] eta 0:02:08 lr 0.001795 wd 0.0500 time 0.3737 (0.3828) data time 0.0008 (0.0027) model time 0.3728 (0.3812) loss 4.3129 (3.4846) grad_norm 1.3122 (1.2966) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][300/625] eta 0:02:04 lr 0.001794 wd 0.0500 time 0.3823 (0.3827) data time 0.0011 (0.0027) model time 0.3812 (0.3811) loss 3.9425 (3.4824) grad_norm 1.1533 (1.2989) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][310/625] eta 0:02:00 lr 0.001794 wd 0.0500 time 0.3909 (0.3833) data time 0.0011 (0.0026) model time 0.3898 (0.3818) loss 3.7656 (3.4784) grad_norm 1.4311 (1.2973) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][320/625] eta 0:01:56 lr 0.001794 wd 0.0500 time 0.3815 (0.3831) data time 0.0008 (0.0026) model time 0.3808 (0.3816) loss 3.5756 (3.4833) grad_norm 1.4742 (1.2978) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][330/625] eta 0:01:53 lr 0.001794 wd 0.0500 time 0.5894 (0.3836) data time 0.0010 (0.0025) model time 0.5884 (0.3822) loss 3.4951 (3.4801) grad_norm 0.8327 (1.2962) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][340/625] eta 0:01:49 lr 0.001794 wd 0.0500 time 0.3802 (0.3852) data time 0.0011 (0.0025) model time 0.3791 (0.3841) loss 3.6207 (3.4827) grad_norm 1.4432 (1.2937) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][350/625] eta 0:01:45 lr 0.001794 wd 0.0500 time 0.3760 (0.3850) data time 0.0008 (0.0024) model time 0.3752 (0.3838) loss 2.5791 (3.4831) grad_norm 1.7086 (1.2970) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][360/625] eta 0:01:41 lr 0.001794 wd 0.0500 time 0.3784 (0.3847) data time 0.0008 (0.0024) model time 0.3776 (0.3836) loss 2.2475 (3.4740) grad_norm 2.5268 (1.3048) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][370/625] eta 0:01:38 lr 0.001794 wd 0.0500 time 0.3765 (0.3845) data time 0.0008 (0.0024) model time 0.3757 (0.3834) loss 3.2805 (3.4778) grad_norm 1.2185 (1.3070) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][380/625] eta 0:01:34 lr 0.001794 wd 0.0500 time 0.3825 (0.3844) data time 0.0010 (0.0023) model time 0.3815 (0.3832) loss 3.7846 (3.4812) grad_norm 1.9294 (1.3141) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][390/625] eta 0:01:30 lr 0.001793 wd 0.0500 time 0.3735 (0.3843) data time 0.0008 (0.0023) model time 0.3727 (0.3830) loss 3.7860 (3.4842) grad_norm 1.0006 (1.3174) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][400/625] eta 0:01:26 lr 0.001793 wd 0.0500 time 0.3939 (0.3841) data time 0.0011 (0.0023) model time 0.3928 (0.3829) loss 2.3209 (3.4822) grad_norm 0.9746 (1.3134) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][410/625] eta 0:01:22 lr 0.001793 wd 0.0500 time 0.3759 (0.3840) data time 0.0009 (0.0022) model time 0.3750 (0.3827) loss 3.5587 (3.4816) grad_norm 0.9643 (1.3063) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][420/625] eta 0:01:18 lr 0.001793 wd 0.0500 time 0.3854 (0.3838) data time 0.0011 (0.0022) model time 0.3843 (0.3826) loss 3.6449 (3.4838) grad_norm 0.9804 (1.3068) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][430/625] eta 0:01:14 lr 0.001793 wd 0.0500 time 0.3764 (0.3837) data time 0.0011 (0.0022) model time 0.3753 (0.3824) loss 4.2083 (3.4850) grad_norm 1.5459 (1.3079) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][440/625] eta 0:01:10 lr 0.001793 wd 0.0500 time 0.3778 (0.3836) data time 0.0009 (0.0022) model time 0.3769 (0.3823) loss 3.7111 (3.4858) grad_norm 1.4789 (1.3065) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][450/625] eta 0:01:07 lr 0.001793 wd 0.0500 time 0.5610 (0.3839) data time 0.0011 (0.0022) model time 0.5599 (0.3826) loss 3.9488 (3.4852) grad_norm 3.0913 (1.3183) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][460/625] eta 0:01:03 lr 0.001793 wd 0.0500 time 0.3727 (0.3846) data time 0.0008 (0.0021) model time 0.3719 (0.3835) loss 2.8846 (3.4818) grad_norm 1.1266 (1.3204) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][470/625] eta 0:00:59 lr 0.001793 wd 0.0500 time 0.3792 (0.3845) data time 0.0009 (0.0021) model time 0.3784 (0.3833) loss 2.6618 (3.4796) grad_norm 1.1431 (1.3169) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][480/625] eta 0:00:55 lr 0.001792 wd 0.0500 time 0.3771 (0.3843) data time 0.0012 (0.0021) model time 0.3759 (0.3831) loss 2.8013 (3.4750) grad_norm 1.7814 (1.3150) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][490/625] eta 0:00:51 lr 0.001792 wd 0.0500 time 0.4070 (0.3847) data time 0.0012 (0.0021) model time 0.4059 (0.3835) loss 3.7305 (3.4721) grad_norm 1.0018 (1.3126) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][500/625] eta 0:00:48 lr 0.001792 wd 0.0500 time 0.3696 (0.3845) data time 0.0009 (0.0021) model time 0.3687 (0.3833) loss 2.1505 (3.4674) grad_norm 0.9448 (1.3112) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][510/625] eta 0:00:44 lr 0.001792 wd 0.0500 time 0.3749 (0.3844) data time 0.0012 (0.0020) model time 0.3738 (0.3832) loss 3.8522 (3.4734) grad_norm 1.6386 (1.3171) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][520/625] eta 0:00:40 lr 0.001792 wd 0.0500 time 0.3826 (0.3847) data time 0.0008 (0.0020) model time 0.3818 (0.3835) loss 2.2290 (3.4695) grad_norm 1.2839 (1.3180) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][530/625] eta 0:00:36 lr 0.001792 wd 0.0500 time 0.3756 (0.3845) data time 0.0011 (0.0020) model time 0.3745 (0.3833) loss 3.2638 (3.4667) grad_norm 1.8372 (1.3170) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][540/625] eta 0:00:32 lr 0.001792 wd 0.0500 time 0.3755 (0.3844) data time 0.0008 (0.0020) model time 0.3747 (0.3832) loss 2.4763 (3.4626) grad_norm 1.1366 (1.3169) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:17:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][550/625] eta 0:00:28 lr 0.001792 wd 0.0500 time 0.3733 (0.3843) data time 0.0009 (0.0020) model time 0.3725 (0.3830) loss 3.6442 (3.4628) grad_norm 2.0635 (1.3209) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][560/625] eta 0:00:24 lr 0.001792 wd 0.0500 time 0.3840 (0.3842) data time 0.0011 (0.0020) model time 0.3829 (0.3829) loss 4.0429 (3.4632) grad_norm 1.0072 (1.3183) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][570/625] eta 0:00:21 lr 0.001791 wd 0.0500 time 0.3749 (0.3840) data time 0.0012 (0.0020) model time 0.3737 (0.3828) loss 3.6797 (3.4677) grad_norm 0.9715 (1.3174) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][580/625] eta 0:00:17 lr 0.001791 wd 0.0500 time 0.3753 (0.3839) data time 0.0012 (0.0019) model time 0.3741 (0.3826) loss 3.1246 (3.4625) grad_norm 1.1804 (1.3161) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][590/625] eta 0:00:13 lr 0.001791 wd 0.0500 time 0.3746 (0.3838) data time 0.0011 (0.0019) model time 0.3735 (0.3825) loss 3.0191 (3.4602) grad_norm 1.4469 (1.3145) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][600/625] eta 0:00:09 lr 0.001791 wd 0.0500 time 0.3771 (0.3837) data time 0.0011 (0.0019) model time 0.3760 (0.3824) loss 3.7924 (3.4594) grad_norm 2.1812 (1.3192) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][610/625] eta 0:00:05 lr 0.001791 wd 0.0500 time 0.3704 (0.3839) data time 0.0008 (0.0019) model time 0.3696 (0.3826) loss 3.2525 (3.4588) grad_norm 1.6416 (1.3197) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [78/300][620/625] eta 0:00:01 lr 0.001791 wd 0.0500 time 0.3675 (0.3836) data time 0.0006 (0.0019) model time 0.3669 (0.3824) loss 3.4753 (3.4594) grad_norm 1.5811 (1.3189) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 78 training takes 0:03:59 [2024-08-26 08:18:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:18:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:18:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.510 (0.510) Loss 0.6377 (0.6377) Acc@1 85.449 (85.449) Acc@5 97.510 (97.510) Mem 11225MB [2024-08-26 08:18:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.145) Loss 1.1670 (0.8141) Acc@1 72.119 (80.975) Acc@5 91.650 (95.952) Mem 11225MB [2024-08-26 08:18:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.2676 (0.9797) Acc@1 69.873 (77.181) Acc@5 90.381 (93.834) Mem 11225MB [2024-08-26 08:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.963 Acc@5 93.808 [2024-08-26 08:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.0% [2024-08-26 08:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 76.96% [2024-08-26 08:18:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 08:18:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 08:18:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.507 (0.507) Loss 0.5654 (0.5654) Acc@1 88.037 (88.037) Acc@5 98.145 (98.145) Mem 11225MB [2024-08-26 08:18:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 0.9473 (0.7065) Acc@1 76.709 (83.798) Acc@5 94.775 (96.955) Mem 11225MB [2024-08-26 08:18:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.1152 (0.8650) Acc@1 72.119 (79.811) Acc@5 92.480 (95.140) Mem 11225MB [2024-08-26 08:18:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.521 Acc@5 95.088 [2024-08-26 08:18:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.5% [2024-08-26 08:18:36 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.52% [2024-08-26 08:18:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:18:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:18:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][0/625] eta 0:08:01 lr 0.001791 wd 0.0500 time 0.7711 (0.7711) data time 0.4347 (0.4347) model time 0.0000 (0.0000) loss 3.5178 (3.5178) grad_norm 1.2633 (1.2633) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][10/625] eta 0:04:13 lr 0.001791 wd 0.0500 time 0.3839 (0.4116) data time 0.0009 (0.0406) model time 0.0000 (0.0000) loss 4.2404 (3.5537) grad_norm 1.1178 (1.1826) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][20/625] eta 0:04:03 lr 0.001791 wd 0.0500 time 0.5135 (0.4020) data time 0.0009 (0.0218) model time 0.0000 (0.0000) loss 3.6718 (3.6027) grad_norm 1.6911 (1.3567) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][30/625] eta 0:04:01 lr 0.001791 wd 0.0500 time 0.3781 (0.4062) data time 0.0011 (0.0152) model time 0.0000 (0.0000) loss 3.2391 (3.4952) grad_norm 1.0783 (1.2981) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][40/625] eta 0:03:53 lr 0.001790 wd 0.0500 time 0.3784 (0.3986) data time 0.0010 (0.0117) model time 0.0000 (0.0000) loss 3.5230 (3.4769) grad_norm 1.4611 (1.3158) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:18:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][50/625] eta 0:03:46 lr 0.001790 wd 0.0500 time 0.3724 (0.3941) data time 0.0008 (0.0097) model time 0.0000 (0.0000) loss 3.4960 (3.4625) grad_norm 1.2447 (1.2990) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][60/625] eta 0:03:41 lr 0.001790 wd 0.0500 time 0.3760 (0.3917) data time 0.0011 (0.0083) model time 0.3749 (0.3784) loss 3.4453 (3.5175) grad_norm 1.2653 (1.3025) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][70/625] eta 0:03:36 lr 0.001790 wd 0.0500 time 0.3731 (0.3897) data time 0.0011 (0.0073) model time 0.3720 (0.3773) loss 3.8177 (3.4969) grad_norm 1.4482 (1.2978) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][80/625] eta 0:03:31 lr 0.001790 wd 0.0500 time 0.3765 (0.3879) data time 0.0008 (0.0065) model time 0.3756 (0.3764) loss 2.6770 (3.5166) grad_norm 1.1157 (1.2938) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][90/625] eta 0:03:27 lr 0.001790 wd 0.0500 time 0.3780 (0.3869) data time 0.0011 (0.0059) model time 0.3769 (0.3767) loss 3.8854 (3.4876) grad_norm 0.9035 (1.3090) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][100/625] eta 0:03:22 lr 0.001790 wd 0.0500 time 0.3776 (0.3860) data time 0.0010 (0.0054) model time 0.3767 (0.3767) loss 3.7142 (3.4591) grad_norm 1.5707 (1.3228) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][110/625] eta 0:03:18 lr 0.001790 wd 0.0500 time 0.3811 (0.3853) data time 0.0012 (0.0050) model time 0.3799 (0.3767) loss 3.8264 (3.4352) grad_norm 1.6312 (1.3338) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][120/625] eta 0:03:14 lr 0.001790 wd 0.0500 time 0.3770 (0.3846) data time 0.0008 (0.0047) model time 0.3762 (0.3767) loss 2.5126 (3.4229) grad_norm 1.1014 (1.3260) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][130/625] eta 0:03:10 lr 0.001789 wd 0.0500 time 0.3855 (0.3843) data time 0.0008 (0.0044) model time 0.3847 (0.3770) loss 3.6084 (3.4268) grad_norm 1.2370 (1.3415) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][140/625] eta 0:03:06 lr 0.001789 wd 0.0500 time 0.3816 (0.3851) data time 0.0011 (0.0042) model time 0.3805 (0.3790) loss 3.8011 (3.4516) grad_norm 1.7296 (1.3442) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][150/625] eta 0:03:02 lr 0.001789 wd 0.0500 time 0.3775 (0.3847) data time 0.0011 (0.0040) model time 0.3765 (0.3788) loss 3.0844 (3.4529) grad_norm 1.0178 (1.3435) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][160/625] eta 0:02:58 lr 0.001789 wd 0.0500 time 0.3796 (0.3843) data time 0.0011 (0.0038) model time 0.3785 (0.3786) loss 3.6105 (3.4553) grad_norm 1.6898 (1.3504) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][170/625] eta 0:02:55 lr 0.001789 wd 0.0500 time 0.3784 (0.3850) data time 0.0009 (0.0037) model time 0.3775 (0.3800) loss 3.9232 (3.4566) grad_norm 1.3606 (1.3593) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][180/625] eta 0:02:51 lr 0.001789 wd 0.0500 time 0.3783 (0.3847) data time 0.0011 (0.0035) model time 0.3772 (0.3799) loss 3.6681 (3.4612) grad_norm 0.9103 (1.3525) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][190/625] eta 0:02:47 lr 0.001789 wd 0.0500 time 0.3799 (0.3844) data time 0.0010 (0.0034) model time 0.3789 (0.3798) loss 3.3126 (3.4482) grad_norm 1.2137 (1.3746) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][200/625] eta 0:02:43 lr 0.001789 wd 0.0500 time 0.3756 (0.3841) data time 0.0009 (0.0033) model time 0.3747 (0.3797) loss 4.0434 (3.4473) grad_norm 1.4063 (1.3736) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][210/625] eta 0:02:39 lr 0.001789 wd 0.0500 time 0.3768 (0.3839) data time 0.0009 (0.0032) model time 0.3759 (0.3795) loss 3.2832 (3.4540) grad_norm 1.2055 (1.3652) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][220/625] eta 0:02:35 lr 0.001788 wd 0.0500 time 0.3777 (0.3836) data time 0.0010 (0.0031) model time 0.3767 (0.3793) loss 3.8682 (3.4629) grad_norm 1.3641 (1.3735) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][230/625] eta 0:02:31 lr 0.001788 wd 0.0500 time 0.5962 (0.3843) data time 0.0011 (0.0030) model time 0.5951 (0.3804) loss 2.9575 (3.4591) grad_norm 1.1286 (1.3649) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][240/625] eta 0:02:28 lr 0.001788 wd 0.0500 time 0.3806 (0.3849) data time 0.0011 (0.0029) model time 0.3795 (0.3814) loss 2.5495 (3.4540) grad_norm 1.7051 (1.3684) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][250/625] eta 0:02:24 lr 0.001788 wd 0.0500 time 0.3844 (0.3846) data time 0.0010 (0.0029) model time 0.3834 (0.3811) loss 3.1369 (3.4583) grad_norm 1.4725 (1.3688) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][260/625] eta 0:02:20 lr 0.001788 wd 0.0500 time 0.3870 (0.3844) data time 0.0011 (0.0028) model time 0.3858 (0.3809) loss 3.0477 (3.4524) grad_norm 1.5875 (1.3643) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][270/625] eta 0:02:16 lr 0.001788 wd 0.0500 time 0.3812 (0.3840) data time 0.0011 (0.0027) model time 0.3801 (0.3806) loss 3.3587 (3.4495) grad_norm 1.9468 (1.3593) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][280/625] eta 0:02:12 lr 0.001788 wd 0.0500 time 0.3814 (0.3838) data time 0.0011 (0.0027) model time 0.3803 (0.3804) loss 3.8272 (3.4497) grad_norm 1.6429 (1.3668) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][290/625] eta 0:02:08 lr 0.001788 wd 0.0500 time 0.3830 (0.3843) data time 0.0011 (0.0026) model time 0.3820 (0.3811) loss 2.6974 (3.4494) grad_norm 1.0267 (1.3622) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][300/625] eta 0:02:04 lr 0.001788 wd 0.0500 time 0.3767 (0.3842) data time 0.0011 (0.0026) model time 0.3755 (0.3810) loss 3.5903 (3.4496) grad_norm 1.0514 (1.3593) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][310/625] eta 0:02:00 lr 0.001787 wd 0.0500 time 0.3797 (0.3839) data time 0.0010 (0.0025) model time 0.3787 (0.3808) loss 3.2086 (3.4431) grad_norm 1.2625 (1.3597) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][320/625] eta 0:01:57 lr 0.001787 wd 0.0500 time 0.3750 (0.3843) data time 0.0008 (0.0025) model time 0.3741 (0.3813) loss 3.2042 (3.4410) grad_norm 1.1644 (1.3517) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][330/625] eta 0:01:53 lr 0.001787 wd 0.0500 time 0.3781 (0.3841) data time 0.0010 (0.0025) model time 0.3771 (0.3811) loss 2.7287 (3.4405) grad_norm 1.4668 (1.3458) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][340/625] eta 0:01:49 lr 0.001787 wd 0.0500 time 0.3739 (0.3838) data time 0.0012 (0.0024) model time 0.3728 (0.3809) loss 3.7410 (3.4528) grad_norm 0.9188 (1.3401) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][350/625] eta 0:01:45 lr 0.001787 wd 0.0500 time 0.3722 (0.3836) data time 0.0010 (0.0024) model time 0.3711 (0.3807) loss 3.4825 (3.4566) grad_norm 1.2582 (1.3364) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][360/625] eta 0:01:41 lr 0.001787 wd 0.0500 time 0.3773 (0.3835) data time 0.0009 (0.0023) model time 0.3764 (0.3806) loss 3.9515 (3.4680) grad_norm 1.4493 (1.3413) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:20:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][370/625] eta 0:01:37 lr 0.001787 wd 0.0500 time 0.3689 (0.3833) data time 0.0009 (0.0023) model time 0.3680 (0.3804) loss 3.9519 (3.4703) grad_norm 1.0913 (1.3387) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][380/625] eta 0:01:33 lr 0.001787 wd 0.0500 time 0.3712 (0.3831) data time 0.0011 (0.0023) model time 0.3702 (0.3803) loss 3.4288 (3.4725) grad_norm 1.3445 (1.3326) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][390/625] eta 0:01:29 lr 0.001787 wd 0.0500 time 0.3721 (0.3829) data time 0.0009 (0.0022) model time 0.3712 (0.3801) loss 3.6307 (3.4713) grad_norm 0.9571 (1.3312) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][400/625] eta 0:01:26 lr 0.001787 wd 0.0500 time 0.3741 (0.3828) data time 0.0011 (0.0022) model time 0.3730 (0.3800) loss 3.0556 (3.4657) grad_norm 1.3418 (1.3295) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][410/625] eta 0:01:22 lr 0.001786 wd 0.0500 time 0.3785 (0.3827) data time 0.0009 (0.0022) model time 0.3776 (0.3799) loss 4.2791 (3.4623) grad_norm 1.4529 (1.3289) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][420/625] eta 0:01:18 lr 0.001786 wd 0.0500 time 0.3956 (0.3836) data time 0.0010 (0.0022) model time 0.3946 (0.3810) loss 2.9578 (3.4586) grad_norm 0.9780 (1.3299) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][430/625] eta 0:01:14 lr 0.001786 wd 0.0500 time 0.3739 (0.3835) data time 0.0011 (0.0021) model time 0.3728 (0.3809) loss 3.9679 (3.4599) grad_norm 1.0742 (1.3260) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][440/625] eta 0:01:10 lr 0.001786 wd 0.0500 time 0.3772 (0.3837) data time 0.0011 (0.0021) model time 0.3761 (0.3812) loss 2.8836 (3.4568) grad_norm 1.6248 (1.3264) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][450/625] eta 0:01:07 lr 0.001786 wd 0.0500 time 0.3793 (0.3836) data time 0.0012 (0.0021) model time 0.3782 (0.3812) loss 3.9350 (3.4602) grad_norm 1.1128 (1.3285) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][460/625] eta 0:01:03 lr 0.001786 wd 0.0500 time 0.3743 (0.3835) data time 0.0008 (0.0021) model time 0.3735 (0.3810) loss 4.0560 (3.4661) grad_norm 1.2666 (1.3239) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][470/625] eta 0:00:59 lr 0.001786 wd 0.0500 time 0.3802 (0.3834) data time 0.0010 (0.0021) model time 0.3792 (0.3810) loss 3.6194 (3.4633) grad_norm 1.1983 (1.3223) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][480/625] eta 0:00:55 lr 0.001786 wd 0.0500 time 0.3753 (0.3834) data time 0.0008 (0.0020) model time 0.3745 (0.3809) loss 3.0325 (3.4573) grad_norm 1.0991 (1.3217) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][490/625] eta 0:00:51 lr 0.001786 wd 0.0500 time 0.3777 (0.3833) data time 0.0012 (0.0020) model time 0.3765 (0.3809) loss 3.5469 (3.4578) grad_norm 1.1949 (1.3187) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][500/625] eta 0:00:47 lr 0.001785 wd 0.0500 time 0.3827 (0.3836) data time 0.0009 (0.0020) model time 0.3818 (0.3813) loss 3.7719 (3.4587) grad_norm 1.1596 (1.3165) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][510/625] eta 0:00:44 lr 0.001785 wd 0.0500 time 0.3849 (0.3835) data time 0.0011 (0.0020) model time 0.3837 (0.3812) loss 3.6806 (3.4547) grad_norm 1.2142 (1.3134) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:21:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][520/625] eta 0:00:40 lr 0.001785 wd 0.0500 time 0.3749 (0.3834) data time 0.0011 (0.0020) model time 0.3739 (0.3811) loss 3.7304 (3.4567) grad_norm 1.2458 (1.3131) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][530/625] eta 0:00:36 lr 0.001785 wd 0.0500 time 0.3780 (0.3833) data time 0.0011 (0.0020) model time 0.3770 (0.3810) loss 3.5314 (3.4599) grad_norm 1.1494 (1.3091) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][540/625] eta 0:00:32 lr 0.001785 wd 0.0500 time 0.3773 (0.3839) data time 0.0010 (0.0019) model time 0.3763 (0.3817) loss 3.1209 (3.4648) grad_norm 1.3097 (1.3093) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][550/625] eta 0:00:28 lr 0.001785 wd 0.0500 time 0.3834 (0.3838) data time 0.0008 (0.0019) model time 0.3826 (0.3816) loss 4.1942 (3.4672) grad_norm 1.6718 (1.3095) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][560/625] eta 0:00:24 lr 0.001785 wd 0.0500 time 0.3730 (0.3837) data time 0.0011 (0.0019) model time 0.3719 (0.3815) loss 3.6761 (3.4708) grad_norm 1.6933 (1.3106) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][570/625] eta 0:00:21 lr 0.001785 wd 0.0500 time 0.3741 (0.3835) data time 0.0009 (0.0019) model time 0.3732 (0.3814) loss 3.2676 (3.4706) grad_norm 1.0511 (1.3091) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][580/625] eta 0:00:17 lr 0.001785 wd 0.0500 time 0.3736 (0.3834) data time 0.0011 (0.0019) model time 0.3725 (0.3813) loss 3.2674 (3.4746) grad_norm 1.3574 (1.3083) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][590/625] eta 0:00:13 lr 0.001784 wd 0.0500 time 0.3766 (0.3834) data time 0.0012 (0.0019) model time 0.3754 (0.3812) loss 3.5569 (3.4750) grad_norm 1.7067 (1.3102) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][600/625] eta 0:00:09 lr 0.001784 wd 0.0500 time 0.3757 (0.3833) data time 0.0009 (0.0019) model time 0.3748 (0.3811) loss 3.9339 (3.4754) grad_norm 1.5321 (1.3146) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][610/625] eta 0:00:05 lr 0.001784 wd 0.0500 time 0.3675 (0.3831) data time 0.0006 (0.0019) model time 0.3669 (0.3810) loss 2.2374 (3.4706) grad_norm 1.0987 (1.3133) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [79/300][620/625] eta 0:00:01 lr 0.001784 wd 0.0500 time 0.3680 (0.3828) data time 0.0008 (0.0018) model time 0.3672 (0.3807) loss 3.1141 (3.4684) grad_norm 1.7996 (1.3146) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 79 training takes 0:03:59 [2024-08-26 08:22:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:22:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:22:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 0.6479 (0.6479) Acc@1 85.840 (85.840) Acc@5 97.168 (97.168) Mem 11225MB [2024-08-26 08:22:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 1.0713 (0.8127) Acc@1 74.756 (80.988) Acc@5 93.262 (96.094) Mem 11225MB [2024-08-26 08:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.2109 (0.9833) Acc@1 70.361 (77.176) Acc@5 92.188 (93.952) Mem 11225MB [2024-08-26 08:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.895 Acc@5 93.880 [2024-08-26 08:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.9% [2024-08-26 08:22:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.885 (0.885) Loss 0.5654 (0.5654) Acc@1 88.086 (88.086) Acc@5 98.193 (98.193) Mem 11225MB [2024-08-26 08:22:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.181) Loss 0.9478 (0.7064) Acc@1 76.953 (83.807) Acc@5 94.629 (96.973) Mem 11225MB [2024-08-26 08:22:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.140) Loss 1.1123 (0.8634) Acc@1 72.119 (79.855) Acc@5 92.480 (95.152) Mem 11225MB [2024-08-26 08:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.565 Acc@5 95.098 [2024-08-26 08:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.6% [2024-08-26 08:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.56% [2024-08-26 08:22:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:22:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:22:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][0/625] eta 0:08:05 lr 0.001784 wd 0.0500 time 0.7773 (0.7773) data time 0.4480 (0.4480) model time 0.0000 (0.0000) loss 3.8350 (3.8350) grad_norm 2.2683 (2.2683) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][10/625] eta 0:04:14 lr 0.001784 wd 0.0500 time 0.3788 (0.4134) data time 0.0011 (0.0418) model time 0.0000 (0.0000) loss 3.4491 (3.3018) grad_norm 1.3178 (1.2430) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][20/625] eta 0:03:59 lr 0.001784 wd 0.0500 time 0.3760 (0.3957) data time 0.0009 (0.0224) model time 0.0000 (0.0000) loss 3.3218 (3.4237) grad_norm 1.1993 (1.2137) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:22:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][30/625] eta 0:03:51 lr 0.001784 wd 0.0500 time 0.3779 (0.3897) data time 0.0008 (0.0156) model time 0.0000 (0.0000) loss 4.1907 (3.4966) grad_norm 1.2144 (1.2497) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][40/625] eta 0:03:46 lr 0.001784 wd 0.0500 time 0.3774 (0.3869) data time 0.0008 (0.0120) model time 0.0000 (0.0000) loss 4.0423 (3.4849) grad_norm 1.0160 (1.4905) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][50/625] eta 0:03:41 lr 0.001783 wd 0.0500 time 0.3724 (0.3852) data time 0.0009 (0.0099) model time 0.0000 (0.0000) loss 4.3446 (3.4541) grad_norm 0.9498 (1.4070) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][60/625] eta 0:03:36 lr 0.001783 wd 0.0500 time 0.3729 (0.3836) data time 0.0011 (0.0085) model time 0.3718 (0.3743) loss 4.2189 (3.4307) grad_norm 1.7541 (1.3971) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][70/625] eta 0:03:36 lr 0.001783 wd 0.0500 time 0.3734 (0.3909) data time 0.0008 (0.0075) model time 0.3725 (0.4044) loss 2.8781 (3.4223) grad_norm 1.6195 (1.4311) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][80/625] eta 0:03:32 lr 0.001783 wd 0.0500 time 0.3641 (0.3894) data time 0.0008 (0.0067) model time 0.3633 (0.3953) loss 3.5461 (3.3973) grad_norm 0.9561 (1.4111) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][90/625] eta 0:03:27 lr 0.001783 wd 0.0500 time 0.3724 (0.3883) data time 0.0011 (0.0061) model time 0.3713 (0.3910) loss 3.9110 (3.4163) grad_norm 1.1911 (1.3865) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][100/625] eta 0:03:23 lr 0.001783 wd 0.0500 time 0.3725 (0.3871) data time 0.0008 (0.0056) model time 0.3717 (0.3878) loss 3.9958 (3.4258) grad_norm 1.3350 (1.3612) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][110/625] eta 0:03:19 lr 0.001783 wd 0.0500 time 0.3803 (0.3865) data time 0.0011 (0.0052) model time 0.3792 (0.3864) loss 3.1169 (3.4286) grad_norm 1.4463 (1.3894) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][120/625] eta 0:03:14 lr 0.001783 wd 0.0500 time 0.3712 (0.3858) data time 0.0011 (0.0049) model time 0.3701 (0.3850) loss 3.6742 (3.4467) grad_norm 0.9901 (1.4015) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][130/625] eta 0:03:10 lr 0.001783 wd 0.0500 time 0.3823 (0.3853) data time 0.0011 (0.0046) model time 0.3812 (0.3842) loss 3.7198 (3.4284) grad_norm 1.2905 (1.3874) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][140/625] eta 0:03:06 lr 0.001782 wd 0.0500 time 0.3740 (0.3848) data time 0.0011 (0.0043) model time 0.3729 (0.3834) loss 3.8734 (3.4200) grad_norm 1.0621 (1.3750) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][150/625] eta 0:03:02 lr 0.001782 wd 0.0500 time 0.3715 (0.3842) data time 0.0008 (0.0041) model time 0.3707 (0.3826) loss 3.7308 (3.4119) grad_norm 1.3611 (1.3636) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][160/625] eta 0:02:58 lr 0.001782 wd 0.0500 time 0.3740 (0.3838) data time 0.0009 (0.0039) model time 0.3730 (0.3820) loss 4.1053 (3.4131) grad_norm 0.9936 (1.3536) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][170/625] eta 0:02:54 lr 0.001782 wd 0.0500 time 0.3680 (0.3843) data time 0.0009 (0.0038) model time 0.3671 (0.3827) loss 4.0920 (3.4253) grad_norm 1.3155 (1.3435) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][180/625] eta 0:02:50 lr 0.001782 wd 0.0500 time 0.3779 (0.3839) data time 0.0011 (0.0036) model time 0.3768 (0.3822) loss 3.1363 (3.4151) grad_norm 1.0705 (1.3388) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:23:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][190/625] eta 0:02:46 lr 0.001782 wd 0.0500 time 0.3748 (0.3835) data time 0.0010 (0.0035) model time 0.3738 (0.3817) loss 2.8545 (3.4119) grad_norm 1.3612 (1.3364) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][200/625] eta 0:02:42 lr 0.001782 wd 0.0500 time 0.3735 (0.3832) data time 0.0008 (0.0034) model time 0.3727 (0.3813) loss 2.3099 (3.4175) grad_norm 1.1816 (1.3415) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][210/625] eta 0:02:38 lr 0.001782 wd 0.0500 time 0.3691 (0.3828) data time 0.0009 (0.0033) model time 0.3682 (0.3809) loss 4.3905 (3.4222) grad_norm 1.2747 (1.3446) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][220/625] eta 0:02:34 lr 0.001782 wd 0.0500 time 0.3792 (0.3826) data time 0.0008 (0.0032) model time 0.3784 (0.3807) loss 3.6397 (3.4155) grad_norm 1.2842 (1.3458) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][230/625] eta 0:02:31 lr 0.001781 wd 0.0500 time 0.3739 (0.3823) data time 0.0008 (0.0031) model time 0.3730 (0.3804) loss 2.5546 (3.4048) grad_norm 1.5521 (1.3500) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][240/625] eta 0:02:27 lr 0.001781 wd 0.0500 time 0.3897 (0.3823) data time 0.0009 (0.0030) model time 0.3888 (0.3804) loss 3.4680 (3.4007) grad_norm 0.9959 (1.3513) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][250/625] eta 0:02:23 lr 0.001781 wd 0.0500 time 0.5629 (0.3837) data time 0.0008 (0.0029) model time 0.5621 (0.3822) loss 4.2365 (3.3996) grad_norm 1.2603 (1.3448) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][260/625] eta 0:02:19 lr 0.001781 wd 0.0500 time 0.3818 (0.3835) data time 0.0010 (0.0029) model time 0.3808 (0.3819) loss 3.8986 (3.4040) grad_norm 1.1238 (1.3446) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][270/625] eta 0:02:16 lr 0.001781 wd 0.0500 time 0.3837 (0.3833) data time 0.0011 (0.0028) model time 0.3826 (0.3817) loss 3.7392 (3.3986) grad_norm 1.1969 (1.3395) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][280/625] eta 0:02:12 lr 0.001781 wd 0.0500 time 0.5919 (0.3847) data time 0.0010 (0.0027) model time 0.5909 (0.3834) loss 2.7225 (3.3893) grad_norm 1.5825 (1.3343) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][290/625] eta 0:02:08 lr 0.001781 wd 0.0500 time 0.3774 (0.3851) data time 0.0011 (0.0027) model time 0.3763 (0.3839) loss 3.2909 (3.3860) grad_norm 1.7331 (1.3403) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][300/625] eta 0:02:05 lr 0.001781 wd 0.0500 time 0.3776 (0.3848) data time 0.0008 (0.0026) model time 0.3768 (0.3836) loss 4.6533 (3.3833) grad_norm 1.4858 (1.3349) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][310/625] eta 0:02:01 lr 0.001781 wd 0.0500 time 0.3777 (0.3846) data time 0.0008 (0.0026) model time 0.3769 (0.3834) loss 3.9596 (3.3903) grad_norm 1.2276 (1.3329) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][320/625] eta 0:01:57 lr 0.001780 wd 0.0500 time 0.3714 (0.3844) data time 0.0010 (0.0025) model time 0.3704 (0.3831) loss 3.3681 (3.3905) grad_norm 1.7345 (1.3295) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][330/625] eta 0:01:53 lr 0.001780 wd 0.0500 time 0.3725 (0.3842) data time 0.0008 (0.0025) model time 0.3717 (0.3828) loss 2.7503 (3.3853) grad_norm 1.0146 (1.3246) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:24:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][340/625] eta 0:01:49 lr 0.001780 wd 0.0500 time 0.3760 (0.3840) data time 0.0010 (0.0025) model time 0.3750 (0.3826) loss 3.7764 (3.3862) grad_norm 1.2926 (1.3202) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][350/625] eta 0:01:45 lr 0.001780 wd 0.0500 time 0.3726 (0.3838) data time 0.0010 (0.0024) model time 0.3716 (0.3824) loss 2.2963 (3.3889) grad_norm 1.5246 (1.3247) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][360/625] eta 0:01:41 lr 0.001780 wd 0.0500 time 0.3768 (0.3836) data time 0.0008 (0.0024) model time 0.3760 (0.3822) loss 4.3165 (3.3911) grad_norm 0.8155 (1.3227) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][370/625] eta 0:01:38 lr 0.001780 wd 0.0500 time 0.5918 (0.3846) data time 0.0011 (0.0023) model time 0.5907 (0.3834) loss 3.5499 (3.3918) grad_norm 1.0587 (1.3182) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][380/625] eta 0:01:34 lr 0.001780 wd 0.0500 time 0.3749 (0.3844) data time 0.0012 (0.0023) model time 0.3737 (0.3832) loss 2.1924 (3.3931) grad_norm 1.0864 (1.3155) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][390/625] eta 0:01:30 lr 0.001780 wd 0.0500 time 0.3742 (0.3842) data time 0.0010 (0.0023) model time 0.3732 (0.3830) loss 3.7715 (3.3949) grad_norm 1.3189 (1.3178) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][400/625] eta 0:01:26 lr 0.001780 wd 0.0500 time 0.3788 (0.3841) data time 0.0008 (0.0023) model time 0.3780 (0.3828) loss 3.3474 (3.3932) grad_norm 1.3593 (1.3172) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][410/625] eta 0:01:22 lr 0.001779 wd 0.0500 time 0.3726 (0.3840) data time 0.0011 (0.0022) model time 0.3716 (0.3826) loss 3.0234 (3.3914) grad_norm 1.5278 (1.3198) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][420/625] eta 0:01:18 lr 0.001779 wd 0.0500 time 0.3731 (0.3838) data time 0.0011 (0.0022) model time 0.3720 (0.3825) loss 3.4164 (3.3945) grad_norm 1.7216 (1.3225) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][430/625] eta 0:01:14 lr 0.001779 wd 0.0500 time 0.3813 (0.3837) data time 0.0008 (0.0022) model time 0.3805 (0.3824) loss 2.3300 (3.3991) grad_norm 0.8812 (1.3215) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][440/625] eta 0:01:10 lr 0.001779 wd 0.0500 time 0.3735 (0.3836) data time 0.0008 (0.0022) model time 0.3726 (0.3822) loss 3.6795 (3.4051) grad_norm 0.8995 (1.3166) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][450/625] eta 0:01:07 lr 0.001779 wd 0.0500 time 0.3788 (0.3834) data time 0.0009 (0.0021) model time 0.3779 (0.3821) loss 3.4165 (3.4032) grad_norm 1.6583 (1.3131) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][460/625] eta 0:01:03 lr 0.001779 wd 0.0500 time 0.3740 (0.3838) data time 0.0011 (0.0021) model time 0.3729 (0.3824) loss 3.3817 (3.4094) grad_norm 1.2235 (1.3141) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][470/625] eta 0:00:59 lr 0.001779 wd 0.0500 time 0.3816 (0.3841) data time 0.0008 (0.0021) model time 0.3808 (0.3828) loss 3.8513 (3.4136) grad_norm 1.8689 (1.3167) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 08:25:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][480/625] eta 0:00:55 lr 0.001779 wd 0.0500 time 0.3750 (0.3840) data time 0.0008 (0.0021) model time 0.3741 (0.3827) loss 3.3325 (3.4155) grad_norm 1.0545 (1.3163) loss_scale 4096.0000 (2052.2578) mem 11225MB [2024-08-26 08:25:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][490/625] eta 0:00:51 lr 0.001779 wd 0.0500 time 0.3927 (0.3839) data time 0.0008 (0.0020) model time 0.3919 (0.3826) loss 3.9876 (3.4170) grad_norm 1.2301 (1.3134) loss_scale 4096.0000 (2093.8819) mem 11225MB [2024-08-26 08:25:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][500/625] eta 0:00:48 lr 0.001778 wd 0.0500 time 0.3724 (0.3841) data time 0.0014 (0.0020) model time 0.3711 (0.3828) loss 3.0431 (3.4148) grad_norm 1.2320 (1.3134) loss_scale 4096.0000 (2133.8443) mem 11225MB [2024-08-26 08:26:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][510/625] eta 0:00:44 lr 0.001778 wd 0.0500 time 0.3906 (0.3840) data time 0.0011 (0.0020) model time 0.3895 (0.3827) loss 4.0816 (3.4174) grad_norm 1.1959 (1.3131) loss_scale 4096.0000 (2172.2427) mem 11225MB [2024-08-26 08:26:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][520/625] eta 0:00:40 lr 0.001778 wd 0.0500 time 0.3866 (0.3839) data time 0.0011 (0.0020) model time 0.3855 (0.3826) loss 3.6931 (3.4241) grad_norm 1.1230 (1.3119) loss_scale 4096.0000 (2209.1670) mem 11225MB [2024-08-26 08:26:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][530/625] eta 0:00:36 lr 0.001778 wd 0.0500 time 0.3790 (0.3837) data time 0.0009 (0.0020) model time 0.3782 (0.3824) loss 4.2014 (3.4291) grad_norm 0.9747 (1.3116) loss_scale 4096.0000 (2244.7006) mem 11225MB [2024-08-26 08:26:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][540/625] eta 0:00:32 lr 0.001778 wd 0.0500 time 0.3776 (0.3836) data time 0.0011 (0.0020) model time 0.3765 (0.3823) loss 3.0832 (3.4310) grad_norm 1.1790 (1.3139) loss_scale 4096.0000 (2278.9205) mem 11225MB [2024-08-26 08:26:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][550/625] eta 0:00:28 lr 0.001778 wd 0.0500 time 0.3740 (0.3835) data time 0.0009 (0.0019) model time 0.3731 (0.3822) loss 3.9885 (3.4363) grad_norm 1.1931 (1.3114) loss_scale 4096.0000 (2311.8984) mem 11225MB [2024-08-26 08:26:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][560/625] eta 0:00:24 lr 0.001778 wd 0.0500 time 0.3745 (0.3834) data time 0.0009 (0.0019) model time 0.3736 (0.3821) loss 2.6566 (3.4363) grad_norm 1.0447 (1.3101) loss_scale 4096.0000 (2343.7005) mem 11225MB [2024-08-26 08:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][570/625] eta 0:00:21 lr 0.001778 wd 0.0500 time 0.3888 (0.3834) data time 0.0008 (0.0019) model time 0.3880 (0.3821) loss 3.6599 (3.4401) grad_norm 1.3381 (1.3113) loss_scale 4096.0000 (2374.3888) mem 11225MB [2024-08-26 08:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][580/625] eta 0:00:17 lr 0.001778 wd 0.0500 time 0.3733 (0.3843) data time 0.0011 (0.0019) model time 0.3722 (0.3831) loss 3.5675 (3.4414) grad_norm 1.0795 (1.3097) loss_scale 4096.0000 (2404.0207) mem 11225MB [2024-08-26 08:26:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][590/625] eta 0:00:13 lr 0.001777 wd 0.0500 time 0.3841 (0.3846) data time 0.0011 (0.0019) model time 0.3830 (0.3834) loss 3.3903 (3.4480) grad_norm 1.2571 (1.3098) loss_scale 4096.0000 (2432.6497) mem 11225MB [2024-08-26 08:26:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][600/625] eta 0:00:09 lr 0.001777 wd 0.0500 time 0.3897 (0.3845) data time 0.0008 (0.0019) model time 0.3889 (0.3833) loss 3.6529 (3.4488) grad_norm 1.5277 (1.3170) loss_scale 4096.0000 (2460.3261) mem 11225MB [2024-08-26 08:26:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][610/625] eta 0:00:05 lr 0.001777 wd 0.0500 time 0.3692 (0.3844) data time 0.0006 (0.0019) model time 0.3686 (0.3831) loss 3.1529 (3.4492) grad_norm 1.2522 (1.3158) loss_scale 4096.0000 (2487.0966) mem 11225MB [2024-08-26 08:26:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [80/300][620/625] eta 0:00:01 lr 0.001777 wd 0.0500 time 0.3705 (0.3842) data time 0.0008 (0.0019) model time 0.3697 (0.3829) loss 2.9471 (3.4494) grad_norm 1.1230 (1.3136) loss_scale 4096.0000 (2513.0048) mem 11225MB [2024-08-26 08:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 80 training takes 0:04:00 [2024-08-26 08:26:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:26:46 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:26:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.512 (0.512) Loss 0.6543 (0.6543) Acc@1 85.059 (85.059) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 08:26:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.1133 (0.8181) Acc@1 73.291 (81.143) Acc@5 93.213 (96.222) Mem 11225MB [2024-08-26 08:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.2549 (0.9881) Acc@1 70.654 (77.244) Acc@5 90.039 (93.945) Mem 11225MB [2024-08-26 08:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.015 Acc@5 93.866 [2024-08-26 08:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.0% [2024-08-26 08:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 77.02% [2024-08-26 08:26:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 08:26:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 08:26:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.5645 (0.5645) Acc@1 87.939 (87.939) Acc@5 98.242 (98.242) Mem 11225MB [2024-08-26 08:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 0.9443 (0.7050) Acc@1 76.904 (83.798) Acc@5 94.775 (96.995) Mem 11225MB [2024-08-26 08:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.1104 (0.8616) Acc@1 72.412 (79.911) Acc@5 92.725 (95.208) Mem 11225MB [2024-08-26 08:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.609 Acc@5 95.152 [2024-08-26 08:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.6% [2024-08-26 08:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.61% [2024-08-26 08:26:53 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:26:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:26:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][0/625] eta 0:07:54 lr 0.001777 wd 0.0500 time 0.7586 (0.7586) data time 0.4242 (0.4242) model time 0.0000 (0.0000) loss 3.6550 (3.6550) grad_norm 1.5931 (1.5931) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:26:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][10/625] eta 0:04:13 lr 0.001777 wd 0.0500 time 0.3761 (0.4118) data time 0.0008 (0.0396) model time 0.0000 (0.0000) loss 4.6028 (3.7994) grad_norm 1.1218 (1.4129) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][20/625] eta 0:03:59 lr 0.001777 wd 0.0500 time 0.3775 (0.3959) data time 0.0008 (0.0213) model time 0.0000 (0.0000) loss 3.8732 (3.5784) grad_norm 1.0044 (1.3885) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][30/625] eta 0:03:51 lr 0.001777 wd 0.0500 time 0.3750 (0.3898) data time 0.0011 (0.0148) model time 0.0000 (0.0000) loss 3.9141 (3.5437) grad_norm 1.4888 (1.3986) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][40/625] eta 0:03:46 lr 0.001777 wd 0.0500 time 0.3817 (0.3867) data time 0.0011 (0.0115) model time 0.0000 (0.0000) loss 3.8970 (3.5205) grad_norm 0.9839 (1.4109) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][50/625] eta 0:03:41 lr 0.001776 wd 0.0500 time 0.3684 (0.3844) data time 0.0014 (0.0095) model time 0.0000 (0.0000) loss 2.4067 (3.4525) grad_norm 1.3022 (1.3684) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][60/625] eta 0:03:36 lr 0.001776 wd 0.0500 time 0.3802 (0.3833) data time 0.0008 (0.0081) model time 0.3793 (0.3766) loss 2.2563 (3.3964) grad_norm 1.7628 (1.3509) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][70/625] eta 0:03:32 lr 0.001776 wd 0.0500 time 0.3806 (0.3826) data time 0.0009 (0.0071) model time 0.3797 (0.3769) loss 2.7679 (3.3941) grad_norm 1.7247 (1.3859) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][80/625] eta 0:03:28 lr 0.001776 wd 0.0500 time 0.3656 (0.3819) data time 0.0009 (0.0064) model time 0.3647 (0.3765) loss 3.3350 (3.3945) grad_norm 1.2070 (1.3672) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][90/625] eta 0:03:24 lr 0.001776 wd 0.0500 time 0.3767 (0.3814) data time 0.0010 (0.0058) model time 0.3757 (0.3765) loss 3.6994 (3.4112) grad_norm 1.6062 (1.3904) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][100/625] eta 0:03:21 lr 0.001776 wd 0.0500 time 0.3777 (0.3845) data time 0.0011 (0.0054) model time 0.3766 (0.3835) loss 3.9094 (3.4326) grad_norm 1.0168 (1.3733) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][110/625] eta 0:03:17 lr 0.001776 wd 0.0500 time 0.3777 (0.3839) data time 0.0008 (0.0050) model time 0.3769 (0.3823) loss 3.8577 (3.4495) grad_norm 1.1495 (1.3649) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][120/625] eta 0:03:13 lr 0.001776 wd 0.0500 time 0.3753 (0.3834) data time 0.0011 (0.0047) model time 0.3742 (0.3816) loss 3.8878 (3.4744) grad_norm 1.7120 (1.3671) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][130/625] eta 0:03:09 lr 0.001776 wd 0.0500 time 0.3776 (0.3829) data time 0.0011 (0.0044) model time 0.3765 (0.3808) loss 3.0364 (3.4557) grad_norm 1.1653 (1.3617) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][140/625] eta 0:03:05 lr 0.001775 wd 0.0500 time 0.3739 (0.3824) data time 0.0011 (0.0042) model time 0.3728 (0.3801) loss 2.5363 (3.4495) grad_norm 1.4014 (1.3633) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][150/625] eta 0:03:01 lr 0.001775 wd 0.0500 time 0.3789 (0.3820) data time 0.0008 (0.0040) model time 0.3781 (0.3796) loss 2.4646 (3.4642) grad_norm 0.9133 (1.3492) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:27:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][160/625] eta 0:02:58 lr 0.001775 wd 0.0500 time 0.3772 (0.3844) data time 0.0008 (0.0038) model time 0.3764 (0.3833) loss 2.1674 (3.4469) grad_norm 1.0908 (1.3503) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][170/625] eta 0:02:55 lr 0.001775 wd 0.0500 time 0.3740 (0.3849) data time 0.0009 (0.0036) model time 0.3731 (0.3840) loss 3.1816 (3.4430) grad_norm 1.4795 (1.3401) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][180/625] eta 0:02:51 lr 0.001775 wd 0.0500 time 0.3795 (0.3845) data time 0.0008 (0.0035) model time 0.3786 (0.3834) loss 4.0891 (3.4468) grad_norm 1.8628 (1.3461) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][190/625] eta 0:02:47 lr 0.001775 wd 0.0500 time 0.3794 (0.3842) data time 0.0011 (0.0034) model time 0.3783 (0.3830) loss 3.3871 (3.4517) grad_norm 1.1528 (1.3559) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][200/625] eta 0:02:43 lr 0.001775 wd 0.0500 time 0.3777 (0.3840) data time 0.0008 (0.0032) model time 0.3769 (0.3827) loss 3.2819 (3.4579) grad_norm 1.3448 (1.3520) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][210/625] eta 0:02:39 lr 0.001775 wd 0.0500 time 0.3784 (0.3838) data time 0.0008 (0.0031) model time 0.3775 (0.3825) loss 3.5515 (3.4523) grad_norm 1.5194 (1.3563) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][220/625] eta 0:02:35 lr 0.001775 wd 0.0500 time 0.3749 (0.3834) data time 0.0009 (0.0030) model time 0.3740 (0.3820) loss 3.4258 (3.4521) grad_norm 1.3134 (1.3680) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][230/625] eta 0:02:31 lr 0.001774 wd 0.0500 time 0.3691 (0.3831) data time 0.0009 (0.0030) model time 0.3681 (0.3816) loss 3.4575 (3.4467) grad_norm 1.0467 (1.3681) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][240/625] eta 0:02:27 lr 0.001774 wd 0.0500 time 0.3785 (0.3828) data time 0.0010 (0.0029) model time 0.3775 (0.3813) loss 2.8788 (3.4458) grad_norm 1.5123 (1.3607) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][250/625] eta 0:02:23 lr 0.001774 wd 0.0500 time 0.3720 (0.3827) data time 0.0012 (0.0028) model time 0.3708 (0.3811) loss 3.1750 (3.4483) grad_norm 1.3730 (1.3649) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][260/625] eta 0:02:19 lr 0.001774 wd 0.0500 time 0.3768 (0.3825) data time 0.0009 (0.0028) model time 0.3760 (0.3809) loss 2.6360 (3.4384) grad_norm 1.1067 (1.3640) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][270/625] eta 0:02:15 lr 0.001774 wd 0.0500 time 0.3729 (0.3823) data time 0.0009 (0.0027) model time 0.3719 (0.3807) loss 3.6513 (3.4374) grad_norm 1.1838 (1.3519) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][280/625] eta 0:02:11 lr 0.001774 wd 0.0500 time 0.3827 (0.3822) data time 0.0008 (0.0026) model time 0.3819 (0.3806) loss 2.8662 (3.4357) grad_norm 0.8268 (1.3462) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][290/625] eta 0:02:08 lr 0.001774 wd 0.0500 time 0.3755 (0.3828) data time 0.0009 (0.0026) model time 0.3747 (0.3813) loss 4.3525 (3.4329) grad_norm 1.3735 (1.3500) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][300/625] eta 0:02:04 lr 0.001774 wd 0.0500 time 0.3785 (0.3827) data time 0.0011 (0.0025) model time 0.3774 (0.3812) loss 2.6995 (3.4358) grad_norm 1.3809 (1.3507) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][310/625] eta 0:02:00 lr 0.001774 wd 0.0500 time 0.3716 (0.3839) data time 0.0009 (0.0025) model time 0.3707 (0.3826) loss 2.9457 (3.4342) grad_norm 0.9037 (1.3468) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:28:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][320/625] eta 0:01:57 lr 0.001773 wd 0.0500 time 0.3793 (0.3837) data time 0.0008 (0.0025) model time 0.3785 (0.3824) loss 3.0527 (3.4247) grad_norm 0.9288 (1.3386) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][330/625] eta 0:01:53 lr 0.001773 wd 0.0500 time 0.3761 (0.3836) data time 0.0012 (0.0024) model time 0.3749 (0.3823) loss 3.5785 (3.4273) grad_norm 1.1265 (1.3413) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][340/625] eta 0:01:49 lr 0.001773 wd 0.0500 time 0.3705 (0.3839) data time 0.0011 (0.0024) model time 0.3694 (0.3827) loss 3.8089 (3.4334) grad_norm 1.2628 (1.3363) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][350/625] eta 0:01:45 lr 0.001773 wd 0.0500 time 0.3757 (0.3843) data time 0.0011 (0.0023) model time 0.3746 (0.3831) loss 2.7327 (3.4237) grad_norm 1.3344 (1.3442) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][360/625] eta 0:01:41 lr 0.001773 wd 0.0500 time 0.3785 (0.3840) data time 0.0008 (0.0023) model time 0.3777 (0.3828) loss 3.4726 (3.4266) grad_norm 1.4494 (1.3395) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][370/625] eta 0:01:37 lr 0.001773 wd 0.0500 time 0.3854 (0.3839) data time 0.0011 (0.0023) model time 0.3843 (0.3827) loss 3.7117 (3.4243) grad_norm 1.3271 (1.3355) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][380/625] eta 0:01:34 lr 0.001773 wd 0.0500 time 0.3880 (0.3837) data time 0.0011 (0.0022) model time 0.3869 (0.3825) loss 3.8571 (3.4299) grad_norm 1.2191 (1.3356) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][390/625] eta 0:01:30 lr 0.001773 wd 0.0500 time 0.3761 (0.3836) data time 0.0012 (0.0022) model time 0.3750 (0.3824) loss 3.2943 (3.4348) grad_norm 1.1043 (1.3314) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][400/625] eta 0:01:26 lr 0.001773 wd 0.0500 time 0.3763 (0.3834) data time 0.0011 (0.0022) model time 0.3752 (0.3822) loss 4.0529 (3.4406) grad_norm 1.1271 (1.3271) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][410/625] eta 0:01:22 lr 0.001772 wd 0.0500 time 0.3819 (0.3833) data time 0.0011 (0.0022) model time 0.3808 (0.3821) loss 3.6725 (3.4437) grad_norm 3.8804 (1.3346) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][420/625] eta 0:01:18 lr 0.001772 wd 0.0500 time 0.3829 (0.3832) data time 0.0008 (0.0021) model time 0.3821 (0.3819) loss 3.0094 (3.4462) grad_norm 1.0659 (1.3391) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][430/625] eta 0:01:14 lr 0.001772 wd 0.0500 time 0.3804 (0.3831) data time 0.0008 (0.0021) model time 0.3796 (0.3818) loss 3.1755 (3.4407) grad_norm 1.5129 (1.3375) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][440/625] eta 0:01:10 lr 0.001772 wd 0.0500 time 0.3806 (0.3831) data time 0.0011 (0.0021) model time 0.3795 (0.3818) loss 3.1928 (3.4355) grad_norm 1.1265 (1.3356) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][450/625] eta 0:01:07 lr 0.001772 wd 0.0500 time 0.3814 (0.3830) data time 0.0010 (0.0021) model time 0.3804 (0.3817) loss 2.6962 (3.4368) grad_norm 1.4598 (1.3353) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][460/625] eta 0:01:03 lr 0.001772 wd 0.0500 time 0.5871 (0.3839) data time 0.0010 (0.0020) model time 0.5860 (0.3827) loss 2.3674 (3.4339) grad_norm 0.9996 (1.3309) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][470/625] eta 0:00:59 lr 0.001772 wd 0.0500 time 0.3785 (0.3837) data time 0.0013 (0.0020) model time 0.3773 (0.3825) loss 3.5678 (3.4301) grad_norm 1.0446 (1.3294) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:29:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][480/625] eta 0:00:55 lr 0.001772 wd 0.0500 time 0.3787 (0.3836) data time 0.0011 (0.0020) model time 0.3777 (0.3824) loss 2.8743 (3.4297) grad_norm 1.0823 (1.3309) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][490/625] eta 0:00:51 lr 0.001772 wd 0.0500 time 0.3819 (0.3843) data time 0.0011 (0.0020) model time 0.3808 (0.3832) loss 2.7463 (3.4277) grad_norm 1.2481 (1.3333) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][500/625] eta 0:00:48 lr 0.001771 wd 0.0500 time 0.3840 (0.3846) data time 0.0011 (0.0020) model time 0.3829 (0.3835) loss 3.1609 (3.4285) grad_norm 1.1139 (1.3308) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][510/625] eta 0:00:44 lr 0.001771 wd 0.0500 time 0.3789 (0.3844) data time 0.0009 (0.0020) model time 0.3780 (0.3833) loss 3.9592 (3.4281) grad_norm 1.2815 (1.3292) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][520/625] eta 0:00:40 lr 0.001771 wd 0.0500 time 0.3815 (0.3843) data time 0.0011 (0.0020) model time 0.3803 (0.3832) loss 2.2184 (3.4255) grad_norm 1.4093 (1.3330) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][530/625] eta 0:00:36 lr 0.001771 wd 0.0500 time 0.3795 (0.3842) data time 0.0012 (0.0019) model time 0.3784 (0.3831) loss 3.5955 (3.4235) grad_norm 0.8822 (1.3324) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][540/625] eta 0:00:32 lr 0.001771 wd 0.0500 time 0.3820 (0.3841) data time 0.0009 (0.0019) model time 0.3811 (0.3829) loss 3.0050 (3.4220) grad_norm 1.0435 (1.3309) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][550/625] eta 0:00:28 lr 0.001771 wd 0.0500 time 0.3789 (0.3840) data time 0.0012 (0.0019) model time 0.3778 (0.3828) loss 3.6413 (3.4197) grad_norm 1.2916 (1.3327) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][560/625] eta 0:00:24 lr 0.001771 wd 0.0500 time 0.3768 (0.3839) data time 0.0008 (0.0019) model time 0.3759 (0.3827) loss 3.4043 (3.4194) grad_norm 1.3452 (1.3336) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][570/625] eta 0:00:21 lr 0.001771 wd 0.0500 time 0.3864 (0.3838) data time 0.0011 (0.0019) model time 0.3853 (0.3826) loss 3.4522 (3.4188) grad_norm 1.1366 (1.3343) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][580/625] eta 0:00:17 lr 0.001770 wd 0.0500 time 0.3772 (0.3837) data time 0.0009 (0.0019) model time 0.3764 (0.3825) loss 3.2506 (3.4176) grad_norm 1.2154 (1.3326) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][590/625] eta 0:00:13 lr 0.001770 wd 0.0500 time 0.3745 (0.3836) data time 0.0011 (0.0019) model time 0.3735 (0.3823) loss 3.7952 (3.4172) grad_norm 0.9922 (1.3313) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][600/625] eta 0:00:09 lr 0.001770 wd 0.0500 time 0.3728 (0.3834) data time 0.0011 (0.0018) model time 0.3718 (0.3822) loss 3.6140 (3.4180) grad_norm 1.0950 (1.3295) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][610/625] eta 0:00:05 lr 0.001770 wd 0.0500 time 0.3677 (0.3840) data time 0.0005 (0.0018) model time 0.3672 (0.3828) loss 3.5281 (3.4204) grad_norm 1.1713 (1.3291) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [81/300][620/625] eta 0:00:01 lr 0.001770 wd 0.0500 time 0.3664 (0.3837) data time 0.0005 (0.0018) model time 0.3659 (0.3825) loss 4.3067 (3.4229) grad_norm 1.1147 (1.3341) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 81 training takes 0:03:59 [2024-08-26 08:30:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:30:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:30:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.497 (0.497) Loss 0.7012 (0.7012) Acc@1 84.570 (84.570) Acc@5 96.582 (96.582) Mem 11225MB [2024-08-26 08:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 1.1611 (0.8258) Acc@1 72.607 (81.148) Acc@5 92.725 (96.009) Mem 11225MB [2024-08-26 08:30:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.2969 (1.0069) Acc@1 70.508 (77.200) Acc@5 90.723 (93.794) Mem 11225MB [2024-08-26 08:30:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.943 Acc@5 93.826 [2024-08-26 08:30:58 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.9% [2024-08-26 08:30:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.861 (0.861) Loss 0.5635 (0.5635) Acc@1 88.037 (88.037) Acc@5 98.193 (98.193) Mem 11225MB [2024-08-26 08:31:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.178) Loss 0.9429 (0.7041) Acc@1 77.197 (83.904) Acc@5 94.873 (97.008) Mem 11225MB [2024-08-26 08:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.139) Loss 1.1094 (0.8597) Acc@1 72.266 (80.013) Acc@5 92.627 (95.229) Mem 11225MB [2024-08-26 08:31:02 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.692 Acc@5 95.176 [2024-08-26 08:31:02 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.7% [2024-08-26 08:31:02 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.69% [2024-08-26 08:31:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:31:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:31:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][0/625] eta 0:08:17 lr 0.001770 wd 0.0500 time 0.7964 (0.7964) data time 0.4597 (0.4597) model time 0.0000 (0.0000) loss 2.8366 (2.8366) grad_norm 1.1832 (1.1832) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][10/625] eta 0:04:26 lr 0.001770 wd 0.0500 time 0.3799 (0.4331) data time 0.0008 (0.0427) model time 0.0000 (0.0000) loss 2.5657 (3.3511) grad_norm 0.9355 (1.1736) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][20/625] eta 0:04:06 lr 0.001770 wd 0.0500 time 0.3764 (0.4075) data time 0.0011 (0.0229) model time 0.0000 (0.0000) loss 2.6090 (3.2146) grad_norm 1.1266 (1.2144) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][30/625] eta 0:04:04 lr 0.001770 wd 0.0500 time 0.3715 (0.4106) data time 0.0008 (0.0159) model time 0.0000 (0.0000) loss 3.0947 (3.2495) grad_norm 2.0032 (1.2432) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][40/625] eta 0:03:55 lr 0.001770 wd 0.0500 time 0.3813 (0.4026) data time 0.0011 (0.0123) model time 0.0000 (0.0000) loss 3.7741 (3.2319) grad_norm 1.4475 (1.2526) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][50/625] eta 0:03:48 lr 0.001769 wd 0.0500 time 0.3781 (0.3974) data time 0.0008 (0.0101) model time 0.0000 (0.0000) loss 3.4587 (3.2565) grad_norm 0.8416 (1.2751) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][60/625] eta 0:03:42 lr 0.001769 wd 0.0500 time 0.3813 (0.3942) data time 0.0012 (0.0086) model time 0.3801 (0.3766) loss 3.1396 (3.2433) grad_norm 1.2292 (1.2620) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][70/625] eta 0:03:37 lr 0.001769 wd 0.0500 time 0.3764 (0.3918) data time 0.0012 (0.0076) model time 0.3752 (0.3763) loss 3.3548 (3.2629) grad_norm 1.4058 (1.2606) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][80/625] eta 0:03:32 lr 0.001769 wd 0.0500 time 0.3797 (0.3902) data time 0.0008 (0.0068) model time 0.3789 (0.3767) loss 3.4382 (3.2959) grad_norm 1.2082 (1.2781) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][90/625] eta 0:03:27 lr 0.001769 wd 0.0500 time 0.3767 (0.3887) data time 0.0008 (0.0062) model time 0.3759 (0.3763) loss 3.7119 (3.3179) grad_norm 1.3466 (1.2781) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][100/625] eta 0:03:23 lr 0.001769 wd 0.0500 time 0.3799 (0.3875) data time 0.0011 (0.0057) model time 0.3788 (0.3762) loss 4.4446 (3.3416) grad_norm 0.9348 (1.2829) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][110/625] eta 0:03:19 lr 0.001769 wd 0.0500 time 0.3737 (0.3866) data time 0.0008 (0.0053) model time 0.3729 (0.3762) loss 3.6419 (3.3684) grad_norm 1.4869 (1.2873) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][120/625] eta 0:03:15 lr 0.001769 wd 0.0500 time 0.6047 (0.3879) data time 0.0010 (0.0049) model time 0.6036 (0.3799) loss 3.2509 (3.3454) grad_norm 1.7925 (1.2907) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][130/625] eta 0:03:11 lr 0.001768 wd 0.0500 time 0.3787 (0.3869) data time 0.0008 (0.0046) model time 0.3779 (0.3791) loss 3.8336 (3.3503) grad_norm 1.1052 (1.2976) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:31:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][140/625] eta 0:03:07 lr 0.001768 wd 0.0500 time 0.3801 (0.3863) data time 0.0009 (0.0044) model time 0.3792 (0.3789) loss 3.8221 (3.3536) grad_norm 1.8969 (1.2997) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][150/625] eta 0:03:03 lr 0.001768 wd 0.0500 time 0.3770 (0.3858) data time 0.0009 (0.0042) model time 0.3761 (0.3787) loss 3.4119 (3.3708) grad_norm 1.8541 (1.3176) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][160/625] eta 0:02:59 lr 0.001768 wd 0.0500 time 0.3768 (0.3852) data time 0.0008 (0.0040) model time 0.3760 (0.3784) loss 3.9551 (3.3881) grad_norm 1.5181 (1.3229) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][170/625] eta 0:02:56 lr 0.001768 wd 0.0500 time 0.3756 (0.3870) data time 0.0008 (0.0038) model time 0.3748 (0.3814) loss 2.3908 (3.3753) grad_norm 1.2413 (1.3142) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][180/625] eta 0:02:52 lr 0.001768 wd 0.0500 time 0.3789 (0.3866) data time 0.0007 (0.0036) model time 0.3782 (0.3812) loss 4.2637 (3.3992) grad_norm 1.9732 (1.3194) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][190/625] eta 0:02:47 lr 0.001768 wd 0.0500 time 0.3780 (0.3861) data time 0.0008 (0.0035) model time 0.3772 (0.3809) loss 4.1054 (3.4048) grad_norm 0.7551 (1.3156) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][200/625] eta 0:02:43 lr 0.001768 wd 0.0500 time 0.3834 (0.3857) data time 0.0008 (0.0034) model time 0.3826 (0.3806) loss 4.3558 (3.4117) grad_norm 1.2751 (1.3121) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][210/625] eta 0:02:39 lr 0.001768 wd 0.0500 time 0.3739 (0.3853) data time 0.0011 (0.0033) model time 0.3729 (0.3804) loss 2.8344 (3.4130) grad_norm 1.1839 (1.3002) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][220/625] eta 0:02:35 lr 0.001767 wd 0.0500 time 0.3871 (0.3850) data time 0.0010 (0.0032) model time 0.3861 (0.3802) loss 3.4944 (3.4112) grad_norm 1.2968 (1.2969) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][230/625] eta 0:02:31 lr 0.001767 wd 0.0500 time 0.3771 (0.3847) data time 0.0007 (0.0031) model time 0.3763 (0.3800) loss 3.4434 (3.4160) grad_norm 1.2057 (1.2942) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][240/625] eta 0:02:28 lr 0.001767 wd 0.0500 time 0.3788 (0.3863) data time 0.0010 (0.0030) model time 0.3778 (0.3823) loss 3.4957 (3.4248) grad_norm 1.1196 (1.2953) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][250/625] eta 0:02:24 lr 0.001767 wd 0.0500 time 0.3753 (0.3860) data time 0.0008 (0.0029) model time 0.3745 (0.3820) loss 3.6035 (3.4285) grad_norm 0.9232 (1.2893) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][260/625] eta 0:02:20 lr 0.001767 wd 0.0500 time 0.3788 (0.3858) data time 0.0010 (0.0029) model time 0.3778 (0.3818) loss 3.9806 (3.4272) grad_norm 0.9494 (1.2848) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][270/625] eta 0:02:16 lr 0.001767 wd 0.0500 time 0.3704 (0.3855) data time 0.0010 (0.0028) model time 0.3694 (0.3816) loss 2.6375 (3.4317) grad_norm 0.9480 (1.2820) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][280/625] eta 0:02:12 lr 0.001767 wd 0.0500 time 0.3755 (0.3852) data time 0.0008 (0.0027) model time 0.3747 (0.3814) loss 4.0514 (3.4301) grad_norm 0.9744 (1.2780) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][290/625] eta 0:02:09 lr 0.001767 wd 0.0500 time 0.3763 (0.3863) data time 0.0011 (0.0027) model time 0.3751 (0.3829) loss 2.4093 (3.4206) grad_norm 1.3617 (1.2847) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:32:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][300/625] eta 0:02:05 lr 0.001767 wd 0.0500 time 0.3715 (0.3859) data time 0.0011 (0.0026) model time 0.3704 (0.3825) loss 3.0423 (3.4199) grad_norm 1.1327 (1.2845) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][310/625] eta 0:02:01 lr 0.001766 wd 0.0500 time 0.3711 (0.3857) data time 0.0009 (0.0026) model time 0.3702 (0.3823) loss 4.2795 (3.4233) grad_norm 1.2995 (1.2791) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][320/625] eta 0:01:57 lr 0.001766 wd 0.0500 time 0.3742 (0.3854) data time 0.0011 (0.0025) model time 0.3731 (0.3821) loss 3.1541 (3.4287) grad_norm 0.9864 (1.2791) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][330/625] eta 0:01:53 lr 0.001766 wd 0.0500 time 0.5768 (0.3858) data time 0.0011 (0.0025) model time 0.5757 (0.3825) loss 2.9639 (3.4268) grad_norm 1.0934 (1.2839) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][340/625] eta 0:01:49 lr 0.001766 wd 0.0500 time 0.3871 (0.3855) data time 0.0010 (0.0024) model time 0.3861 (0.3823) loss 2.9504 (3.4263) grad_norm 0.9634 (1.2821) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][350/625] eta 0:01:45 lr 0.001766 wd 0.0500 time 0.3766 (0.3854) data time 0.0010 (0.0024) model time 0.3755 (0.3822) loss 3.9235 (3.4270) grad_norm 1.9385 (1.2846) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][360/625] eta 0:01:42 lr 0.001766 wd 0.0500 time 0.3712 (0.3851) data time 0.0011 (0.0024) model time 0.3701 (0.3820) loss 3.6300 (3.4192) grad_norm 0.8992 (1.2881) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][370/625] eta 0:01:38 lr 0.001766 wd 0.0500 time 0.3739 (0.3849) data time 0.0011 (0.0023) model time 0.3728 (0.3818) loss 3.6505 (3.4237) grad_norm 0.9937 (1.2860) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][380/625] eta 0:01:34 lr 0.001766 wd 0.0500 time 0.3728 (0.3846) data time 0.0011 (0.0023) model time 0.3718 (0.3816) loss 2.9462 (3.4290) grad_norm 1.1514 (1.2852) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][390/625] eta 0:01:30 lr 0.001766 wd 0.0500 time 0.3750 (0.3844) data time 0.0008 (0.0023) model time 0.3741 (0.3814) loss 3.9724 (3.4334) grad_norm 1.4101 (1.2851) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][400/625] eta 0:01:26 lr 0.001765 wd 0.0500 time 0.3869 (0.3842) data time 0.0011 (0.0022) model time 0.3858 (0.3812) loss 3.4427 (3.4379) grad_norm 1.2648 (1.2875) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][410/625] eta 0:01:22 lr 0.001765 wd 0.0500 time 0.3767 (0.3840) data time 0.0009 (0.0022) model time 0.3759 (0.3810) loss 3.5146 (3.4366) grad_norm 1.3726 (1.2850) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][420/625] eta 0:01:18 lr 0.001765 wd 0.0500 time 0.5936 (0.3849) data time 0.0008 (0.0022) model time 0.5928 (0.3820) loss 2.3398 (3.4361) grad_norm 1.6128 (1.2851) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][430/625] eta 0:01:15 lr 0.001765 wd 0.0500 time 0.3795 (0.3847) data time 0.0011 (0.0022) model time 0.3784 (0.3819) loss 3.7911 (3.4461) grad_norm 2.1126 (1.2869) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][440/625] eta 0:01:11 lr 0.001765 wd 0.0500 time 0.3740 (0.3845) data time 0.0011 (0.0021) model time 0.3729 (0.3817) loss 3.5647 (3.4434) grad_norm 0.9918 (1.2909) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:33:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][450/625] eta 0:01:07 lr 0.001765 wd 0.0500 time 0.3726 (0.3844) data time 0.0008 (0.0021) model time 0.3717 (0.3816) loss 4.7257 (3.4456) grad_norm 1.0078 (1.2900) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][460/625] eta 0:01:03 lr 0.001765 wd 0.0500 time 0.3756 (0.3841) data time 0.0013 (0.0021) model time 0.3743 (0.3814) loss 3.3750 (3.4483) grad_norm 1.1422 (1.2878) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][470/625] eta 0:00:59 lr 0.001765 wd 0.0500 time 0.3750 (0.3839) data time 0.0008 (0.0021) model time 0.3742 (0.3812) loss 2.8591 (3.4373) grad_norm 1.0209 (1.2878) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][480/625] eta 0:00:55 lr 0.001764 wd 0.0500 time 0.3733 (0.3838) data time 0.0008 (0.0021) model time 0.3725 (0.3811) loss 4.2364 (3.4408) grad_norm 1.5190 (1.2849) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][490/625] eta 0:00:51 lr 0.001764 wd 0.0500 time 0.3769 (0.3836) data time 0.0008 (0.0020) model time 0.3761 (0.3809) loss 3.7374 (3.4343) grad_norm 1.6292 (1.2895) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][500/625] eta 0:00:48 lr 0.001764 wd 0.0500 time 0.3698 (0.3843) data time 0.0009 (0.0020) model time 0.3689 (0.3817) loss 2.5272 (3.4341) grad_norm 1.4501 (1.3026) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][510/625] eta 0:00:44 lr 0.001764 wd 0.0500 time 0.3749 (0.3841) data time 0.0009 (0.0020) model time 0.3741 (0.3816) loss 2.9574 (3.4372) grad_norm 0.9889 (1.3046) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][520/625] eta 0:00:40 lr 0.001764 wd 0.0500 time 0.3811 (0.3844) data time 0.0011 (0.0020) model time 0.3800 (0.3818) loss 3.7939 (3.4439) grad_norm 1.0838 (1.3057) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][530/625] eta 0:00:36 lr 0.001764 wd 0.0500 time 0.3726 (0.3842) data time 0.0011 (0.0020) model time 0.3715 (0.3817) loss 3.7825 (3.4473) grad_norm 1.2334 (1.3060) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][540/625] eta 0:00:32 lr 0.001764 wd 0.0500 time 0.3697 (0.3848) data time 0.0008 (0.0020) model time 0.3690 (0.3824) loss 3.4814 (3.4491) grad_norm 1.2773 (1.3065) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][550/625] eta 0:00:28 lr 0.001764 wd 0.0500 time 0.3895 (0.3847) data time 0.0008 (0.0019) model time 0.3887 (0.3823) loss 3.3983 (3.4439) grad_norm 1.3797 (1.3047) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][560/625] eta 0:00:24 lr 0.001764 wd 0.0500 time 0.3770 (0.3845) data time 0.0010 (0.0019) model time 0.3760 (0.3821) loss 2.9682 (3.4433) grad_norm 1.4956 (1.3055) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][570/625] eta 0:00:21 lr 0.001763 wd 0.0500 time 0.3738 (0.3844) data time 0.0011 (0.0019) model time 0.3726 (0.3820) loss 3.9067 (3.4419) grad_norm 0.9234 (1.3051) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][580/625] eta 0:00:17 lr 0.001763 wd 0.0500 time 0.3921 (0.3843) data time 0.0008 (0.0019) model time 0.3913 (0.3819) loss 2.4897 (3.4353) grad_norm 1.0715 (1.3026) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][590/625] eta 0:00:13 lr 0.001763 wd 0.0500 time 0.3772 (0.3842) data time 0.0008 (0.0019) model time 0.3763 (0.3818) loss 3.9566 (3.4392) grad_norm 1.2760 (1.3034) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][600/625] eta 0:00:09 lr 0.001763 wd 0.0500 time 0.3729 (0.3840) data time 0.0008 (0.0019) model time 0.3720 (0.3817) loss 4.1589 (3.4404) grad_norm 1.2054 (1.3028) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:34:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][610/625] eta 0:00:05 lr 0.001763 wd 0.0500 time 0.3657 (0.3839) data time 0.0006 (0.0019) model time 0.3651 (0.3816) loss 3.2318 (3.4372) grad_norm 1.3310 (1.3016) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [82/300][620/625] eta 0:00:01 lr 0.001763 wd 0.0500 time 0.3634 (0.3836) data time 0.0008 (0.0018) model time 0.3626 (0.3813) loss 3.6535 (3.4338) grad_norm 1.3779 (1.2999) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:03 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 82 training takes 0:03:59 [2024-08-26 08:35:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:35:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:35:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.486 (0.486) Loss 0.6777 (0.6777) Acc@1 85.254 (85.254) Acc@5 97.314 (97.314) Mem 11225MB [2024-08-26 08:35:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.142) Loss 1.1201 (0.8272) Acc@1 73.779 (81.436) Acc@5 93.359 (96.156) Mem 11225MB [2024-08-26 08:35:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.2695 (0.9929) Acc@1 70.410 (77.500) Acc@5 90.918 (94.036) Mem 11225MB [2024-08-26 08:35:07 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.315 Acc@5 94.020 [2024-08-26 08:35:07 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.3% [2024-08-26 08:35:07 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 77.32% [2024-08-26 08:35:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 08:35:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 08:35:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 0.5615 (0.5615) Acc@1 88.184 (88.184) Acc@5 98.145 (98.145) Mem 11225MB [2024-08-26 08:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.141) Loss 0.9409 (0.7027) Acc@1 77.246 (83.940) Acc@5 94.775 (97.004) Mem 11225MB [2024-08-26 08:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.1074 (0.8577) Acc@1 72.510 (80.080) Acc@5 92.529 (95.215) Mem 11225MB [2024-08-26 08:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.774 Acc@5 95.168 [2024-08-26 08:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.8% [2024-08-26 08:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.77% [2024-08-26 08:35:11 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:35:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:35:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][0/625] eta 0:08:01 lr 0.001763 wd 0.0500 time 0.7697 (0.7697) data time 0.4094 (0.4094) model time 0.0000 (0.0000) loss 3.8148 (3.8148) grad_norm 1.1489 (1.1489) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][10/625] eta 0:04:12 lr 0.001763 wd 0.0500 time 0.3812 (0.4113) data time 0.0008 (0.0382) model time 0.0000 (0.0000) loss 4.0342 (3.5085) grad_norm 1.3728 (1.2534) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][20/625] eta 0:03:58 lr 0.001763 wd 0.0500 time 0.3745 (0.3948) data time 0.0008 (0.0205) model time 0.0000 (0.0000) loss 3.7869 (3.5042) grad_norm 3.0151 (1.4835) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][30/625] eta 0:03:51 lr 0.001762 wd 0.0500 time 0.3721 (0.3892) data time 0.0011 (0.0142) model time 0.0000 (0.0000) loss 4.0025 (3.4628) grad_norm 1.1630 (1.4305) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][40/625] eta 0:03:46 lr 0.001762 wd 0.0500 time 0.3848 (0.3864) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 3.2195 (3.4327) grad_norm 1.0840 (1.3813) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][50/625] eta 0:03:41 lr 0.001762 wd 0.0500 time 0.3808 (0.3845) data time 0.0008 (0.0091) model time 0.0000 (0.0000) loss 2.7927 (3.4619) grad_norm 1.5781 (1.3656) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][60/625] eta 0:03:38 lr 0.001762 wd 0.0500 time 0.3726 (0.3865) data time 0.0008 (0.0078) model time 0.3718 (0.3955) loss 3.6797 (3.4649) grad_norm 1.3845 (1.3719) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][70/625] eta 0:03:37 lr 0.001762 wd 0.0500 time 0.5971 (0.3914) data time 0.0008 (0.0069) model time 0.5963 (0.4078) loss 3.4499 (3.4726) grad_norm 1.3411 (1.3486) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][80/625] eta 0:03:32 lr 0.001762 wd 0.0500 time 0.3746 (0.3895) data time 0.0009 (0.0062) model time 0.3737 (0.3968) loss 3.5570 (3.5079) grad_norm 1.5706 (1.3490) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][90/625] eta 0:03:27 lr 0.001762 wd 0.0500 time 0.3741 (0.3880) data time 0.0012 (0.0056) model time 0.3730 (0.3912) loss 3.6797 (3.4970) grad_norm 0.9827 (1.3575) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][100/625] eta 0:03:23 lr 0.001762 wd 0.0500 time 0.3712 (0.3868) data time 0.0008 (0.0052) model time 0.3704 (0.3879) loss 3.2929 (3.4720) grad_norm 1.2146 (1.3552) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][110/625] eta 0:03:18 lr 0.001762 wd 0.0500 time 0.3725 (0.3858) data time 0.0008 (0.0048) model time 0.3717 (0.3857) loss 2.5331 (3.4719) grad_norm 1.0160 (1.3439) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:35:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][120/625] eta 0:03:14 lr 0.001761 wd 0.0500 time 0.3738 (0.3850) data time 0.0011 (0.0045) model time 0.3727 (0.3842) loss 3.6685 (3.4783) grad_norm 1.8627 (1.3637) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][130/625] eta 0:03:10 lr 0.001761 wd 0.0500 time 0.3734 (0.3845) data time 0.0012 (0.0042) model time 0.3722 (0.3834) loss 2.4493 (3.4344) grad_norm 1.1436 (1.3713) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][140/625] eta 0:03:06 lr 0.001761 wd 0.0500 time 0.3713 (0.3852) data time 0.0012 (0.0040) model time 0.3701 (0.3845) loss 3.4979 (3.4305) grad_norm 2.1951 (1.3787) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][150/625] eta 0:03:02 lr 0.001761 wd 0.0500 time 0.3801 (0.3846) data time 0.0009 (0.0038) model time 0.3792 (0.3835) loss 4.3861 (3.4333) grad_norm 1.3499 (1.3781) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][160/625] eta 0:02:58 lr 0.001761 wd 0.0500 time 0.3778 (0.3841) data time 0.0009 (0.0037) model time 0.3769 (0.3827) loss 2.7703 (3.4294) grad_norm 1.4547 (1.3712) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][170/625] eta 0:02:54 lr 0.001761 wd 0.0500 time 0.3793 (0.3836) data time 0.0012 (0.0035) model time 0.3781 (0.3820) loss 4.1587 (3.4251) grad_norm 1.5842 (1.3606) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][180/625] eta 0:02:50 lr 0.001761 wd 0.0500 time 0.3755 (0.3832) data time 0.0008 (0.0034) model time 0.3747 (0.3816) loss 4.0197 (3.4234) grad_norm 1.2065 (1.3561) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][190/625] eta 0:02:46 lr 0.001761 wd 0.0500 time 0.3842 (0.3830) data time 0.0008 (0.0033) model time 0.3834 (0.3813) loss 3.6619 (3.4166) grad_norm 0.8471 (1.3440) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][200/625] eta 0:02:42 lr 0.001760 wd 0.0500 time 0.3742 (0.3827) data time 0.0013 (0.0032) model time 0.3728 (0.3810) loss 3.3265 (3.4208) grad_norm 1.0857 (1.3409) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][210/625] eta 0:02:38 lr 0.001760 wd 0.0500 time 0.3839 (0.3825) data time 0.0008 (0.0031) model time 0.3831 (0.3808) loss 3.7788 (3.4236) grad_norm 1.1322 (1.3464) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][220/625] eta 0:02:34 lr 0.001760 wd 0.0500 time 0.3729 (0.3822) data time 0.0011 (0.0030) model time 0.3718 (0.3804) loss 3.0929 (3.4300) grad_norm 1.4746 (1.3463) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][230/625] eta 0:02:30 lr 0.001760 wd 0.0500 time 0.3837 (0.3820) data time 0.0008 (0.0029) model time 0.3829 (0.3802) loss 3.8262 (3.4331) grad_norm 1.1509 (1.3461) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][240/625] eta 0:02:27 lr 0.001760 wd 0.0500 time 0.3797 (0.3819) data time 0.0011 (0.0028) model time 0.3785 (0.3800) loss 2.7360 (3.4291) grad_norm 1.4028 (1.3426) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][250/625] eta 0:02:23 lr 0.001760 wd 0.0500 time 0.5533 (0.3832) data time 0.0008 (0.0028) model time 0.5525 (0.3817) loss 3.4976 (3.4300) grad_norm 1.1893 (1.3408) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][260/625] eta 0:02:19 lr 0.001760 wd 0.0500 time 0.3841 (0.3830) data time 0.0011 (0.0027) model time 0.3830 (0.3815) loss 3.3599 (3.4223) grad_norm 1.2816 (1.3346) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:36:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][270/625] eta 0:02:15 lr 0.001760 wd 0.0500 time 0.3820 (0.3828) data time 0.0011 (0.0026) model time 0.3809 (0.3812) loss 2.7316 (3.4171) grad_norm 1.8139 (1.3338) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][280/625] eta 0:02:12 lr 0.001760 wd 0.0500 time 0.3800 (0.3833) data time 0.0009 (0.0026) model time 0.3791 (0.3819) loss 3.0564 (3.4148) grad_norm 1.8115 (1.3370) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][290/625] eta 0:02:08 lr 0.001759 wd 0.0500 time 0.3774 (0.3838) data time 0.0011 (0.0025) model time 0.3763 (0.3825) loss 3.6045 (3.4209) grad_norm 0.8777 (1.3402) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][300/625] eta 0:02:04 lr 0.001759 wd 0.0500 time 0.3754 (0.3836) data time 0.0021 (0.0025) model time 0.3733 (0.3823) loss 3.6468 (3.4211) grad_norm 1.1621 (1.3369) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][310/625] eta 0:02:00 lr 0.001759 wd 0.0500 time 0.3720 (0.3834) data time 0.0010 (0.0025) model time 0.3710 (0.3820) loss 3.3192 (3.4307) grad_norm 3.3181 (1.3460) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][320/625] eta 0:01:57 lr 0.001759 wd 0.0500 time 0.3759 (0.3837) data time 0.0012 (0.0024) model time 0.3747 (0.3824) loss 2.9347 (3.4321) grad_norm 1.0226 (1.3494) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][330/625] eta 0:01:53 lr 0.001759 wd 0.0500 time 0.3831 (0.3836) data time 0.0012 (0.0024) model time 0.3820 (0.3822) loss 3.2311 (3.4231) grad_norm 1.2934 (1.3481) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][340/625] eta 0:01:49 lr 0.001759 wd 0.0500 time 0.3889 (0.3834) data time 0.0009 (0.0023) model time 0.3881 (0.3821) loss 3.1065 (3.4250) grad_norm 1.5962 (1.3472) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][350/625] eta 0:01:45 lr 0.001759 wd 0.0500 time 0.3777 (0.3833) data time 0.0011 (0.0023) model time 0.3766 (0.3819) loss 4.1975 (3.4195) grad_norm 1.0158 (1.3480) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][360/625] eta 0:01:41 lr 0.001759 wd 0.0500 time 0.3699 (0.3830) data time 0.0008 (0.0023) model time 0.3691 (0.3817) loss 3.5789 (3.4179) grad_norm 1.4226 (1.3451) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][370/625] eta 0:01:37 lr 0.001759 wd 0.0500 time 0.5780 (0.3840) data time 0.0010 (0.0022) model time 0.5770 (0.3827) loss 3.5147 (3.4207) grad_norm 1.0608 (1.3420) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][380/625] eta 0:01:34 lr 0.001758 wd 0.0500 time 0.3750 (0.3837) data time 0.0008 (0.0022) model time 0.3742 (0.3825) loss 3.6873 (3.4236) grad_norm 1.0129 (1.3397) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][390/625] eta 0:01:30 lr 0.001758 wd 0.0500 time 0.3802 (0.3835) data time 0.0009 (0.0022) model time 0.3793 (0.3822) loss 4.0754 (3.4288) grad_norm 1.0457 (1.3378) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][400/625] eta 0:01:26 lr 0.001758 wd 0.0500 time 0.3819 (0.3835) data time 0.0009 (0.0022) model time 0.3810 (0.3822) loss 2.9281 (3.4273) grad_norm 1.5503 (1.3409) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][410/625] eta 0:01:22 lr 0.001758 wd 0.0500 time 0.3756 (0.3833) data time 0.0012 (0.0021) model time 0.3744 (0.3820) loss 3.5654 (3.4296) grad_norm 0.9254 (1.3369) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][420/625] eta 0:01:18 lr 0.001758 wd 0.0500 time 0.3722 (0.3832) data time 0.0011 (0.0021) model time 0.3711 (0.3819) loss 4.0130 (3.4289) grad_norm 1.6154 (1.3329) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:37:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][430/625] eta 0:01:14 lr 0.001758 wd 0.0500 time 0.3809 (0.3830) data time 0.0008 (0.0021) model time 0.3801 (0.3817) loss 4.2068 (3.4310) grad_norm 1.3178 (1.3335) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][440/625] eta 0:01:10 lr 0.001758 wd 0.0500 time 0.3569 (0.3834) data time 0.0008 (0.0021) model time 0.3561 (0.3821) loss 3.3047 (3.4306) grad_norm 1.0473 (1.3350) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][450/625] eta 0:01:07 lr 0.001758 wd 0.0500 time 0.3758 (0.3832) data time 0.0011 (0.0020) model time 0.3747 (0.3819) loss 3.7469 (3.4308) grad_norm 1.1533 (1.3341) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][460/625] eta 0:01:03 lr 0.001757 wd 0.0500 time 0.5707 (0.3835) data time 0.0009 (0.0020) model time 0.5699 (0.3822) loss 2.8178 (3.4277) grad_norm 0.9804 (1.3307) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][470/625] eta 0:00:59 lr 0.001757 wd 0.0500 time 0.3775 (0.3838) data time 0.0010 (0.0020) model time 0.3765 (0.3826) loss 3.7511 (3.4220) grad_norm 1.4080 (1.3263) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][480/625] eta 0:00:55 lr 0.001757 wd 0.0500 time 0.3756 (0.3837) data time 0.0010 (0.0020) model time 0.3746 (0.3825) loss 2.6906 (3.4207) grad_norm 1.5764 (1.3266) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][490/625] eta 0:00:51 lr 0.001757 wd 0.0500 time 0.3758 (0.3836) data time 0.0011 (0.0020) model time 0.3747 (0.3824) loss 3.1632 (3.4207) grad_norm 0.9445 (1.3280) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][500/625] eta 0:00:47 lr 0.001757 wd 0.0500 time 0.3738 (0.3835) data time 0.0011 (0.0020) model time 0.3727 (0.3822) loss 2.7731 (3.4205) grad_norm 1.1119 (1.3251) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][510/625] eta 0:00:44 lr 0.001757 wd 0.0500 time 0.3785 (0.3834) data time 0.0011 (0.0019) model time 0.3774 (0.3821) loss 3.7905 (3.4141) grad_norm 1.8550 (1.3277) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][520/625] eta 0:00:40 lr 0.001757 wd 0.0500 time 0.3757 (0.3832) data time 0.0012 (0.0019) model time 0.3745 (0.3819) loss 3.2085 (3.4088) grad_norm 1.1437 (1.3246) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][530/625] eta 0:00:36 lr 0.001757 wd 0.0500 time 0.3752 (0.3831) data time 0.0012 (0.0019) model time 0.3740 (0.3818) loss 3.8929 (3.4060) grad_norm 1.0785 (1.3275) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][540/625] eta 0:00:32 lr 0.001757 wd 0.0500 time 0.3784 (0.3830) data time 0.0009 (0.0019) model time 0.3776 (0.3817) loss 4.3088 (3.4062) grad_norm 1.1462 (1.3225) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][550/625] eta 0:00:28 lr 0.001756 wd 0.0500 time 0.3787 (0.3829) data time 0.0011 (0.0019) model time 0.3776 (0.3816) loss 3.7162 (3.4054) grad_norm 1.6034 (1.3278) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][560/625] eta 0:00:24 lr 0.001756 wd 0.0500 time 0.3787 (0.3828) data time 0.0011 (0.0019) model time 0.3776 (0.3815) loss 3.4738 (3.4043) grad_norm 1.2900 (1.3276) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][570/625] eta 0:00:21 lr 0.001756 wd 0.0500 time 0.5832 (0.3831) data time 0.0009 (0.0019) model time 0.5823 (0.3819) loss 2.7933 (3.4026) grad_norm 1.0459 (1.3267) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][580/625] eta 0:00:17 lr 0.001756 wd 0.0500 time 0.5854 (0.3838) data time 0.0011 (0.0018) model time 0.5843 (0.3826) loss 3.5365 (3.4081) grad_norm 1.0818 (1.3280) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][590/625] eta 0:00:13 lr 0.001756 wd 0.0500 time 0.3764 (0.3840) data time 0.0010 (0.0018) model time 0.3753 (0.3828) loss 2.9596 (3.4079) grad_norm 1.4019 (1.3260) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:39:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][600/625] eta 0:00:09 lr 0.001756 wd 0.0500 time 0.3743 (0.3839) data time 0.0008 (0.0018) model time 0.3734 (0.3827) loss 4.3175 (3.4111) grad_norm 1.7437 (1.3300) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][610/625] eta 0:00:05 lr 0.001756 wd 0.0500 time 0.3636 (0.3837) data time 0.0006 (0.0018) model time 0.3630 (0.3825) loss 4.0666 (3.4118) grad_norm 1.1893 (1.3318) loss_scale 8192.0000 (4136.2226) mem 11225MB [2024-08-26 08:39:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [83/300][620/625] eta 0:00:01 lr 0.001756 wd 0.0500 time 0.3611 (0.3834) data time 0.0008 (0.0018) model time 0.3603 (0.3822) loss 2.9659 (3.4142) grad_norm 1.1359 (1.3346) loss_scale 8192.0000 (4201.5330) mem 11225MB [2024-08-26 08:39:12 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 83 training takes 0:03:59 [2024-08-26 08:39:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:39:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.6714 (0.6714) Acc@1 85.400 (85.400) Acc@5 97.461 (97.461) Mem 11225MB [2024-08-26 08:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.0928 (0.8098) Acc@1 74.023 (81.286) Acc@5 93.652 (96.249) Mem 11225MB [2024-08-26 08:39:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.2441 (0.9850) Acc@1 70.801 (77.279) Acc@5 90.381 (93.957) Mem 11225MB [2024-08-26 08:39:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.031 Acc@5 93.914 [2024-08-26 08:39:16 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.0% [2024-08-26 08:39:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.815 (0.815) Loss 0.5605 (0.5605) Acc@1 87.939 (87.939) Acc@5 98.242 (98.242) Mem 11225MB [2024-08-26 08:39:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.173) Loss 0.9390 (0.7018) Acc@1 77.393 (83.958) Acc@5 94.824 (97.026) Mem 11225MB [2024-08-26 08:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.135) Loss 1.1055 (0.8565) Acc@1 72.559 (80.108) Acc@5 92.578 (95.240) Mem 11225MB [2024-08-26 08:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.796 Acc@5 95.200 [2024-08-26 08:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.8% [2024-08-26 08:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.80% [2024-08-26 08:39:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:39:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:39:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][0/625] eta 0:07:51 lr 0.001756 wd 0.0500 time 0.7540 (0.7540) data time 0.4201 (0.4201) model time 0.0000 (0.0000) loss 4.2953 (4.2953) grad_norm 1.5996 (1.5996) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][10/625] eta 0:04:12 lr 0.001755 wd 0.0500 time 0.3792 (0.4113) data time 0.0008 (0.0393) model time 0.0000 (0.0000) loss 3.6779 (3.5920) grad_norm 1.2342 (1.2945) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][20/625] eta 0:03:59 lr 0.001755 wd 0.0500 time 0.3784 (0.3954) data time 0.0008 (0.0212) model time 0.0000 (0.0000) loss 2.5092 (3.4270) grad_norm 0.9469 (1.2953) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][30/625] eta 0:03:51 lr 0.001755 wd 0.0500 time 0.3808 (0.3897) data time 0.0008 (0.0147) model time 0.0000 (0.0000) loss 4.0815 (3.5370) grad_norm 1.0367 (1.2545) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][40/625] eta 0:03:46 lr 0.001755 wd 0.0500 time 0.3758 (0.3865) data time 0.0011 (0.0114) model time 0.0000 (0.0000) loss 3.7251 (3.5461) grad_norm 1.1373 (1.2633) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][50/625] eta 0:03:41 lr 0.001755 wd 0.0500 time 0.3757 (0.3846) data time 0.0011 (0.0093) model time 0.0000 (0.0000) loss 3.8268 (3.4894) grad_norm 1.1151 (1.2747) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][60/625] eta 0:03:36 lr 0.001755 wd 0.0500 time 0.3792 (0.3832) data time 0.0008 (0.0080) model time 0.3783 (0.3753) loss 3.2484 (3.4867) grad_norm 1.0977 (1.2565) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][70/625] eta 0:03:32 lr 0.001755 wd 0.0500 time 0.3770 (0.3824) data time 0.0008 (0.0070) model time 0.3762 (0.3758) loss 2.5070 (3.4751) grad_norm 1.2178 (1.2580) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][80/625] eta 0:03:27 lr 0.001755 wd 0.0500 time 0.3737 (0.3816) data time 0.0011 (0.0063) model time 0.3726 (0.3754) loss 3.0499 (3.4332) grad_norm 1.2529 (1.2541) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][90/625] eta 0:03:23 lr 0.001754 wd 0.0500 time 0.3707 (0.3812) data time 0.0010 (0.0058) model time 0.3696 (0.3756) loss 3.5495 (3.4301) grad_norm 1.1770 (1.2447) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:39:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][100/625] eta 0:03:22 lr 0.001754 wd 0.0500 time 0.3942 (0.3851) data time 0.0008 (0.0053) model time 0.3933 (0.3845) loss 4.3929 (3.4162) grad_norm 2.0808 (1.2505) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][110/625] eta 0:03:17 lr 0.001754 wd 0.0500 time 0.3736 (0.3844) data time 0.0008 (0.0049) model time 0.3728 (0.3830) loss 2.0711 (3.4101) grad_norm 1.0143 (1.2536) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][120/625] eta 0:03:13 lr 0.001754 wd 0.0500 time 0.3709 (0.3837) data time 0.0012 (0.0046) model time 0.3697 (0.3819) loss 3.6646 (3.4259) grad_norm 1.4649 (1.2568) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][130/625] eta 0:03:09 lr 0.001754 wd 0.0500 time 0.3907 (0.3834) data time 0.0011 (0.0044) model time 0.3897 (0.3815) loss 3.4464 (3.4436) grad_norm 1.3968 (1.2521) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][140/625] eta 0:03:05 lr 0.001754 wd 0.0500 time 0.3702 (0.3829) data time 0.0011 (0.0041) model time 0.3690 (0.3808) loss 3.8071 (3.4441) grad_norm 0.9821 (1.2610) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][150/625] eta 0:03:02 lr 0.001754 wd 0.0500 time 0.3818 (0.3838) data time 0.0008 (0.0039) model time 0.3810 (0.3822) loss 3.5035 (3.4452) grad_norm 1.1045 (1.2594) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][160/625] eta 0:02:58 lr 0.001754 wd 0.0500 time 0.3783 (0.3848) data time 0.0012 (0.0038) model time 0.3772 (0.3838) loss 3.4192 (3.4530) grad_norm 1.2745 (1.2862) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][170/625] eta 0:02:55 lr 0.001754 wd 0.0500 time 0.3722 (0.3851) data time 0.0011 (0.0036) model time 0.3711 (0.3841) loss 3.9195 (3.4517) grad_norm 1.0561 (1.2847) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][180/625] eta 0:02:51 lr 0.001753 wd 0.0500 time 0.3695 (0.3845) data time 0.0009 (0.0035) model time 0.3686 (0.3833) loss 4.0944 (3.4596) grad_norm 1.3380 (1.2847) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][190/625] eta 0:02:47 lr 0.001753 wd 0.0500 time 0.3725 (0.3841) data time 0.0009 (0.0033) model time 0.3717 (0.3827) loss 3.8850 (3.4683) grad_norm 1.3841 (1.2817) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][200/625] eta 0:02:43 lr 0.001753 wd 0.0500 time 0.3743 (0.3837) data time 0.0010 (0.0032) model time 0.3733 (0.3823) loss 3.8416 (3.4796) grad_norm 2.1380 (1.2941) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][210/625] eta 0:02:39 lr 0.001753 wd 0.0500 time 0.3834 (0.3834) data time 0.0011 (0.0031) model time 0.3823 (0.3819) loss 3.5378 (3.4774) grad_norm 1.2969 (1.3013) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][220/625] eta 0:02:35 lr 0.001753 wd 0.0500 time 0.3801 (0.3831) data time 0.0011 (0.0030) model time 0.3791 (0.3815) loss 3.5871 (3.4785) grad_norm 1.3147 (1.2997) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][230/625] eta 0:02:31 lr 0.001753 wd 0.0500 time 0.3740 (0.3828) data time 0.0011 (0.0030) model time 0.3729 (0.3811) loss 3.1754 (3.4848) grad_norm 1.0601 (1.3019) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][240/625] eta 0:02:27 lr 0.001753 wd 0.0500 time 0.3740 (0.3825) data time 0.0011 (0.0029) model time 0.3729 (0.3808) loss 2.6078 (3.4912) grad_norm 0.7634 (1.2944) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:40:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][250/625] eta 0:02:23 lr 0.001753 wd 0.0500 time 0.3737 (0.3822) data time 0.0012 (0.0028) model time 0.3725 (0.3805) loss 3.1078 (3.4730) grad_norm 1.4011 (1.2910) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:41:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][260/625] eta 0:02:19 lr 0.001752 wd 0.0500 time 0.3758 (0.3819) data time 0.0011 (0.0028) model time 0.3748 (0.3801) loss 3.3366 (3.4772) grad_norm 1.3898 (inf) loss_scale 4096.0000 (8066.4521) mem 11225MB [2024-08-26 08:41:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][270/625] eta 0:02:15 lr 0.001752 wd 0.0500 time 0.3787 (0.3818) data time 0.0012 (0.0027) model time 0.3775 (0.3799) loss 3.5890 (3.4733) grad_norm 1.1026 (inf) loss_scale 4096.0000 (7919.9410) mem 11225MB [2024-08-26 08:41:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][280/625] eta 0:02:11 lr 0.001752 wd 0.0500 time 0.3721 (0.3816) data time 0.0011 (0.0026) model time 0.3710 (0.3798) loss 3.3591 (3.4809) grad_norm 1.2339 (inf) loss_scale 4096.0000 (7783.8577) mem 11225MB [2024-08-26 08:41:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][290/625] eta 0:02:08 lr 0.001752 wd 0.0500 time 0.3751 (0.3821) data time 0.0012 (0.0026) model time 0.3739 (0.3804) loss 3.7223 (3.4794) grad_norm 2.2971 (inf) loss_scale 4096.0000 (7657.1271) mem 11225MB [2024-08-26 08:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][300/625] eta 0:02:04 lr 0.001752 wd 0.0500 time 0.3775 (0.3820) data time 0.0011 (0.0025) model time 0.3764 (0.3803) loss 2.5518 (3.4746) grad_norm 1.1809 (inf) loss_scale 4096.0000 (7538.8173) mem 11225MB [2024-08-26 08:41:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][310/625] eta 0:02:00 lr 0.001752 wd 0.0500 time 0.3896 (0.3833) data time 0.0009 (0.0025) model time 0.3887 (0.3819) loss 3.9925 (3.4732) grad_norm 2.1280 (inf) loss_scale 4096.0000 (7428.1158) mem 11225MB [2024-08-26 08:41:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][320/625] eta 0:01:56 lr 0.001752 wd 0.0500 time 0.3689 (0.3832) data time 0.0012 (0.0025) model time 0.3678 (0.3817) loss 3.4705 (3.4763) grad_norm 1.3282 (inf) loss_scale 4096.0000 (7324.3115) mem 11225MB [2024-08-26 08:41:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][330/625] eta 0:01:52 lr 0.001752 wd 0.0500 time 0.3674 (0.3830) data time 0.0008 (0.0024) model time 0.3665 (0.3815) loss 4.1677 (3.4787) grad_norm 1.3645 (inf) loss_scale 4096.0000 (7226.7795) mem 11225MB [2024-08-26 08:41:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][340/625] eta 0:01:49 lr 0.001752 wd 0.0500 time 0.3720 (0.3834) data time 0.0008 (0.0024) model time 0.3712 (0.3820) loss 3.7363 (3.4773) grad_norm 0.9672 (inf) loss_scale 4096.0000 (7134.9677) mem 11225MB [2024-08-26 08:41:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][350/625] eta 0:01:45 lr 0.001751 wd 0.0500 time 0.3744 (0.3838) data time 0.0011 (0.0023) model time 0.3733 (0.3825) loss 3.3141 (3.4815) grad_norm 1.3680 (inf) loss_scale 4096.0000 (7048.3875) mem 11225MB [2024-08-26 08:41:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][360/625] eta 0:01:41 lr 0.001751 wd 0.0500 time 0.3799 (0.3837) data time 0.0008 (0.0023) model time 0.3791 (0.3824) loss 3.6060 (3.4834) grad_norm 1.2132 (inf) loss_scale 4096.0000 (6966.6039) mem 11225MB [2024-08-26 08:41:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][370/625] eta 0:01:37 lr 0.001751 wd 0.0500 time 0.3804 (0.3835) data time 0.0008 (0.0023) model time 0.3796 (0.3822) loss 3.9273 (3.4847) grad_norm 1.5123 (inf) loss_scale 4096.0000 (6889.2291) mem 11225MB [2024-08-26 08:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][380/625] eta 0:01:33 lr 0.001751 wd 0.0500 time 0.3693 (0.3834) data time 0.0011 (0.0022) model time 0.3682 (0.3821) loss 3.7089 (3.4843) grad_norm 0.8895 (inf) loss_scale 4096.0000 (6815.9160) mem 11225MB [2024-08-26 08:41:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][390/625] eta 0:01:30 lr 0.001751 wd 0.0500 time 0.3802 (0.3833) data time 0.0011 (0.0022) model time 0.3791 (0.3819) loss 3.2656 (3.4825) grad_norm 1.1466 (inf) loss_scale 4096.0000 (6746.3529) mem 11225MB [2024-08-26 08:41:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][400/625] eta 0:01:26 lr 0.001751 wd 0.0500 time 0.3830 (0.3832) data time 0.0008 (0.0022) model time 0.3822 (0.3818) loss 2.6669 (3.4845) grad_norm 1.1108 (inf) loss_scale 4096.0000 (6680.2594) mem 11225MB [2024-08-26 08:41:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][410/625] eta 0:01:22 lr 0.001751 wd 0.0500 time 0.3765 (0.3831) data time 0.0009 (0.0022) model time 0.3757 (0.3817) loss 2.1421 (3.4822) grad_norm 1.6083 (inf) loss_scale 4096.0000 (6617.3820) mem 11225MB [2024-08-26 08:42:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][420/625] eta 0:01:18 lr 0.001751 wd 0.0500 time 0.3771 (0.3829) data time 0.0010 (0.0021) model time 0.3762 (0.3816) loss 3.9407 (3.4751) grad_norm 1.2873 (inf) loss_scale 4096.0000 (6557.4917) mem 11225MB [2024-08-26 08:42:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][430/625] eta 0:01:14 lr 0.001750 wd 0.0500 time 0.3759 (0.3828) data time 0.0011 (0.0021) model time 0.3748 (0.3815) loss 3.3109 (3.4758) grad_norm 1.3471 (inf) loss_scale 4096.0000 (6500.3805) mem 11225MB [2024-08-26 08:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][440/625] eta 0:01:10 lr 0.001750 wd 0.0500 time 0.3794 (0.3827) data time 0.0008 (0.0021) model time 0.3786 (0.3813) loss 2.5208 (3.4680) grad_norm 1.4207 (inf) loss_scale 4096.0000 (6445.8594) mem 11225MB [2024-08-26 08:42:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][450/625] eta 0:01:06 lr 0.001750 wd 0.0500 time 0.3828 (0.3826) data time 0.0008 (0.0021) model time 0.3820 (0.3812) loss 2.3903 (3.4649) grad_norm 1.3198 (inf) loss_scale 4096.0000 (6393.7561) mem 11225MB [2024-08-26 08:42:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][460/625] eta 0:01:03 lr 0.001750 wd 0.0500 time 0.5823 (0.3834) data time 0.0008 (0.0021) model time 0.5814 (0.3821) loss 2.9928 (3.4667) grad_norm 1.6406 (inf) loss_scale 4096.0000 (6343.9132) mem 11225MB [2024-08-26 08:42:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][470/625] eta 0:00:59 lr 0.001750 wd 0.0500 time 0.3739 (0.3833) data time 0.0011 (0.0020) model time 0.3728 (0.3820) loss 3.6187 (3.4698) grad_norm 1.1664 (inf) loss_scale 4096.0000 (6296.1868) mem 11225MB [2024-08-26 08:42:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][480/625] eta 0:00:55 lr 0.001750 wd 0.0500 time 0.3782 (0.3832) data time 0.0011 (0.0020) model time 0.3771 (0.3819) loss 3.6683 (3.4652) grad_norm 1.2320 (inf) loss_scale 4096.0000 (6250.4449) mem 11225MB [2024-08-26 08:42:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][490/625] eta 0:00:51 lr 0.001750 wd 0.0500 time 0.3741 (0.3838) data time 0.0008 (0.0020) model time 0.3733 (0.3826) loss 3.5093 (3.4652) grad_norm 1.4111 (inf) loss_scale 4096.0000 (6206.5662) mem 11225MB [2024-08-26 08:42:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][500/625] eta 0:00:48 lr 0.001750 wd 0.0500 time 0.3690 (0.3841) data time 0.0009 (0.0020) model time 0.3681 (0.3829) loss 3.3235 (3.4638) grad_norm 1.1956 (inf) loss_scale 4096.0000 (6164.4391) mem 11225MB [2024-08-26 08:42:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][510/625] eta 0:00:44 lr 0.001750 wd 0.0500 time 0.3752 (0.3840) data time 0.0011 (0.0020) model time 0.3741 (0.3827) loss 3.5408 (3.4626) grad_norm 0.7919 (inf) loss_scale 4096.0000 (6123.9609) mem 11225MB [2024-08-26 08:42:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][520/625] eta 0:00:40 lr 0.001749 wd 0.0500 time 0.3811 (0.3838) data time 0.0009 (0.0019) model time 0.3803 (0.3826) loss 2.5133 (3.4617) grad_norm 1.2040 (inf) loss_scale 4096.0000 (6085.0365) mem 11225MB [2024-08-26 08:42:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][530/625] eta 0:00:36 lr 0.001749 wd 0.0500 time 0.3770 (0.3837) data time 0.0009 (0.0019) model time 0.3761 (0.3825) loss 2.4266 (3.4570) grad_norm 1.7791 (inf) loss_scale 4096.0000 (6047.5782) mem 11225MB [2024-08-26 08:42:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][540/625] eta 0:00:32 lr 0.001749 wd 0.0500 time 0.3745 (0.3836) data time 0.0009 (0.0019) model time 0.3737 (0.3823) loss 4.0804 (3.4586) grad_norm 1.6000 (inf) loss_scale 4096.0000 (6011.5046) mem 11225MB [2024-08-26 08:42:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][550/625] eta 0:00:28 lr 0.001749 wd 0.0500 time 0.3793 (0.3835) data time 0.0011 (0.0019) model time 0.3782 (0.3822) loss 2.4265 (3.4635) grad_norm 1.0776 (inf) loss_scale 4096.0000 (5976.7405) mem 11225MB [2024-08-26 08:42:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][560/625] eta 0:00:24 lr 0.001749 wd 0.0500 time 0.3771 (0.3834) data time 0.0012 (0.0019) model time 0.3759 (0.3821) loss 3.0682 (3.4617) grad_norm 1.1306 (inf) loss_scale 4096.0000 (5943.2157) mem 11225MB [2024-08-26 08:42:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][570/625] eta 0:00:21 lr 0.001749 wd 0.0500 time 0.3773 (0.3833) data time 0.0011 (0.0019) model time 0.3762 (0.3820) loss 2.6716 (3.4547) grad_norm 1.1433 (inf) loss_scale 4096.0000 (5910.8651) mem 11225MB [2024-08-26 08:43:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][580/625] eta 0:00:17 lr 0.001749 wd 0.0500 time 0.3852 (0.3832) data time 0.0009 (0.0019) model time 0.3844 (0.3819) loss 3.3397 (3.4578) grad_norm 1.4811 (inf) loss_scale 4096.0000 (5879.6282) mem 11225MB [2024-08-26 08:43:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][590/625] eta 0:00:13 lr 0.001749 wd 0.0500 time 0.3793 (0.3831) data time 0.0011 (0.0019) model time 0.3782 (0.3819) loss 3.6595 (3.4552) grad_norm 1.2047 (inf) loss_scale 4096.0000 (5849.4484) mem 11225MB [2024-08-26 08:43:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][600/625] eta 0:00:09 lr 0.001748 wd 0.0500 time 0.3750 (0.3830) data time 0.0008 (0.0018) model time 0.3741 (0.3817) loss 4.5828 (3.4593) grad_norm 1.2941 (inf) loss_scale 4096.0000 (5820.2729) mem 11225MB [2024-08-26 08:43:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][610/625] eta 0:00:05 lr 0.001748 wd 0.0500 time 0.3672 (0.3836) data time 0.0008 (0.0018) model time 0.3664 (0.3823) loss 2.9282 (3.4577) grad_norm 1.0654 (inf) loss_scale 4096.0000 (5792.0524) mem 11225MB [2024-08-26 08:43:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [84/300][620/625] eta 0:00:01 lr 0.001748 wd 0.0500 time 0.3700 (0.3833) data time 0.0008 (0.0018) model time 0.3692 (0.3821) loss 3.0509 (3.4578) grad_norm 1.1690 (inf) loss_scale 4096.0000 (5764.7407) mem 11225MB [2024-08-26 08:43:20 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 84 training takes 0:03:59 [2024-08-26 08:43:20 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:43:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:43:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.6890 (0.6890) Acc@1 85.059 (85.059) Acc@5 97.217 (97.217) Mem 11225MB [2024-08-26 08:43:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.141) Loss 1.0977 (0.8319) Acc@1 74.414 (81.254) Acc@5 92.432 (95.974) Mem 11225MB [2024-08-26 08:43:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.2871 (1.0065) Acc@1 69.922 (77.081) Acc@5 90.820 (93.927) Mem 11225MB [2024-08-26 08:43:24 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.891 Acc@5 93.910 [2024-08-26 08:43:24 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.9% [2024-08-26 08:43:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.920 (0.920) Loss 0.5610 (0.5610) Acc@1 87.939 (87.939) Acc@5 98.193 (98.193) Mem 11225MB [2024-08-26 08:43:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.183) Loss 0.9365 (0.7008) Acc@1 77.344 (83.993) Acc@5 94.873 (97.030) Mem 11225MB [2024-08-26 08:43:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.141) Loss 1.1016 (0.8547) Acc@1 72.559 (80.143) Acc@5 92.432 (95.264) Mem 11225MB [2024-08-26 08:43:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.850 Acc@5 95.228 [2024-08-26 08:43:27 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.9% [2024-08-26 08:43:27 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.85% [2024-08-26 08:43:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:43:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:43:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][0/625] eta 0:07:55 lr 0.001748 wd 0.0500 time 0.7607 (0.7607) data time 0.4261 (0.4261) model time 0.0000 (0.0000) loss 3.7878 (3.7878) grad_norm 1.7145 (1.7145) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][10/625] eta 0:04:25 lr 0.001748 wd 0.0500 time 0.3770 (0.4312) data time 0.0008 (0.0399) model time 0.0000 (0.0000) loss 4.2570 (3.7452) grad_norm 1.1653 (1.7047) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:43:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][20/625] eta 0:04:10 lr 0.001748 wd 0.0500 time 0.3717 (0.4148) data time 0.0011 (0.0214) model time 0.0000 (0.0000) loss 3.6021 (3.6765) grad_norm 1.1668 (1.5421) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:43:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][30/625] eta 0:04:02 lr 0.001748 wd 0.0500 time 0.3704 (0.4080) data time 0.0010 (0.0149) model time 0.0000 (0.0000) loss 3.7020 (3.5687) grad_norm 0.9349 (1.4309) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:43:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][40/625] eta 0:03:54 lr 0.001748 wd 0.0500 time 0.3771 (0.4003) data time 0.0011 (0.0115) model time 0.0000 (0.0000) loss 3.6374 (3.5950) grad_norm 1.8802 (1.3969) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:43:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][50/625] eta 0:03:47 lr 0.001748 wd 0.0500 time 0.3771 (0.3954) data time 0.0011 (0.0094) model time 0.0000 (0.0000) loss 2.6654 (3.5378) grad_norm 1.1252 (1.3632) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:43:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][60/625] eta 0:03:41 lr 0.001747 wd 0.0500 time 0.3720 (0.3924) data time 0.0012 (0.0081) model time 0.3708 (0.3759) loss 3.6956 (3.5153) grad_norm 1.4762 (1.3727) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:43:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][70/625] eta 0:03:36 lr 0.001747 wd 0.0500 time 0.3763 (0.3904) data time 0.0011 (0.0071) model time 0.3753 (0.3764) loss 3.6458 (3.5027) grad_norm 1.0276 (1.3490) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][80/625] eta 0:03:31 lr 0.001747 wd 0.0500 time 0.3734 (0.3886) data time 0.0013 (0.0064) model time 0.3722 (0.3760) loss 3.7573 (3.4779) grad_norm 1.0373 (1.3350) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][90/625] eta 0:03:27 lr 0.001747 wd 0.0500 time 0.3695 (0.3870) data time 0.0011 (0.0058) model time 0.3684 (0.3751) loss 3.4776 (3.4394) grad_norm 1.0019 (1.3244) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][100/625] eta 0:03:22 lr 0.001747 wd 0.0500 time 0.3769 (0.3858) data time 0.0011 (0.0054) model time 0.3759 (0.3749) loss 3.2710 (3.4263) grad_norm 1.1151 (1.3049) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][110/625] eta 0:03:18 lr 0.001747 wd 0.0500 time 0.3772 (0.3852) data time 0.0008 (0.0050) model time 0.3763 (0.3753) loss 3.8642 (3.4215) grad_norm 1.8896 (1.3135) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][120/625] eta 0:03:15 lr 0.001747 wd 0.0500 time 0.6075 (0.3864) data time 0.0008 (0.0047) model time 0.6066 (0.3788) loss 2.2944 (3.3929) grad_norm 1.2479 (1.3147) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][130/625] eta 0:03:10 lr 0.001747 wd 0.0500 time 0.3726 (0.3854) data time 0.0008 (0.0044) model time 0.3717 (0.3779) loss 3.9887 (3.3864) grad_norm 1.1244 (1.3181) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][140/625] eta 0:03:06 lr 0.001747 wd 0.0500 time 0.3823 (0.3848) data time 0.0011 (0.0041) model time 0.3812 (0.3776) loss 3.9076 (3.4070) grad_norm 1.6279 (1.3256) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][150/625] eta 0:03:02 lr 0.001746 wd 0.0500 time 0.3782 (0.3843) data time 0.0011 (0.0040) model time 0.3770 (0.3775) loss 3.4734 (3.4058) grad_norm 1.3003 (1.3320) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][160/625] eta 0:02:58 lr 0.001746 wd 0.0500 time 0.3793 (0.3839) data time 0.0009 (0.0038) model time 0.3784 (0.3775) loss 3.4390 (3.4045) grad_norm 0.9585 (1.3215) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][170/625] eta 0:02:55 lr 0.001746 wd 0.0500 time 0.3727 (0.3858) data time 0.0009 (0.0036) model time 0.3718 (0.3806) loss 3.0153 (3.4196) grad_norm 0.9293 (1.3161) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][180/625] eta 0:02:51 lr 0.001746 wd 0.0500 time 0.3784 (0.3853) data time 0.0008 (0.0035) model time 0.3776 (0.3803) loss 2.3509 (3.4175) grad_norm 1.0993 (1.3129) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][190/625] eta 0:02:47 lr 0.001746 wd 0.0500 time 0.3835 (0.3850) data time 0.0011 (0.0034) model time 0.3824 (0.3801) loss 3.9100 (3.4159) grad_norm 1.2834 (1.3238) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][200/625] eta 0:02:43 lr 0.001746 wd 0.0500 time 0.3759 (0.3846) data time 0.0009 (0.0032) model time 0.3750 (0.3799) loss 3.2593 (3.4176) grad_norm 2.1965 (1.3284) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][210/625] eta 0:02:39 lr 0.001746 wd 0.0500 time 0.3720 (0.3843) data time 0.0010 (0.0031) model time 0.3710 (0.3797) loss 3.5631 (3.4130) grad_norm 1.1252 (1.3242) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][220/625] eta 0:02:35 lr 0.001746 wd 0.0500 time 0.3775 (0.3840) data time 0.0009 (0.0030) model time 0.3766 (0.3795) loss 3.3468 (3.4080) grad_norm 1.4436 (1.3236) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:44:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][230/625] eta 0:02:32 lr 0.001745 wd 0.0500 time 0.5875 (0.3851) data time 0.0010 (0.0030) model time 0.5865 (0.3811) loss 3.7393 (3.4125) grad_norm 1.0000 (1.3150) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][240/625] eta 0:02:28 lr 0.001745 wd 0.0500 time 0.3830 (0.3857) data time 0.0009 (0.0029) model time 0.3821 (0.3820) loss 4.0164 (3.4235) grad_norm 1.2029 (1.3108) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][250/625] eta 0:02:24 lr 0.001745 wd 0.0500 time 0.3769 (0.3855) data time 0.0012 (0.0028) model time 0.3757 (0.3818) loss 3.3959 (3.4260) grad_norm 1.2692 (1.3056) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][260/625] eta 0:02:20 lr 0.001745 wd 0.0500 time 0.3764 (0.3851) data time 0.0009 (0.0028) model time 0.3755 (0.3814) loss 3.1531 (3.4239) grad_norm 1.3444 (1.3079) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][270/625] eta 0:02:16 lr 0.001745 wd 0.0500 time 0.3680 (0.3849) data time 0.0009 (0.0027) model time 0.3672 (0.3813) loss 3.1248 (3.4253) grad_norm 1.4530 (1.3048) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][280/625] eta 0:02:12 lr 0.001745 wd 0.0500 time 0.3771 (0.3846) data time 0.0009 (0.0026) model time 0.3762 (0.3810) loss 2.4265 (3.4161) grad_norm 1.0384 (1.2995) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][290/625] eta 0:02:09 lr 0.001745 wd 0.0500 time 0.3768 (0.3858) data time 0.0011 (0.0026) model time 0.3757 (0.3826) loss 3.8129 (3.4229) grad_norm 1.7309 (1.3009) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][300/625] eta 0:02:05 lr 0.001745 wd 0.0500 time 0.3708 (0.3856) data time 0.0012 (0.0025) model time 0.3696 (0.3824) loss 3.5101 (3.4272) grad_norm 1.5222 (1.3094) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][310/625] eta 0:02:01 lr 0.001745 wd 0.0500 time 0.3792 (0.3853) data time 0.0008 (0.0025) model time 0.3784 (0.3822) loss 4.3944 (3.4291) grad_norm 0.9366 (1.3188) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][320/625] eta 0:01:57 lr 0.001744 wd 0.0500 time 0.3753 (0.3850) data time 0.0008 (0.0025) model time 0.3746 (0.3819) loss 3.5486 (3.4323) grad_norm 1.1323 (1.3217) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][330/625] eta 0:01:53 lr 0.001744 wd 0.0500 time 0.3777 (0.3848) data time 0.0009 (0.0024) model time 0.3768 (0.3817) loss 3.2443 (3.4309) grad_norm 1.1278 (1.3183) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][340/625] eta 0:01:49 lr 0.001744 wd 0.0500 time 0.3788 (0.3851) data time 0.0009 (0.0024) model time 0.3779 (0.3822) loss 2.5379 (3.4233) grad_norm 2.3460 (1.3209) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][350/625] eta 0:01:45 lr 0.001744 wd 0.0500 time 0.3757 (0.3849) data time 0.0011 (0.0023) model time 0.3746 (0.3820) loss 3.1821 (3.4285) grad_norm 1.3546 (1.3212) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][360/625] eta 0:01:41 lr 0.001744 wd 0.0500 time 0.3748 (0.3847) data time 0.0011 (0.0023) model time 0.3737 (0.3818) loss 2.8662 (3.4369) grad_norm 1.6991 (1.3206) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][370/625] eta 0:01:38 lr 0.001744 wd 0.0500 time 0.3753 (0.3844) data time 0.0011 (0.0023) model time 0.3742 (0.3816) loss 3.9607 (3.4362) grad_norm 1.1911 (1.3194) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][380/625] eta 0:01:34 lr 0.001744 wd 0.0500 time 0.3749 (0.3843) data time 0.0012 (0.0022) model time 0.3737 (0.3814) loss 3.3419 (3.4342) grad_norm 1.0568 (1.3186) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:45:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][390/625] eta 0:01:30 lr 0.001744 wd 0.0500 time 0.3800 (0.3841) data time 0.0009 (0.0022) model time 0.3792 (0.3813) loss 4.1405 (3.4350) grad_norm 1.3460 (1.3168) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][400/625] eta 0:01:26 lr 0.001743 wd 0.0500 time 0.3765 (0.3840) data time 0.0010 (0.0022) model time 0.3755 (0.3811) loss 2.7864 (3.4306) grad_norm 1.3974 (1.3159) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][410/625] eta 0:01:22 lr 0.001743 wd 0.0500 time 0.3747 (0.3838) data time 0.0011 (0.0022) model time 0.3735 (0.3810) loss 3.3767 (3.4336) grad_norm 1.2312 (1.3201) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][420/625] eta 0:01:18 lr 0.001743 wd 0.0500 time 0.3775 (0.3850) data time 0.0009 (0.0021) model time 0.3767 (0.3824) loss 3.1904 (3.4295) grad_norm 1.1140 (1.3158) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][430/625] eta 0:01:15 lr 0.001743 wd 0.0500 time 0.3781 (0.3849) data time 0.0009 (0.0021) model time 0.3772 (0.3823) loss 4.2329 (3.4362) grad_norm 1.1544 (1.3195) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][440/625] eta 0:01:11 lr 0.001743 wd 0.0500 time 0.3789 (0.3847) data time 0.0008 (0.0021) model time 0.3780 (0.3822) loss 3.8056 (3.4406) grad_norm 1.2809 (1.3162) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][450/625] eta 0:01:07 lr 0.001743 wd 0.0500 time 0.3797 (0.3846) data time 0.0009 (0.0021) model time 0.3788 (0.3820) loss 2.5309 (3.4391) grad_norm 2.5711 (1.3184) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][460/625] eta 0:01:03 lr 0.001743 wd 0.0500 time 0.3749 (0.3844) data time 0.0008 (0.0021) model time 0.3741 (0.3819) loss 3.2486 (3.4391) grad_norm 1.0397 (1.3150) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][470/625] eta 0:00:59 lr 0.001743 wd 0.0500 time 0.3743 (0.3842) data time 0.0011 (0.0020) model time 0.3732 (0.3817) loss 3.6709 (3.4387) grad_norm 1.1820 (1.3105) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][480/625] eta 0:00:55 lr 0.001742 wd 0.0500 time 0.3740 (0.3841) data time 0.0011 (0.0020) model time 0.3729 (0.3816) loss 3.8675 (3.4427) grad_norm 1.3103 (1.3100) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][490/625] eta 0:00:51 lr 0.001742 wd 0.0500 time 0.3745 (0.3839) data time 0.0009 (0.0020) model time 0.3736 (0.3815) loss 2.5536 (3.4417) grad_norm 1.1967 (1.3118) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][500/625] eta 0:00:48 lr 0.001742 wd 0.0500 time 0.3767 (0.3847) data time 0.0008 (0.0020) model time 0.3759 (0.3824) loss 4.1571 (3.4454) grad_norm 0.9446 (1.3103) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][510/625] eta 0:00:44 lr 0.001742 wd 0.0500 time 0.3814 (0.3846) data time 0.0011 (0.0020) model time 0.3803 (0.3823) loss 3.3586 (3.4526) grad_norm 0.8690 (1.3079) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][520/625] eta 0:00:40 lr 0.001742 wd 0.0500 time 0.3755 (0.3848) data time 0.0008 (0.0020) model time 0.3747 (0.3825) loss 3.9272 (3.4567) grad_norm 1.4222 (1.3088) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][530/625] eta 0:00:36 lr 0.001742 wd 0.0500 time 0.3663 (0.3846) data time 0.0011 (0.0019) model time 0.3652 (0.3823) loss 3.7100 (3.4485) grad_norm 0.8704 (1.3033) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:46:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][540/625] eta 0:00:32 lr 0.001742 wd 0.0500 time 0.3766 (0.3853) data time 0.0011 (0.0019) model time 0.3755 (0.3831) loss 3.9100 (3.4520) grad_norm 0.9634 (1.3041) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][550/625] eta 0:00:28 lr 0.001742 wd 0.0500 time 0.3696 (0.3851) data time 0.0009 (0.0019) model time 0.3686 (0.3829) loss 2.0578 (3.4489) grad_norm 0.9631 (1.3034) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][560/625] eta 0:00:25 lr 0.001742 wd 0.0500 time 0.3765 (0.3850) data time 0.0009 (0.0019) model time 0.3755 (0.3829) loss 4.1774 (3.4507) grad_norm 1.2024 (1.3011) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][570/625] eta 0:00:21 lr 0.001741 wd 0.0500 time 0.3798 (0.3849) data time 0.0008 (0.0019) model time 0.3790 (0.3828) loss 2.5571 (3.4504) grad_norm 2.4768 (1.3044) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][580/625] eta 0:00:17 lr 0.001741 wd 0.0500 time 0.3801 (0.3848) data time 0.0008 (0.0019) model time 0.3792 (0.3827) loss 3.2891 (3.4509) grad_norm 1.4221 (1.3085) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][590/625] eta 0:00:13 lr 0.001741 wd 0.0500 time 0.3751 (0.3847) data time 0.0011 (0.0019) model time 0.3740 (0.3826) loss 3.5739 (3.4503) grad_norm 0.9851 (1.3079) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][600/625] eta 0:00:09 lr 0.001741 wd 0.0500 time 0.3793 (0.3846) data time 0.0008 (0.0018) model time 0.3785 (0.3825) loss 3.6684 (3.4499) grad_norm 1.4138 (1.3070) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][610/625] eta 0:00:05 lr 0.001741 wd 0.0500 time 0.3635 (0.3844) data time 0.0005 (0.0018) model time 0.3629 (0.3823) loss 3.9146 (3.4463) grad_norm 1.6625 (1.3098) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [85/300][620/625] eta 0:00:01 lr 0.001741 wd 0.0500 time 0.3661 (0.3842) data time 0.0008 (0.0018) model time 0.3653 (0.3820) loss 2.3802 (3.4464) grad_norm 1.2087 (1.3119) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:29 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 85 training takes 0:04:00 [2024-08-26 08:47:29 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:47:30 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:47:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.7212 (0.7212) Acc@1 84.570 (84.570) Acc@5 97.217 (97.217) Mem 11225MB [2024-08-26 08:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.1172 (0.8314) Acc@1 74.609 (81.343) Acc@5 92.822 (96.147) Mem 11225MB [2024-08-26 08:47:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.2178 (0.9946) Acc@1 72.021 (77.537) Acc@5 91.309 (94.069) Mem 11225MB [2024-08-26 08:47:33 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.455 Acc@5 94.080 [2024-08-26 08:47:33 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.5% [2024-08-26 08:47:33 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 77.46% [2024-08-26 08:47:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 08:47:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 08:47:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.513 (0.513) Loss 0.5610 (0.5610) Acc@1 87.939 (87.939) Acc@5 98.291 (98.291) Mem 11225MB [2024-08-26 08:47:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 0.9326 (0.6997) Acc@1 77.588 (84.086) Acc@5 94.873 (97.066) Mem 11225MB [2024-08-26 08:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.0986 (0.8533) Acc@1 72.412 (80.213) Acc@5 92.578 (95.299) Mem 11225MB [2024-08-26 08:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.910 Acc@5 95.276 [2024-08-26 08:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.9% [2024-08-26 08:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.91% [2024-08-26 08:47:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:47:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:47:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][0/625] eta 0:07:47 lr 0.001741 wd 0.0500 time 0.7479 (0.7479) data time 0.4089 (0.4089) model time 0.0000 (0.0000) loss 2.4690 (2.4690) grad_norm 1.1203 (1.1203) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][10/625] eta 0:04:12 lr 0.001741 wd 0.0500 time 0.3867 (0.4110) data time 0.0011 (0.0383) model time 0.0000 (0.0000) loss 3.2597 (3.3877) grad_norm 1.5256 (1.1964) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][20/625] eta 0:03:58 lr 0.001740 wd 0.0500 time 0.3719 (0.3947) data time 0.0008 (0.0206) model time 0.0000 (0.0000) loss 2.7647 (3.3785) grad_norm 2.4302 (1.2498) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][30/625] eta 0:03:51 lr 0.001740 wd 0.0500 time 0.3774 (0.3890) data time 0.0011 (0.0143) model time 0.0000 (0.0000) loss 3.4010 (3.4205) grad_norm 1.1295 (1.2595) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][40/625] eta 0:03:45 lr 0.001740 wd 0.0500 time 0.3810 (0.3862) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 3.9609 (3.4195) grad_norm 0.9256 (1.2762) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:47:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][50/625] eta 0:03:41 lr 0.001740 wd 0.0500 time 0.3810 (0.3850) data time 0.0008 (0.0091) model time 0.0000 (0.0000) loss 4.1459 (3.4923) grad_norm 1.4446 (1.2612) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][60/625] eta 0:03:38 lr 0.001740 wd 0.0500 time 0.5887 (0.3872) data time 0.0009 (0.0078) model time 0.5878 (0.3977) loss 4.2652 (3.5139) grad_norm 1.4390 (1.2409) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][70/625] eta 0:03:38 lr 0.001740 wd 0.0500 time 0.3841 (0.3940) data time 0.0011 (0.0069) model time 0.3830 (0.4158) loss 3.5668 (3.4883) grad_norm 1.4942 (1.2492) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][80/625] eta 0:03:33 lr 0.001740 wd 0.0500 time 0.3842 (0.3919) data time 0.0009 (0.0062) model time 0.3833 (0.4026) loss 3.4944 (3.5029) grad_norm 1.7264 (1.2909) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][90/625] eta 0:03:29 lr 0.001740 wd 0.0500 time 0.3815 (0.3907) data time 0.0011 (0.0057) model time 0.3804 (0.3969) loss 3.7482 (3.4898) grad_norm 1.3866 (1.3007) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][100/625] eta 0:03:24 lr 0.001740 wd 0.0500 time 0.3735 (0.3893) data time 0.0012 (0.0052) model time 0.3723 (0.3925) loss 3.8122 (3.4767) grad_norm 1.2528 (1.2834) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][110/625] eta 0:03:20 lr 0.001739 wd 0.0500 time 0.3725 (0.3884) data time 0.0011 (0.0048) model time 0.3714 (0.3901) loss 3.0579 (3.4453) grad_norm 1.0114 (1.2671) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][120/625] eta 0:03:15 lr 0.001739 wd 0.0500 time 0.3737 (0.3876) data time 0.0008 (0.0045) model time 0.3728 (0.3883) loss 3.0533 (3.4278) grad_norm 0.9729 (1.2745) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][130/625] eta 0:03:11 lr 0.001739 wd 0.0500 time 0.3838 (0.3871) data time 0.0011 (0.0043) model time 0.3826 (0.3873) loss 2.8422 (3.4365) grad_norm 1.2064 (1.2683) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][140/625] eta 0:03:07 lr 0.001739 wd 0.0500 time 0.3777 (0.3866) data time 0.0011 (0.0041) model time 0.3766 (0.3863) loss 3.3293 (3.4313) grad_norm 0.8877 (1.2578) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][150/625] eta 0:03:03 lr 0.001739 wd 0.0500 time 0.3724 (0.3872) data time 0.0011 (0.0039) model time 0.3714 (0.3872) loss 3.3535 (3.4091) grad_norm 1.4397 (1.2504) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][160/625] eta 0:02:59 lr 0.001739 wd 0.0500 time 0.3839 (0.3868) data time 0.0011 (0.0037) model time 0.3828 (0.3864) loss 3.6030 (3.4143) grad_norm 1.0896 (1.2526) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][170/625] eta 0:02:55 lr 0.001739 wd 0.0500 time 0.3776 (0.3862) data time 0.0011 (0.0035) model time 0.3765 (0.3856) loss 2.5048 (3.4159) grad_norm 1.2178 (1.2587) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][180/625] eta 0:02:51 lr 0.001739 wd 0.0500 time 0.3847 (0.3857) data time 0.0008 (0.0034) model time 0.3838 (0.3848) loss 2.1623 (3.4055) grad_norm 0.9984 (1.2587) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][190/625] eta 0:02:47 lr 0.001738 wd 0.0500 time 0.3716 (0.3852) data time 0.0012 (0.0033) model time 0.3704 (0.3841) loss 3.5094 (3.4142) grad_norm 1.0697 (1.2530) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:48:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][200/625] eta 0:02:43 lr 0.001738 wd 0.0500 time 0.3797 (0.3849) data time 0.0011 (0.0032) model time 0.3786 (0.3837) loss 2.5321 (3.4079) grad_norm 1.6577 (1.2538) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][210/625] eta 0:02:39 lr 0.001738 wd 0.0500 time 0.3800 (0.3846) data time 0.0008 (0.0031) model time 0.3792 (0.3833) loss 4.2301 (3.3978) grad_norm 1.2161 (1.2596) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][220/625] eta 0:02:35 lr 0.001738 wd 0.0500 time 0.3691 (0.3843) data time 0.0009 (0.0030) model time 0.3683 (0.3829) loss 3.1144 (3.3938) grad_norm 1.4310 (1.2675) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][230/625] eta 0:02:31 lr 0.001738 wd 0.0500 time 0.3803 (0.3840) data time 0.0011 (0.0029) model time 0.3793 (0.3826) loss 3.7041 (3.3861) grad_norm 1.4694 (1.2717) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][240/625] eta 0:02:27 lr 0.001738 wd 0.0500 time 0.3775 (0.3838) data time 0.0011 (0.0029) model time 0.3764 (0.3823) loss 3.8283 (3.3988) grad_norm 0.9117 (1.2726) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][250/625] eta 0:02:24 lr 0.001738 wd 0.0500 time 0.3802 (0.3852) data time 0.0011 (0.0028) model time 0.3791 (0.3840) loss 2.7079 (3.3974) grad_norm 1.3357 (1.2766) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][260/625] eta 0:02:20 lr 0.001738 wd 0.0500 time 0.3912 (0.3849) data time 0.0008 (0.0027) model time 0.3904 (0.3837) loss 2.6751 (3.3909) grad_norm 1.3916 (1.2773) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][270/625] eta 0:02:16 lr 0.001737 wd 0.0500 time 0.3772 (0.3846) data time 0.0012 (0.0027) model time 0.3760 (0.3834) loss 3.4309 (3.3917) grad_norm 1.1398 (1.2744) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][280/625] eta 0:02:13 lr 0.001737 wd 0.0500 time 0.3817 (0.3861) data time 0.0008 (0.0026) model time 0.3809 (0.3851) loss 4.1093 (3.3942) grad_norm 0.9878 (1.2872) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][290/625] eta 0:02:09 lr 0.001737 wd 0.0500 time 0.3815 (0.3859) data time 0.0008 (0.0026) model time 0.3807 (0.3849) loss 3.2114 (3.3966) grad_norm 1.3197 (1.2855) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][300/625] eta 0:02:05 lr 0.001737 wd 0.0500 time 0.3735 (0.3856) data time 0.0011 (0.0025) model time 0.3725 (0.3846) loss 3.6036 (3.3945) grad_norm 1.2649 (1.2868) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][310/625] eta 0:02:01 lr 0.001737 wd 0.0500 time 0.3744 (0.3854) data time 0.0010 (0.0025) model time 0.3733 (0.3843) loss 3.4408 (3.4007) grad_norm 1.2253 (1.2888) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][320/625] eta 0:01:57 lr 0.001737 wd 0.0500 time 0.3847 (0.3852) data time 0.0011 (0.0024) model time 0.3837 (0.3841) loss 3.6039 (3.3979) grad_norm 1.1366 (1.2961) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][330/625] eta 0:01:53 lr 0.001737 wd 0.0500 time 0.6182 (0.3857) data time 0.0009 (0.0024) model time 0.6173 (0.3846) loss 3.8340 (3.3981) grad_norm 0.8391 (1.2929) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][340/625] eta 0:01:49 lr 0.001737 wd 0.0500 time 0.3766 (0.3853) data time 0.0011 (0.0024) model time 0.3755 (0.3842) loss 3.9182 (3.4068) grad_norm 1.6913 (1.2935) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][350/625] eta 0:01:45 lr 0.001737 wd 0.0500 time 0.3787 (0.3851) data time 0.0011 (0.0023) model time 0.3777 (0.3839) loss 3.3387 (3.4107) grad_norm 0.8805 (1.2941) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:49:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][360/625] eta 0:01:41 lr 0.001736 wd 0.0500 time 0.3784 (0.3849) data time 0.0009 (0.0023) model time 0.3775 (0.3837) loss 2.8117 (3.4114) grad_norm 0.9902 (1.2879) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][370/625] eta 0:01:38 lr 0.001736 wd 0.0500 time 0.3813 (0.3858) data time 0.0008 (0.0023) model time 0.3805 (0.3847) loss 3.6686 (3.4111) grad_norm 1.3900 (1.2869) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][380/625] eta 0:01:34 lr 0.001736 wd 0.0500 time 0.3786 (0.3855) data time 0.0011 (0.0022) model time 0.3775 (0.3844) loss 3.7594 (3.4121) grad_norm 0.8210 (1.2857) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][390/625] eta 0:01:30 lr 0.001736 wd 0.0500 time 0.3766 (0.3853) data time 0.0011 (0.0022) model time 0.3756 (0.3842) loss 3.4765 (3.4123) grad_norm 1.5363 (1.2871) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][400/625] eta 0:01:26 lr 0.001736 wd 0.0500 time 0.3663 (0.3851) data time 0.0008 (0.0022) model time 0.3654 (0.3839) loss 2.4728 (3.4121) grad_norm 1.4839 (1.2920) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][410/625] eta 0:01:22 lr 0.001736 wd 0.0500 time 0.3675 (0.3849) data time 0.0009 (0.0022) model time 0.3666 (0.3837) loss 3.7176 (3.4151) grad_norm 1.6263 (1.2925) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][420/625] eta 0:01:18 lr 0.001736 wd 0.0500 time 0.3797 (0.3847) data time 0.0009 (0.0021) model time 0.3788 (0.3835) loss 4.3823 (3.4199) grad_norm 1.0226 (1.2875) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][430/625] eta 0:01:14 lr 0.001736 wd 0.0500 time 0.3774 (0.3846) data time 0.0009 (0.0021) model time 0.3765 (0.3833) loss 3.3701 (3.4214) grad_norm 1.0640 (1.2863) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][440/625] eta 0:01:11 lr 0.001735 wd 0.0500 time 0.3821 (0.3844) data time 0.0011 (0.0021) model time 0.3810 (0.3832) loss 3.4858 (3.4212) grad_norm 1.2560 (1.3045) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][450/625] eta 0:01:07 lr 0.001735 wd 0.0500 time 0.3345 (0.3847) data time 0.0009 (0.0021) model time 0.3336 (0.3835) loss 3.0210 (3.4208) grad_norm 0.9915 (1.3063) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][460/625] eta 0:01:03 lr 0.001735 wd 0.0500 time 0.5759 (0.3849) data time 0.0009 (0.0020) model time 0.5749 (0.3838) loss 3.1526 (3.4193) grad_norm 1.7781 (1.3047) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][470/625] eta 0:00:59 lr 0.001735 wd 0.0500 time 0.3743 (0.3852) data time 0.0011 (0.0020) model time 0.3732 (0.3841) loss 3.3697 (3.4182) grad_norm 1.2381 (1.3059) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][480/625] eta 0:00:55 lr 0.001735 wd 0.0500 time 0.3769 (0.3850) data time 0.0011 (0.0020) model time 0.3758 (0.3839) loss 3.3709 (3.4202) grad_norm 1.3553 (1.3068) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][490/625] eta 0:00:51 lr 0.001735 wd 0.0500 time 0.3804 (0.3849) data time 0.0009 (0.0020) model time 0.3795 (0.3837) loss 3.5634 (3.4198) grad_norm 1.0554 (1.3055) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][500/625] eta 0:00:48 lr 0.001735 wd 0.0500 time 0.3831 (0.3848) data time 0.0008 (0.0020) model time 0.3822 (0.3836) loss 4.3886 (3.4271) grad_norm 1.0159 (1.3047) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][510/625] eta 0:00:44 lr 0.001735 wd 0.0500 time 0.3703 (0.3847) data time 0.0008 (0.0020) model time 0.3695 (0.3835) loss 3.9549 (3.4236) grad_norm 1.1294 (1.3019) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:50:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][520/625] eta 0:00:40 lr 0.001734 wd 0.0500 time 0.3815 (0.3846) data time 0.0011 (0.0019) model time 0.3804 (0.3834) loss 3.2958 (3.4211) grad_norm 0.9373 (1.3021) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][530/625] eta 0:00:36 lr 0.001734 wd 0.0500 time 0.3719 (0.3844) data time 0.0011 (0.0019) model time 0.3709 (0.3832) loss 3.9809 (3.4239) grad_norm 2.1452 (1.2997) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][540/625] eta 0:00:32 lr 0.001734 wd 0.0500 time 0.3775 (0.3842) data time 0.0009 (0.0019) model time 0.3766 (0.3830) loss 3.5627 (3.4218) grad_norm 1.3062 (1.3004) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][550/625] eta 0:00:28 lr 0.001734 wd 0.0500 time 0.3887 (0.3841) data time 0.0009 (0.0019) model time 0.3878 (0.3829) loss 4.4455 (3.4208) grad_norm 1.7739 (1.3041) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][560/625] eta 0:00:24 lr 0.001734 wd 0.0500 time 0.3834 (0.3840) data time 0.0011 (0.0019) model time 0.3823 (0.3827) loss 3.6287 (3.4197) grad_norm 1.0522 (1.3046) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][570/625] eta 0:00:21 lr 0.001734 wd 0.0500 time 0.3742 (0.3839) data time 0.0008 (0.0019) model time 0.3734 (0.3826) loss 4.0029 (3.4164) grad_norm 0.8453 (1.3011) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][580/625] eta 0:00:17 lr 0.001734 wd 0.0500 time 0.5969 (0.3851) data time 0.0008 (0.0019) model time 0.5960 (0.3840) loss 3.8304 (3.4191) grad_norm 1.8462 (1.3024) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][590/625] eta 0:00:13 lr 0.001734 wd 0.0500 time 0.3774 (0.3850) data time 0.0008 (0.0018) model time 0.3766 (0.3839) loss 3.5707 (3.4214) grad_norm 0.8532 (1.3045) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][600/625] eta 0:00:09 lr 0.001734 wd 0.0500 time 0.3657 (0.3849) data time 0.0009 (0.0018) model time 0.3648 (0.3837) loss 4.0395 (3.4211) grad_norm 1.0457 (1.3033) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][610/625] eta 0:00:05 lr 0.001733 wd 0.0500 time 0.3623 (0.3847) data time 0.0008 (0.0018) model time 0.3615 (0.3835) loss 3.9728 (3.4233) grad_norm 1.0325 (1.3029) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [86/300][620/625] eta 0:00:01 lr 0.001733 wd 0.0500 time 0.3685 (0.3845) data time 0.0005 (0.0018) model time 0.3679 (0.3833) loss 4.3329 (3.4237) grad_norm 1.1143 (1.3036) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:39 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 86 training takes 0:04:00 [2024-08-26 08:51:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:51:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:51:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.508 (0.508) Loss 0.6270 (0.6270) Acc@1 85.400 (85.400) Acc@5 97.168 (97.168) Mem 11225MB [2024-08-26 08:51:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.0850 (0.7680) Acc@1 74.609 (81.800) Acc@5 93.066 (96.245) Mem 11225MB [2024-08-26 08:51:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.2402 (0.9415) Acc@1 70.068 (77.879) Acc@5 90.625 (94.120) Mem 11225MB [2024-08-26 08:51:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.529 Acc@5 94.080 [2024-08-26 08:51:43 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.5% [2024-08-26 08:51:43 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 77.53% [2024-08-26 08:51:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 08:51:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 08:51:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 0.5605 (0.5605) Acc@1 88.037 (88.037) Acc@5 98.291 (98.291) Mem 11225MB [2024-08-26 08:51:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.9312 (0.6982) Acc@1 77.539 (84.086) Acc@5 94.971 (97.066) Mem 11225MB [2024-08-26 08:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.0986 (0.8512) Acc@1 72.217 (80.215) Acc@5 92.529 (95.322) Mem 11225MB [2024-08-26 08:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.924 Acc@5 95.294 [2024-08-26 08:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 79.9% [2024-08-26 08:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.92% [2024-08-26 08:51:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:51:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:51:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][0/625] eta 0:07:53 lr 0.001733 wd 0.0500 time 0.7577 (0.7577) data time 0.4187 (0.4187) model time 0.0000 (0.0000) loss 2.9913 (2.9913) grad_norm 1.0951 (1.0951) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][10/625] eta 0:04:13 lr 0.001733 wd 0.0500 time 0.3814 (0.4126) data time 0.0008 (0.0391) model time 0.0000 (0.0000) loss 4.7411 (3.4343) grad_norm 1.3172 (1.2784) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:51:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][20/625] eta 0:03:59 lr 0.001733 wd 0.0500 time 0.3835 (0.3951) data time 0.0011 (0.0210) model time 0.0000 (0.0000) loss 3.9923 (3.4655) grad_norm 1.1807 (1.2049) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][30/625] eta 0:03:51 lr 0.001733 wd 0.0500 time 0.3721 (0.3892) data time 0.0011 (0.0146) model time 0.0000 (0.0000) loss 3.5496 (3.4680) grad_norm 1.2957 (1.2684) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][40/625] eta 0:03:46 lr 0.001733 wd 0.0500 time 0.3818 (0.3869) data time 0.0012 (0.0113) model time 0.0000 (0.0000) loss 3.3593 (3.4300) grad_norm 1.4534 (1.3124) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][50/625] eta 0:03:41 lr 0.001733 wd 0.0500 time 0.3712 (0.3851) data time 0.0008 (0.0093) model time 0.0000 (0.0000) loss 3.9943 (3.4149) grad_norm 1.4179 (1.2979) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][60/625] eta 0:03:37 lr 0.001732 wd 0.0500 time 0.3779 (0.3842) data time 0.0009 (0.0080) model time 0.3770 (0.3785) loss 2.1997 (3.3556) grad_norm 1.0411 (1.2875) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][70/625] eta 0:03:32 lr 0.001732 wd 0.0500 time 0.3803 (0.3836) data time 0.0008 (0.0070) model time 0.3795 (0.3786) loss 3.7296 (3.4111) grad_norm 1.2067 (1.2865) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][80/625] eta 0:03:28 lr 0.001732 wd 0.0500 time 0.3733 (0.3827) data time 0.0011 (0.0063) model time 0.3722 (0.3776) loss 2.9032 (3.4282) grad_norm 1.3346 (1.4129) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][90/625] eta 0:03:24 lr 0.001732 wd 0.0500 time 0.3779 (0.3822) data time 0.0011 (0.0057) model time 0.3768 (0.3773) loss 3.9940 (3.4424) grad_norm 1.2027 (1.4118) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][100/625] eta 0:03:24 lr 0.001732 wd 0.0500 time 0.5574 (0.3895) data time 0.0011 (0.0053) model time 0.5563 (0.3928) loss 3.5715 (3.4490) grad_norm 0.8975 (1.3823) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][110/625] eta 0:03:19 lr 0.001732 wd 0.0500 time 0.3779 (0.3881) data time 0.0008 (0.0049) model time 0.3771 (0.3895) loss 2.9449 (3.4612) grad_norm 1.6710 (1.3778) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][120/625] eta 0:03:15 lr 0.001732 wd 0.0500 time 0.3784 (0.3872) data time 0.0010 (0.0046) model time 0.3774 (0.3875) loss 3.4563 (3.4647) grad_norm 0.9747 (1.3743) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][130/625] eta 0:03:11 lr 0.001732 wd 0.0500 time 0.3778 (0.3865) data time 0.0009 (0.0044) model time 0.3769 (0.3862) loss 4.1273 (3.4662) grad_norm 1.6196 (1.3655) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][140/625] eta 0:03:07 lr 0.001731 wd 0.0500 time 0.3740 (0.3858) data time 0.0011 (0.0041) model time 0.3728 (0.3849) loss 3.5062 (3.4569) grad_norm 1.1440 (1.3764) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][150/625] eta 0:03:03 lr 0.001731 wd 0.0500 time 0.5845 (0.3866) data time 0.0011 (0.0039) model time 0.5834 (0.3862) loss 3.6999 (3.4703) grad_norm 1.2643 (1.3671) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][160/625] eta 0:03:00 lr 0.001731 wd 0.0500 time 0.3783 (0.3875) data time 0.0011 (0.0038) model time 0.3772 (0.3874) loss 3.6862 (3.4842) grad_norm 1.0484 (1.3644) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][170/625] eta 0:02:56 lr 0.001731 wd 0.0500 time 0.3744 (0.3868) data time 0.0011 (0.0036) model time 0.3733 (0.3864) loss 2.9763 (3.4907) grad_norm 1.3083 (1.3607) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:52:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][180/625] eta 0:02:51 lr 0.001731 wd 0.0500 time 0.3789 (0.3864) data time 0.0010 (0.0035) model time 0.3779 (0.3857) loss 3.5409 (3.4878) grad_norm 1.0226 (1.3532) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][190/625] eta 0:02:47 lr 0.001731 wd 0.0500 time 0.3806 (0.3860) data time 0.0011 (0.0034) model time 0.3795 (0.3851) loss 3.1049 (3.4784) grad_norm 0.9382 (1.3425) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][200/625] eta 0:02:43 lr 0.001731 wd 0.0500 time 0.3753 (0.3856) data time 0.0008 (0.0033) model time 0.3745 (0.3846) loss 3.6401 (3.4732) grad_norm 1.1020 (1.3338) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][210/625] eta 0:02:39 lr 0.001731 wd 0.0500 time 0.3760 (0.3852) data time 0.0008 (0.0032) model time 0.3751 (0.3841) loss 4.1436 (3.4689) grad_norm 0.9581 (1.3234) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][220/625] eta 0:02:35 lr 0.001731 wd 0.0500 time 0.3820 (0.3849) data time 0.0008 (0.0031) model time 0.3812 (0.3837) loss 3.8649 (3.4771) grad_norm 1.8416 (1.3227) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][230/625] eta 0:02:31 lr 0.001730 wd 0.0500 time 0.3820 (0.3846) data time 0.0008 (0.0030) model time 0.3812 (0.3833) loss 2.4667 (3.4545) grad_norm 1.2846 (1.3269) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][240/625] eta 0:02:27 lr 0.001730 wd 0.0500 time 0.3724 (0.3844) data time 0.0010 (0.0029) model time 0.3713 (0.3830) loss 2.9334 (3.4574) grad_norm 1.0315 (1.3170) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][250/625] eta 0:02:23 lr 0.001730 wd 0.0500 time 0.3770 (0.3840) data time 0.0012 (0.0028) model time 0.3759 (0.3825) loss 3.2389 (3.4610) grad_norm 0.9124 (1.3169) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][260/625] eta 0:02:20 lr 0.001730 wd 0.0500 time 0.3823 (0.3837) data time 0.0011 (0.0028) model time 0.3812 (0.3822) loss 2.6274 (3.4594) grad_norm 0.8932 (1.3129) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][270/625] eta 0:02:16 lr 0.001730 wd 0.0500 time 0.3812 (0.3835) data time 0.0010 (0.0027) model time 0.3802 (0.3820) loss 3.8379 (3.4564) grad_norm 1.0631 (1.3102) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][280/625] eta 0:02:12 lr 0.001730 wd 0.0500 time 0.3803 (0.3833) data time 0.0008 (0.0027) model time 0.3795 (0.3818) loss 2.8115 (3.4523) grad_norm 1.2776 (1.3089) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][290/625] eta 0:02:08 lr 0.001730 wd 0.0500 time 0.3778 (0.3831) data time 0.0008 (0.0026) model time 0.3769 (0.3815) loss 4.1859 (3.4508) grad_norm 1.0209 (1.3072) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][300/625] eta 0:02:04 lr 0.001730 wd 0.0500 time 0.3799 (0.3830) data time 0.0011 (0.0026) model time 0.3788 (0.3814) loss 3.8091 (3.4516) grad_norm 1.1401 (1.3048) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][310/625] eta 0:02:01 lr 0.001729 wd 0.0500 time 0.3776 (0.3845) data time 0.0008 (0.0025) model time 0.3768 (0.3832) loss 3.4780 (3.4527) grad_norm 1.1623 (1.2994) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][320/625] eta 0:01:57 lr 0.001729 wd 0.0500 time 0.3777 (0.3849) data time 0.0009 (0.0025) model time 0.3768 (0.3837) loss 2.3235 (3.4393) grad_norm 1.4787 (1.3033) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:53:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][330/625] eta 0:01:53 lr 0.001729 wd 0.0500 time 0.3869 (0.3847) data time 0.0011 (0.0024) model time 0.3859 (0.3835) loss 3.6796 (3.4434) grad_norm 1.2034 (1.2992) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][340/625] eta 0:01:50 lr 0.001729 wd 0.0500 time 0.3754 (0.3864) data time 0.0008 (0.0024) model time 0.3746 (0.3855) loss 2.4299 (3.4412) grad_norm 1.2250 (1.2974) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][350/625] eta 0:01:46 lr 0.001729 wd 0.0500 time 0.3714 (0.3861) data time 0.0011 (0.0024) model time 0.3703 (0.3852) loss 3.7731 (3.4387) grad_norm 1.1866 (1.2929) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:54:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][360/625] eta 0:01:42 lr 0.001729 wd 0.0500 time 0.3816 (0.3859) data time 0.0008 (0.0023) model time 0.3808 (0.3848) loss 3.4917 (3.4431) grad_norm 1.3421 (1.2891) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:54:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][370/625] eta 0:01:38 lr 0.001729 wd 0.0500 time 0.3761 (0.3856) data time 0.0011 (0.0023) model time 0.3750 (0.3845) loss 3.8510 (3.4458) grad_norm 1.1354 (1.2874) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 08:54:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][380/625] eta 0:01:34 lr 0.001729 wd 0.0500 time 0.3724 (0.3853) data time 0.0011 (0.0023) model time 0.3713 (0.3842) loss 3.3033 (3.4519) grad_norm 1.7604 (1.2979) loss_scale 8192.0000 (4128.2520) mem 11225MB [2024-08-26 08:54:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][390/625] eta 0:01:30 lr 0.001728 wd 0.0500 time 0.3798 (0.3852) data time 0.0011 (0.0022) model time 0.3787 (0.3840) loss 3.3890 (3.4528) grad_norm 1.0787 (1.2964) loss_scale 8192.0000 (4232.1841) mem 11225MB [2024-08-26 08:54:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][400/625] eta 0:01:26 lr 0.001728 wd 0.0500 time 0.3757 (0.3850) data time 0.0011 (0.0022) model time 0.3746 (0.3838) loss 2.8636 (3.4407) grad_norm 1.1596 (1.2938) loss_scale 8192.0000 (4330.9327) mem 11225MB [2024-08-26 08:54:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][410/625] eta 0:01:22 lr 0.001728 wd 0.0500 time 0.3780 (0.3849) data time 0.0014 (0.0022) model time 0.3766 (0.3837) loss 3.2300 (3.4383) grad_norm 1.0098 (1.2927) loss_scale 8192.0000 (4424.8759) mem 11225MB [2024-08-26 08:54:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][420/625] eta 0:01:18 lr 0.001728 wd 0.0500 time 0.3705 (0.3847) data time 0.0011 (0.0022) model time 0.3694 (0.3834) loss 3.1075 (3.4389) grad_norm 1.0755 (1.2965) loss_scale 8192.0000 (4514.3563) mem 11225MB [2024-08-26 08:54:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][430/625] eta 0:01:14 lr 0.001728 wd 0.0500 time 0.3772 (0.3845) data time 0.0010 (0.0021) model time 0.3762 (0.3832) loss 3.6051 (3.4319) grad_norm 1.1320 (1.2991) loss_scale 8192.0000 (4599.6845) mem 11225MB [2024-08-26 08:54:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][440/625] eta 0:01:11 lr 0.001728 wd 0.0500 time 0.3783 (0.3843) data time 0.0011 (0.0021) model time 0.3772 (0.3830) loss 2.7858 (3.4295) grad_norm 1.7611 (1.3016) loss_scale 8192.0000 (4681.1429) mem 11225MB [2024-08-26 08:54:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][450/625] eta 0:01:07 lr 0.001728 wd 0.0500 time 0.3709 (0.3841) data time 0.0008 (0.0021) model time 0.3700 (0.3828) loss 2.7298 (3.4221) grad_norm 1.9911 (1.3064) loss_scale 8192.0000 (4758.9889) mem 11225MB [2024-08-26 08:54:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][460/625] eta 0:01:03 lr 0.001728 wd 0.0500 time 0.3698 (0.3852) data time 0.0009 (0.0021) model time 0.3689 (0.3841) loss 3.2157 (3.4305) grad_norm 0.8448 (1.3058) loss_scale 8192.0000 (4833.4577) mem 11225MB [2024-08-26 08:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][470/625] eta 0:00:59 lr 0.001727 wd 0.0500 time 0.3716 (0.3850) data time 0.0008 (0.0021) model time 0.3708 (0.3839) loss 3.8389 (3.4306) grad_norm 1.0260 (1.3020) loss_scale 8192.0000 (4904.7643) mem 11225MB [2024-08-26 08:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][480/625] eta 0:00:55 lr 0.001727 wd 0.0500 time 0.3754 (0.3848) data time 0.0008 (0.0020) model time 0.3745 (0.3836) loss 3.0406 (3.4303) grad_norm 1.0347 (1.2995) loss_scale 8192.0000 (4973.1060) mem 11225MB [2024-08-26 08:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][490/625] eta 0:00:52 lr 0.001727 wd 0.0500 time 0.3733 (0.3854) data time 0.0011 (0.0020) model time 0.3722 (0.3842) loss 3.8231 (3.4321) grad_norm 1.4822 (1.3010) loss_scale 8192.0000 (5038.6640) mem 11225MB [2024-08-26 08:55:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][500/625] eta 0:00:48 lr 0.001727 wd 0.0500 time 0.3788 (0.3856) data time 0.0011 (0.0020) model time 0.3777 (0.3845) loss 3.5922 (3.4307) grad_norm 1.0575 (1.3012) loss_scale 8192.0000 (5101.6048) mem 11225MB [2024-08-26 08:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][510/625] eta 0:00:44 lr 0.001727 wd 0.0500 time 0.3678 (0.3854) data time 0.0013 (0.0020) model time 0.3666 (0.3843) loss 2.6248 (3.4314) grad_norm 1.2787 (1.3010) loss_scale 8192.0000 (5162.0822) mem 11225MB [2024-08-26 08:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][520/625] eta 0:00:40 lr 0.001727 wd 0.0500 time 0.3732 (0.3853) data time 0.0011 (0.0020) model time 0.3721 (0.3841) loss 2.1172 (3.4282) grad_norm 1.6121 (1.3017) loss_scale 8192.0000 (5220.2380) mem 11225MB [2024-08-26 08:55:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][530/625] eta 0:00:36 lr 0.001727 wd 0.0500 time 0.3728 (0.3851) data time 0.0012 (0.0020) model time 0.3716 (0.3839) loss 3.3304 (3.4265) grad_norm 1.2593 (1.3005) loss_scale 8192.0000 (5276.2034) mem 11225MB [2024-08-26 08:55:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][540/625] eta 0:00:32 lr 0.001727 wd 0.0500 time 0.3782 (0.3849) data time 0.0008 (0.0019) model time 0.3774 (0.3837) loss 2.4535 (3.4275) grad_norm 1.7768 (1.2993) loss_scale 8192.0000 (5330.0998) mem 11225MB [2024-08-26 08:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][550/625] eta 0:00:28 lr 0.001726 wd 0.0500 time 0.3801 (0.3848) data time 0.0008 (0.0019) model time 0.3793 (0.3836) loss 3.6841 (3.4285) grad_norm 1.3472 (1.2977) loss_scale 8192.0000 (5382.0399) mem 11225MB [2024-08-26 08:55:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][560/625] eta 0:00:25 lr 0.001726 wd 0.0500 time 0.3730 (0.3847) data time 0.0009 (0.0019) model time 0.3721 (0.3835) loss 3.7553 (3.4333) grad_norm 1.1814 (1.2937) loss_scale 8192.0000 (5432.1283) mem 11225MB [2024-08-26 08:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][570/625] eta 0:00:21 lr 0.001726 wd 0.0500 time 0.3754 (0.3845) data time 0.0009 (0.0019) model time 0.3745 (0.3833) loss 3.3721 (3.4272) grad_norm 1.4937 (1.2929) loss_scale 8192.0000 (5480.4623) mem 11225MB [2024-08-26 08:55:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][580/625] eta 0:00:17 lr 0.001726 wd 0.0500 time 0.3768 (0.3844) data time 0.0014 (0.0019) model time 0.3754 (0.3832) loss 3.8114 (3.4289) grad_norm 0.9611 (1.2937) loss_scale 8192.0000 (5527.1325) mem 11225MB [2024-08-26 08:55:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][590/625] eta 0:00:13 lr 0.001726 wd 0.0500 time 0.3774 (0.3843) data time 0.0008 (0.0019) model time 0.3765 (0.3830) loss 3.8371 (3.4245) grad_norm 1.4018 (1.2952) loss_scale 8192.0000 (5572.2234) mem 11225MB [2024-08-26 08:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][600/625] eta 0:00:09 lr 0.001726 wd 0.0500 time 0.3710 (0.3842) data time 0.0011 (0.0019) model time 0.3699 (0.3829) loss 2.7733 (3.4211) grad_norm 1.2907 (1.2956) loss_scale 8192.0000 (5615.8136) mem 11225MB [2024-08-26 08:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][610/625] eta 0:00:05 lr 0.001726 wd 0.0500 time 0.3640 (0.3847) data time 0.0009 (0.0019) model time 0.3632 (0.3835) loss 3.9159 (3.4182) grad_norm 1.2374 (1.2945) loss_scale 8192.0000 (5657.9771) mem 11225MB [2024-08-26 08:55:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [87/300][620/625] eta 0:00:01 lr 0.001726 wd 0.0500 time 0.3658 (0.3848) data time 0.0008 (0.0018) model time 0.3650 (0.3836) loss 3.5417 (3.4197) grad_norm 1.1972 (1.2960) loss_scale 8192.0000 (5698.7826) mem 11225MB [2024-08-26 08:55:49 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 87 training takes 0:04:00 [2024-08-26 08:55:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:55:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 08:55:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.520 (0.520) Loss 0.6812 (0.6812) Acc@1 85.791 (85.791) Acc@5 97.412 (97.412) Mem 11225MB [2024-08-26 08:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.145) Loss 1.0771 (0.8228) Acc@1 75.098 (81.543) Acc@5 93.164 (96.236) Mem 11225MB [2024-08-26 08:55:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.2393 (1.0003) Acc@1 71.191 (77.297) Acc@5 91.260 (94.138) Mem 11225MB [2024-08-26 08:55:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.111 Acc@5 94.142 [2024-08-26 08:55:53 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.1% [2024-08-26 08:55:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.811 (0.811) Loss 0.5596 (0.5596) Acc@1 88.281 (88.281) Acc@5 98.291 (98.291) Mem 11225MB [2024-08-26 08:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.175) Loss 0.9302 (0.6969) Acc@1 77.441 (84.104) Acc@5 95.117 (97.084) Mem 11225MB [2024-08-26 08:55:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 1.0986 (0.8496) Acc@1 72.510 (80.276) Acc@5 92.676 (95.361) Mem 11225MB [2024-08-26 08:55:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.986 Acc@5 95.329 [2024-08-26 08:55:56 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.0% [2024-08-26 08:55:56 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 79.99% [2024-08-26 08:55:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 08:55:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 08:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][0/625] eta 0:08:20 lr 0.001726 wd 0.0500 time 0.8008 (0.8008) data time 0.4572 (0.4572) model time 0.0000 (0.0000) loss 2.9524 (2.9524) grad_norm 1.9019 (1.9019) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][10/625] eta 0:04:15 lr 0.001725 wd 0.0500 time 0.3770 (0.4148) data time 0.0008 (0.0427) model time 0.0000 (0.0000) loss 3.0544 (3.4317) grad_norm 1.2488 (1.4527) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][20/625] eta 0:03:59 lr 0.001725 wd 0.0500 time 0.3716 (0.3961) data time 0.0008 (0.0236) model time 0.0000 (0.0000) loss 4.1200 (3.2722) grad_norm 1.1556 (1.3329) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][30/625] eta 0:04:03 lr 0.001725 wd 0.0500 time 0.3730 (0.4088) data time 0.0008 (0.0163) model time 0.0000 (0.0000) loss 3.3992 (3.2686) grad_norm 1.3565 (1.3284) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][40/625] eta 0:03:54 lr 0.001725 wd 0.0500 time 0.3802 (0.4009) data time 0.0011 (0.0126) model time 0.0000 (0.0000) loss 3.9685 (3.3292) grad_norm 1.6073 (1.3272) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][50/625] eta 0:03:47 lr 0.001725 wd 0.0500 time 0.3742 (0.3957) data time 0.0011 (0.0104) model time 0.0000 (0.0000) loss 3.4990 (3.3506) grad_norm 1.4897 (1.3435) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][60/625] eta 0:03:41 lr 0.001725 wd 0.0500 time 0.3856 (0.3924) data time 0.0010 (0.0089) model time 0.3846 (0.3745) loss 3.1860 (3.3556) grad_norm 1.0775 (1.3121) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][70/625] eta 0:03:36 lr 0.001725 wd 0.0500 time 0.3778 (0.3902) data time 0.0008 (0.0078) model time 0.3770 (0.3749) loss 2.8727 (3.3163) grad_norm 1.0379 (1.3015) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][80/625] eta 0:03:31 lr 0.001725 wd 0.0500 time 0.3753 (0.3887) data time 0.0011 (0.0070) model time 0.3742 (0.3757) loss 3.3963 (3.3256) grad_norm 1.3196 (1.3100) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][90/625] eta 0:03:27 lr 0.001724 wd 0.0500 time 0.3744 (0.3873) data time 0.0009 (0.0063) model time 0.3734 (0.3755) loss 3.4811 (3.2966) grad_norm 1.0369 (1.3036) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][100/625] eta 0:03:22 lr 0.001724 wd 0.0500 time 0.3738 (0.3863) data time 0.0011 (0.0058) model time 0.3728 (0.3757) loss 3.9018 (3.2903) grad_norm 1.3056 (1.2900) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][110/625] eta 0:03:18 lr 0.001724 wd 0.0500 time 0.3761 (0.3855) data time 0.0008 (0.0054) model time 0.3753 (0.3756) loss 4.1775 (3.3148) grad_norm 1.7643 (1.3050) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][120/625] eta 0:03:14 lr 0.001724 wd 0.0500 time 0.3303 (0.3858) data time 0.0010 (0.0050) model time 0.3293 (0.3775) loss 3.4939 (3.3363) grad_norm 1.1786 (1.3093) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][130/625] eta 0:03:10 lr 0.001724 wd 0.0500 time 0.3744 (0.3853) data time 0.0008 (0.0047) model time 0.3736 (0.3776) loss 3.8252 (3.3542) grad_norm 1.3103 (1.3173) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][140/625] eta 0:03:06 lr 0.001724 wd 0.0500 time 0.3748 (0.3847) data time 0.0011 (0.0045) model time 0.3737 (0.3774) loss 3.5269 (3.3688) grad_norm 1.6798 (1.3205) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][150/625] eta 0:03:02 lr 0.001724 wd 0.0500 time 0.3842 (0.3843) data time 0.0011 (0.0043) model time 0.3831 (0.3773) loss 2.8981 (3.3851) grad_norm 1.5915 (1.3208) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][160/625] eta 0:02:58 lr 0.001724 wd 0.0500 time 0.3745 (0.3839) data time 0.0009 (0.0041) model time 0.3736 (0.3772) loss 2.4703 (3.3677) grad_norm 1.1582 (1.3160) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][170/625] eta 0:02:55 lr 0.001723 wd 0.0500 time 0.3787 (0.3860) data time 0.0009 (0.0039) model time 0.3778 (0.3807) loss 3.5509 (3.3868) grad_norm 0.8952 (1.3227) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][180/625] eta 0:02:51 lr 0.001723 wd 0.0500 time 0.3813 (0.3855) data time 0.0009 (0.0038) model time 0.3804 (0.3803) loss 2.3787 (3.3779) grad_norm 1.2582 (1.3198) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][190/625] eta 0:02:47 lr 0.001723 wd 0.0500 time 0.3852 (0.3851) data time 0.0011 (0.0036) model time 0.3841 (0.3801) loss 3.8418 (3.3803) grad_norm 1.5183 (1.3151) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:57:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][200/625] eta 0:02:43 lr 0.001723 wd 0.0500 time 0.4280 (0.3849) data time 0.0011 (0.0035) model time 0.4269 (0.3801) loss 3.0461 (3.3699) grad_norm 1.2946 (1.3349) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][210/625] eta 0:02:39 lr 0.001723 wd 0.0500 time 0.3688 (0.3846) data time 0.0009 (0.0035) model time 0.3679 (0.3797) loss 3.7903 (3.3774) grad_norm 1.0706 (1.3449) loss_scale 8192.0000 (8192.0000) mem 11225MB [2024-08-26 08:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][220/625] eta 0:02:35 lr 0.001723 wd 0.0500 time 0.3762 (0.3844) data time 0.0011 (0.0034) model time 0.3751 (0.3797) loss 2.6808 (3.3796) grad_norm 1.6955 (inf) loss_scale 4096.0000 (8025.1946) mem 11225MB [2024-08-26 08:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][230/625] eta 0:02:31 lr 0.001723 wd 0.0500 time 0.3833 (0.3841) data time 0.0008 (0.0033) model time 0.3825 (0.3796) loss 2.8566 (3.3887) grad_norm 1.3327 (inf) loss_scale 2048.0000 (7775.3074) mem 11225MB [2024-08-26 08:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][240/625] eta 0:02:28 lr 0.001723 wd 0.0500 time 0.3349 (0.3865) data time 0.0012 (0.0032) model time 0.3337 (0.3828) loss 2.7480 (3.3890) grad_norm 1.1533 (inf) loss_scale 2048.0000 (7537.6598) mem 11225MB [2024-08-26 08:57:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][250/625] eta 0:02:24 lr 0.001722 wd 0.0500 time 0.3812 (0.3862) data time 0.0008 (0.0031) model time 0.3804 (0.3826) loss 2.5015 (3.3749) grad_norm 1.7990 (inf) loss_scale 2048.0000 (7318.9482) mem 11225MB [2024-08-26 08:57:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][260/625] eta 0:02:20 lr 0.001722 wd 0.0500 time 0.3810 (0.3860) data time 0.0011 (0.0030) model time 0.3799 (0.3825) loss 3.6892 (3.3729) grad_norm 0.9785 (inf) loss_scale 2048.0000 (7116.9962) mem 11225MB [2024-08-26 08:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][270/625] eta 0:02:16 lr 0.001722 wd 0.0500 time 0.3808 (0.3859) data time 0.0011 (0.0030) model time 0.3798 (0.3824) loss 3.6877 (3.3757) grad_norm 1.3089 (inf) loss_scale 2048.0000 (6929.9483) mem 11225MB [2024-08-26 08:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][280/625] eta 0:02:13 lr 0.001722 wd 0.0500 time 0.3741 (0.3857) data time 0.0009 (0.0029) model time 0.3732 (0.3822) loss 4.3725 (3.3838) grad_norm 1.2514 (inf) loss_scale 2048.0000 (6756.2135) mem 11225MB [2024-08-26 08:57:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][290/625] eta 0:02:09 lr 0.001722 wd 0.0500 time 0.3848 (0.3869) data time 0.0008 (0.0028) model time 0.3839 (0.3838) loss 4.0550 (3.3730) grad_norm 1.2769 (inf) loss_scale 2048.0000 (6594.4192) mem 11225MB [2024-08-26 08:57:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][300/625] eta 0:02:05 lr 0.001722 wd 0.0500 time 0.3780 (0.3866) data time 0.0009 (0.0028) model time 0.3772 (0.3835) loss 2.5160 (3.3757) grad_norm 1.8347 (inf) loss_scale 2048.0000 (6443.3754) mem 11225MB [2024-08-26 08:57:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][310/625] eta 0:02:01 lr 0.001722 wd 0.0500 time 0.3848 (0.3864) data time 0.0009 (0.0027) model time 0.3839 (0.3834) loss 2.5851 (3.3668) grad_norm 0.9079 (inf) loss_scale 2048.0000 (6302.0450) mem 11225MB [2024-08-26 08:58:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][320/625] eta 0:01:57 lr 0.001722 wd 0.0500 time 0.3982 (0.3862) data time 0.0011 (0.0027) model time 0.3971 (0.3832) loss 4.2032 (3.3605) grad_norm 0.9841 (inf) loss_scale 2048.0000 (6169.5202) mem 11225MB [2024-08-26 08:58:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][330/625] eta 0:01:53 lr 0.001722 wd 0.0500 time 0.3702 (0.3860) data time 0.0009 (0.0026) model time 0.3694 (0.3830) loss 3.5211 (3.3630) grad_norm 1.2342 (inf) loss_scale 2048.0000 (6045.0030) mem 11225MB [2024-08-26 08:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][340/625] eta 0:01:49 lr 0.001721 wd 0.0500 time 0.3830 (0.3859) data time 0.0008 (0.0026) model time 0.3822 (0.3830) loss 3.7673 (3.3645) grad_norm 1.1329 (inf) loss_scale 2048.0000 (5927.7889) mem 11225MB [2024-08-26 08:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][350/625] eta 0:01:46 lr 0.001721 wd 0.0500 time 0.3909 (0.3858) data time 0.0010 (0.0026) model time 0.3900 (0.3829) loss 3.6215 (3.3634) grad_norm 1.4402 (inf) loss_scale 2048.0000 (5817.2536) mem 11225MB [2024-08-26 08:58:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][360/625] eta 0:01:42 lr 0.001721 wd 0.0500 time 0.3698 (0.3856) data time 0.0011 (0.0025) model time 0.3687 (0.3827) loss 3.5997 (3.3650) grad_norm 2.0453 (inf) loss_scale 2048.0000 (5712.8421) mem 11225MB [2024-08-26 08:58:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][370/625] eta 0:01:38 lr 0.001721 wd 0.0500 time 0.3785 (0.3854) data time 0.0008 (0.0025) model time 0.3777 (0.3825) loss 2.3756 (3.3721) grad_norm 0.8280 (inf) loss_scale 2048.0000 (5614.0593) mem 11225MB [2024-08-26 08:58:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][380/625] eta 0:01:34 lr 0.001721 wd 0.0500 time 0.3815 (0.3852) data time 0.0011 (0.0025) model time 0.3804 (0.3824) loss 3.5967 (3.3740) grad_norm 1.2097 (inf) loss_scale 2048.0000 (5520.4619) mem 11225MB [2024-08-26 08:58:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][390/625] eta 0:01:30 lr 0.001721 wd 0.0500 time 0.3841 (0.3851) data time 0.0011 (0.0024) model time 0.3831 (0.3822) loss 3.8041 (3.3785) grad_norm 1.3539 (inf) loss_scale 2048.0000 (5431.6522) mem 11225MB [2024-08-26 08:58:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][400/625] eta 0:01:26 lr 0.001721 wd 0.0500 time 0.3874 (0.3849) data time 0.0012 (0.0024) model time 0.3862 (0.3821) loss 3.3892 (3.3796) grad_norm 0.8475 (inf) loss_scale 2048.0000 (5347.2718) mem 11225MB [2024-08-26 08:58:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][410/625] eta 0:01:22 lr 0.001721 wd 0.0500 time 0.3780 (0.3848) data time 0.0011 (0.0024) model time 0.3769 (0.3820) loss 3.5062 (3.3802) grad_norm 1.4887 (inf) loss_scale 2048.0000 (5266.9976) mem 11225MB [2024-08-26 08:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][420/625] eta 0:01:19 lr 0.001720 wd 0.0500 time 0.5813 (0.3857) data time 0.0008 (0.0023) model time 0.5804 (0.3831) loss 4.2542 (3.3832) grad_norm 0.9930 (inf) loss_scale 2048.0000 (5190.5368) mem 11225MB [2024-08-26 08:58:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][430/625] eta 0:01:15 lr 0.001720 wd 0.0500 time 0.3764 (0.3856) data time 0.0011 (0.0023) model time 0.3753 (0.3830) loss 3.6272 (3.3793) grad_norm 1.7443 (inf) loss_scale 2048.0000 (5117.6241) mem 11225MB [2024-08-26 08:58:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][440/625] eta 0:01:11 lr 0.001720 wd 0.0500 time 0.3721 (0.3854) data time 0.0009 (0.0023) model time 0.3712 (0.3829) loss 2.5603 (3.3837) grad_norm 1.5472 (inf) loss_scale 2048.0000 (5048.0181) mem 11225MB [2024-08-26 08:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][450/625] eta 0:01:07 lr 0.001720 wd 0.0500 time 0.3727 (0.3857) data time 0.0008 (0.0023) model time 0.3719 (0.3832) loss 3.4112 (3.3865) grad_norm 1.4673 (inf) loss_scale 2048.0000 (4981.4989) mem 11225MB [2024-08-26 08:58:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][460/625] eta 0:01:03 lr 0.001720 wd 0.0500 time 0.3730 (0.3855) data time 0.0008 (0.0022) model time 0.3722 (0.3830) loss 4.4596 (3.3878) grad_norm 1.2537 (inf) loss_scale 2048.0000 (4917.8655) mem 11225MB [2024-08-26 08:58:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][470/625] eta 0:00:59 lr 0.001720 wd 0.0500 time 0.3849 (0.3853) data time 0.0010 (0.0022) model time 0.3838 (0.3829) loss 2.9235 (3.3899) grad_norm 1.0992 (inf) loss_scale 2048.0000 (4856.9342) mem 11225MB [2024-08-26 08:59:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][480/625] eta 0:00:55 lr 0.001720 wd 0.0500 time 0.3726 (0.3851) data time 0.0008 (0.0022) model time 0.3718 (0.3826) loss 3.7144 (3.3954) grad_norm 1.1621 (inf) loss_scale 2048.0000 (4798.5364) mem 11225MB [2024-08-26 08:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][490/625] eta 0:00:51 lr 0.001720 wd 0.0500 time 0.3767 (0.3849) data time 0.0008 (0.0022) model time 0.3759 (0.3825) loss 2.8865 (3.3902) grad_norm 1.1334 (inf) loss_scale 2048.0000 (4742.5173) mem 11225MB [2024-08-26 08:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][500/625] eta 0:00:48 lr 0.001719 wd 0.0500 time 0.3801 (0.3860) data time 0.0008 (0.0021) model time 0.3793 (0.3837) loss 4.3847 (3.3921) grad_norm 1.3003 (inf) loss_scale 2048.0000 (4688.7345) mem 11225MB [2024-08-26 08:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][510/625] eta 0:00:44 lr 0.001719 wd 0.0500 time 0.3758 (0.3859) data time 0.0009 (0.0021) model time 0.3749 (0.3836) loss 2.9968 (3.3912) grad_norm 1.3796 (inf) loss_scale 2048.0000 (4637.0568) mem 11225MB [2024-08-26 08:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][520/625] eta 0:00:40 lr 0.001719 wd 0.0500 time 0.3761 (0.3857) data time 0.0009 (0.0021) model time 0.3752 (0.3834) loss 3.1475 (3.3909) grad_norm 0.9351 (inf) loss_scale 2048.0000 (4587.3628) mem 11225MB [2024-08-26 08:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][530/625] eta 0:00:36 lr 0.001719 wd 0.0500 time 0.3723 (0.3856) data time 0.0009 (0.0021) model time 0.3714 (0.3833) loss 2.7282 (3.3923) grad_norm 1.7041 (inf) loss_scale 2048.0000 (4539.5405) mem 11225MB [2024-08-26 08:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][540/625] eta 0:00:32 lr 0.001719 wd 0.0500 time 0.3810 (0.3862) data time 0.0009 (0.0021) model time 0.3801 (0.3840) loss 2.7866 (3.3930) grad_norm 1.1059 (inf) loss_scale 2048.0000 (4493.4861) mem 11225MB [2024-08-26 08:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][550/625] eta 0:00:28 lr 0.001719 wd 0.0500 time 0.3759 (0.3861) data time 0.0008 (0.0021) model time 0.3752 (0.3839) loss 3.0962 (3.3931) grad_norm 1.0335 (inf) loss_scale 2048.0000 (4449.1034) mem 11225MB [2024-08-26 08:59:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][560/625] eta 0:00:25 lr 0.001719 wd 0.0500 time 0.3736 (0.3860) data time 0.0009 (0.0020) model time 0.3728 (0.3838) loss 4.1612 (3.3980) grad_norm 1.1770 (inf) loss_scale 2048.0000 (4406.3030) mem 11225MB [2024-08-26 08:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][570/625] eta 0:00:21 lr 0.001719 wd 0.0500 time 0.3817 (0.3858) data time 0.0011 (0.0020) model time 0.3806 (0.3836) loss 3.6924 (3.3998) grad_norm 1.3569 (inf) loss_scale 2048.0000 (4365.0018) mem 11225MB [2024-08-26 08:59:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][580/625] eta 0:00:17 lr 0.001718 wd 0.0500 time 0.3759 (0.3857) data time 0.0011 (0.0020) model time 0.3748 (0.3835) loss 3.5887 (3.3979) grad_norm 1.4120 (inf) loss_scale 2048.0000 (4325.1222) mem 11225MB [2024-08-26 08:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][590/625] eta 0:00:13 lr 0.001718 wd 0.0500 time 0.3772 (0.3855) data time 0.0008 (0.0020) model time 0.3764 (0.3833) loss 3.3331 (3.4012) grad_norm 1.5956 (inf) loss_scale 2048.0000 (4286.5922) mem 11225MB [2024-08-26 08:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][600/625] eta 0:00:09 lr 0.001718 wd 0.0500 time 0.3766 (0.3853) data time 0.0011 (0.0020) model time 0.3756 (0.3832) loss 3.0965 (3.4018) grad_norm 1.1143 (inf) loss_scale 2048.0000 (4249.3444) mem 11225MB [2024-08-26 08:59:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][610/625] eta 0:00:05 lr 0.001718 wd 0.0500 time 0.3681 (0.3852) data time 0.0006 (0.0020) model time 0.3675 (0.3830) loss 4.0716 (3.4003) grad_norm 0.9175 (inf) loss_scale 2048.0000 (4213.3159) mem 11225MB [2024-08-26 08:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [88/300][620/625] eta 0:00:01 lr 0.001718 wd 0.0500 time 0.3699 (0.3849) data time 0.0008 (0.0019) model time 0.3691 (0.3828) loss 3.5404 (3.4043) grad_norm 1.6574 (inf) loss_scale 2048.0000 (4178.4477) mem 11225MB [2024-08-26 08:59:58 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 88 training takes 0:04:00 [2024-08-26 08:59:58 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 08:59:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.6519 (0.6519) Acc@1 84.668 (84.668) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 09:00:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.146) Loss 1.0869 (0.7956) Acc@1 75.146 (81.374) Acc@5 93.311 (96.205) Mem 11225MB [2024-08-26 09:00:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.2197 (0.9611) Acc@1 71.289 (77.732) Acc@5 91.895 (94.294) Mem 11225MB [2024-08-26 09:00:02 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.515 Acc@5 94.272 [2024-08-26 09:00:02 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.5% [2024-08-26 09:00:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.823 (0.823) Loss 0.5591 (0.5591) Acc@1 88.379 (88.379) Acc@5 98.242 (98.242) Mem 11225MB [2024-08-26 09:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.179) Loss 0.9287 (0.6961) Acc@1 77.588 (84.135) Acc@5 95.068 (97.066) Mem 11225MB [2024-08-26 09:00:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.138) Loss 1.0947 (0.8482) Acc@1 72.607 (80.346) Acc@5 92.773 (95.359) Mem 11225MB [2024-08-26 09:00:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.058 Acc@5 95.327 [2024-08-26 09:00:05 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.1% [2024-08-26 09:00:05 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.06% [2024-08-26 09:00:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:00:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][0/625] eta 0:08:20 lr 0.001718 wd 0.0500 time 0.8002 (0.8002) data time 0.4660 (0.4660) model time 0.0000 (0.0000) loss 3.4333 (3.4333) grad_norm 1.3879 (1.3879) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][10/625] eta 0:04:20 lr 0.001718 wd 0.0500 time 0.3760 (0.4232) data time 0.0013 (0.0435) model time 0.0000 (0.0000) loss 3.8072 (3.5665) grad_norm 1.2445 (1.3597) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][20/625] eta 0:04:02 lr 0.001718 wd 0.0500 time 0.3760 (0.4011) data time 0.0009 (0.0233) model time 0.0000 (0.0000) loss 4.0128 (3.5960) grad_norm 1.3604 (1.3098) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][30/625] eta 0:03:54 lr 0.001717 wd 0.0500 time 0.3832 (0.3942) data time 0.0011 (0.0162) model time 0.0000 (0.0000) loss 3.4679 (3.4774) grad_norm 1.2890 (1.2883) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][40/625] eta 0:03:48 lr 0.001717 wd 0.0500 time 0.3796 (0.3898) data time 0.0008 (0.0125) model time 0.0000 (0.0000) loss 3.6979 (3.4566) grad_norm 1.7205 (1.3541) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][50/625] eta 0:03:43 lr 0.001717 wd 0.0500 time 0.3710 (0.3879) data time 0.0011 (0.0103) model time 0.0000 (0.0000) loss 2.2092 (3.4638) grad_norm 1.3925 (1.3693) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][60/625] eta 0:03:40 lr 0.001717 wd 0.0500 time 0.5890 (0.3898) data time 0.0010 (0.0088) model time 0.5880 (0.3980) loss 3.1056 (3.4400) grad_norm 1.2993 (1.3852) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][70/625] eta 0:03:40 lr 0.001717 wd 0.0500 time 0.5901 (0.3967) data time 0.0011 (0.0077) model time 0.5890 (0.4180) loss 3.9400 (3.4409) grad_norm 1.4609 (1.3801) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][80/625] eta 0:03:34 lr 0.001717 wd 0.0500 time 0.3766 (0.3943) data time 0.0011 (0.0069) model time 0.3755 (0.4038) loss 2.9062 (3.4483) grad_norm 0.8745 (1.3578) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][90/625] eta 0:03:29 lr 0.001717 wd 0.0500 time 0.3734 (0.3923) data time 0.0011 (0.0063) model time 0.3723 (0.3966) loss 2.8228 (3.4322) grad_norm 1.4661 (1.3852) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][100/625] eta 0:03:25 lr 0.001717 wd 0.0500 time 0.3773 (0.3922) data time 0.0011 (0.0058) model time 0.3762 (0.3953) loss 3.9213 (3.4288) grad_norm 1.5251 (1.3889) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][110/625] eta 0:03:21 lr 0.001716 wd 0.0500 time 0.3742 (0.3908) data time 0.0011 (0.0054) model time 0.3731 (0.3920) loss 3.4264 (3.4476) grad_norm 1.4027 (1.3953) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][120/625] eta 0:03:16 lr 0.001716 wd 0.0500 time 0.3753 (0.3897) data time 0.0010 (0.0050) model time 0.3743 (0.3898) loss 2.6275 (3.4574) grad_norm 0.9593 (1.3794) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:00:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][130/625] eta 0:03:12 lr 0.001716 wd 0.0500 time 0.3967 (0.3889) data time 0.0008 (0.0047) model time 0.3959 (0.3883) loss 4.7905 (3.4385) grad_norm 1.3461 (1.3772) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][140/625] eta 0:03:08 lr 0.001716 wd 0.0500 time 0.3788 (0.3881) data time 0.0009 (0.0045) model time 0.3779 (0.3870) loss 3.6939 (3.4541) grad_norm 1.4680 (1.3665) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][150/625] eta 0:03:03 lr 0.001716 wd 0.0500 time 0.3807 (0.3873) data time 0.0012 (0.0042) model time 0.3795 (0.3858) loss 3.9801 (3.4509) grad_norm 1.6740 (1.3732) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][160/625] eta 0:02:59 lr 0.001716 wd 0.0500 time 0.3671 (0.3867) data time 0.0012 (0.0040) model time 0.3659 (0.3849) loss 3.4797 (3.4614) grad_norm 1.3745 (1.3732) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][170/625] eta 0:02:55 lr 0.001716 wd 0.0500 time 0.3787 (0.3862) data time 0.0010 (0.0039) model time 0.3778 (0.3842) loss 3.6779 (3.4497) grad_norm 1.0761 (1.3772) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][180/625] eta 0:02:51 lr 0.001716 wd 0.0500 time 0.3733 (0.3856) data time 0.0008 (0.0037) model time 0.3725 (0.3835) loss 4.2506 (3.4492) grad_norm 1.2268 (1.3754) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][190/625] eta 0:02:47 lr 0.001715 wd 0.0500 time 0.3816 (0.3851) data time 0.0009 (0.0036) model time 0.3807 (0.3830) loss 4.1846 (3.4703) grad_norm 1.0925 (1.3884) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][200/625] eta 0:02:43 lr 0.001715 wd 0.0500 time 0.3791 (0.3847) data time 0.0010 (0.0035) model time 0.3780 (0.3824) loss 3.5310 (3.4649) grad_norm 1.7715 (1.3877) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][210/625] eta 0:02:39 lr 0.001715 wd 0.0500 time 0.3736 (0.3846) data time 0.0008 (0.0034) model time 0.3727 (0.3824) loss 2.9558 (3.4589) grad_norm 1.0856 (1.3806) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][220/625] eta 0:02:35 lr 0.001715 wd 0.0500 time 0.3774 (0.3843) data time 0.0008 (0.0033) model time 0.3766 (0.3820) loss 4.2724 (3.4665) grad_norm 1.1501 (1.3728) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][230/625] eta 0:02:31 lr 0.001715 wd 0.0500 time 0.3776 (0.3840) data time 0.0011 (0.0032) model time 0.3764 (0.3817) loss 3.8075 (3.4677) grad_norm 1.2490 (1.3720) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][240/625] eta 0:02:27 lr 0.001715 wd 0.0500 time 0.3715 (0.3837) data time 0.0011 (0.0031) model time 0.3704 (0.3814) loss 3.3393 (3.4623) grad_norm 1.3053 (1.3794) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][250/625] eta 0:02:24 lr 0.001715 wd 0.0500 time 0.3783 (0.3851) data time 0.0009 (0.0030) model time 0.3775 (0.3832) loss 2.7706 (3.4585) grad_norm 1.3671 (1.3850) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][260/625] eta 0:02:20 lr 0.001715 wd 0.0500 time 0.3704 (0.3848) data time 0.0011 (0.0029) model time 0.3693 (0.3829) loss 2.4339 (3.4454) grad_norm 1.7577 (1.3789) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][270/625] eta 0:02:16 lr 0.001714 wd 0.0500 time 0.3799 (0.3848) data time 0.0008 (0.0029) model time 0.3791 (0.3829) loss 4.1922 (3.4497) grad_norm 1.1101 (1.3798) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][280/625] eta 0:02:12 lr 0.001714 wd 0.0500 time 0.3870 (0.3853) data time 0.0009 (0.0028) model time 0.3861 (0.3836) loss 3.4899 (3.4552) grad_norm 1.6706 (1.3786) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:01:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][290/625] eta 0:02:08 lr 0.001714 wd 0.0500 time 0.3723 (0.3850) data time 0.0011 (0.0027) model time 0.3712 (0.3832) loss 3.2987 (3.4577) grad_norm 2.4349 (1.3888) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][300/625] eta 0:02:05 lr 0.001714 wd 0.0500 time 0.3653 (0.3848) data time 0.0012 (0.0027) model time 0.3641 (0.3830) loss 2.9400 (3.4511) grad_norm 0.9888 (1.3901) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][310/625] eta 0:02:01 lr 0.001714 wd 0.0500 time 0.3843 (0.3851) data time 0.0008 (0.0026) model time 0.3835 (0.3834) loss 2.4850 (3.4435) grad_norm 1.6246 (1.3848) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][320/625] eta 0:01:57 lr 0.001714 wd 0.0500 time 0.3736 (0.3850) data time 0.0009 (0.0026) model time 0.3728 (0.3833) loss 3.8630 (3.4403) grad_norm 1.1587 (1.3817) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][330/625] eta 0:01:53 lr 0.001714 wd 0.0500 time 0.3744 (0.3847) data time 0.0009 (0.0025) model time 0.3736 (0.3830) loss 4.1302 (3.4349) grad_norm 1.3474 (1.3861) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][340/625] eta 0:01:49 lr 0.001714 wd 0.0500 time 0.3910 (0.3846) data time 0.0008 (0.0025) model time 0.3902 (0.3828) loss 2.0056 (3.4308) grad_norm 1.3715 (1.3872) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][350/625] eta 0:01:45 lr 0.001713 wd 0.0500 time 0.3758 (0.3844) data time 0.0008 (0.0025) model time 0.3750 (0.3827) loss 3.4546 (3.4346) grad_norm 1.2968 (1.3925) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][360/625] eta 0:01:41 lr 0.001713 wd 0.0500 time 0.3756 (0.3843) data time 0.0009 (0.0024) model time 0.3746 (0.3826) loss 3.4800 (3.4358) grad_norm 1.3672 (1.3900) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][370/625] eta 0:01:38 lr 0.001713 wd 0.0500 time 0.3790 (0.3853) data time 0.0010 (0.0024) model time 0.3780 (0.3837) loss 3.3105 (3.4359) grad_norm 1.3144 (1.3897) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][380/625] eta 0:01:34 lr 0.001713 wd 0.0500 time 0.3640 (0.3851) data time 0.0010 (0.0024) model time 0.3630 (0.3835) loss 3.7393 (3.4287) grad_norm 1.0966 (1.3834) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][390/625] eta 0:01:30 lr 0.001713 wd 0.0500 time 0.3815 (0.3850) data time 0.0011 (0.0023) model time 0.3804 (0.3835) loss 3.6476 (3.4323) grad_norm 2.2436 (1.3853) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][400/625] eta 0:01:26 lr 0.001713 wd 0.0500 time 0.3819 (0.3849) data time 0.0010 (0.0023) model time 0.3809 (0.3833) loss 3.4998 (3.4376) grad_norm 1.1965 (1.3833) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][410/625] eta 0:01:22 lr 0.001713 wd 0.0500 time 0.3687 (0.3848) data time 0.0009 (0.0023) model time 0.3678 (0.3832) loss 3.9287 (3.4347) grad_norm 1.0460 (1.3791) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][420/625] eta 0:01:18 lr 0.001713 wd 0.0500 time 0.3818 (0.3846) data time 0.0011 (0.0022) model time 0.3807 (0.3830) loss 3.4803 (3.4315) grad_norm 1.2705 (1.3741) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][430/625] eta 0:01:14 lr 0.001712 wd 0.0500 time 0.3872 (0.3845) data time 0.0009 (0.0022) model time 0.3863 (0.3829) loss 3.9512 (3.4351) grad_norm 1.0262 (1.3695) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:02:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][440/625] eta 0:01:11 lr 0.001712 wd 0.0500 time 0.3753 (0.3843) data time 0.0011 (0.0022) model time 0.3743 (0.3827) loss 2.9614 (3.4276) grad_norm 1.4943 (1.3672) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][450/625] eta 0:01:07 lr 0.001712 wd 0.0500 time 0.3750 (0.3842) data time 0.0008 (0.0022) model time 0.3742 (0.3825) loss 4.0064 (3.4302) grad_norm 1.2802 (1.3661) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][460/625] eta 0:01:03 lr 0.001712 wd 0.0500 time 0.3763 (0.3840) data time 0.0011 (0.0022) model time 0.3752 (0.3824) loss 3.5572 (3.4366) grad_norm 1.4779 (1.3667) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][470/625] eta 0:00:59 lr 0.001712 wd 0.0500 time 0.3740 (0.3847) data time 0.0009 (0.0021) model time 0.3732 (0.3832) loss 2.6702 (3.4329) grad_norm 1.2629 (1.3674) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][480/625] eta 0:00:55 lr 0.001712 wd 0.0500 time 0.3767 (0.3846) data time 0.0011 (0.0021) model time 0.3756 (0.3830) loss 3.7461 (3.4284) grad_norm 1.4980 (1.3636) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][490/625] eta 0:00:51 lr 0.001712 wd 0.0500 time 0.6140 (0.3849) data time 0.0009 (0.0021) model time 0.6131 (0.3834) loss 2.7551 (3.4287) grad_norm 1.0316 (1.3589) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][500/625] eta 0:00:48 lr 0.001712 wd 0.0500 time 0.3729 (0.3847) data time 0.0011 (0.0021) model time 0.3718 (0.3832) loss 3.8407 (3.4355) grad_norm 1.1046 (1.3559) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][510/625] eta 0:00:44 lr 0.001711 wd 0.0500 time 0.3764 (0.3846) data time 0.0009 (0.0021) model time 0.3755 (0.3831) loss 4.3936 (3.4331) grad_norm 1.0311 (1.3589) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][520/625] eta 0:00:40 lr 0.001711 wd 0.0500 time 0.3869 (0.3846) data time 0.0010 (0.0021) model time 0.3859 (0.3830) loss 3.8825 (3.4311) grad_norm 1.0812 (1.3579) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][530/625] eta 0:00:36 lr 0.001711 wd 0.0500 time 0.3694 (0.3844) data time 0.0011 (0.0020) model time 0.3684 (0.3829) loss 3.7743 (3.4236) grad_norm 1.1724 (1.3529) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][540/625] eta 0:00:32 lr 0.001711 wd 0.0500 time 0.3794 (0.3843) data time 0.0011 (0.0020) model time 0.3783 (0.3827) loss 2.7050 (3.4265) grad_norm 0.9160 (1.3558) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][550/625] eta 0:00:28 lr 0.001711 wd 0.0500 time 0.3811 (0.3842) data time 0.0008 (0.0020) model time 0.3802 (0.3826) loss 2.8012 (3.4284) grad_norm 1.5061 (1.3579) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][560/625] eta 0:00:24 lr 0.001711 wd 0.0500 time 0.3737 (0.3841) data time 0.0012 (0.0020) model time 0.3725 (0.3825) loss 3.3974 (3.4260) grad_norm 1.2324 (1.3560) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][570/625] eta 0:00:21 lr 0.001711 wd 0.0500 time 0.3744 (0.3839) data time 0.0009 (0.0020) model time 0.3736 (0.3824) loss 2.9965 (3.4197) grad_norm 1.3675 (1.3572) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][580/625] eta 0:00:17 lr 0.001711 wd 0.0500 time 0.3750 (0.3846) data time 0.0008 (0.0020) model time 0.3742 (0.3831) loss 4.6521 (3.4215) grad_norm 1.3492 (1.3579) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][590/625] eta 0:00:13 lr 0.001710 wd 0.0500 time 0.3758 (0.3851) data time 0.0009 (0.0019) model time 0.3749 (0.3837) loss 3.8167 (3.4247) grad_norm 1.4668 (1.3552) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:03:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][600/625] eta 0:00:09 lr 0.001710 wd 0.0500 time 0.3770 (0.3850) data time 0.0012 (0.0019) model time 0.3758 (0.3836) loss 3.4274 (3.4290) grad_norm 1.0921 (1.3527) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][610/625] eta 0:00:05 lr 0.001710 wd 0.0500 time 0.3318 (0.3852) data time 0.0007 (0.0019) model time 0.3311 (0.3838) loss 2.7565 (3.4301) grad_norm 1.9120 (1.3539) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [89/300][620/625] eta 0:00:01 lr 0.001710 wd 0.0500 time 0.3660 (0.3849) data time 0.0008 (0.0019) model time 0.3652 (0.3834) loss 3.5767 (3.4362) grad_norm 1.9742 (1.3549) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 89 training takes 0:04:00 [2024-08-26 09:04:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:04:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:04:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.507 (0.507) Loss 0.6392 (0.6392) Acc@1 85.791 (85.791) Acc@5 97.705 (97.705) Mem 11225MB [2024-08-26 09:04:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.1406 (0.8052) Acc@1 72.852 (81.672) Acc@5 92.822 (96.280) Mem 11225MB [2024-08-26 09:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.2812 (0.9773) Acc@1 69.873 (77.490) Acc@5 90.088 (94.055) Mem 11225MB [2024-08-26 09:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.357 Acc@5 94.014 [2024-08-26 09:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.4% [2024-08-26 09:04:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.821 (0.821) Loss 0.5586 (0.5586) Acc@1 88.232 (88.232) Acc@5 98.193 (98.193) Mem 11225MB [2024-08-26 09:04:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.176) Loss 0.9272 (0.6953) Acc@1 77.832 (84.189) Acc@5 95.068 (97.053) Mem 11225MB [2024-08-26 09:04:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.0938 (0.8470) Acc@1 72.852 (80.357) Acc@5 92.822 (95.359) Mem 11225MB [2024-08-26 09:04:14 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.086 Acc@5 95.331 [2024-08-26 09:04:14 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.1% [2024-08-26 09:04:14 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.09% [2024-08-26 09:04:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:04:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:04:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][0/625] eta 0:07:54 lr 0.001710 wd 0.0500 time 0.7586 (0.7586) data time 0.4065 (0.4065) model time 0.0000 (0.0000) loss 2.5529 (2.5529) grad_norm 1.4666 (1.4666) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][10/625] eta 0:04:13 lr 0.001710 wd 0.0500 time 0.3839 (0.4118) data time 0.0008 (0.0379) model time 0.0000 (0.0000) loss 4.0862 (3.3404) grad_norm 2.0044 (1.3930) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][20/625] eta 0:03:59 lr 0.001710 wd 0.0500 time 0.3767 (0.3956) data time 0.0008 (0.0204) model time 0.0000 (0.0000) loss 4.4685 (3.5290) grad_norm 0.9786 (1.3458) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][30/625] eta 0:03:51 lr 0.001710 wd 0.0500 time 0.3709 (0.3894) data time 0.0010 (0.0142) model time 0.0000 (0.0000) loss 3.9947 (3.5567) grad_norm 1.2652 (1.3244) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][40/625] eta 0:03:46 lr 0.001710 wd 0.0500 time 0.3789 (0.3867) data time 0.0011 (0.0110) model time 0.0000 (0.0000) loss 2.5003 (3.5177) grad_norm 0.9982 (1.3024) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][50/625] eta 0:03:41 lr 0.001709 wd 0.0500 time 0.3790 (0.3850) data time 0.0010 (0.0091) model time 0.0000 (0.0000) loss 4.0037 (3.4543) grad_norm 1.1027 (1.2937) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][60/625] eta 0:03:36 lr 0.001709 wd 0.0500 time 0.3733 (0.3835) data time 0.0008 (0.0078) model time 0.3725 (0.3746) loss 3.1326 (3.4475) grad_norm 1.0183 (1.3688) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][70/625] eta 0:03:32 lr 0.001709 wd 0.0500 time 0.3800 (0.3825) data time 0.0011 (0.0068) model time 0.3790 (0.3752) loss 2.7395 (3.4443) grad_norm 1.3431 (1.3796) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][80/625] eta 0:03:28 lr 0.001709 wd 0.0500 time 0.3771 (0.3820) data time 0.0008 (0.0061) model time 0.3763 (0.3759) loss 3.9007 (3.4501) grad_norm 1.2189 (1.3790) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][90/625] eta 0:03:24 lr 0.001709 wd 0.0500 time 0.3819 (0.3817) data time 0.0009 (0.0056) model time 0.3810 (0.3763) loss 3.6477 (3.4550) grad_norm 1.8945 (1.3764) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][100/625] eta 0:03:22 lr 0.001709 wd 0.0500 time 0.3769 (0.3852) data time 0.0011 (0.0052) model time 0.3758 (0.3843) loss 2.7139 (3.4358) grad_norm 0.9381 (1.3550) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:04:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][110/625] eta 0:03:18 lr 0.001709 wd 0.0500 time 0.3793 (0.3847) data time 0.0009 (0.0048) model time 0.3784 (0.3834) loss 2.8352 (3.4231) grad_norm 1.0014 (1.3262) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][120/625] eta 0:03:14 lr 0.001709 wd 0.0500 time 0.3740 (0.3859) data time 0.0008 (0.0045) model time 0.3731 (0.3854) loss 3.7634 (3.4166) grad_norm 1.3192 (1.3075) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][130/625] eta 0:03:10 lr 0.001708 wd 0.0500 time 0.3823 (0.3855) data time 0.0011 (0.0042) model time 0.3812 (0.3847) loss 3.4121 (3.4199) grad_norm 2.0288 (1.3143) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][140/625] eta 0:03:06 lr 0.001708 wd 0.0500 time 0.3770 (0.3851) data time 0.0011 (0.0040) model time 0.3759 (0.3840) loss 3.0487 (3.4450) grad_norm 1.0403 (1.3194) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][150/625] eta 0:03:02 lr 0.001708 wd 0.0500 time 0.3754 (0.3846) data time 0.0008 (0.0038) model time 0.3746 (0.3833) loss 3.8898 (3.4533) grad_norm 1.2286 (1.3180) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][160/625] eta 0:02:59 lr 0.001708 wd 0.0500 time 0.3730 (0.3869) data time 0.0011 (0.0037) model time 0.3719 (0.3866) loss 3.0694 (3.4653) grad_norm 0.8958 (1.3052) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][170/625] eta 0:02:55 lr 0.001708 wd 0.0500 time 0.3803 (0.3864) data time 0.0008 (0.0035) model time 0.3795 (0.3859) loss 3.0596 (3.4517) grad_norm 1.1616 (1.2995) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][180/625] eta 0:02:51 lr 0.001708 wd 0.0500 time 0.3797 (0.3859) data time 0.0010 (0.0034) model time 0.3787 (0.3852) loss 4.0504 (3.4437) grad_norm 1.4492 (1.3033) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][190/625] eta 0:02:47 lr 0.001708 wd 0.0500 time 0.3759 (0.3855) data time 0.0008 (0.0032) model time 0.3751 (0.3845) loss 3.3398 (3.4319) grad_norm 2.0558 (1.3131) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][200/625] eta 0:02:43 lr 0.001708 wd 0.0500 time 0.3805 (0.3851) data time 0.0008 (0.0031) model time 0.3797 (0.3841) loss 4.0323 (3.4463) grad_norm 0.9895 (1.3087) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][210/625] eta 0:02:39 lr 0.001707 wd 0.0500 time 0.3739 (0.3848) data time 0.0011 (0.0030) model time 0.3728 (0.3836) loss 3.4529 (3.4535) grad_norm 1.5690 (1.3106) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][220/625] eta 0:02:35 lr 0.001707 wd 0.0500 time 0.3768 (0.3844) data time 0.0008 (0.0030) model time 0.3760 (0.3831) loss 4.1037 (3.4638) grad_norm 1.1777 (1.3103) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][230/625] eta 0:02:31 lr 0.001707 wd 0.0500 time 0.3762 (0.3841) data time 0.0008 (0.0029) model time 0.3754 (0.3827) loss 3.4502 (3.4672) grad_norm 1.1338 (1.3107) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][240/625] eta 0:02:28 lr 0.001707 wd 0.0500 time 0.3830 (0.3846) data time 0.0014 (0.0028) model time 0.3816 (0.3833) loss 3.1768 (3.4677) grad_norm 1.0508 (1.3205) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][250/625] eta 0:02:24 lr 0.001707 wd 0.0500 time 0.3754 (0.3843) data time 0.0011 (0.0028) model time 0.3743 (0.3829) loss 2.6521 (3.4660) grad_norm 0.8917 (1.3220) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][260/625] eta 0:02:20 lr 0.001707 wd 0.0500 time 0.3806 (0.3840) data time 0.0008 (0.0027) model time 0.3799 (0.3826) loss 4.1331 (3.4543) grad_norm 0.9755 (1.3135) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][270/625] eta 0:02:16 lr 0.001707 wd 0.0500 time 0.3768 (0.3838) data time 0.0008 (0.0026) model time 0.3760 (0.3824) loss 3.6475 (3.4468) grad_norm 1.1456 (1.3098) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][280/625] eta 0:02:12 lr 0.001707 wd 0.0500 time 0.3688 (0.3836) data time 0.0009 (0.0026) model time 0.3679 (0.3821) loss 4.0099 (3.4518) grad_norm 1.1633 (1.3124) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][290/625] eta 0:02:08 lr 0.001706 wd 0.0500 time 0.3828 (0.3833) data time 0.0008 (0.0025) model time 0.3820 (0.3818) loss 3.7799 (3.4485) grad_norm 1.2468 (1.3073) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][300/625] eta 0:02:04 lr 0.001706 wd 0.0500 time 0.3793 (0.3831) data time 0.0011 (0.0025) model time 0.3782 (0.3816) loss 2.7367 (3.4485) grad_norm 0.9678 (1.3024) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][310/625] eta 0:02:00 lr 0.001706 wd 0.0500 time 0.3741 (0.3837) data time 0.0011 (0.0024) model time 0.3730 (0.3823) loss 3.4881 (3.4526) grad_norm 1.5919 (1.3035) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][320/625] eta 0:01:56 lr 0.001706 wd 0.0500 time 0.3800 (0.3835) data time 0.0011 (0.0024) model time 0.3789 (0.3820) loss 3.0076 (3.4414) grad_norm 1.0095 (1.3047) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][330/625] eta 0:01:53 lr 0.001706 wd 0.0500 time 0.3804 (0.3834) data time 0.0011 (0.0024) model time 0.3793 (0.3820) loss 4.0130 (3.4389) grad_norm 1.3121 (1.3011) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][340/625] eta 0:01:49 lr 0.001706 wd 0.0500 time 0.3770 (0.3838) data time 0.0010 (0.0023) model time 0.3760 (0.3825) loss 2.8974 (3.4462) grad_norm 0.8982 (1.2996) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][350/625] eta 0:01:45 lr 0.001706 wd 0.0500 time 0.3777 (0.3836) data time 0.0008 (0.0023) model time 0.3769 (0.3823) loss 3.6538 (3.4453) grad_norm 2.4457 (1.3063) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][360/625] eta 0:01:41 lr 0.001705 wd 0.0500 time 0.3817 (0.3835) data time 0.0008 (0.0023) model time 0.3809 (0.3821) loss 3.4171 (3.4482) grad_norm 1.5600 (1.3178) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][370/625] eta 0:01:37 lr 0.001705 wd 0.0500 time 0.3757 (0.3834) data time 0.0010 (0.0022) model time 0.3747 (0.3820) loss 3.3237 (3.4444) grad_norm 1.0320 (1.3272) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][380/625] eta 0:01:33 lr 0.001705 wd 0.0500 time 0.3748 (0.3833) data time 0.0008 (0.0022) model time 0.3740 (0.3819) loss 3.0753 (3.4448) grad_norm 1.0352 (1.3225) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][390/625] eta 0:01:30 lr 0.001705 wd 0.0500 time 0.3824 (0.3832) data time 0.0008 (0.0022) model time 0.3816 (0.3818) loss 2.7060 (3.4310) grad_norm 1.2989 (1.3200) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][400/625] eta 0:01:26 lr 0.001705 wd 0.0500 time 0.3806 (0.3832) data time 0.0008 (0.0021) model time 0.3798 (0.3818) loss 3.0651 (3.4333) grad_norm 1.5784 (1.3225) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][410/625] eta 0:01:22 lr 0.001705 wd 0.0500 time 0.3792 (0.3830) data time 0.0010 (0.0021) model time 0.3782 (0.3816) loss 4.2489 (3.4365) grad_norm 0.9216 (1.3186) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][420/625] eta 0:01:18 lr 0.001705 wd 0.0500 time 0.3859 (0.3829) data time 0.0008 (0.0021) model time 0.3851 (0.3815) loss 4.3857 (3.4389) grad_norm 1.5501 (1.3208) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][430/625] eta 0:01:14 lr 0.001705 wd 0.0500 time 0.3780 (0.3829) data time 0.0008 (0.0021) model time 0.3772 (0.3814) loss 4.1023 (3.4364) grad_norm 1.4183 (1.3232) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][440/625] eta 0:01:10 lr 0.001704 wd 0.0500 time 0.3792 (0.3827) data time 0.0011 (0.0021) model time 0.3781 (0.3813) loss 3.2842 (3.4384) grad_norm 1.2905 (1.3195) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][450/625] eta 0:01:07 lr 0.001704 wd 0.0500 time 0.3731 (0.3830) data time 0.0011 (0.0020) model time 0.3720 (0.3816) loss 2.5071 (3.4366) grad_norm 1.1757 (1.3208) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][460/625] eta 0:01:03 lr 0.001704 wd 0.0500 time 0.3736 (0.3838) data time 0.0011 (0.0020) model time 0.3725 (0.3825) loss 3.6123 (3.4369) grad_norm 1.1917 (1.3198) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][470/625] eta 0:00:59 lr 0.001704 wd 0.0500 time 0.3819 (0.3836) data time 0.0008 (0.0020) model time 0.3811 (0.3823) loss 4.0197 (3.4408) grad_norm 1.3935 (1.3168) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][480/625] eta 0:00:55 lr 0.001704 wd 0.0500 time 0.3819 (0.3835) data time 0.0008 (0.0020) model time 0.3811 (0.3822) loss 3.6921 (3.4389) grad_norm 1.2155 (1.3175) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][490/625] eta 0:00:51 lr 0.001704 wd 0.0500 time 0.3791 (0.3838) data time 0.0009 (0.0020) model time 0.3782 (0.3826) loss 2.8239 (3.4407) grad_norm 1.9057 (1.3201) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][500/625] eta 0:00:47 lr 0.001704 wd 0.0500 time 0.3740 (0.3837) data time 0.0011 (0.0019) model time 0.3729 (0.3824) loss 2.9884 (3.4383) grad_norm 1.4217 (1.3174) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][510/625] eta 0:00:44 lr 0.001704 wd 0.0500 time 0.3794 (0.3836) data time 0.0010 (0.0019) model time 0.3784 (0.3823) loss 3.2879 (3.4385) grad_norm 2.8768 (1.3220) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][520/625] eta 0:00:40 lr 0.001703 wd 0.0500 time 0.3665 (0.3835) data time 0.0012 (0.0019) model time 0.3653 (0.3822) loss 4.2267 (3.4384) grad_norm 1.0216 (1.3213) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][530/625] eta 0:00:36 lr 0.001703 wd 0.0500 time 0.3787 (0.3834) data time 0.0011 (0.0019) model time 0.3775 (0.3821) loss 3.1624 (3.4396) grad_norm 1.3663 (1.3230) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][540/625] eta 0:00:32 lr 0.001703 wd 0.0500 time 0.3852 (0.3832) data time 0.0010 (0.0019) model time 0.3843 (0.3819) loss 3.6032 (3.4419) grad_norm 1.2660 (1.3227) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][550/625] eta 0:00:28 lr 0.001703 wd 0.0500 time 0.3801 (0.3831) data time 0.0010 (0.0019) model time 0.3790 (0.3818) loss 3.9749 (3.4479) grad_norm 1.0664 (1.3233) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][560/625] eta 0:00:24 lr 0.001703 wd 0.0500 time 0.3775 (0.3830) data time 0.0010 (0.0019) model time 0.3764 (0.3817) loss 3.6600 (3.4477) grad_norm 1.7421 (1.3218) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][570/625] eta 0:00:21 lr 0.001703 wd 0.0500 time 0.3728 (0.3829) data time 0.0009 (0.0018) model time 0.3719 (0.3816) loss 2.4588 (3.4435) grad_norm 1.6862 (1.3250) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:07:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][580/625] eta 0:00:17 lr 0.001703 wd 0.0500 time 0.3769 (0.3828) data time 0.0008 (0.0018) model time 0.3761 (0.3814) loss 3.3682 (3.4448) grad_norm 1.3264 (1.3250) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][590/625] eta 0:00:13 lr 0.001703 wd 0.0500 time 0.3743 (0.3829) data time 0.0009 (0.0018) model time 0.3735 (0.3815) loss 3.9987 (3.4443) grad_norm 1.6183 (1.3248) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][600/625] eta 0:00:09 lr 0.001702 wd 0.0500 time 0.3777 (0.3828) data time 0.0008 (0.0018) model time 0.3769 (0.3815) loss 3.5138 (3.4469) grad_norm 1.3219 (1.3261) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][610/625] eta 0:00:05 lr 0.001702 wd 0.0500 time 0.3668 (0.3830) data time 0.0008 (0.0018) model time 0.3660 (0.3817) loss 3.1298 (3.4500) grad_norm 1.4330 (1.3390) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [90/300][620/625] eta 0:00:01 lr 0.001702 wd 0.0500 time 0.3661 (0.3828) data time 0.0008 (0.0018) model time 0.3653 (0.3814) loss 3.5795 (3.4543) grad_norm 1.0216 (1.3375) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:15 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 90 training takes 0:03:59 [2024-08-26 09:08:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:08:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:08:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 0.7251 (0.7251) Acc@1 84.814 (84.814) Acc@5 97.217 (97.217) Mem 11225MB [2024-08-26 09:08:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 1.1377 (0.8504) Acc@1 73.682 (81.357) Acc@5 92.529 (96.165) Mem 11225MB [2024-08-26 09:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.2715 (1.0107) Acc@1 70.361 (77.472) Acc@5 91.357 (94.222) Mem 11225MB [2024-08-26 09:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.351 Acc@5 94.174 [2024-08-26 09:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.4% [2024-08-26 09:08:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.831 (0.831) Loss 0.5576 (0.5576) Acc@1 88.184 (88.184) Acc@5 98.291 (98.291) Mem 11225MB [2024-08-26 09:08:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.178) Loss 0.9263 (0.6946) Acc@1 77.881 (84.193) Acc@5 95.117 (97.088) Mem 11225MB [2024-08-26 09:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.137) Loss 1.0938 (0.8453) Acc@1 72.998 (80.355) Acc@5 92.871 (95.401) Mem 11225MB [2024-08-26 09:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.102 Acc@5 95.367 [2024-08-26 09:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.1% [2024-08-26 09:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.10% [2024-08-26 09:08:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:08:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:08:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][0/625] eta 0:07:46 lr 0.001702 wd 0.0500 time 0.7470 (0.7470) data time 0.3920 (0.3920) model time 0.0000 (0.0000) loss 4.2806 (4.2806) grad_norm 1.0386 (1.0386) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][10/625] eta 0:04:12 lr 0.001702 wd 0.0500 time 0.3972 (0.4106) data time 0.0010 (0.0367) model time 0.0000 (0.0000) loss 2.7871 (3.6639) grad_norm 1.2929 (1.2915) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][20/625] eta 0:03:59 lr 0.001702 wd 0.0500 time 0.3839 (0.3950) data time 0.0011 (0.0197) model time 0.0000 (0.0000) loss 3.2022 (3.5185) grad_norm 1.0947 (1.3529) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][30/625] eta 0:03:59 lr 0.001702 wd 0.0500 time 0.3880 (0.4023) data time 0.0011 (0.0137) model time 0.0000 (0.0000) loss 3.6347 (3.3876) grad_norm 1.6642 (1.3287) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][40/625] eta 0:03:51 lr 0.001702 wd 0.0500 time 0.3765 (0.3962) data time 0.0009 (0.0106) model time 0.0000 (0.0000) loss 3.9874 (3.4391) grad_norm 1.1029 (1.3360) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][50/625] eta 0:03:45 lr 0.001702 wd 0.0500 time 0.3799 (0.3929) data time 0.0008 (0.0088) model time 0.0000 (0.0000) loss 4.0913 (3.4530) grad_norm 2.1272 (1.3584) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][60/625] eta 0:03:40 lr 0.001701 wd 0.0500 time 0.3768 (0.3906) data time 0.0009 (0.0076) model time 0.3760 (0.3771) loss 3.6252 (3.4492) grad_norm 1.1618 (1.3972) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][70/625] eta 0:03:35 lr 0.001701 wd 0.0500 time 0.3756 (0.3890) data time 0.0008 (0.0067) model time 0.3748 (0.3777) loss 2.6831 (3.4116) grad_norm 1.3505 (1.3804) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][80/625] eta 0:03:31 lr 0.001701 wd 0.0500 time 0.3748 (0.3873) data time 0.0009 (0.0060) model time 0.3740 (0.3765) loss 3.6685 (3.4092) grad_norm 1.5538 (1.3731) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][90/625] eta 0:03:26 lr 0.001701 wd 0.0500 time 0.3722 (0.3861) data time 0.0011 (0.0055) model time 0.3711 (0.3763) loss 3.4229 (3.3825) grad_norm 1.5007 (1.3676) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][100/625] eta 0:03:23 lr 0.001701 wd 0.0500 time 0.3730 (0.3867) data time 0.0012 (0.0051) model time 0.3718 (0.3791) loss 3.8050 (3.4159) grad_norm 1.0476 (1.3555) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][110/625] eta 0:03:18 lr 0.001701 wd 0.0500 time 0.3795 (0.3858) data time 0.0008 (0.0047) model time 0.3787 (0.3785) loss 3.5278 (3.4263) grad_norm 1.5365 (1.3505) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][120/625] eta 0:03:14 lr 0.001701 wd 0.0500 time 0.3810 (0.3850) data time 0.0008 (0.0044) model time 0.3802 (0.3780) loss 3.0450 (3.4150) grad_norm 1.1573 (1.3546) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][130/625] eta 0:03:10 lr 0.001700 wd 0.0500 time 0.3757 (0.3844) data time 0.0008 (0.0042) model time 0.3749 (0.3778) loss 3.2731 (3.4232) grad_norm 1.0155 (1.3375) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][140/625] eta 0:03:06 lr 0.001700 wd 0.0500 time 0.3756 (0.3841) data time 0.0011 (0.0040) model time 0.3745 (0.3780) loss 3.3097 (3.4203) grad_norm 0.9291 (1.3232) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][150/625] eta 0:03:02 lr 0.001700 wd 0.0500 time 0.3785 (0.3839) data time 0.0008 (0.0038) model time 0.3776 (0.3780) loss 3.2938 (3.4211) grad_norm 1.4491 (1.3206) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][160/625] eta 0:02:58 lr 0.001700 wd 0.0500 time 0.3787 (0.3835) data time 0.0008 (0.0036) model time 0.3779 (0.3780) loss 4.0617 (3.4258) grad_norm 1.1505 (1.3204) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][170/625] eta 0:02:54 lr 0.001700 wd 0.0500 time 0.5691 (0.3842) data time 0.0009 (0.0035) model time 0.5682 (0.3794) loss 3.5712 (3.4316) grad_norm 1.1707 (1.3115) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][180/625] eta 0:02:50 lr 0.001700 wd 0.0500 time 0.3856 (0.3838) data time 0.0010 (0.0033) model time 0.3846 (0.3791) loss 2.6790 (3.4346) grad_norm 1.3603 (1.3063) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][190/625] eta 0:02:46 lr 0.001700 wd 0.0500 time 0.3817 (0.3835) data time 0.0008 (0.0032) model time 0.3810 (0.3789) loss 3.0689 (3.4242) grad_norm 1.7612 (1.3084) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][200/625] eta 0:02:42 lr 0.001700 wd 0.0500 time 0.3785 (0.3832) data time 0.0010 (0.0031) model time 0.3775 (0.3787) loss 3.9273 (3.4210) grad_norm 1.8592 (1.3066) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][210/625] eta 0:02:38 lr 0.001699 wd 0.0500 time 0.3892 (0.3830) data time 0.0008 (0.0030) model time 0.3884 (0.3786) loss 4.2400 (3.4251) grad_norm 1.1780 (1.3141) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][220/625] eta 0:02:35 lr 0.001699 wd 0.0500 time 0.3718 (0.3828) data time 0.0011 (0.0029) model time 0.3707 (0.3786) loss 3.4452 (3.4272) grad_norm 1.1976 (1.3182) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][230/625] eta 0:02:31 lr 0.001699 wd 0.0500 time 0.3764 (0.3825) data time 0.0008 (0.0029) model time 0.3756 (0.3784) loss 4.0671 (3.4289) grad_norm 1.1958 (1.3137) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][240/625] eta 0:02:27 lr 0.001699 wd 0.0500 time 0.3712 (0.3840) data time 0.0008 (0.0028) model time 0.3704 (0.3805) loss 4.2413 (3.4379) grad_norm 1.1572 (1.3138) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][250/625] eta 0:02:23 lr 0.001699 wd 0.0500 time 0.3770 (0.3838) data time 0.0011 (0.0027) model time 0.3759 (0.3803) loss 2.9060 (3.4282) grad_norm 1.3078 (1.3188) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][260/625] eta 0:02:19 lr 0.001699 wd 0.0500 time 0.3799 (0.3835) data time 0.0010 (0.0027) model time 0.3789 (0.3800) loss 3.7019 (3.4335) grad_norm 1.3169 (1.3143) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][270/625] eta 0:02:16 lr 0.001699 wd 0.0500 time 0.3732 (0.3833) data time 0.0009 (0.0026) model time 0.3723 (0.3799) loss 3.9589 (3.4302) grad_norm 1.6814 (1.3188) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][280/625] eta 0:02:12 lr 0.001699 wd 0.0500 time 0.3749 (0.3831) data time 0.0010 (0.0025) model time 0.3739 (0.3798) loss 3.9041 (3.4349) grad_norm 1.8171 (1.3249) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][290/625] eta 0:02:08 lr 0.001698 wd 0.0500 time 0.5803 (0.3837) data time 0.0011 (0.0025) model time 0.5791 (0.3806) loss 3.5005 (3.4375) grad_norm 1.3892 (1.3247) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][300/625] eta 0:02:04 lr 0.001698 wd 0.0500 time 0.3754 (0.3836) data time 0.0010 (0.0025) model time 0.3744 (0.3805) loss 3.4366 (3.4389) grad_norm 1.2168 (1.3176) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][310/625] eta 0:02:00 lr 0.001698 wd 0.0500 time 0.3739 (0.3840) data time 0.0011 (0.0024) model time 0.3729 (0.3811) loss 3.9435 (3.4392) grad_norm 1.3518 (1.3121) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][320/625] eta 0:01:57 lr 0.001698 wd 0.0500 time 0.3805 (0.3838) data time 0.0008 (0.0024) model time 0.3796 (0.3810) loss 4.0547 (3.4468) grad_norm 1.6718 (1.3084) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][330/625] eta 0:01:53 lr 0.001698 wd 0.0500 time 0.3751 (0.3837) data time 0.0010 (0.0023) model time 0.3740 (0.3809) loss 3.7419 (3.4520) grad_norm 1.5426 (1.3139) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][340/625] eta 0:01:49 lr 0.001698 wd 0.0500 time 0.3748 (0.3835) data time 0.0009 (0.0023) model time 0.3739 (0.3807) loss 3.8106 (3.4511) grad_norm 1.0599 (1.3202) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:10:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][350/625] eta 0:01:45 lr 0.001698 wd 0.0500 time 0.3741 (0.3833) data time 0.0009 (0.0023) model time 0.3733 (0.3805) loss 2.8301 (3.4500) grad_norm 1.1431 (1.3196) loss_scale 4096.0000 (2071.3390) mem 11225MB [2024-08-26 09:10:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][360/625] eta 0:01:41 lr 0.001698 wd 0.0500 time 0.3758 (0.3832) data time 0.0011 (0.0022) model time 0.3747 (0.3804) loss 3.6413 (3.4578) grad_norm 1.4793 (1.3261) loss_scale 4096.0000 (2127.4238) mem 11225MB [2024-08-26 09:10:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][370/625] eta 0:01:37 lr 0.001697 wd 0.0500 time 0.3837 (0.3831) data time 0.0013 (0.0022) model time 0.3824 (0.3803) loss 3.8540 (3.4673) grad_norm 1.3923 (1.3285) loss_scale 4096.0000 (2180.4852) mem 11225MB [2024-08-26 09:10:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][380/625] eta 0:01:33 lr 0.001697 wd 0.0500 time 0.3730 (0.3829) data time 0.0009 (0.0022) model time 0.3721 (0.3802) loss 3.1936 (3.4608) grad_norm 1.8026 (1.3269) loss_scale 4096.0000 (2230.7612) mem 11225MB [2024-08-26 09:10:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][390/625] eta 0:01:29 lr 0.001697 wd 0.0500 time 0.3738 (0.3828) data time 0.0011 (0.0022) model time 0.3727 (0.3802) loss 3.3941 (3.4557) grad_norm 1.2193 (1.3374) loss_scale 4096.0000 (2278.4655) mem 11225MB [2024-08-26 09:10:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][400/625] eta 0:01:26 lr 0.001697 wd 0.0500 time 0.3786 (0.3827) data time 0.0010 (0.0021) model time 0.3776 (0.3801) loss 2.9161 (3.4578) grad_norm 1.2006 (1.3354) loss_scale 4096.0000 (2323.7905) mem 11225MB [2024-08-26 09:11:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][410/625] eta 0:01:22 lr 0.001697 wd 0.0500 time 0.3785 (0.3826) data time 0.0011 (0.0021) model time 0.3774 (0.3800) loss 3.7891 (3.4531) grad_norm 1.0308 (1.3297) loss_scale 4096.0000 (2366.9100) mem 11225MB [2024-08-26 09:11:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][420/625] eta 0:01:18 lr 0.001697 wd 0.0500 time 0.3760 (0.3835) data time 0.0008 (0.0021) model time 0.3751 (0.3811) loss 2.1815 (3.4504) grad_norm 1.6248 (1.3314) loss_scale 4096.0000 (2407.9810) mem 11225MB [2024-08-26 09:11:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][430/625] eta 0:01:14 lr 0.001697 wd 0.0500 time 0.3741 (0.3834) data time 0.0011 (0.0021) model time 0.3730 (0.3810) loss 3.9513 (3.4511) grad_norm 1.3032 (1.3283) loss_scale 4096.0000 (2447.1462) mem 11225MB [2024-08-26 09:11:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][440/625] eta 0:01:10 lr 0.001697 wd 0.0500 time 0.3798 (0.3833) data time 0.0011 (0.0020) model time 0.3788 (0.3809) loss 2.8814 (3.4546) grad_norm 1.0787 (1.3275) loss_scale 4096.0000 (2484.5351) mem 11225MB [2024-08-26 09:11:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][450/625] eta 0:01:07 lr 0.001696 wd 0.0500 time 0.3786 (0.3831) data time 0.0011 (0.0020) model time 0.3775 (0.3807) loss 3.8918 (3.4583) grad_norm 1.7335 (1.3328) loss_scale 4096.0000 (2520.2661) mem 11225MB [2024-08-26 09:11:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][460/625] eta 0:01:03 lr 0.001696 wd 0.0500 time 0.3812 (0.3830) data time 0.0011 (0.0020) model time 0.3802 (0.3806) loss 2.8746 (3.4518) grad_norm 1.4219 (1.3329) loss_scale 4096.0000 (2554.4469) mem 11225MB [2024-08-26 09:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][470/625] eta 0:00:59 lr 0.001696 wd 0.0500 time 0.3843 (0.3829) data time 0.0012 (0.0020) model time 0.3832 (0.3806) loss 3.9031 (3.4531) grad_norm 1.2504 (1.3296) loss_scale 4096.0000 (2587.1762) mem 11225MB [2024-08-26 09:11:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][480/625] eta 0:00:55 lr 0.001696 wd 0.0500 time 0.3801 (0.3828) data time 0.0010 (0.0020) model time 0.3791 (0.3805) loss 2.3236 (3.4506) grad_norm 1.0869 (1.3308) loss_scale 4096.0000 (2618.5447) mem 11225MB [2024-08-26 09:11:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][490/625] eta 0:00:51 lr 0.001696 wd 0.0500 time 0.3792 (0.3831) data time 0.0011 (0.0020) model time 0.3780 (0.3808) loss 2.6470 (3.4514) grad_norm 1.6089 (1.3308) loss_scale 4096.0000 (2648.6354) mem 11225MB [2024-08-26 09:11:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][500/625] eta 0:00:47 lr 0.001696 wd 0.0500 time 0.3809 (0.3835) data time 0.0008 (0.0019) model time 0.3801 (0.3813) loss 2.5367 (3.4481) grad_norm 2.5497 (1.3372) loss_scale 4096.0000 (2677.5250) mem 11225MB [2024-08-26 09:11:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][510/625] eta 0:00:44 lr 0.001696 wd 0.0500 time 0.3772 (0.3834) data time 0.0008 (0.0019) model time 0.3763 (0.3811) loss 4.4356 (3.4533) grad_norm 1.1970 (1.3366) loss_scale 4096.0000 (2705.2838) mem 11225MB [2024-08-26 09:11:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][520/625] eta 0:00:40 lr 0.001695 wd 0.0500 time 0.3725 (0.3832) data time 0.0011 (0.0019) model time 0.3714 (0.3810) loss 3.9918 (3.4506) grad_norm 0.9126 (1.3345) loss_scale 4096.0000 (2731.9770) mem 11225MB [2024-08-26 09:11:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][530/625] eta 0:00:36 lr 0.001695 wd 0.0500 time 0.3820 (0.3831) data time 0.0009 (0.0019) model time 0.3811 (0.3809) loss 3.6787 (3.4498) grad_norm 2.1501 (1.3334) loss_scale 4096.0000 (2757.6648) mem 11225MB [2024-08-26 09:11:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][540/625] eta 0:00:32 lr 0.001695 wd 0.0500 time 0.3741 (0.3837) data time 0.0010 (0.0019) model time 0.3730 (0.3816) loss 2.8102 (3.4455) grad_norm 1.1932 (1.3391) loss_scale 4096.0000 (2782.4030) mem 11225MB [2024-08-26 09:11:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][550/625] eta 0:00:28 lr 0.001695 wd 0.0500 time 0.3817 (0.3836) data time 0.0008 (0.0019) model time 0.3809 (0.3815) loss 4.2061 (3.4445) grad_norm 0.8495 (1.3374) loss_scale 4096.0000 (2806.2432) mem 11225MB [2024-08-26 09:11:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][560/625] eta 0:00:24 lr 0.001695 wd 0.0500 time 0.3755 (0.3835) data time 0.0010 (0.0019) model time 0.3744 (0.3814) loss 3.1797 (3.4434) grad_norm 0.8934 (1.3373) loss_scale 4096.0000 (2829.2335) mem 11225MB [2024-08-26 09:12:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][570/625] eta 0:00:21 lr 0.001695 wd 0.0500 time 0.3780 (0.3834) data time 0.0011 (0.0018) model time 0.3769 (0.3813) loss 2.9507 (3.4409) grad_norm 1.2097 (1.3344) loss_scale 4096.0000 (2851.4186) mem 11225MB [2024-08-26 09:12:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][580/625] eta 0:00:17 lr 0.001695 wd 0.0500 time 0.3752 (0.3833) data time 0.0014 (0.0018) model time 0.3738 (0.3812) loss 3.7215 (3.4455) grad_norm 2.1770 (1.3354) loss_scale 4096.0000 (2872.8399) mem 11225MB [2024-08-26 09:12:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][590/625] eta 0:00:13 lr 0.001695 wd 0.0500 time 0.3759 (0.3831) data time 0.0009 (0.0018) model time 0.3750 (0.3811) loss 3.3290 (3.4481) grad_norm 1.2062 (1.3380) loss_scale 4096.0000 (2893.5364) mem 11225MB [2024-08-26 09:12:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][600/625] eta 0:00:09 lr 0.001694 wd 0.0500 time 0.3738 (0.3830) data time 0.0011 (0.0018) model time 0.3728 (0.3809) loss 3.6144 (3.4471) grad_norm 1.2114 (1.3345) loss_scale 4096.0000 (2913.5441) mem 11225MB [2024-08-26 09:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][610/625] eta 0:00:05 lr 0.001694 wd 0.0500 time 0.3651 (0.3832) data time 0.0006 (0.0018) model time 0.3645 (0.3811) loss 3.0318 (3.4496) grad_norm 0.9810 (1.3332) loss_scale 4096.0000 (2932.8969) mem 11225MB [2024-08-26 09:12:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [91/300][620/625] eta 0:00:01 lr 0.001694 wd 0.0500 time 0.3596 (0.3829) data time 0.0005 (0.0018) model time 0.3590 (0.3808) loss 3.3728 (3.4517) grad_norm 1.3484 (1.3323) loss_scale 4096.0000 (2951.6264) mem 11225MB [2024-08-26 09:12:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 91 training takes 0:03:59 [2024-08-26 09:12:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:12:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.498 (0.498) Loss 0.6587 (0.6587) Acc@1 85.693 (85.693) Acc@5 97.705 (97.705) Mem 11225MB [2024-08-26 09:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 1.0928 (0.8173) Acc@1 74.316 (81.410) Acc@5 93.848 (96.254) Mem 11225MB [2024-08-26 09:12:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.1953 (0.9800) Acc@1 71.729 (77.648) Acc@5 91.895 (94.199) Mem 11225MB [2024-08-26 09:12:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.389 Acc@5 94.178 [2024-08-26 09:12:27 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.4% [2024-08-26 09:12:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.835 (0.835) Loss 0.5571 (0.5571) Acc@1 88.135 (88.135) Acc@5 98.193 (98.193) Mem 11225MB [2024-08-26 09:12:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.176) Loss 0.9253 (0.6939) Acc@1 77.637 (84.184) Acc@5 95.020 (97.084) Mem 11225MB [2024-08-26 09:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.0918 (0.8441) Acc@1 73.145 (80.411) Acc@5 93.018 (95.412) Mem 11225MB [2024-08-26 09:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.148 Acc@5 95.389 [2024-08-26 09:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.1% [2024-08-26 09:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.15% [2024-08-26 09:12:30 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:12:31 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:12:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][0/625] eta 0:08:17 lr 0.001694 wd 0.0500 time 0.7963 (0.7963) data time 0.4580 (0.4580) model time 0.0000 (0.0000) loss 3.4837 (3.4837) grad_norm 1.2025 (1.2025) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:12:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][10/625] eta 0:04:15 lr 0.001694 wd 0.0500 time 0.3763 (0.4153) data time 0.0008 (0.0427) model time 0.0000 (0.0000) loss 2.9408 (3.3916) grad_norm 1.0654 (1.1708) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:12:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][20/625] eta 0:03:59 lr 0.001694 wd 0.0500 time 0.3617 (0.3966) data time 0.0009 (0.0229) model time 0.0000 (0.0000) loss 3.4105 (3.3450) grad_norm 1.5599 (1.3004) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:12:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][30/625] eta 0:03:52 lr 0.001694 wd 0.0500 time 0.3760 (0.3912) data time 0.0008 (0.0158) model time 0.0000 (0.0000) loss 3.0171 (3.3483) grad_norm 2.1687 (1.3211) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:12:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][40/625] eta 0:03:47 lr 0.001694 wd 0.0500 time 0.3746 (0.3883) data time 0.0009 (0.0123) model time 0.0000 (0.0000) loss 2.9663 (3.2862) grad_norm 1.5577 (1.3351) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:12:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][50/625] eta 0:03:42 lr 0.001694 wd 0.0500 time 0.3750 (0.3862) data time 0.0008 (0.0101) model time 0.0000 (0.0000) loss 2.7781 (3.2820) grad_norm 1.1062 (1.3350) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:12:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][60/625] eta 0:03:37 lr 0.001693 wd 0.0500 time 0.3776 (0.3853) data time 0.0010 (0.0086) model time 0.3765 (0.3799) loss 3.0872 (3.3089) grad_norm 1.2450 (1.3450) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:12:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][70/625] eta 0:03:37 lr 0.001693 wd 0.0500 time 0.3961 (0.3914) data time 0.0008 (0.0075) model time 0.3953 (0.4036) loss 3.2305 (3.3512) grad_norm 1.1814 (1.3529) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][80/625] eta 0:03:32 lr 0.001693 wd 0.0500 time 0.3689 (0.3900) data time 0.0008 (0.0068) model time 0.3681 (0.3952) loss 2.8516 (3.3731) grad_norm 1.4233 (1.3852) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][90/625] eta 0:03:28 lr 0.001693 wd 0.0500 time 0.3772 (0.3892) data time 0.0011 (0.0063) model time 0.3761 (0.3917) loss 2.6433 (3.3658) grad_norm 1.5486 (1.4141) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][100/625] eta 0:03:24 lr 0.001693 wd 0.0500 time 0.3813 (0.3889) data time 0.0008 (0.0058) model time 0.3805 (0.3902) loss 3.7603 (3.3820) grad_norm 1.0486 (1.3949) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][110/625] eta 0:03:19 lr 0.001693 wd 0.0500 time 0.3631 (0.3883) data time 0.0010 (0.0055) model time 0.3620 (0.3886) loss 3.4177 (3.3939) grad_norm 1.2725 (1.3838) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][120/625] eta 0:03:16 lr 0.001693 wd 0.0500 time 0.5691 (0.3894) data time 0.0008 (0.0052) model time 0.5682 (0.3901) loss 2.4631 (3.3785) grad_norm 1.1984 (1.3752) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][130/625] eta 0:03:12 lr 0.001692 wd 0.0500 time 0.3975 (0.3890) data time 0.0011 (0.0050) model time 0.3964 (0.3891) loss 3.9259 (3.3866) grad_norm 1.2322 (1.3613) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][140/625] eta 0:03:08 lr 0.001692 wd 0.0500 time 0.3584 (0.3882) data time 0.0008 (0.0047) model time 0.3576 (0.3877) loss 2.0708 (3.3846) grad_norm 1.3997 (1.3634) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][150/625] eta 0:03:04 lr 0.001692 wd 0.0500 time 0.3791 (0.3878) data time 0.0008 (0.0045) model time 0.3783 (0.3869) loss 3.6437 (3.3764) grad_norm 1.8588 (1.3770) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][160/625] eta 0:03:00 lr 0.001692 wd 0.0500 time 0.3835 (0.3873) data time 0.0009 (0.0043) model time 0.3826 (0.3861) loss 4.0281 (3.4000) grad_norm 1.1771 (1.3831) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][170/625] eta 0:02:55 lr 0.001692 wd 0.0500 time 0.3669 (0.3867) data time 0.0008 (0.0042) model time 0.3661 (0.3853) loss 3.5426 (3.4041) grad_norm 1.4272 (1.3815) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][180/625] eta 0:02:51 lr 0.001692 wd 0.0500 time 0.3828 (0.3863) data time 0.0009 (0.0040) model time 0.3819 (0.3847) loss 3.9061 (3.4175) grad_norm 1.1276 (1.3983) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][190/625] eta 0:02:47 lr 0.001692 wd 0.0500 time 0.4025 (0.3860) data time 0.0009 (0.0039) model time 0.4016 (0.3844) loss 3.6915 (3.4219) grad_norm 1.3382 (1.4021) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][200/625] eta 0:02:43 lr 0.001692 wd 0.0500 time 0.3713 (0.3857) data time 0.0011 (0.0038) model time 0.3702 (0.3839) loss 2.3312 (3.4292) grad_norm 1.0383 (1.3844) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][210/625] eta 0:02:39 lr 0.001691 wd 0.0500 time 0.3759 (0.3854) data time 0.0011 (0.0037) model time 0.3748 (0.3835) loss 3.9519 (3.4281) grad_norm 1.3220 (1.3837) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][220/625] eta 0:02:36 lr 0.001691 wd 0.0500 time 0.3919 (0.3853) data time 0.0011 (0.0036) model time 0.3908 (0.3833) loss 3.3036 (3.4247) grad_norm 1.7648 (1.4062) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][230/625] eta 0:02:32 lr 0.001691 wd 0.0500 time 0.3684 (0.3850) data time 0.0010 (0.0035) model time 0.3674 (0.3831) loss 3.3772 (3.4307) grad_norm 0.9636 (1.4123) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][240/625] eta 0:02:28 lr 0.001691 wd 0.0500 time 0.3812 (0.3856) data time 0.0011 (0.0034) model time 0.3802 (0.3838) loss 3.7558 (3.4305) grad_norm 1.0057 (1.3991) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][250/625] eta 0:02:25 lr 0.001691 wd 0.0500 time 0.5821 (0.3869) data time 0.0011 (0.0033) model time 0.5811 (0.3855) loss 3.0429 (3.4198) grad_norm 1.0678 (1.3839) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][260/625] eta 0:02:21 lr 0.001691 wd 0.0500 time 0.3793 (0.3866) data time 0.0008 (0.0032) model time 0.3785 (0.3851) loss 3.0613 (3.4112) grad_norm 1.2540 (1.3779) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][270/625] eta 0:02:17 lr 0.001691 wd 0.0500 time 0.3828 (0.3863) data time 0.0008 (0.0032) model time 0.3820 (0.3848) loss 4.1189 (3.4061) grad_norm 1.0836 (1.3718) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][280/625] eta 0:02:13 lr 0.001691 wd 0.0500 time 0.3841 (0.3879) data time 0.0009 (0.0031) model time 0.3833 (0.3867) loss 4.1419 (3.4141) grad_norm 0.9995 (1.3699) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][290/625] eta 0:02:09 lr 0.001690 wd 0.0500 time 0.3700 (0.3875) data time 0.0009 (0.0030) model time 0.3691 (0.3863) loss 2.2863 (3.4194) grad_norm 1.3298 (1.3653) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][300/625] eta 0:02:05 lr 0.001690 wd 0.0500 time 0.3802 (0.3872) data time 0.0008 (0.0030) model time 0.3794 (0.3859) loss 3.0277 (3.4135) grad_norm 1.4820 (1.3587) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][310/625] eta 0:02:01 lr 0.001690 wd 0.0500 time 0.3768 (0.3869) data time 0.0017 (0.0029) model time 0.3751 (0.3855) loss 2.5800 (3.4148) grad_norm 2.0383 (1.3607) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][320/625] eta 0:01:57 lr 0.001690 wd 0.0500 time 0.3759 (0.3866) data time 0.0009 (0.0029) model time 0.3750 (0.3852) loss 3.4528 (3.4245) grad_norm 1.2858 (1.3670) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][330/625] eta 0:01:53 lr 0.001690 wd 0.0500 time 0.3832 (0.3864) data time 0.0011 (0.0028) model time 0.3821 (0.3849) loss 3.7504 (3.4232) grad_norm 2.4445 (1.3686) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][340/625] eta 0:01:50 lr 0.001690 wd 0.0500 time 0.3892 (0.3862) data time 0.0011 (0.0028) model time 0.3881 (0.3847) loss 3.6902 (3.4299) grad_norm 1.2596 (1.3666) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][350/625] eta 0:01:46 lr 0.001690 wd 0.0500 time 0.3726 (0.3859) data time 0.0009 (0.0027) model time 0.3717 (0.3844) loss 3.9513 (3.4354) grad_norm 1.2968 (1.3603) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][360/625] eta 0:01:42 lr 0.001690 wd 0.0500 time 0.3757 (0.3858) data time 0.0012 (0.0027) model time 0.3745 (0.3842) loss 2.6832 (3.4311) grad_norm 1.5192 (1.3597) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][370/625] eta 0:01:38 lr 0.001689 wd 0.0500 time 0.3752 (0.3867) data time 0.0008 (0.0026) model time 0.3744 (0.3853) loss 3.2205 (3.4348) grad_norm 1.4001 (1.3544) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:14:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][380/625] eta 0:01:34 lr 0.001689 wd 0.0500 time 0.3842 (0.3865) data time 0.0008 (0.0026) model time 0.3833 (0.3852) loss 3.8074 (3.4372) grad_norm 1.3060 (1.3481) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][390/625] eta 0:01:30 lr 0.001689 wd 0.0500 time 0.3756 (0.3863) data time 0.0012 (0.0026) model time 0.3744 (0.3849) loss 2.7930 (3.4427) grad_norm 1.1392 (1.3458) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][400/625] eta 0:01:26 lr 0.001689 wd 0.0500 time 0.3766 (0.3861) data time 0.0010 (0.0025) model time 0.3756 (0.3847) loss 3.7255 (3.4433) grad_norm 1.2298 (1.3479) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][410/625] eta 0:01:22 lr 0.001689 wd 0.0500 time 0.3760 (0.3859) data time 0.0012 (0.0025) model time 0.3749 (0.3845) loss 3.1758 (3.4415) grad_norm 0.9702 (1.3442) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][420/625] eta 0:01:19 lr 0.001689 wd 0.0500 time 0.3852 (0.3858) data time 0.0008 (0.0024) model time 0.3844 (0.3843) loss 3.7316 (3.4404) grad_norm 1.1333 (1.3384) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][430/625] eta 0:01:15 lr 0.001689 wd 0.0500 time 0.3735 (0.3856) data time 0.0008 (0.0024) model time 0.3727 (0.3841) loss 3.6385 (3.4440) grad_norm 1.4727 (1.3372) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][440/625] eta 0:01:11 lr 0.001688 wd 0.0500 time 0.3751 (0.3854) data time 0.0013 (0.0024) model time 0.3739 (0.3839) loss 3.2826 (3.4477) grad_norm 1.3290 (1.3369) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][450/625] eta 0:01:07 lr 0.001688 wd 0.0500 time 0.3345 (0.3856) data time 0.0010 (0.0024) model time 0.3335 (0.3842) loss 3.8407 (3.4471) grad_norm 1.3561 (1.3363) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][460/625] eta 0:01:03 lr 0.001688 wd 0.0500 time 0.5457 (0.3858) data time 0.0009 (0.0023) model time 0.5448 (0.3844) loss 4.4124 (3.4495) grad_norm 1.2550 (1.3354) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][470/625] eta 0:00:59 lr 0.001688 wd 0.0500 time 0.3756 (0.3861) data time 0.0013 (0.0023) model time 0.3744 (0.3847) loss 2.7487 (3.4470) grad_norm 1.0600 (1.3328) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][480/625] eta 0:00:55 lr 0.001688 wd 0.0500 time 0.3745 (0.3859) data time 0.0008 (0.0023) model time 0.3737 (0.3845) loss 3.5893 (3.4421) grad_norm 1.3064 (1.3300) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][490/625] eta 0:00:52 lr 0.001688 wd 0.0500 time 0.3776 (0.3857) data time 0.0013 (0.0023) model time 0.3764 (0.3843) loss 3.0287 (3.4388) grad_norm 1.0573 (1.3268) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][500/625] eta 0:00:48 lr 0.001688 wd 0.0500 time 0.3818 (0.3856) data time 0.0008 (0.0022) model time 0.3810 (0.3842) loss 4.2020 (3.4352) grad_norm 1.5369 (1.3238) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][510/625] eta 0:00:44 lr 0.001688 wd 0.0500 time 0.3741 (0.3855) data time 0.0010 (0.0022) model time 0.3731 (0.3840) loss 3.3055 (3.4383) grad_norm 1.2370 (1.3231) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][520/625] eta 0:00:40 lr 0.001687 wd 0.0500 time 0.3819 (0.3853) data time 0.0011 (0.0022) model time 0.3809 (0.3839) loss 3.5977 (3.4372) grad_norm 1.3893 (1.3258) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][530/625] eta 0:00:36 lr 0.001687 wd 0.0500 time 0.3827 (0.3852) data time 0.0011 (0.0022) model time 0.3816 (0.3837) loss 2.8621 (3.4314) grad_norm 1.2926 (1.3241) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][540/625] eta 0:00:32 lr 0.001687 wd 0.0500 time 0.3786 (0.3851) data time 0.0008 (0.0022) model time 0.3778 (0.3836) loss 3.5321 (3.4344) grad_norm 1.0491 (1.3276) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][550/625] eta 0:00:28 lr 0.001687 wd 0.0500 time 0.3804 (0.3850) data time 0.0011 (0.0021) model time 0.3793 (0.3835) loss 3.7285 (3.4340) grad_norm 1.3965 (1.3268) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][560/625] eta 0:00:25 lr 0.001687 wd 0.0500 time 0.3717 (0.3849) data time 0.0011 (0.0021) model time 0.3705 (0.3834) loss 3.5028 (3.4333) grad_norm 1.2503 (1.3290) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][570/625] eta 0:00:21 lr 0.001687 wd 0.0500 time 0.3808 (0.3848) data time 0.0008 (0.0021) model time 0.3800 (0.3833) loss 4.2050 (3.4388) grad_norm 1.2568 (1.3297) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][580/625] eta 0:00:17 lr 0.001687 wd 0.0500 time 0.5864 (0.3854) data time 0.0008 (0.0021) model time 0.5856 (0.3840) loss 2.0907 (3.4375) grad_norm 1.0772 (1.3286) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][590/625] eta 0:00:13 lr 0.001687 wd 0.0500 time 0.3730 (0.3856) data time 0.0011 (0.0021) model time 0.3719 (0.3843) loss 3.2143 (3.4347) grad_norm 1.9823 (1.3319) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][600/625] eta 0:00:09 lr 0.001686 wd 0.0500 time 0.3829 (0.3855) data time 0.0008 (0.0021) model time 0.3821 (0.3841) loss 2.1981 (3.4334) grad_norm 1.6909 (1.3429) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][610/625] eta 0:00:05 lr 0.001686 wd 0.0500 time 0.3684 (0.3854) data time 0.0008 (0.0021) model time 0.3677 (0.3840) loss 3.9627 (3.4298) grad_norm 1.0970 (1.3466) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [92/300][620/625] eta 0:00:01 lr 0.001686 wd 0.0500 time 0.3668 (0.3851) data time 0.0008 (0.0020) model time 0.3660 (0.3837) loss 3.5129 (3.4286) grad_norm 1.1377 (1.3439) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:32 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 92 training takes 0:04:00 [2024-08-26 09:16:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:16:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.533 (0.533) Loss 0.6997 (0.6997) Acc@1 85.400 (85.400) Acc@5 97.168 (97.168) Mem 11225MB [2024-08-26 09:16:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.146) Loss 1.1104 (0.8759) Acc@1 73.926 (80.558) Acc@5 93.018 (95.827) Mem 11225MB [2024-08-26 09:16:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.2686 (1.0402) Acc@1 70.996 (76.818) Acc@5 90.967 (93.771) Mem 11225MB [2024-08-26 09:16:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 76.637 Acc@5 93.766 [2024-08-26 09:16:36 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 76.6% [2024-08-26 09:16:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.864 (0.864) Loss 0.5557 (0.5557) Acc@1 88.184 (88.184) Acc@5 98.242 (98.242) Mem 11225MB [2024-08-26 09:16:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.181) Loss 0.9253 (0.6932) Acc@1 77.539 (84.260) Acc@5 95.020 (97.097) Mem 11225MB [2024-08-26 09:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.139) Loss 1.0918 (0.8431) Acc@1 73.145 (80.480) Acc@5 93.018 (95.443) Mem 11225MB [2024-08-26 09:16:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.218 Acc@5 95.423 [2024-08-26 09:16:40 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.2% [2024-08-26 09:16:40 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.22% [2024-08-26 09:16:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:16:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:16:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][0/625] eta 0:08:16 lr 0.001686 wd 0.0500 time 0.7937 (0.7937) data time 0.4561 (0.4561) model time 0.0000 (0.0000) loss 3.4557 (3.4557) grad_norm 1.3930 (1.3930) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][10/625] eta 0:04:16 lr 0.001686 wd 0.0500 time 0.3857 (0.4169) data time 0.0011 (0.0426) model time 0.0000 (0.0000) loss 3.4708 (3.7894) grad_norm 1.2886 (1.2566) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][20/625] eta 0:04:03 lr 0.001686 wd 0.0500 time 0.3591 (0.4023) data time 0.0008 (0.0228) model time 0.0000 (0.0000) loss 2.8766 (3.5435) grad_norm 1.5342 (1.2822) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][30/625] eta 0:03:56 lr 0.001686 wd 0.0500 time 0.3814 (0.3971) data time 0.0008 (0.0165) model time 0.0000 (0.0000) loss 3.6090 (3.5110) grad_norm 1.4288 (1.3085) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:16:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][40/625] eta 0:03:51 lr 0.001686 wd 0.0500 time 0.3999 (0.3952) data time 0.0011 (0.0130) model time 0.0000 (0.0000) loss 3.4511 (3.4344) grad_norm 1.1779 (1.3138) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][50/625] eta 0:03:45 lr 0.001685 wd 0.0500 time 0.3661 (0.3922) data time 0.0009 (0.0107) model time 0.0000 (0.0000) loss 2.2504 (3.3952) grad_norm 1.1931 (1.3092) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][60/625] eta 0:03:41 lr 0.001685 wd 0.0500 time 0.3874 (0.3912) data time 0.0009 (0.0093) model time 0.3866 (0.3839) loss 3.8052 (3.4104) grad_norm 1.3538 (1.3047) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][70/625] eta 0:03:36 lr 0.001685 wd 0.0500 time 0.3857 (0.3900) data time 0.0011 (0.0081) model time 0.3846 (0.3826) loss 4.0726 (3.4234) grad_norm 1.4301 (1.2957) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][80/625] eta 0:03:32 lr 0.001685 wd 0.0500 time 0.3772 (0.3891) data time 0.0011 (0.0073) model time 0.3761 (0.3824) loss 3.8416 (3.4301) grad_norm 1.2079 (1.2932) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][90/625] eta 0:03:27 lr 0.001685 wd 0.0500 time 0.3771 (0.3884) data time 0.0011 (0.0066) model time 0.3760 (0.3820) loss 2.9890 (3.4345) grad_norm 1.8230 (1.3153) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][100/625] eta 0:03:25 lr 0.001685 wd 0.0500 time 0.3866 (0.3915) data time 0.0011 (0.0061) model time 0.3855 (0.3894) loss 3.3737 (3.4383) grad_norm 1.2986 (1.3390) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][110/625] eta 0:03:21 lr 0.001685 wd 0.0500 time 0.3802 (0.3904) data time 0.0011 (0.0056) model time 0.3790 (0.3876) loss 3.0596 (3.4133) grad_norm 1.0475 (1.3238) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][120/625] eta 0:03:16 lr 0.001685 wd 0.0500 time 0.3782 (0.3895) data time 0.0009 (0.0052) model time 0.3773 (0.3862) loss 4.0479 (3.4295) grad_norm 1.7051 (1.3221) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][130/625] eta 0:03:12 lr 0.001684 wd 0.0500 time 0.3815 (0.3884) data time 0.0010 (0.0049) model time 0.3804 (0.3847) loss 3.5185 (3.4401) grad_norm 1.7911 (inf) loss_scale 2048.0000 (3986.5649) mem 11225MB [2024-08-26 09:17:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][140/625] eta 0:03:08 lr 0.001684 wd 0.0500 time 0.3675 (0.3878) data time 0.0008 (0.0047) model time 0.3667 (0.3841) loss 3.2684 (3.4418) grad_norm 1.3214 (inf) loss_scale 2048.0000 (3849.0780) mem 11225MB [2024-08-26 09:17:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][150/625] eta 0:03:03 lr 0.001684 wd 0.0500 time 0.3767 (0.3871) data time 0.0009 (0.0044) model time 0.3758 (0.3833) loss 2.5544 (3.4249) grad_norm 1.1974 (inf) loss_scale 2048.0000 (3729.8013) mem 11225MB [2024-08-26 09:17:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][160/625] eta 0:03:00 lr 0.001684 wd 0.0500 time 0.5772 (0.3892) data time 0.0013 (0.0042) model time 0.5759 (0.3865) loss 3.3398 (3.4216) grad_norm 1.3061 (inf) loss_scale 2048.0000 (3625.3416) mem 11225MB [2024-08-26 09:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][170/625] eta 0:02:56 lr 0.001684 wd 0.0500 time 0.3784 (0.3885) data time 0.0011 (0.0041) model time 0.3773 (0.3857) loss 3.8931 (3.4155) grad_norm 1.0290 (inf) loss_scale 2048.0000 (3533.0994) mem 11225MB [2024-08-26 09:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][180/625] eta 0:02:52 lr 0.001684 wd 0.0500 time 0.3775 (0.3880) data time 0.0008 (0.0039) model time 0.3767 (0.3851) loss 3.1654 (3.4084) grad_norm 1.1312 (inf) loss_scale 2048.0000 (3451.0497) mem 11225MB [2024-08-26 09:17:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][190/625] eta 0:02:48 lr 0.001684 wd 0.0500 time 0.3912 (0.3876) data time 0.0011 (0.0038) model time 0.3901 (0.3847) loss 2.8899 (3.4030) grad_norm 1.4763 (inf) loss_scale 2048.0000 (3377.5916) mem 11225MB [2024-08-26 09:17:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][200/625] eta 0:02:44 lr 0.001683 wd 0.0500 time 0.3715 (0.3870) data time 0.0012 (0.0036) model time 0.3703 (0.3840) loss 3.4767 (3.4170) grad_norm 1.5091 (inf) loss_scale 2048.0000 (3311.4428) mem 11225MB [2024-08-26 09:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][210/625] eta 0:02:40 lr 0.001683 wd 0.0500 time 0.3738 (0.3865) data time 0.0008 (0.0035) model time 0.3730 (0.3835) loss 3.0293 (3.4293) grad_norm 2.0432 (inf) loss_scale 2048.0000 (3251.5640) mem 11225MB [2024-08-26 09:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][220/625] eta 0:02:36 lr 0.001683 wd 0.0500 time 0.3872 (0.3862) data time 0.0010 (0.0034) model time 0.3862 (0.3832) loss 3.2877 (3.4414) grad_norm 1.0754 (inf) loss_scale 2048.0000 (3197.1041) mem 11225MB [2024-08-26 09:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][230/625] eta 0:02:32 lr 0.001683 wd 0.0500 time 0.3772 (0.3858) data time 0.0011 (0.0033) model time 0.3761 (0.3827) loss 3.8658 (3.4515) grad_norm 1.3084 (inf) loss_scale 2048.0000 (3147.3593) mem 11225MB [2024-08-26 09:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][240/625] eta 0:02:28 lr 0.001683 wd 0.0500 time 0.3787 (0.3853) data time 0.0011 (0.0032) model time 0.3776 (0.3822) loss 4.0225 (3.4683) grad_norm 1.4734 (inf) loss_scale 2048.0000 (3101.7427) mem 11225MB [2024-08-26 09:18:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][250/625] eta 0:02:24 lr 0.001683 wd 0.0500 time 0.3798 (0.3850) data time 0.0008 (0.0031) model time 0.3790 (0.3819) loss 3.1282 (3.4651) grad_norm 1.5490 (inf) loss_scale 2048.0000 (3059.7610) mem 11225MB [2024-08-26 09:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][260/625] eta 0:02:20 lr 0.001683 wd 0.0500 time 0.3823 (0.3847) data time 0.0009 (0.0031) model time 0.3815 (0.3817) loss 4.0852 (3.4640) grad_norm 1.1161 (inf) loss_scale 2048.0000 (3020.9962) mem 11225MB [2024-08-26 09:18:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][270/625] eta 0:02:16 lr 0.001683 wd 0.0500 time 0.3814 (0.3845) data time 0.0008 (0.0030) model time 0.3805 (0.3815) loss 4.1037 (3.4647) grad_norm 1.7120 (inf) loss_scale 2048.0000 (2985.0923) mem 11225MB [2024-08-26 09:18:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][280/625] eta 0:02:12 lr 0.001682 wd 0.0500 time 0.3762 (0.3843) data time 0.0009 (0.0029) model time 0.3753 (0.3813) loss 3.9676 (3.4604) grad_norm 1.7270 (inf) loss_scale 2048.0000 (2951.7438) mem 11225MB [2024-08-26 09:18:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][290/625] eta 0:02:08 lr 0.001682 wd 0.0500 time 0.3884 (0.3841) data time 0.0008 (0.0029) model time 0.3876 (0.3812) loss 3.7604 (3.4602) grad_norm 1.3955 (inf) loss_scale 2048.0000 (2920.6873) mem 11225MB [2024-08-26 09:18:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][300/625] eta 0:02:04 lr 0.001682 wd 0.0500 time 0.3771 (0.3839) data time 0.0011 (0.0028) model time 0.3760 (0.3810) loss 3.8616 (3.4633) grad_norm 2.7666 (inf) loss_scale 2048.0000 (2891.6944) mem 11225MB [2024-08-26 09:18:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][310/625] eta 0:02:01 lr 0.001682 wd 0.0500 time 0.3748 (0.3854) data time 0.0008 (0.0027) model time 0.3740 (0.3829) loss 2.3632 (3.4632) grad_norm 2.2386 (inf) loss_scale 2048.0000 (2864.5659) mem 11225MB [2024-08-26 09:18:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][320/625] eta 0:01:57 lr 0.001682 wd 0.0500 time 0.3771 (0.3852) data time 0.0010 (0.0027) model time 0.3761 (0.3827) loss 3.1648 (3.4596) grad_norm 1.2605 (inf) loss_scale 2048.0000 (2839.1277) mem 11225MB [2024-08-26 09:18:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][330/625] eta 0:01:53 lr 0.001682 wd 0.0500 time 0.3822 (0.3850) data time 0.0011 (0.0026) model time 0.3811 (0.3824) loss 3.5932 (3.4633) grad_norm 1.2282 (inf) loss_scale 2048.0000 (2815.2266) mem 11225MB [2024-08-26 09:18:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][340/625] eta 0:01:49 lr 0.001682 wd 0.0500 time 0.3796 (0.3854) data time 0.0011 (0.0026) model time 0.3785 (0.3830) loss 3.3441 (3.4611) grad_norm 1.3817 (inf) loss_scale 2048.0000 (2792.7273) mem 11225MB [2024-08-26 09:18:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][350/625] eta 0:01:46 lr 0.001682 wd 0.0500 time 0.3813 (0.3857) data time 0.0008 (0.0026) model time 0.3805 (0.3834) loss 3.4442 (3.4586) grad_norm 0.9242 (inf) loss_scale 2048.0000 (2771.5100) mem 11225MB [2024-08-26 09:19:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][360/625] eta 0:01:42 lr 0.001681 wd 0.0500 time 0.3771 (0.3854) data time 0.0008 (0.0025) model time 0.3764 (0.3831) loss 2.9503 (3.4495) grad_norm 1.4691 (inf) loss_scale 2048.0000 (2751.4681) mem 11225MB [2024-08-26 09:19:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][370/625] eta 0:01:38 lr 0.001681 wd 0.0500 time 0.3733 (0.3852) data time 0.0010 (0.0025) model time 0.3723 (0.3829) loss 3.6533 (3.4523) grad_norm 1.2577 (inf) loss_scale 2048.0000 (2732.5067) mem 11225MB [2024-08-26 09:19:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][380/625] eta 0:01:34 lr 0.001681 wd 0.0500 time 0.3837 (0.3850) data time 0.0010 (0.0024) model time 0.3826 (0.3827) loss 3.7814 (3.4531) grad_norm 1.3413 (inf) loss_scale 2048.0000 (2714.5407) mem 11225MB [2024-08-26 09:19:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][390/625] eta 0:01:30 lr 0.001681 wd 0.0500 time 0.3771 (0.3848) data time 0.0009 (0.0024) model time 0.3763 (0.3825) loss 2.8385 (3.4524) grad_norm 1.0429 (inf) loss_scale 2048.0000 (2697.4936) mem 11225MB [2024-08-26 09:19:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][400/625] eta 0:01:26 lr 0.001681 wd 0.0500 time 0.3739 (0.3845) data time 0.0011 (0.0024) model time 0.3728 (0.3822) loss 3.3633 (3.4448) grad_norm 1.2583 (inf) loss_scale 2048.0000 (2681.2968) mem 11225MB [2024-08-26 09:19:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][410/625] eta 0:01:22 lr 0.001681 wd 0.0500 time 0.3775 (0.3843) data time 0.0009 (0.0023) model time 0.3766 (0.3820) loss 4.1682 (3.4470) grad_norm 1.7660 (inf) loss_scale 2048.0000 (2665.8881) mem 11225MB [2024-08-26 09:19:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][420/625] eta 0:01:18 lr 0.001681 wd 0.0500 time 0.3858 (0.3842) data time 0.0008 (0.0023) model time 0.3850 (0.3819) loss 3.7225 (3.4461) grad_norm 0.9003 (inf) loss_scale 2048.0000 (2651.2114) mem 11225MB [2024-08-26 09:19:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][430/625] eta 0:01:14 lr 0.001680 wd 0.0500 time 0.3785 (0.3841) data time 0.0011 (0.0023) model time 0.3774 (0.3818) loss 3.5457 (3.4483) grad_norm 1.4758 (inf) loss_scale 2048.0000 (2637.2158) mem 11225MB [2024-08-26 09:19:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][440/625] eta 0:01:11 lr 0.001680 wd 0.0500 time 0.3770 (0.3840) data time 0.0008 (0.0023) model time 0.3762 (0.3817) loss 2.1137 (3.4457) grad_norm 1.2082 (inf) loss_scale 2048.0000 (2623.8549) mem 11225MB [2024-08-26 09:19:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][450/625] eta 0:01:07 lr 0.001680 wd 0.0500 time 0.3778 (0.3838) data time 0.0011 (0.0022) model time 0.3767 (0.3816) loss 3.9107 (3.4439) grad_norm 1.1241 (inf) loss_scale 2048.0000 (2611.0865) mem 11225MB [2024-08-26 09:19:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][460/625] eta 0:01:03 lr 0.001680 wd 0.0500 time 0.3847 (0.3842) data time 0.0011 (0.0022) model time 0.3836 (0.3820) loss 3.5947 (3.4443) grad_norm 1.0912 (inf) loss_scale 2048.0000 (2598.8720) mem 11225MB [2024-08-26 09:19:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][470/625] eta 0:00:59 lr 0.001680 wd 0.0500 time 0.3777 (0.3845) data time 0.0009 (0.0022) model time 0.3768 (0.3824) loss 2.7065 (3.4373) grad_norm 1.0996 (inf) loss_scale 2048.0000 (2587.1762) mem 11225MB [2024-08-26 09:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][480/625] eta 0:00:55 lr 0.001680 wd 0.0500 time 0.3735 (0.3843) data time 0.0009 (0.0022) model time 0.3726 (0.3822) loss 2.5019 (3.4381) grad_norm 1.6436 (inf) loss_scale 2048.0000 (2575.9667) mem 11225MB [2024-08-26 09:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][490/625] eta 0:00:51 lr 0.001680 wd 0.0500 time 0.3794 (0.3850) data time 0.0008 (0.0022) model time 0.3785 (0.3829) loss 3.4343 (3.4403) grad_norm 1.3252 (inf) loss_scale 2048.0000 (2565.2138) mem 11225MB [2024-08-26 09:19:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][500/625] eta 0:00:48 lr 0.001680 wd 0.0500 time 0.3861 (0.3848) data time 0.0008 (0.0021) model time 0.3853 (0.3828) loss 2.8000 (3.4407) grad_norm 1.2536 (inf) loss_scale 2048.0000 (2554.8902) mem 11225MB [2024-08-26 09:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][510/625] eta 0:00:44 lr 0.001679 wd 0.0500 time 0.3818 (0.3847) data time 0.0009 (0.0021) model time 0.3809 (0.3826) loss 4.0702 (3.4433) grad_norm 1.4316 (inf) loss_scale 2048.0000 (2544.9706) mem 11225MB [2024-08-26 09:20:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][520/625] eta 0:00:40 lr 0.001679 wd 0.0500 time 0.3756 (0.3845) data time 0.0014 (0.0021) model time 0.3742 (0.3825) loss 3.3082 (3.4455) grad_norm 1.3385 (inf) loss_scale 2048.0000 (2535.4319) mem 11225MB [2024-08-26 09:20:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][530/625] eta 0:00:36 lr 0.001679 wd 0.0500 time 0.3838 (0.3844) data time 0.0009 (0.0021) model time 0.3830 (0.3824) loss 3.0866 (3.4487) grad_norm 1.4801 (inf) loss_scale 2048.0000 (2526.2524) mem 11225MB [2024-08-26 09:20:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][540/625] eta 0:00:32 lr 0.001679 wd 0.0500 time 0.3798 (0.3843) data time 0.0014 (0.0021) model time 0.3784 (0.3823) loss 3.4820 (3.4513) grad_norm 1.8426 (inf) loss_scale 2048.0000 (2517.4122) mem 11225MB [2024-08-26 09:20:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][550/625] eta 0:00:28 lr 0.001679 wd 0.0500 time 0.3825 (0.3842) data time 0.0012 (0.0021) model time 0.3813 (0.3822) loss 3.0162 (3.4503) grad_norm 1.7951 (inf) loss_scale 2048.0000 (2508.8929) mem 11225MB [2024-08-26 09:20:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][560/625] eta 0:00:24 lr 0.001679 wd 0.0500 time 0.3782 (0.3841) data time 0.0008 (0.0020) model time 0.3774 (0.3821) loss 3.1427 (3.4466) grad_norm 1.4034 (inf) loss_scale 2048.0000 (2500.6774) mem 11225MB [2024-08-26 09:20:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][570/625] eta 0:00:21 lr 0.001679 wd 0.0500 time 0.3892 (0.3840) data time 0.0011 (0.0020) model time 0.3881 (0.3821) loss 2.6691 (3.4395) grad_norm 1.0798 (inf) loss_scale 2048.0000 (2492.7496) mem 11225MB [2024-08-26 09:20:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][580/625] eta 0:00:17 lr 0.001679 wd 0.0500 time 0.3827 (0.3839) data time 0.0011 (0.0020) model time 0.3816 (0.3820) loss 3.2847 (3.4419) grad_norm 1.6358 (inf) loss_scale 2048.0000 (2485.0947) mem 11225MB [2024-08-26 09:20:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][590/625] eta 0:00:13 lr 0.001678 wd 0.0500 time 0.3731 (0.3838) data time 0.0010 (0.0020) model time 0.3722 (0.3819) loss 3.2892 (3.4429) grad_norm 0.9961 (inf) loss_scale 2048.0000 (2477.6988) mem 11225MB [2024-08-26 09:20:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][600/625] eta 0:00:09 lr 0.001678 wd 0.0500 time 0.3825 (0.3837) data time 0.0010 (0.0020) model time 0.3815 (0.3818) loss 3.1952 (3.4446) grad_norm 1.1702 (inf) loss_scale 2048.0000 (2470.5491) mem 11225MB [2024-08-26 09:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][610/625] eta 0:00:05 lr 0.001678 wd 0.0500 time 0.3673 (0.3846) data time 0.0008 (0.0020) model time 0.3665 (0.3827) loss 3.8436 (3.4396) grad_norm 1.0287 (inf) loss_scale 2048.0000 (2463.6334) mem 11225MB [2024-08-26 09:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [93/300][620/625] eta 0:00:01 lr 0.001678 wd 0.0500 time 0.3668 (0.3843) data time 0.0008 (0.0020) model time 0.3660 (0.3824) loss 3.2125 (3.4402) grad_norm 0.9080 (inf) loss_scale 2048.0000 (2456.9404) mem 11225MB [2024-08-26 09:20:41 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 93 training takes 0:04:00 [2024-08-26 09:20:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:20:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:20:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.525 (0.525) Loss 0.6060 (0.6060) Acc@1 86.182 (86.182) Acc@5 97.900 (97.900) Mem 11225MB [2024-08-26 09:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.145) Loss 1.1143 (0.7806) Acc@1 73.730 (81.650) Acc@5 92.383 (96.276) Mem 11225MB [2024-08-26 09:20:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.096 (0.122) Loss 1.2129 (0.9520) Acc@1 71.875 (77.765) Acc@5 91.504 (94.220) Mem 11225MB [2024-08-26 09:20:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.611 Acc@5 94.178 [2024-08-26 09:20:45 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.6% [2024-08-26 09:20:45 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 77.61% [2024-08-26 09:20:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 09:20:46 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 09:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.513 (0.513) Loss 0.5552 (0.5552) Acc@1 87.988 (87.988) Acc@5 98.242 (98.242) Mem 11225MB [2024-08-26 09:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 0.9243 (0.6925) Acc@1 77.539 (84.197) Acc@5 94.971 (97.088) Mem 11225MB [2024-08-26 09:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.0879 (0.8419) Acc@1 73.242 (80.464) Acc@5 92.969 (95.445) Mem 11225MB [2024-08-26 09:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.194 Acc@5 95.429 [2024-08-26 09:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.2% [2024-08-26 09:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][0/625] eta 0:13:22 lr 0.001678 wd 0.0500 time 1.2834 (1.2834) data time 0.7895 (0.7895) model time 0.0000 (0.0000) loss 4.0761 (4.0761) grad_norm 1.1815 (1.1815) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:20:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][10/625] eta 0:04:43 lr 0.001678 wd 0.0500 time 0.3804 (0.4610) data time 0.0009 (0.0728) model time 0.0000 (0.0000) loss 2.4844 (3.1994) grad_norm 0.9451 (1.2518) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:20:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][20/625] eta 0:04:15 lr 0.001678 wd 0.0500 time 0.3828 (0.4218) data time 0.0008 (0.0387) model time 0.0000 (0.0000) loss 2.2938 (3.3431) grad_norm 1.7730 (1.3530) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][30/625] eta 0:04:09 lr 0.001678 wd 0.0500 time 0.3756 (0.4199) data time 0.0009 (0.0266) model time 0.0000 (0.0000) loss 3.4949 (3.3267) grad_norm 1.3478 (1.3502) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][40/625] eta 0:03:59 lr 0.001677 wd 0.0500 time 0.3765 (0.4095) data time 0.0009 (0.0204) model time 0.0000 (0.0000) loss 3.8666 (3.3599) grad_norm 1.6915 (1.3457) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][50/625] eta 0:03:52 lr 0.001677 wd 0.0500 time 0.3745 (0.4038) data time 0.0009 (0.0166) model time 0.0000 (0.0000) loss 3.8520 (3.3782) grad_norm 1.5685 (1.3451) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][60/625] eta 0:03:45 lr 0.001677 wd 0.0500 time 0.3804 (0.3993) data time 0.0008 (0.0141) model time 0.3795 (0.3749) loss 3.3311 (3.3955) grad_norm 1.4675 (1.3564) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][70/625] eta 0:03:39 lr 0.001677 wd 0.0500 time 0.3708 (0.3960) data time 0.0010 (0.0123) model time 0.3697 (0.3750) loss 4.0323 (3.4613) grad_norm 1.9214 (1.3507) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][80/625] eta 0:03:34 lr 0.001677 wd 0.0500 time 0.3761 (0.3938) data time 0.0009 (0.0109) model time 0.3752 (0.3757) loss 3.1000 (3.4476) grad_norm 1.4126 (1.3483) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][90/625] eta 0:03:30 lr 0.001677 wd 0.0500 time 0.3777 (0.3926) data time 0.0008 (0.0098) model time 0.3769 (0.3771) loss 4.0668 (3.4916) grad_norm 1.0390 (1.3405) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][100/625] eta 0:03:25 lr 0.001677 wd 0.0500 time 0.3745 (0.3915) data time 0.0010 (0.0090) model time 0.3735 (0.3778) loss 3.4366 (3.4966) grad_norm 1.0798 (1.3250) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][110/625] eta 0:03:21 lr 0.001676 wd 0.0500 time 0.5614 (0.3921) data time 0.0011 (0.0083) model time 0.5602 (0.3810) loss 2.7322 (3.4761) grad_norm 2.1457 (1.3357) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][120/625] eta 0:03:17 lr 0.001676 wd 0.0500 time 0.3775 (0.3905) data time 0.0008 (0.0077) model time 0.3767 (0.3796) loss 2.4278 (3.4531) grad_norm 1.5015 (1.3360) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][130/625] eta 0:03:12 lr 0.001676 wd 0.0500 time 0.3796 (0.3895) data time 0.0008 (0.0072) model time 0.3788 (0.3793) loss 3.7520 (3.4562) grad_norm 1.3448 (1.3353) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][140/625] eta 0:03:08 lr 0.001676 wd 0.0500 time 0.3744 (0.3888) data time 0.0012 (0.0067) model time 0.3731 (0.3792) loss 3.6137 (3.4456) grad_norm 1.0746 (1.3273) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][150/625] eta 0:03:04 lr 0.001676 wd 0.0500 time 0.3921 (0.3883) data time 0.0012 (0.0064) model time 0.3909 (0.3792) loss 3.7807 (3.4449) grad_norm 1.2043 (1.3226) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][160/625] eta 0:03:00 lr 0.001676 wd 0.0500 time 0.3786 (0.3876) data time 0.0008 (0.0061) model time 0.3777 (0.3789) loss 3.1541 (3.4488) grad_norm 1.2573 (1.3294) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:21:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][170/625] eta 0:02:57 lr 0.001676 wd 0.0500 time 0.3759 (0.3893) data time 0.0011 (0.0058) model time 0.3749 (0.3820) loss 3.6545 (3.4332) grad_norm 1.1611 (1.3207) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][180/625] eta 0:02:52 lr 0.001676 wd 0.0500 time 0.3732 (0.3886) data time 0.0011 (0.0055) model time 0.3721 (0.3814) loss 3.6287 (3.4218) grad_norm 1.5847 (1.3206) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][190/625] eta 0:02:48 lr 0.001675 wd 0.0500 time 0.3830 (0.3883) data time 0.0011 (0.0053) model time 0.3819 (0.3815) loss 3.9433 (3.4119) grad_norm 1.3878 (1.3241) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][200/625] eta 0:02:44 lr 0.001675 wd 0.0500 time 0.3805 (0.3878) data time 0.0011 (0.0051) model time 0.3795 (0.3812) loss 4.0439 (3.4153) grad_norm 1.5554 (1.3316) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][210/625] eta 0:02:40 lr 0.001675 wd 0.0500 time 0.3775 (0.3873) data time 0.0011 (0.0049) model time 0.3764 (0.3808) loss 3.6248 (3.4175) grad_norm 1.8642 (1.3416) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][220/625] eta 0:02:36 lr 0.001675 wd 0.0500 time 0.3753 (0.3869) data time 0.0011 (0.0047) model time 0.3742 (0.3806) loss 2.4058 (3.4198) grad_norm 1.3109 (1.3360) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][230/625] eta 0:02:33 lr 0.001675 wd 0.0500 time 0.6082 (0.3874) data time 0.0008 (0.0046) model time 0.6074 (0.3816) loss 3.9172 (3.4056) grad_norm 1.3601 (1.3287) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][240/625] eta 0:02:29 lr 0.001675 wd 0.0500 time 0.3818 (0.3879) data time 0.0008 (0.0044) model time 0.3810 (0.3825) loss 3.6003 (3.4044) grad_norm 1.1666 (1.3209) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][250/625] eta 0:02:25 lr 0.001675 wd 0.0500 time 0.3751 (0.3875) data time 0.0010 (0.0043) model time 0.3741 (0.3822) loss 3.6418 (3.4074) grad_norm 1.3490 (1.3221) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][260/625] eta 0:02:21 lr 0.001675 wd 0.0500 time 0.3803 (0.3872) data time 0.0011 (0.0042) model time 0.3792 (0.3819) loss 3.6877 (3.3931) grad_norm 1.3271 (1.3307) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][270/625] eta 0:02:17 lr 0.001674 wd 0.0500 time 0.3772 (0.3868) data time 0.0008 (0.0041) model time 0.3763 (0.3817) loss 3.2952 (3.3985) grad_norm 1.6684 (1.3304) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][280/625] eta 0:02:13 lr 0.001674 wd 0.0500 time 0.3731 (0.3865) data time 0.0010 (0.0040) model time 0.3721 (0.3815) loss 3.8396 (3.4008) grad_norm 1.4977 (1.3410) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][290/625] eta 0:02:09 lr 0.001674 wd 0.0500 time 0.3705 (0.3875) data time 0.0011 (0.0039) model time 0.3694 (0.3828) loss 3.6917 (3.3997) grad_norm 1.3913 (1.3395) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][300/625] eta 0:02:05 lr 0.001674 wd 0.0500 time 0.3765 (0.3872) data time 0.0009 (0.0038) model time 0.3756 (0.3827) loss 2.7823 (3.3962) grad_norm 1.2561 (1.3382) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][310/625] eta 0:02:01 lr 0.001674 wd 0.0500 time 0.3719 (0.3869) data time 0.0008 (0.0037) model time 0.3710 (0.3824) loss 3.8701 (3.4057) grad_norm 1.5361 (1.3357) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][320/625] eta 0:01:57 lr 0.001674 wd 0.0500 time 0.3776 (0.3866) data time 0.0011 (0.0036) model time 0.3765 (0.3822) loss 2.8781 (3.3968) grad_norm 1.8308 (1.3395) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:22:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][330/625] eta 0:01:53 lr 0.001674 wd 0.0500 time 0.3762 (0.3863) data time 0.0010 (0.0035) model time 0.3752 (0.3819) loss 3.5975 (3.3929) grad_norm 1.0040 (1.3362) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][340/625] eta 0:01:50 lr 0.001673 wd 0.0500 time 0.3764 (0.3860) data time 0.0008 (0.0035) model time 0.3756 (0.3818) loss 3.9156 (3.3906) grad_norm 0.9063 (1.3388) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][350/625] eta 0:01:46 lr 0.001673 wd 0.0500 time 0.3727 (0.3858) data time 0.0011 (0.0034) model time 0.3717 (0.3816) loss 3.6636 (3.3929) grad_norm 1.4424 (1.3342) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][360/625] eta 0:01:42 lr 0.001673 wd 0.0500 time 0.3793 (0.3856) data time 0.0007 (0.0033) model time 0.3785 (0.3815) loss 3.5272 (3.3898) grad_norm 0.7440 (1.3465) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][370/625] eta 0:01:38 lr 0.001673 wd 0.0500 time 0.3766 (0.3854) data time 0.0008 (0.0033) model time 0.3757 (0.3813) loss 3.8104 (3.3923) grad_norm 0.9488 (1.3416) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][380/625] eta 0:01:34 lr 0.001673 wd 0.0500 time 0.3783 (0.3852) data time 0.0010 (0.0032) model time 0.3773 (0.3812) loss 3.8249 (3.3894) grad_norm 1.5762 (1.3374) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][390/625] eta 0:01:30 lr 0.001673 wd 0.0500 time 0.3774 (0.3851) data time 0.0008 (0.0032) model time 0.3766 (0.3812) loss 2.6946 (3.3903) grad_norm 1.0022 (1.3337) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][400/625] eta 0:01:26 lr 0.001673 wd 0.0500 time 0.3780 (0.3850) data time 0.0009 (0.0031) model time 0.3771 (0.3811) loss 2.3051 (3.3917) grad_norm 1.4415 (1.3344) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][410/625] eta 0:01:22 lr 0.001673 wd 0.0500 time 0.3744 (0.3847) data time 0.0012 (0.0031) model time 0.3733 (0.3809) loss 4.0409 (3.3860) grad_norm 1.5423 (1.3345) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][420/625] eta 0:01:19 lr 0.001672 wd 0.0500 time 0.3808 (0.3855) data time 0.0008 (0.0030) model time 0.3800 (0.3819) loss 3.7336 (3.3811) grad_norm 0.7862 (1.3316) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][430/625] eta 0:01:15 lr 0.001672 wd 0.0500 time 0.3765 (0.3854) data time 0.0009 (0.0030) model time 0.3756 (0.3818) loss 3.8582 (3.3834) grad_norm 1.8600 (1.3367) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][440/625] eta 0:01:11 lr 0.001672 wd 0.0500 time 0.5697 (0.3857) data time 0.0011 (0.0029) model time 0.5685 (0.3821) loss 2.5528 (3.3852) grad_norm 1.4365 (1.3424) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][450/625] eta 0:01:07 lr 0.001672 wd 0.0500 time 0.3805 (0.3855) data time 0.0011 (0.0029) model time 0.3794 (0.3820) loss 2.6752 (3.3816) grad_norm 1.0208 (1.3441) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][460/625] eta 0:01:03 lr 0.001672 wd 0.0500 time 0.3776 (0.3854) data time 0.0008 (0.0029) model time 0.3768 (0.3819) loss 4.2530 (3.3843) grad_norm 1.0983 (1.3413) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][470/625] eta 0:00:59 lr 0.001672 wd 0.0500 time 0.3772 (0.3852) data time 0.0011 (0.0028) model time 0.3761 (0.3818) loss 3.9034 (3.3831) grad_norm 1.2527 (1.3421) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][480/625] eta 0:00:55 lr 0.001672 wd 0.0500 time 0.3799 (0.3850) data time 0.0010 (0.0028) model time 0.3789 (0.3816) loss 3.6149 (3.3916) grad_norm 1.4719 (1.3441) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:23:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][490/625] eta 0:00:51 lr 0.001671 wd 0.0500 time 0.3817 (0.3849) data time 0.0008 (0.0027) model time 0.3809 (0.3815) loss 2.4961 (3.3895) grad_norm 1.2868 (1.3400) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][500/625] eta 0:00:48 lr 0.001671 wd 0.0500 time 0.3764 (0.3856) data time 0.0009 (0.0027) model time 0.3755 (0.3824) loss 3.0813 (3.3882) grad_norm 1.2027 (1.3363) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][510/625] eta 0:00:44 lr 0.001671 wd 0.0500 time 0.3761 (0.3854) data time 0.0009 (0.0027) model time 0.3752 (0.3823) loss 3.7416 (3.3966) grad_norm 1.2943 (1.3410) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][520/625] eta 0:00:40 lr 0.001671 wd 0.0500 time 0.3746 (0.3853) data time 0.0011 (0.0027) model time 0.3735 (0.3821) loss 3.8298 (3.3999) grad_norm 1.0300 (1.3383) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][530/625] eta 0:00:36 lr 0.001671 wd 0.0500 time 0.3793 (0.3851) data time 0.0010 (0.0026) model time 0.3782 (0.3819) loss 3.0014 (3.3994) grad_norm 2.0160 (1.3421) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][540/625] eta 0:00:32 lr 0.001671 wd 0.0500 time 0.3782 (0.3857) data time 0.0010 (0.0026) model time 0.3772 (0.3827) loss 3.9792 (3.4027) grad_norm 1.0289 (1.3440) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][550/625] eta 0:00:28 lr 0.001671 wd 0.0500 time 0.3752 (0.3856) data time 0.0013 (0.0026) model time 0.3739 (0.3825) loss 3.8687 (3.4013) grad_norm 1.3838 (1.3449) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][560/625] eta 0:00:25 lr 0.001671 wd 0.0500 time 0.3723 (0.3854) data time 0.0011 (0.0026) model time 0.3712 (0.3824) loss 3.6878 (3.4032) grad_norm 1.3308 (1.3482) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][570/625] eta 0:00:21 lr 0.001670 wd 0.0500 time 0.3875 (0.3852) data time 0.0008 (0.0025) model time 0.3867 (0.3823) loss 3.5054 (3.4064) grad_norm 2.2559 (1.3503) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][580/625] eta 0:00:17 lr 0.001670 wd 0.0500 time 0.3753 (0.3851) data time 0.0011 (0.0025) model time 0.3742 (0.3822) loss 3.1423 (3.4069) grad_norm 0.8280 (1.3481) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][590/625] eta 0:00:13 lr 0.001670 wd 0.0500 time 0.3727 (0.3850) data time 0.0011 (0.0025) model time 0.3716 (0.3820) loss 3.7741 (3.4099) grad_norm 1.9554 (1.3467) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][600/625] eta 0:00:09 lr 0.001670 wd 0.0500 time 0.3732 (0.3849) data time 0.0015 (0.0025) model time 0.3717 (0.3820) loss 3.6710 (3.4147) grad_norm 1.2062 (1.3473) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][610/625] eta 0:00:05 lr 0.001670 wd 0.0500 time 0.3657 (0.3848) data time 0.0005 (0.0024) model time 0.3652 (0.3819) loss 3.5076 (3.4109) grad_norm 1.4746 (1.3599) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [94/300][620/625] eta 0:00:01 lr 0.001670 wd 0.0500 time 0.3646 (0.3845) data time 0.0008 (0.0024) model time 0.3638 (0.3816) loss 3.4426 (3.4123) grad_norm 1.1452 (1.3597) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:24:50 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 94 training takes 0:04:00 [2024-08-26 09:24:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:24:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:24:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.517 (0.517) Loss 0.7041 (0.7041) Acc@1 85.400 (85.400) Acc@5 97.119 (97.119) Mem 11225MB [2024-08-26 09:24:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.146) Loss 1.1152 (0.8524) Acc@1 75.146 (81.428) Acc@5 93.115 (96.080) Mem 11225MB [2024-08-26 09:24:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.2930 (1.0223) Acc@1 70.312 (77.451) Acc@5 91.016 (93.873) Mem 11225MB [2024-08-26 09:24:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.227 Acc@5 93.904 [2024-08-26 09:24:54 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.2% [2024-08-26 09:24:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.840 (0.840) Loss 0.5542 (0.5542) Acc@1 87.793 (87.793) Acc@5 98.242 (98.242) Mem 11225MB [2024-08-26 09:24:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.176) Loss 0.9238 (0.6921) Acc@1 77.832 (84.229) Acc@5 94.873 (97.115) Mem 11225MB [2024-08-26 09:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.0869 (0.8409) Acc@1 73.535 (80.501) Acc@5 92.920 (95.452) Mem 11225MB [2024-08-26 09:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.238 Acc@5 95.451 [2024-08-26 09:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.2% [2024-08-26 09:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.24% [2024-08-26 09:24:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:24:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:24:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][0/625] eta 0:08:37 lr 0.001670 wd 0.0500 time 0.8275 (0.8275) data time 0.4909 (0.4909) model time 0.0000 (0.0000) loss 4.2673 (4.2673) grad_norm 1.7163 (1.7163) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][10/625] eta 0:04:18 lr 0.001670 wd 0.0500 time 0.3794 (0.4197) data time 0.0010 (0.0456) model time 0.0000 (0.0000) loss 3.9613 (3.8899) grad_norm 1.1010 (1.3934) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][20/625] eta 0:04:01 lr 0.001669 wd 0.0500 time 0.3675 (0.3997) data time 0.0011 (0.0244) model time 0.0000 (0.0000) loss 3.4438 (3.8173) grad_norm 1.6486 (1.3804) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][30/625] eta 0:03:53 lr 0.001669 wd 0.0500 time 0.3763 (0.3928) data time 0.0011 (0.0169) model time 0.0000 (0.0000) loss 3.2359 (3.6473) grad_norm 1.3693 (1.3809) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][40/625] eta 0:03:47 lr 0.001669 wd 0.0500 time 0.3747 (0.3892) data time 0.0008 (0.0130) model time 0.0000 (0.0000) loss 2.6512 (3.5310) grad_norm 1.0251 (1.3450) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][50/625] eta 0:03:42 lr 0.001669 wd 0.0500 time 0.3781 (0.3870) data time 0.0011 (0.0107) model time 0.0000 (0.0000) loss 4.0599 (3.5798) grad_norm 1.6642 (1.4143) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][60/625] eta 0:03:40 lr 0.001669 wd 0.0500 time 0.5821 (0.3906) data time 0.0010 (0.0092) model time 0.5810 (0.4079) loss 3.4128 (3.5899) grad_norm 1.5375 (1.4088) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][70/625] eta 0:03:40 lr 0.001669 wd 0.0500 time 0.5470 (0.3971) data time 0.0011 (0.0080) model time 0.5460 (0.4218) loss 3.5842 (3.5235) grad_norm 1.2206 (1.4093) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][80/625] eta 0:03:35 lr 0.001669 wd 0.0500 time 0.3756 (0.3947) data time 0.0009 (0.0071) model time 0.3747 (0.4068) loss 3.4268 (3.4960) grad_norm 1.8353 (1.4362) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][90/625] eta 0:03:30 lr 0.001668 wd 0.0500 time 0.3764 (0.3928) data time 0.0011 (0.0065) model time 0.3753 (0.3992) loss 3.6094 (3.4854) grad_norm 1.9971 (1.4637) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][100/625] eta 0:03:25 lr 0.001668 wd 0.0500 time 0.3758 (0.3913) data time 0.0008 (0.0059) model time 0.3750 (0.3945) loss 3.0293 (3.4921) grad_norm 1.0634 (1.4290) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][110/625] eta 0:03:20 lr 0.001668 wd 0.0500 time 0.3761 (0.3900) data time 0.0010 (0.0055) model time 0.3750 (0.3915) loss 2.5432 (3.4470) grad_norm 1.4989 (1.4350) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][120/625] eta 0:03:16 lr 0.001668 wd 0.0500 time 0.3870 (0.3890) data time 0.0011 (0.0052) model time 0.3859 (0.3893) loss 3.7954 (3.4430) grad_norm 1.3116 (1.4486) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][130/625] eta 0:03:12 lr 0.001668 wd 0.0500 time 0.3773 (0.3882) data time 0.0011 (0.0049) model time 0.3762 (0.3878) loss 3.7205 (3.4589) grad_norm 1.2843 (1.4400) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][140/625] eta 0:03:08 lr 0.001668 wd 0.0500 time 0.3761 (0.3889) data time 0.0012 (0.0046) model time 0.3749 (0.3887) loss 3.3814 (3.4685) grad_norm 1.4999 (1.4138) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:25:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][150/625] eta 0:03:04 lr 0.001668 wd 0.0500 time 0.3858 (0.3883) data time 0.0010 (0.0044) model time 0.3848 (0.3878) loss 2.7281 (3.4604) grad_norm 1.2961 (1.4044) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][160/625] eta 0:03:00 lr 0.001668 wd 0.0500 time 0.3725 (0.3878) data time 0.0009 (0.0042) model time 0.3717 (0.3870) loss 3.5154 (3.4613) grad_norm 0.9870 (1.4000) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][170/625] eta 0:02:56 lr 0.001667 wd 0.0500 time 0.3793 (0.3873) data time 0.0008 (0.0040) model time 0.3785 (0.3862) loss 4.3693 (3.4642) grad_norm 1.4636 (1.3959) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][180/625] eta 0:02:52 lr 0.001667 wd 0.0500 time 0.3742 (0.3868) data time 0.0008 (0.0039) model time 0.3734 (0.3855) loss 3.0986 (3.4348) grad_norm 1.0977 (1.3940) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][190/625] eta 0:02:47 lr 0.001667 wd 0.0500 time 0.3765 (0.3862) data time 0.0011 (0.0037) model time 0.3755 (0.3847) loss 3.5414 (3.4423) grad_norm 1.7374 (1.3925) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][200/625] eta 0:02:43 lr 0.001667 wd 0.0500 time 0.3812 (0.3859) data time 0.0011 (0.0036) model time 0.3801 (0.3843) loss 2.7111 (3.4177) grad_norm 1.1376 (1.3855) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][210/625] eta 0:02:40 lr 0.001667 wd 0.0500 time 0.3838 (0.3855) data time 0.0011 (0.0035) model time 0.3827 (0.3839) loss 2.8404 (3.4169) grad_norm 1.2220 (1.3857) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][220/625] eta 0:02:36 lr 0.001667 wd 0.0500 time 0.3808 (0.3852) data time 0.0011 (0.0034) model time 0.3797 (0.3835) loss 3.6756 (3.4166) grad_norm 1.3401 (1.3925) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][230/625] eta 0:02:32 lr 0.001667 wd 0.0500 time 0.3738 (0.3848) data time 0.0011 (0.0033) model time 0.3726 (0.3830) loss 3.0678 (3.4129) grad_norm 1.9174 (1.3869) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][240/625] eta 0:02:28 lr 0.001666 wd 0.0500 time 0.3732 (0.3846) data time 0.0008 (0.0032) model time 0.3724 (0.3827) loss 4.6716 (3.4187) grad_norm 2.2075 (1.3924) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][250/625] eta 0:02:24 lr 0.001666 wd 0.0500 time 0.3818 (0.3864) data time 0.0011 (0.0031) model time 0.3807 (0.3851) loss 4.0780 (3.4136) grad_norm 1.4006 (1.3861) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][260/625] eta 0:02:21 lr 0.001666 wd 0.0500 time 0.3715 (0.3869) data time 0.0010 (0.0030) model time 0.3705 (0.3856) loss 3.9615 (3.4226) grad_norm 1.1621 (1.3862) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][270/625] eta 0:02:17 lr 0.001666 wd 0.0500 time 0.3762 (0.3865) data time 0.0012 (0.0030) model time 0.3750 (0.3852) loss 3.2442 (3.4129) grad_norm 1.5932 (1.3872) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][280/625] eta 0:02:13 lr 0.001666 wd 0.0500 time 0.5944 (0.3877) data time 0.0013 (0.0029) model time 0.5931 (0.3867) loss 2.5508 (3.4134) grad_norm 1.5654 (1.3898) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][290/625] eta 0:02:09 lr 0.001666 wd 0.0500 time 0.3751 (0.3874) data time 0.0009 (0.0029) model time 0.3743 (0.3862) loss 3.7002 (3.4129) grad_norm 0.9627 (1.3863) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][300/625] eta 0:02:05 lr 0.001666 wd 0.0500 time 0.3743 (0.3869) data time 0.0009 (0.0028) model time 0.3734 (0.3857) loss 2.5480 (3.4127) grad_norm 1.0759 (1.3783) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:26:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][310/625] eta 0:02:01 lr 0.001666 wd 0.0500 time 0.3724 (0.3866) data time 0.0011 (0.0028) model time 0.3713 (0.3853) loss 3.6659 (3.4099) grad_norm 1.7117 (1.3785) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][320/625] eta 0:01:57 lr 0.001665 wd 0.0500 time 0.3776 (0.3863) data time 0.0012 (0.0027) model time 0.3765 (0.3849) loss 3.7295 (3.4129) grad_norm 1.5497 (1.3757) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][330/625] eta 0:01:53 lr 0.001665 wd 0.0500 time 0.3824 (0.3860) data time 0.0009 (0.0027) model time 0.3815 (0.3846) loss 3.3224 (3.4080) grad_norm 1.3773 (1.3747) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][340/625] eta 0:01:49 lr 0.001665 wd 0.0500 time 0.3857 (0.3859) data time 0.0009 (0.0026) model time 0.3848 (0.3845) loss 3.3016 (3.4108) grad_norm 1.0363 (1.3720) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][350/625] eta 0:01:46 lr 0.001665 wd 0.0500 time 0.3816 (0.3857) data time 0.0009 (0.0026) model time 0.3807 (0.3843) loss 2.5400 (3.4087) grad_norm 1.0920 (1.3659) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][360/625] eta 0:01:42 lr 0.001665 wd 0.0500 time 0.3746 (0.3856) data time 0.0012 (0.0026) model time 0.3734 (0.3841) loss 3.1317 (3.4144) grad_norm 1.2674 (1.3607) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][370/625] eta 0:01:38 lr 0.001665 wd 0.0500 time 0.3965 (0.3865) data time 0.0008 (0.0025) model time 0.3957 (0.3852) loss 2.9102 (3.4118) grad_norm 1.7412 (1.3807) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][380/625] eta 0:01:34 lr 0.001665 wd 0.0500 time 0.3760 (0.3864) data time 0.0009 (0.0025) model time 0.3751 (0.3851) loss 4.1975 (3.4098) grad_norm 1.3710 (1.3823) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][390/625] eta 0:01:30 lr 0.001664 wd 0.0500 time 0.3759 (0.3864) data time 0.0009 (0.0025) model time 0.3750 (0.3850) loss 2.7494 (3.4071) grad_norm 1.1929 (1.3826) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][400/625] eta 0:01:26 lr 0.001664 wd 0.0500 time 0.3868 (0.3862) data time 0.0015 (0.0025) model time 0.3852 (0.3848) loss 3.7420 (3.4088) grad_norm 1.0452 (1.3800) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][410/625] eta 0:01:23 lr 0.001664 wd 0.0500 time 0.3764 (0.3861) data time 0.0011 (0.0025) model time 0.3753 (0.3846) loss 3.7587 (3.4142) grad_norm 2.1355 (1.3889) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][420/625] eta 0:01:19 lr 0.001664 wd 0.0500 time 0.3806 (0.3860) data time 0.0011 (0.0024) model time 0.3794 (0.3845) loss 3.4068 (3.4176) grad_norm 1.0118 (1.3884) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][430/625] eta 0:01:15 lr 0.001664 wd 0.0500 time 0.3945 (0.3858) data time 0.0011 (0.0024) model time 0.3934 (0.3844) loss 2.5810 (3.4088) grad_norm 1.1221 (1.3835) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][440/625] eta 0:01:11 lr 0.001664 wd 0.0500 time 0.3799 (0.3857) data time 0.0009 (0.0024) model time 0.3790 (0.3842) loss 2.8648 (3.4121) grad_norm 1.3719 (1.3804) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][450/625] eta 0:01:07 lr 0.001664 wd 0.0500 time 0.3783 (0.3857) data time 0.0013 (0.0024) model time 0.3770 (0.3842) loss 3.7073 (3.4187) grad_norm 1.3132 (1.3797) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:27:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][460/625] eta 0:01:03 lr 0.001664 wd 0.0500 time 0.3800 (0.3856) data time 0.0009 (0.0024) model time 0.3792 (0.3841) loss 3.2972 (3.4165) grad_norm 1.0825 (1.3814) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][470/625] eta 0:00:59 lr 0.001663 wd 0.0500 time 0.3831 (0.3867) data time 0.0011 (0.0023) model time 0.3820 (0.3854) loss 3.7601 (3.4235) grad_norm 1.3400 (1.3778) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][480/625] eta 0:00:56 lr 0.001663 wd 0.0500 time 0.3711 (0.3865) data time 0.0009 (0.0023) model time 0.3703 (0.3851) loss 4.0246 (3.4286) grad_norm 1.6391 (1.3773) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][490/625] eta 0:00:52 lr 0.001663 wd 0.0500 time 0.3774 (0.3863) data time 0.0008 (0.0023) model time 0.3766 (0.3849) loss 3.4299 (3.4346) grad_norm 1.8609 (1.3792) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][500/625] eta 0:00:48 lr 0.001663 wd 0.0500 time 0.3814 (0.3862) data time 0.0010 (0.0023) model time 0.3804 (0.3848) loss 3.7014 (3.4387) grad_norm 0.9955 (1.3792) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][510/625] eta 0:00:44 lr 0.001663 wd 0.0500 time 0.3782 (0.3860) data time 0.0010 (0.0022) model time 0.3772 (0.3846) loss 3.5230 (3.4413) grad_norm 1.4721 (1.3774) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][520/625] eta 0:00:40 lr 0.001663 wd 0.0500 time 0.3731 (0.3859) data time 0.0011 (0.0022) model time 0.3720 (0.3845) loss 3.4410 (3.4430) grad_norm 1.8688 (1.3772) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][530/625] eta 0:00:36 lr 0.001663 wd 0.0500 time 0.3783 (0.3858) data time 0.0008 (0.0022) model time 0.3775 (0.3844) loss 3.5408 (3.4475) grad_norm 1.2656 (1.3728) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][540/625] eta 0:00:32 lr 0.001662 wd 0.0500 time 0.3791 (0.3857) data time 0.0011 (0.0022) model time 0.3780 (0.3842) loss 3.4189 (3.4522) grad_norm 1.2630 (1.3700) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][550/625] eta 0:00:28 lr 0.001662 wd 0.0500 time 0.3878 (0.3856) data time 0.0008 (0.0022) model time 0.3870 (0.3842) loss 3.8107 (3.4552) grad_norm 1.0761 (1.3713) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][560/625] eta 0:00:25 lr 0.001662 wd 0.0500 time 0.3823 (0.3856) data time 0.0008 (0.0021) model time 0.3814 (0.3842) loss 4.0748 (3.4456) grad_norm 1.6933 (1.3783) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][570/625] eta 0:00:21 lr 0.001662 wd 0.0500 time 0.3801 (0.3855) data time 0.0009 (0.0021) model time 0.3792 (0.3841) loss 2.7323 (3.4475) grad_norm 1.1211 (1.3796) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][580/625] eta 0:00:17 lr 0.001662 wd 0.0500 time 0.3818 (0.3861) data time 0.0009 (0.0021) model time 0.3809 (0.3847) loss 3.7316 (3.4466) grad_norm 1.2376 (1.3764) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][590/625] eta 0:00:13 lr 0.001662 wd 0.0500 time 0.3738 (0.3863) data time 0.0009 (0.0021) model time 0.3729 (0.3850) loss 3.8760 (3.4471) grad_norm 1.2480 (1.3753) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][600/625] eta 0:00:09 lr 0.001662 wd 0.0500 time 0.3779 (0.3862) data time 0.0009 (0.0021) model time 0.3770 (0.3848) loss 3.9014 (3.4424) grad_norm 1.4758 (1.3770) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][610/625] eta 0:00:05 lr 0.001662 wd 0.0500 time 0.3694 (0.3861) data time 0.0008 (0.0021) model time 0.3687 (0.3847) loss 3.8012 (3.4433) grad_norm 1.7938 (1.3749) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [95/300][620/625] eta 0:00:01 lr 0.001661 wd 0.0500 time 0.3686 (0.3858) data time 0.0006 (0.0021) model time 0.3681 (0.3844) loss 3.6657 (3.4466) grad_norm 0.9378 (1.3708) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:28:59 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 95 training takes 0:04:01 [2024-08-26 09:28:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:29:01 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:29:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 0.6421 (0.6421) Acc@1 85.791 (85.791) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 09:29:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.0947 (0.8151) Acc@1 72.803 (81.459) Acc@5 93.945 (96.391) Mem 11225MB [2024-08-26 09:29:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.2197 (0.9787) Acc@1 69.971 (77.632) Acc@5 91.406 (94.234) Mem 11225MB [2024-08-26 09:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.451 Acc@5 94.208 [2024-08-26 09:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.5% [2024-08-26 09:29:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.822 (0.822) Loss 0.5518 (0.5518) Acc@1 87.939 (87.939) Acc@5 98.145 (98.145) Mem 11225MB [2024-08-26 09:29:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.177) Loss 0.9214 (0.6906) Acc@1 77.832 (84.286) Acc@5 95.020 (97.137) Mem 11225MB [2024-08-26 09:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.0859 (0.8394) Acc@1 73.438 (80.529) Acc@5 92.871 (95.473) Mem 11225MB [2024-08-26 09:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.262 Acc@5 95.459 [2024-08-26 09:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.3% [2024-08-26 09:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.26% [2024-08-26 09:29:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:29:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:29:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][0/625] eta 0:07:54 lr 0.001661 wd 0.0500 time 0.7589 (0.7589) data time 0.4240 (0.4240) model time 0.0000 (0.0000) loss 3.8630 (3.8630) grad_norm 1.3032 (1.3032) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][10/625] eta 0:04:15 lr 0.001661 wd 0.0500 time 0.3975 (0.4159) data time 0.0008 (0.0396) model time 0.0000 (0.0000) loss 2.8935 (3.4076) grad_norm 1.1427 (1.2512) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][20/625] eta 0:04:01 lr 0.001661 wd 0.0500 time 0.3690 (0.3986) data time 0.0011 (0.0213) model time 0.0000 (0.0000) loss 3.9017 (3.4942) grad_norm 0.8956 (1.2163) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][30/625] eta 0:03:52 lr 0.001661 wd 0.0500 time 0.3752 (0.3914) data time 0.0008 (0.0148) model time 0.0000 (0.0000) loss 4.3002 (3.4891) grad_norm 1.6917 (1.2374) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][40/625] eta 0:03:46 lr 0.001661 wd 0.0500 time 0.3783 (0.3879) data time 0.0009 (0.0115) model time 0.0000 (0.0000) loss 2.8604 (3.4684) grad_norm 1.2196 (1.2440) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][50/625] eta 0:03:42 lr 0.001661 wd 0.0500 time 0.3856 (0.3861) data time 0.0012 (0.0095) model time 0.0000 (0.0000) loss 3.7965 (3.4131) grad_norm 1.9668 (1.2543) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][60/625] eta 0:03:37 lr 0.001661 wd 0.0500 time 0.3725 (0.3846) data time 0.0011 (0.0081) model time 0.3714 (0.3756) loss 3.3564 (3.3929) grad_norm 1.0228 (1.3032) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][70/625] eta 0:03:32 lr 0.001660 wd 0.0500 time 0.3796 (0.3837) data time 0.0011 (0.0071) model time 0.3785 (0.3765) loss 2.9482 (3.3623) grad_norm 1.5801 (1.3135) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][80/625] eta 0:03:28 lr 0.001660 wd 0.0500 time 0.3783 (0.3833) data time 0.0011 (0.0064) model time 0.3772 (0.3773) loss 2.8620 (3.3250) grad_norm 1.6844 (1.3309) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][90/625] eta 0:03:24 lr 0.001660 wd 0.0500 time 0.3830 (0.3827) data time 0.0008 (0.0058) model time 0.3822 (0.3773) loss 3.6873 (3.3494) grad_norm 1.5738 (1.3396) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][100/625] eta 0:03:21 lr 0.001660 wd 0.0500 time 0.3696 (0.3842) data time 0.0009 (0.0053) model time 0.3688 (0.3812) loss 2.7418 (3.3106) grad_norm 1.0615 (1.3224) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][110/625] eta 0:03:18 lr 0.001660 wd 0.0500 time 0.3761 (0.3853) data time 0.0010 (0.0049) model time 0.3751 (0.3835) loss 3.6209 (3.3234) grad_norm 1.2817 (1.3227) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][120/625] eta 0:03:14 lr 0.001660 wd 0.0500 time 0.3758 (0.3855) data time 0.0009 (0.0046) model time 0.3749 (0.3840) loss 3.8714 (3.3478) grad_norm 1.5132 (1.3276) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:29:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][130/625] eta 0:03:10 lr 0.001660 wd 0.0500 time 0.3790 (0.3849) data time 0.0008 (0.0044) model time 0.3782 (0.3831) loss 4.5359 (3.3542) grad_norm 1.8533 (1.3378) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][140/625] eta 0:03:06 lr 0.001659 wd 0.0500 time 0.3728 (0.3845) data time 0.0009 (0.0041) model time 0.3720 (0.3824) loss 3.8899 (3.3750) grad_norm 1.3340 (1.3469) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][150/625] eta 0:03:03 lr 0.001659 wd 0.0500 time 0.5736 (0.3854) data time 0.0012 (0.0039) model time 0.5724 (0.3839) loss 2.5803 (3.3790) grad_norm 1.6750 (1.3600) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][160/625] eta 0:02:59 lr 0.001659 wd 0.0500 time 0.3782 (0.3863) data time 0.0008 (0.0038) model time 0.3774 (0.3853) loss 3.8469 (3.3906) grad_norm 1.2329 (1.3495) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][170/625] eta 0:02:55 lr 0.001659 wd 0.0500 time 0.3754 (0.3858) data time 0.0009 (0.0036) model time 0.3744 (0.3846) loss 4.0341 (3.4086) grad_norm 1.4331 (1.3412) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][180/625] eta 0:02:51 lr 0.001659 wd 0.0500 time 0.3771 (0.3854) data time 0.0009 (0.0035) model time 0.3763 (0.3840) loss 3.2356 (3.4053) grad_norm 1.0258 (1.3335) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][190/625] eta 0:02:47 lr 0.001659 wd 0.0500 time 0.3809 (0.3850) data time 0.0009 (0.0034) model time 0.3800 (0.3835) loss 4.3861 (3.4120) grad_norm 1.4915 (1.3292) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][200/625] eta 0:02:43 lr 0.001659 wd 0.0500 time 0.3767 (0.3847) data time 0.0012 (0.0033) model time 0.3754 (0.3831) loss 3.5876 (3.3977) grad_norm 1.4161 (1.3260) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][210/625] eta 0:02:39 lr 0.001659 wd 0.0500 time 0.3762 (0.3845) data time 0.0012 (0.0032) model time 0.3750 (0.3828) loss 3.5822 (3.3878) grad_norm 1.4215 (1.3204) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][220/625] eta 0:02:35 lr 0.001658 wd 0.0500 time 0.3863 (0.3842) data time 0.0011 (0.0031) model time 0.3852 (0.3825) loss 3.2600 (3.3859) grad_norm 1.4908 (1.3183) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][230/625] eta 0:02:31 lr 0.001658 wd 0.0500 time 0.3744 (0.3839) data time 0.0009 (0.0030) model time 0.3735 (0.3821) loss 2.6383 (3.3713) grad_norm 1.9817 (1.3265) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][240/625] eta 0:02:27 lr 0.001658 wd 0.0500 time 0.3711 (0.3835) data time 0.0011 (0.0029) model time 0.3700 (0.3817) loss 3.3815 (3.3670) grad_norm 1.3082 (1.3286) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:30:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][250/625] eta 0:02:23 lr 0.001658 wd 0.0500 time 0.3798 (0.3833) data time 0.0012 (0.0028) model time 0.3786 (0.3814) loss 3.4196 (3.3639) grad_norm 1.3624 (1.3265) loss_scale 4096.0000 (2064.3187) mem 11225MB [2024-08-26 09:30:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][260/625] eta 0:02:19 lr 0.001658 wd 0.0500 time 0.3776 (0.3832) data time 0.0011 (0.0028) model time 0.3765 (0.3813) loss 3.3966 (3.3706) grad_norm 1.3074 (1.3350) loss_scale 4096.0000 (2142.1609) mem 11225MB [2024-08-26 09:30:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][270/625] eta 0:02:16 lr 0.001658 wd 0.0500 time 0.3849 (0.3833) data time 0.0009 (0.0027) model time 0.3840 (0.3814) loss 4.2838 (3.3640) grad_norm 1.3284 (1.3296) loss_scale 4096.0000 (2214.2583) mem 11225MB [2024-08-26 09:30:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][280/625] eta 0:02:12 lr 0.001658 wd 0.0500 time 0.3986 (0.3833) data time 0.0012 (0.0027) model time 0.3975 (0.3815) loss 3.1840 (3.3699) grad_norm 1.0497 (1.3210) loss_scale 4096.0000 (2281.2242) mem 11225MB [2024-08-26 09:31:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][290/625] eta 0:02:08 lr 0.001657 wd 0.0500 time 0.3724 (0.3832) data time 0.0009 (0.0026) model time 0.3715 (0.3814) loss 3.8736 (3.3759) grad_norm 1.3936 (1.3261) loss_scale 4096.0000 (2343.5876) mem 11225MB [2024-08-26 09:31:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][300/625] eta 0:02:04 lr 0.001657 wd 0.0500 time 0.3777 (0.3830) data time 0.0011 (0.0026) model time 0.3766 (0.3812) loss 3.9234 (3.3854) grad_norm 1.2195 (1.3236) loss_scale 4096.0000 (2401.8073) mem 11225MB [2024-08-26 09:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][310/625] eta 0:02:00 lr 0.001657 wd 0.0500 time 0.3785 (0.3836) data time 0.0008 (0.0025) model time 0.3776 (0.3819) loss 4.1255 (3.3930) grad_norm 1.3270 (1.3245) loss_scale 4096.0000 (2456.2830) mem 11225MB [2024-08-26 09:31:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][320/625] eta 0:01:57 lr 0.001657 wd 0.0500 time 0.3812 (0.3841) data time 0.0011 (0.0025) model time 0.3802 (0.3825) loss 3.6351 (3.3978) grad_norm 1.4594 (1.3301) loss_scale 4096.0000 (2507.3645) mem 11225MB [2024-08-26 09:31:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][330/625] eta 0:01:53 lr 0.001657 wd 0.0500 time 0.3790 (0.3839) data time 0.0009 (0.0025) model time 0.3781 (0.3823) loss 2.3903 (3.4061) grad_norm 1.0263 (1.3251) loss_scale 4096.0000 (2555.3595) mem 11225MB [2024-08-26 09:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][340/625] eta 0:01:49 lr 0.001657 wd 0.0500 time 0.3992 (0.3853) data time 0.0008 (0.0024) model time 0.3983 (0.3840) loss 3.6751 (3.4068) grad_norm 0.9829 (1.3200) loss_scale 4096.0000 (2600.5396) mem 11225MB [2024-08-26 09:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][350/625] eta 0:01:45 lr 0.001657 wd 0.0500 time 0.3748 (0.3852) data time 0.0008 (0.0024) model time 0.3740 (0.3839) loss 2.3984 (3.4143) grad_norm 1.6264 (1.3498) loss_scale 4096.0000 (2643.1453) mem 11225MB [2024-08-26 09:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][360/625] eta 0:01:42 lr 0.001657 wd 0.0500 time 0.3751 (0.3850) data time 0.0011 (0.0024) model time 0.3740 (0.3836) loss 3.5768 (3.4209) grad_norm 1.1861 (1.3544) loss_scale 4096.0000 (2683.3906) mem 11225MB [2024-08-26 09:31:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][370/625] eta 0:01:38 lr 0.001656 wd 0.0500 time 0.3739 (0.3848) data time 0.0009 (0.0023) model time 0.3730 (0.3834) loss 4.4921 (3.4277) grad_norm 0.9337 (1.3533) loss_scale 4096.0000 (2721.4663) mem 11225MB [2024-08-26 09:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][380/625] eta 0:01:34 lr 0.001656 wd 0.0500 time 0.3785 (0.3846) data time 0.0009 (0.0023) model time 0.3776 (0.3832) loss 3.4218 (3.4276) grad_norm 1.5658 (1.3534) loss_scale 4096.0000 (2757.5433) mem 11225MB [2024-08-26 09:31:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][390/625] eta 0:01:30 lr 0.001656 wd 0.0500 time 0.3787 (0.3845) data time 0.0011 (0.0023) model time 0.3777 (0.3830) loss 3.3943 (3.4311) grad_norm 1.4462 (1.3541) loss_scale 4096.0000 (2791.7749) mem 11225MB [2024-08-26 09:31:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][400/625] eta 0:01:26 lr 0.001656 wd 0.0500 time 0.3903 (0.3845) data time 0.0009 (0.0022) model time 0.3894 (0.3831) loss 3.8800 (3.4319) grad_norm 1.0512 (1.3501) loss_scale 4096.0000 (2824.2993) mem 11225MB [2024-08-26 09:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][410/625] eta 0:01:22 lr 0.001656 wd 0.0500 time 0.3700 (0.3845) data time 0.0012 (0.0022) model time 0.3688 (0.3831) loss 3.8457 (3.4337) grad_norm 1.1547 (1.3506) loss_scale 4096.0000 (2855.2409) mem 11225MB [2024-08-26 09:31:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][420/625] eta 0:01:18 lr 0.001656 wd 0.0500 time 0.3800 (0.3844) data time 0.0011 (0.0022) model time 0.3789 (0.3830) loss 3.4580 (3.4344) grad_norm 0.9301 (1.3462) loss_scale 4096.0000 (2884.7126) mem 11225MB [2024-08-26 09:31:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][430/625] eta 0:01:14 lr 0.001656 wd 0.0500 time 0.3791 (0.3843) data time 0.0011 (0.0022) model time 0.3780 (0.3828) loss 3.1157 (3.4279) grad_norm 1.3984 (1.3436) loss_scale 4096.0000 (2912.8167) mem 11225MB [2024-08-26 09:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][440/625] eta 0:01:11 lr 0.001655 wd 0.0500 time 0.3734 (0.3841) data time 0.0008 (0.0022) model time 0.3726 (0.3827) loss 4.0168 (3.4257) grad_norm 1.2593 (1.3452) loss_scale 4096.0000 (2939.6463) mem 11225MB [2024-08-26 09:32:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][450/625] eta 0:01:07 lr 0.001655 wd 0.0500 time 0.3938 (0.3841) data time 0.0008 (0.0021) model time 0.3930 (0.3826) loss 4.1915 (3.4288) grad_norm 1.6001 (1.3444) loss_scale 4096.0000 (2965.2860) mem 11225MB [2024-08-26 09:32:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][460/625] eta 0:01:03 lr 0.001655 wd 0.0500 time 0.3722 (0.3848) data time 0.0008 (0.0021) model time 0.3714 (0.3834) loss 3.7761 (3.4243) grad_norm 1.7746 (1.3438) loss_scale 4096.0000 (2989.8134) mem 11225MB [2024-08-26 09:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][470/625] eta 0:00:59 lr 0.001655 wd 0.0500 time 0.3772 (0.3846) data time 0.0011 (0.0021) model time 0.3761 (0.3832) loss 3.5367 (3.4226) grad_norm 1.2194 (1.3441) loss_scale 4096.0000 (3013.2994) mem 11225MB [2024-08-26 09:32:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][480/625] eta 0:00:55 lr 0.001655 wd 0.0500 time 0.3707 (0.3844) data time 0.0009 (0.0021) model time 0.3698 (0.3830) loss 3.6943 (3.4232) grad_norm 1.2719 (1.3436) loss_scale 4096.0000 (3035.8087) mem 11225MB [2024-08-26 09:32:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][490/625] eta 0:00:51 lr 0.001655 wd 0.0500 time 0.3816 (0.3852) data time 0.0011 (0.0021) model time 0.3805 (0.3839) loss 3.0187 (3.4259) grad_norm 1.0317 (1.3395) loss_scale 4096.0000 (3057.4012) mem 11225MB [2024-08-26 09:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][500/625] eta 0:00:48 lr 0.001655 wd 0.0500 time 0.3313 (0.3853) data time 0.0008 (0.0020) model time 0.3305 (0.3840) loss 3.3866 (3.4248) grad_norm 1.2593 (1.3367) loss_scale 4096.0000 (3078.1317) mem 11225MB [2024-08-26 09:32:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][510/625] eta 0:00:44 lr 0.001655 wd 0.0500 time 0.3787 (0.3852) data time 0.0008 (0.0020) model time 0.3779 (0.3839) loss 2.6293 (3.4246) grad_norm 1.1248 (1.3339) loss_scale 4096.0000 (3098.0509) mem 11225MB [2024-08-26 09:32:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][520/625] eta 0:00:40 lr 0.001654 wd 0.0500 time 0.3811 (0.3851) data time 0.0011 (0.0020) model time 0.3801 (0.3838) loss 3.2815 (3.4219) grad_norm 0.9344 (1.3337) loss_scale 4096.0000 (3117.2054) mem 11225MB [2024-08-26 09:32:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][530/625] eta 0:00:36 lr 0.001654 wd 0.0500 time 0.3835 (0.3849) data time 0.0008 (0.0020) model time 0.3828 (0.3836) loss 2.5909 (3.4208) grad_norm 0.9753 (1.3296) loss_scale 4096.0000 (3135.6384) mem 11225MB [2024-08-26 09:32:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][540/625] eta 0:00:32 lr 0.001654 wd 0.0500 time 0.3765 (0.3848) data time 0.0012 (0.0020) model time 0.3753 (0.3835) loss 3.8335 (3.4253) grad_norm 1.0536 (1.3265) loss_scale 4096.0000 (3153.3900) mem 11225MB [2024-08-26 09:32:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][550/625] eta 0:00:28 lr 0.001654 wd 0.0500 time 0.3726 (0.3847) data time 0.0009 (0.0020) model time 0.3717 (0.3834) loss 3.7528 (3.4245) grad_norm 1.1971 (1.3251) loss_scale 4096.0000 (3170.4973) mem 11225MB [2024-08-26 09:32:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][560/625] eta 0:00:25 lr 0.001654 wd 0.0500 time 0.3827 (0.3847) data time 0.0011 (0.0019) model time 0.3816 (0.3833) loss 3.4621 (3.4278) grad_norm 1.2100 (1.3234) loss_scale 4096.0000 (3186.9947) mem 11225MB [2024-08-26 09:32:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][570/625] eta 0:00:21 lr 0.001654 wd 0.0500 time 0.3773 (0.3845) data time 0.0011 (0.0019) model time 0.3763 (0.3832) loss 3.8924 (3.4272) grad_norm 0.8370 (1.3200) loss_scale 4096.0000 (3202.9142) mem 11225MB [2024-08-26 09:32:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][580/625] eta 0:00:17 lr 0.001654 wd 0.0500 time 0.3807 (0.3845) data time 0.0011 (0.0019) model time 0.3796 (0.3831) loss 3.3600 (3.4210) grad_norm 1.7080 (1.3180) loss_scale 4096.0000 (3218.2857) mem 11225MB [2024-08-26 09:32:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][590/625] eta 0:00:13 lr 0.001653 wd 0.0500 time 0.3778 (0.3845) data time 0.0011 (0.0019) model time 0.3767 (0.3831) loss 3.5554 (3.4164) grad_norm 1.4776 (1.3180) loss_scale 4096.0000 (3233.1371) mem 11225MB [2024-08-26 09:32:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][600/625] eta 0:00:09 lr 0.001653 wd 0.0500 time 0.3782 (0.3844) data time 0.0008 (0.0019) model time 0.3774 (0.3830) loss 3.2973 (3.4163) grad_norm 1.7668 (1.3288) loss_scale 4096.0000 (3247.4942) mem 11225MB [2024-08-26 09:33:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][610/625] eta 0:00:05 lr 0.001653 wd 0.0500 time 0.3702 (0.3851) data time 0.0008 (0.0019) model time 0.3694 (0.3838) loss 3.4415 (3.4211) grad_norm 1.4835 (1.3293) loss_scale 4096.0000 (3261.3813) mem 11225MB [2024-08-26 09:33:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [96/300][620/625] eta 0:00:01 lr 0.001653 wd 0.0500 time 0.3329 (0.3852) data time 0.0009 (0.0019) model time 0.3320 (0.3839) loss 2.0829 (3.4183) grad_norm 1.4414 (1.3287) loss_scale 4096.0000 (3274.8213) mem 11225MB [2024-08-26 09:33:09 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 96 training takes 0:04:00 [2024-08-26 09:33:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:33:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:33:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.528 (0.528) Loss 0.6450 (0.6450) Acc@1 86.133 (86.133) Acc@5 97.314 (97.314) Mem 11225MB [2024-08-26 09:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.148) Loss 1.1445 (0.8122) Acc@1 73.633 (81.889) Acc@5 92.383 (96.183) Mem 11225MB [2024-08-26 09:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.123) Loss 1.2021 (0.9784) Acc@1 72.266 (77.930) Acc@5 91.553 (94.157) Mem 11225MB [2024-08-26 09:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.721 Acc@5 94.166 [2024-08-26 09:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.7% [2024-08-26 09:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 77.72% [2024-08-26 09:33:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 09:33:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 09:33:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.5498 (0.5498) Acc@1 87.939 (87.939) Acc@5 98.145 (98.145) Mem 11225MB [2024-08-26 09:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.149) Loss 0.9194 (0.6891) Acc@1 78.027 (84.291) Acc@5 95.117 (97.141) Mem 11225MB [2024-08-26 09:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 1.0830 (0.8377) Acc@1 73.291 (80.515) Acc@5 92.920 (95.505) Mem 11225MB [2024-08-26 09:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.254 Acc@5 95.493 [2024-08-26 09:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.3% [2024-08-26 09:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][0/625] eta 0:12:44 lr 0.001653 wd 0.0500 time 1.2227 (1.2227) data time 0.8750 (0.8750) model time 0.0000 (0.0000) loss 4.0421 (4.0421) grad_norm 1.3408 (1.3408) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][10/625] eta 0:04:40 lr 0.001653 wd 0.0500 time 0.3763 (0.4560) data time 0.0008 (0.0807) model time 0.0000 (0.0000) loss 3.3460 (3.5437) grad_norm 1.5674 (1.4923) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][20/625] eta 0:04:12 lr 0.001653 wd 0.0500 time 0.3816 (0.4180) data time 0.0011 (0.0428) model time 0.0000 (0.0000) loss 3.5014 (3.4922) grad_norm 1.1040 (1.3726) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][30/625] eta 0:04:05 lr 0.001653 wd 0.0500 time 0.3824 (0.4121) data time 0.0008 (0.0293) model time 0.0000 (0.0000) loss 3.8051 (3.4115) grad_norm 1.6889 (1.3209) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][40/625] eta 0:03:56 lr 0.001652 wd 0.0500 time 0.3748 (0.4040) data time 0.0008 (0.0225) model time 0.0000 (0.0000) loss 3.4079 (3.3477) grad_norm 1.1113 (1.3001) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][50/625] eta 0:03:49 lr 0.001652 wd 0.0500 time 0.3758 (0.3989) data time 0.0008 (0.0183) model time 0.0000 (0.0000) loss 2.3865 (3.2655) grad_norm 1.1321 (1.3103) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][60/625] eta 0:03:43 lr 0.001652 wd 0.0500 time 0.4098 (0.3963) data time 0.0010 (0.0155) model time 0.4088 (0.3818) loss 2.3913 (3.2258) grad_norm 1.3261 (1.3285) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][70/625] eta 0:03:39 lr 0.001652 wd 0.0500 time 0.3821 (0.3950) data time 0.0008 (0.0135) model time 0.3813 (0.3837) loss 2.8946 (3.2202) grad_norm 0.9832 (1.2936) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][80/625] eta 0:03:34 lr 0.001652 wd 0.0500 time 0.3742 (0.3932) data time 0.0010 (0.0120) model time 0.3733 (0.3822) loss 3.7593 (3.2529) grad_norm 1.1589 (1.2897) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][90/625] eta 0:03:29 lr 0.001652 wd 0.0500 time 0.3879 (0.3921) data time 0.0011 (0.0108) model time 0.3868 (0.3822) loss 3.0946 (3.2822) grad_norm 2.3862 (1.3352) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:33:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][100/625] eta 0:03:25 lr 0.001652 wd 0.0500 time 0.3781 (0.3908) data time 0.0008 (0.0099) model time 0.3773 (0.3812) loss 3.3543 (3.2840) grad_norm 1.0103 (1.3245) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][110/625] eta 0:03:20 lr 0.001651 wd 0.0500 time 0.3779 (0.3897) data time 0.0008 (0.0091) model time 0.3771 (0.3806) loss 3.9640 (3.3015) grad_norm 1.3135 (1.3071) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][120/625] eta 0:03:16 lr 0.001651 wd 0.0500 time 0.3918 (0.3888) data time 0.0012 (0.0084) model time 0.3906 (0.3802) loss 3.6544 (3.3397) grad_norm 1.8397 (1.3092) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][130/625] eta 0:03:12 lr 0.001651 wd 0.0500 time 0.3823 (0.3896) data time 0.0008 (0.0079) model time 0.3815 (0.3825) loss 3.5270 (3.3208) grad_norm 1.1800 (1.3125) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][140/625] eta 0:03:08 lr 0.001651 wd 0.0500 time 0.3716 (0.3888) data time 0.0009 (0.0074) model time 0.3707 (0.3818) loss 2.4275 (3.2934) grad_norm 1.5716 (1.3265) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][150/625] eta 0:03:04 lr 0.001651 wd 0.0500 time 0.3788 (0.3880) data time 0.0008 (0.0070) model time 0.3780 (0.3812) loss 2.5601 (3.3096) grad_norm 2.3375 (1.3318) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][160/625] eta 0:03:00 lr 0.001651 wd 0.0500 time 0.3761 (0.3874) data time 0.0009 (0.0066) model time 0.3752 (0.3808) loss 3.8183 (3.3262) grad_norm 1.8360 (1.3394) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][170/625] eta 0:02:57 lr 0.001651 wd 0.0500 time 0.3740 (0.3892) data time 0.0011 (0.0063) model time 0.3729 (0.3838) loss 2.8335 (3.3303) grad_norm 1.5145 (1.3464) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][180/625] eta 0:02:52 lr 0.001651 wd 0.0500 time 0.3743 (0.3884) data time 0.0009 (0.0060) model time 0.3735 (0.3831) loss 2.8728 (3.3383) grad_norm 1.1672 (1.3433) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][190/625] eta 0:02:48 lr 0.001650 wd 0.0500 time 0.3743 (0.3878) data time 0.0011 (0.0058) model time 0.3732 (0.3826) loss 2.4277 (3.3451) grad_norm 0.9847 (1.3393) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][200/625] eta 0:02:44 lr 0.001650 wd 0.0500 time 0.3857 (0.3873) data time 0.0009 (0.0055) model time 0.3848 (0.3822) loss 3.3378 (3.3365) grad_norm 1.3715 (1.3503) loss_scale 4096.0000 (4096.0000) mem 11225MB [2024-08-26 09:34:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][210/625] eta 0:02:40 lr 0.001650 wd 0.0500 time 0.3587 (0.3867) data time 0.0012 (0.0053) model time 0.3574 (0.3817) loss 3.2492 (3.3493) grad_norm inf (inf) loss_scale 2048.0000 (4086.2938) mem 11225MB [2024-08-26 09:34:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][220/625] eta 0:02:36 lr 0.001650 wd 0.0500 time 0.3766 (0.3865) data time 0.0008 (0.0051) model time 0.3758 (0.3816) loss 2.6070 (3.3483) grad_norm 1.2624 (inf) loss_scale 2048.0000 (3994.0633) mem 11225MB [2024-08-26 09:34:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][230/625] eta 0:02:32 lr 0.001650 wd 0.0500 time 0.5896 (0.3870) data time 0.0008 (0.0050) model time 0.5888 (0.3824) loss 3.5108 (3.3513) grad_norm 1.4359 (inf) loss_scale 2048.0000 (3909.8182) mem 11225MB [2024-08-26 09:34:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][240/625] eta 0:02:29 lr 0.001650 wd 0.0500 time 0.3354 (0.3882) data time 0.0011 (0.0048) model time 0.3343 (0.3842) loss 3.2524 (3.3605) grad_norm 0.9692 (inf) loss_scale 2048.0000 (3832.5643) mem 11225MB [2024-08-26 09:34:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][250/625] eta 0:02:25 lr 0.001650 wd 0.0500 time 0.3761 (0.3878) data time 0.0011 (0.0047) model time 0.3750 (0.3838) loss 3.6581 (3.3696) grad_norm 1.2375 (inf) loss_scale 2048.0000 (3761.4661) mem 11225MB [2024-08-26 09:34:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][260/625] eta 0:02:21 lr 0.001649 wd 0.0500 time 0.3817 (0.3875) data time 0.0008 (0.0045) model time 0.3808 (0.3835) loss 4.3420 (3.3698) grad_norm 1.0734 (inf) loss_scale 2048.0000 (3695.8161) mem 11225MB [2024-08-26 09:35:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][270/625] eta 0:02:17 lr 0.001649 wd 0.0500 time 0.3745 (0.3872) data time 0.0012 (0.0044) model time 0.3733 (0.3833) loss 3.9181 (3.3711) grad_norm 1.2729 (inf) loss_scale 2048.0000 (3635.0111) mem 11225MB [2024-08-26 09:35:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][280/625] eta 0:02:13 lr 0.001649 wd 0.0500 time 0.3816 (0.3870) data time 0.0009 (0.0043) model time 0.3807 (0.3831) loss 2.4485 (3.3654) grad_norm 1.8567 (inf) loss_scale 2048.0000 (3578.5338) mem 11225MB [2024-08-26 09:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][290/625] eta 0:02:10 lr 0.001649 wd 0.0500 time 0.3790 (0.3882) data time 0.0011 (0.0042) model time 0.3780 (0.3847) loss 2.8689 (3.3729) grad_norm 0.8761 (inf) loss_scale 2048.0000 (3525.9381) mem 11225MB [2024-08-26 09:35:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][300/625] eta 0:02:06 lr 0.001649 wd 0.0500 time 0.3777 (0.3879) data time 0.0009 (0.0041) model time 0.3768 (0.3844) loss 2.1899 (3.3668) grad_norm 1.0870 (inf) loss_scale 2048.0000 (3476.8372) mem 11225MB [2024-08-26 09:35:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][310/625] eta 0:02:02 lr 0.001649 wd 0.0500 time 0.3725 (0.3876) data time 0.0013 (0.0040) model time 0.3712 (0.3841) loss 3.0037 (3.3645) grad_norm 1.0212 (inf) loss_scale 2048.0000 (3430.8939) mem 11225MB [2024-08-26 09:35:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][320/625] eta 0:01:58 lr 0.001649 wd 0.0500 time 0.3810 (0.3872) data time 0.0009 (0.0039) model time 0.3800 (0.3838) loss 3.1816 (3.3656) grad_norm 1.7671 (inf) loss_scale 2048.0000 (3387.8131) mem 11225MB [2024-08-26 09:35:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][330/625] eta 0:01:54 lr 0.001648 wd 0.0500 time 0.3782 (0.3870) data time 0.0009 (0.0038) model time 0.3773 (0.3837) loss 3.8059 (3.3679) grad_norm 1.4272 (inf) loss_scale 2048.0000 (3347.3353) mem 11225MB [2024-08-26 09:35:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][340/625] eta 0:01:50 lr 0.001648 wd 0.0500 time 0.3664 (0.3868) data time 0.0009 (0.0038) model time 0.3655 (0.3835) loss 2.9107 (3.3704) grad_norm 1.2155 (inf) loss_scale 2048.0000 (3309.2317) mem 11225MB [2024-08-26 09:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][350/625] eta 0:01:46 lr 0.001648 wd 0.0500 time 0.3857 (0.3866) data time 0.0010 (0.0037) model time 0.3847 (0.3833) loss 4.1315 (3.3765) grad_norm 1.0426 (inf) loss_scale 2048.0000 (3273.2991) mem 11225MB [2024-08-26 09:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][360/625] eta 0:01:42 lr 0.001648 wd 0.0500 time 0.3851 (0.3866) data time 0.0008 (0.0037) model time 0.3842 (0.3833) loss 4.1322 (3.3846) grad_norm 1.2569 (inf) loss_scale 2048.0000 (3239.3573) mem 11225MB [2024-08-26 09:35:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][370/625] eta 0:01:38 lr 0.001648 wd 0.0500 time 0.3778 (0.3864) data time 0.0011 (0.0036) model time 0.3768 (0.3832) loss 2.4771 (3.3813) grad_norm 1.3457 (inf) loss_scale 2048.0000 (3207.2453) mem 11225MB [2024-08-26 09:35:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][380/625] eta 0:01:34 lr 0.001648 wd 0.0500 time 0.3803 (0.3862) data time 0.0011 (0.0035) model time 0.3792 (0.3830) loss 3.6437 (3.3886) grad_norm 1.0553 (inf) loss_scale 2048.0000 (3176.8189) mem 11225MB [2024-08-26 09:35:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][390/625] eta 0:01:30 lr 0.001648 wd 0.0500 time 0.3775 (0.3860) data time 0.0009 (0.0035) model time 0.3766 (0.3828) loss 2.4546 (3.3801) grad_norm 1.6599 (inf) loss_scale 2048.0000 (3147.9488) mem 11225MB [2024-08-26 09:35:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][400/625] eta 0:01:26 lr 0.001648 wd 0.0500 time 0.3761 (0.3858) data time 0.0010 (0.0034) model time 0.3751 (0.3827) loss 3.5551 (3.3786) grad_norm 1.3012 (inf) loss_scale 2048.0000 (3120.5187) mem 11225MB [2024-08-26 09:35:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][410/625] eta 0:01:22 lr 0.001647 wd 0.0500 time 0.3695 (0.3856) data time 0.0013 (0.0034) model time 0.3683 (0.3825) loss 4.0735 (3.3811) grad_norm 1.6733 (inf) loss_scale 2048.0000 (3094.4234) mem 11225MB [2024-08-26 09:36:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][420/625] eta 0:01:19 lr 0.001647 wd 0.0500 time 0.3757 (0.3865) data time 0.0008 (0.0033) model time 0.3749 (0.3835) loss 2.9963 (3.3862) grad_norm 1.2951 (inf) loss_scale 2048.0000 (3069.5677) mem 11225MB [2024-08-26 09:36:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][430/625] eta 0:01:15 lr 0.001647 wd 0.0500 time 0.3738 (0.3863) data time 0.0008 (0.0033) model time 0.3730 (0.3833) loss 3.5752 (3.3889) grad_norm 1.3355 (inf) loss_scale 2048.0000 (3045.8654) mem 11225MB [2024-08-26 09:36:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][440/625] eta 0:01:11 lr 0.001647 wd 0.0500 time 0.3692 (0.3861) data time 0.0011 (0.0032) model time 0.3681 (0.3831) loss 3.2314 (3.3919) grad_norm 1.1034 (inf) loss_scale 2048.0000 (3023.2381) mem 11225MB [2024-08-26 09:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][450/625] eta 0:01:07 lr 0.001647 wd 0.0500 time 0.6075 (0.3864) data time 0.0012 (0.0032) model time 0.6063 (0.3836) loss 2.5769 (3.3850) grad_norm 1.1406 (inf) loss_scale 2048.0000 (3001.6142) mem 11225MB [2024-08-26 09:36:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][460/625] eta 0:01:03 lr 0.001647 wd 0.0500 time 0.3824 (0.3861) data time 0.0008 (0.0031) model time 0.3815 (0.3833) loss 3.3986 (3.3907) grad_norm 1.0739 (inf) loss_scale 2048.0000 (2980.9284) mem 11225MB [2024-08-26 09:36:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][470/625] eta 0:00:59 lr 0.001647 wd 0.0500 time 0.3787 (0.3859) data time 0.0009 (0.0031) model time 0.3779 (0.3831) loss 3.7277 (3.3900) grad_norm 1.7699 (inf) loss_scale 2048.0000 (2961.1210) mem 11225MB [2024-08-26 09:36:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][480/625] eta 0:00:55 lr 0.001646 wd 0.0500 time 0.3788 (0.3857) data time 0.0008 (0.0030) model time 0.3779 (0.3830) loss 2.5511 (3.3890) grad_norm 1.5384 (inf) loss_scale 2048.0000 (2942.1372) mem 11225MB [2024-08-26 09:36:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][490/625] eta 0:00:52 lr 0.001646 wd 0.0500 time 0.3805 (0.3856) data time 0.0013 (0.0030) model time 0.3792 (0.3828) loss 3.1728 (3.3934) grad_norm 1.6664 (inf) loss_scale 2048.0000 (2923.9267) mem 11225MB [2024-08-26 09:36:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][500/625] eta 0:00:48 lr 0.001646 wd 0.0500 time 0.3815 (0.3863) data time 0.0010 (0.0030) model time 0.3805 (0.3837) loss 3.7615 (3.3926) grad_norm 1.2877 (inf) loss_scale 2048.0000 (2906.4431) mem 11225MB [2024-08-26 09:36:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][510/625] eta 0:00:44 lr 0.001646 wd 0.0500 time 0.3804 (0.3862) data time 0.0011 (0.0029) model time 0.3793 (0.3835) loss 3.2759 (3.3941) grad_norm 1.0831 (inf) loss_scale 2048.0000 (2889.6438) mem 11225MB [2024-08-26 09:36:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][520/625] eta 0:00:40 lr 0.001646 wd 0.0500 time 0.3851 (0.3860) data time 0.0008 (0.0029) model time 0.3843 (0.3834) loss 3.1343 (3.3898) grad_norm 1.7593 (inf) loss_scale 2048.0000 (2873.4894) mem 11225MB [2024-08-26 09:36:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][530/625] eta 0:00:36 lr 0.001646 wd 0.0500 time 0.3689 (0.3858) data time 0.0008 (0.0029) model time 0.3680 (0.3832) loss 3.8397 (3.3909) grad_norm 1.1039 (inf) loss_scale 2048.0000 (2857.9435) mem 11225MB [2024-08-26 09:36:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][540/625] eta 0:00:32 lr 0.001646 wd 0.0500 time 0.3751 (0.3864) data time 0.0008 (0.0028) model time 0.3743 (0.3839) loss 3.9000 (3.3892) grad_norm 1.2630 (inf) loss_scale 2048.0000 (2842.9723) mem 11225MB [2024-08-26 09:36:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][550/625] eta 0:00:28 lr 0.001645 wd 0.0500 time 0.3840 (0.3862) data time 0.0012 (0.0028) model time 0.3828 (0.3837) loss 3.0591 (3.3827) grad_norm 1.0151 (inf) loss_scale 2048.0000 (2828.5445) mem 11225MB [2024-08-26 09:36:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][560/625] eta 0:00:25 lr 0.001645 wd 0.0500 time 0.3801 (0.3861) data time 0.0008 (0.0028) model time 0.3793 (0.3836) loss 3.7286 (3.3855) grad_norm 3.7730 (inf) loss_scale 2048.0000 (2814.6310) mem 11225MB [2024-08-26 09:36:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][570/625] eta 0:00:21 lr 0.001645 wd 0.0500 time 0.3793 (0.3860) data time 0.0008 (0.0027) model time 0.3785 (0.3835) loss 3.7302 (3.3872) grad_norm 2.1375 (inf) loss_scale 2048.0000 (2801.2049) mem 11225MB [2024-08-26 09:37:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][580/625] eta 0:00:17 lr 0.001645 wd 0.0500 time 0.3783 (0.3858) data time 0.0010 (0.0027) model time 0.3773 (0.3834) loss 3.6566 (3.3872) grad_norm 1.0767 (inf) loss_scale 2048.0000 (2788.2410) mem 11225MB [2024-08-26 09:37:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][590/625] eta 0:00:13 lr 0.001645 wd 0.0500 time 0.3733 (0.3858) data time 0.0009 (0.0027) model time 0.3724 (0.3833) loss 2.6704 (3.3809) grad_norm 1.0381 (inf) loss_scale 2048.0000 (2775.7157) mem 11225MB [2024-08-26 09:37:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][600/625] eta 0:00:09 lr 0.001645 wd 0.0500 time 0.3788 (0.3856) data time 0.0012 (0.0027) model time 0.3776 (0.3832) loss 2.8105 (3.3807) grad_norm 1.3086 (inf) loss_scale 2048.0000 (2763.6073) mem 11225MB [2024-08-26 09:37:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][610/625] eta 0:00:05 lr 0.001645 wd 0.0500 time 0.3656 (0.3854) data time 0.0008 (0.0026) model time 0.3648 (0.3830) loss 3.0899 (3.3816) grad_norm 1.1270 (inf) loss_scale 2048.0000 (2751.8953) mem 11225MB [2024-08-26 09:37:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [97/300][620/625] eta 0:00:01 lr 0.001645 wd 0.0500 time 0.3673 (0.3852) data time 0.0008 (0.0026) model time 0.3665 (0.3827) loss 3.4010 (3.3824) grad_norm 1.5446 (inf) loss_scale 2048.0000 (2740.5604) mem 11225MB [2024-08-26 09:37:18 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 97 training takes 0:04:00 [2024-08-26 09:37:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:37:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.6309 (0.6309) Acc@1 85.791 (85.791) Acc@5 97.266 (97.266) Mem 11225MB [2024-08-26 09:37:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 1.0762 (0.8002) Acc@1 75.000 (81.565) Acc@5 92.969 (96.271) Mem 11225MB [2024-08-26 09:37:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.2266 (0.9626) Acc@1 71.191 (77.741) Acc@5 91.309 (94.308) Mem 11225MB [2024-08-26 09:37:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.573 Acc@5 94.298 [2024-08-26 09:37:22 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.6% [2024-08-26 09:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.833 (0.833) Loss 0.5479 (0.5479) Acc@1 88.232 (88.232) Acc@5 98.096 (98.096) Mem 11225MB [2024-08-26 09:37:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.177) Loss 0.9155 (0.6874) Acc@1 78.027 (84.326) Acc@5 95.068 (97.159) Mem 11225MB [2024-08-26 09:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.0791 (0.8361) Acc@1 73.340 (80.550) Acc@5 92.871 (95.503) Mem 11225MB [2024-08-26 09:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.284 Acc@5 95.479 [2024-08-26 09:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.3% [2024-08-26 09:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.28% [2024-08-26 09:37:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:37:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:37:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][0/625] eta 0:08:16 lr 0.001644 wd 0.0500 time 0.7945 (0.7945) data time 0.4493 (0.4493) model time 0.0000 (0.0000) loss 3.6738 (3.6738) grad_norm 2.3780 (2.3780) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:37:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][10/625] eta 0:04:14 lr 0.001644 wd 0.0500 time 0.3894 (0.4137) data time 0.0014 (0.0419) model time 0.0000 (0.0000) loss 3.6767 (3.4310) grad_norm 1.3027 (1.4098) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][20/625] eta 0:03:59 lr 0.001644 wd 0.0500 time 0.3683 (0.3957) data time 0.0009 (0.0225) model time 0.0000 (0.0000) loss 4.1414 (3.4612) grad_norm 1.3925 (1.3244) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:37:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][30/625] eta 0:03:51 lr 0.001644 wd 0.0500 time 0.3737 (0.3895) data time 0.0014 (0.0157) model time 0.0000 (0.0000) loss 3.0765 (3.4142) grad_norm 1.2677 (1.2978) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][40/625] eta 0:03:46 lr 0.001644 wd 0.0500 time 0.3772 (0.3868) data time 0.0008 (0.0121) model time 0.0000 (0.0000) loss 3.9406 (3.4298) grad_norm 1.1002 (1.2970) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][50/625] eta 0:03:41 lr 0.001644 wd 0.0500 time 0.3788 (0.3848) data time 0.0008 (0.0100) model time 0.0000 (0.0000) loss 3.4006 (3.3798) grad_norm 1.2355 (1.2787) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:37:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][60/625] eta 0:03:38 lr 0.001644 wd 0.0500 time 0.3834 (0.3871) data time 0.0009 (0.0085) model time 0.3824 (0.3977) loss 3.9625 (3.3883) grad_norm 1.1150 (1.2563) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][70/625] eta 0:03:38 lr 0.001644 wd 0.0500 time 0.3822 (0.3943) data time 0.0008 (0.0075) model time 0.3814 (0.4175) loss 3.0587 (3.3770) grad_norm 1.5316 (1.2620) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:37:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][80/625] eta 0:03:33 lr 0.001643 wd 0.0500 time 0.3759 (0.3925) data time 0.0008 (0.0067) model time 0.3751 (0.4045) loss 4.0477 (3.3792) grad_norm 1.2233 (1.2879) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][90/625] eta 0:03:29 lr 0.001643 wd 0.0500 time 0.3747 (0.3910) data time 0.0009 (0.0061) model time 0.3738 (0.3977) loss 3.9046 (3.4187) grad_norm 1.0595 (1.2670) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][100/625] eta 0:03:25 lr 0.001643 wd 0.0500 time 0.5554 (0.3914) data time 0.0009 (0.0056) model time 0.5545 (0.3970) loss 2.3761 (3.3953) grad_norm 1.0082 (1.2772) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][110/625] eta 0:03:20 lr 0.001643 wd 0.0500 time 0.3755 (0.3897) data time 0.0008 (0.0052) model time 0.3747 (0.3926) loss 4.2486 (3.4187) grad_norm 1.4867 (1.2779) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][120/625] eta 0:03:16 lr 0.001643 wd 0.0500 time 0.3762 (0.3887) data time 0.0008 (0.0049) model time 0.3754 (0.3904) loss 4.4726 (3.4247) grad_norm 1.4906 (1.3038) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][130/625] eta 0:03:11 lr 0.001643 wd 0.0500 time 0.3785 (0.3877) data time 0.0011 (0.0046) model time 0.3774 (0.3883) loss 3.4149 (3.4324) grad_norm 1.4426 (1.3411) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][140/625] eta 0:03:07 lr 0.001643 wd 0.0500 time 0.3749 (0.3869) data time 0.0008 (0.0044) model time 0.3740 (0.3869) loss 3.4259 (3.4342) grad_norm 0.9700 (1.3467) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][150/625] eta 0:03:03 lr 0.001642 wd 0.0500 time 0.3744 (0.3862) data time 0.0012 (0.0041) model time 0.3732 (0.3857) loss 3.3476 (3.4312) grad_norm 1.0088 (1.3397) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][160/625] eta 0:02:59 lr 0.001642 wd 0.0500 time 0.3702 (0.3855) data time 0.0011 (0.0040) model time 0.3691 (0.3847) loss 3.0215 (3.4429) grad_norm 0.9504 (1.3403) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][170/625] eta 0:02:55 lr 0.001642 wd 0.0500 time 0.3824 (0.3850) data time 0.0008 (0.0038) model time 0.3815 (0.3839) loss 3.3806 (3.4371) grad_norm 1.9117 (1.3307) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][180/625] eta 0:02:51 lr 0.001642 wd 0.0500 time 0.3766 (0.3846) data time 0.0012 (0.0037) model time 0.3754 (0.3834) loss 2.4258 (3.4238) grad_norm 1.0570 (1.3241) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][190/625] eta 0:02:47 lr 0.001642 wd 0.0500 time 0.3718 (0.3842) data time 0.0009 (0.0035) model time 0.3709 (0.3828) loss 3.0156 (3.4280) grad_norm 1.1929 (1.3133) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][200/625] eta 0:02:43 lr 0.001642 wd 0.0500 time 0.3734 (0.3838) data time 0.0012 (0.0034) model time 0.3722 (0.3823) loss 3.3694 (3.4247) grad_norm 2.2917 (1.3297) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][210/625] eta 0:02:39 lr 0.001642 wd 0.0500 time 0.3807 (0.3835) data time 0.0012 (0.0033) model time 0.3795 (0.3819) loss 2.8222 (3.4212) grad_norm 0.9134 (1.3239) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][220/625] eta 0:02:35 lr 0.001641 wd 0.0500 time 0.3853 (0.3833) data time 0.0009 (0.0032) model time 0.3844 (0.3817) loss 4.0045 (3.4313) grad_norm 0.9449 (1.3140) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][230/625] eta 0:02:31 lr 0.001641 wd 0.0500 time 0.3725 (0.3830) data time 0.0008 (0.0031) model time 0.3717 (0.3813) loss 3.2625 (3.4160) grad_norm 1.7255 (1.3154) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][240/625] eta 0:02:27 lr 0.001641 wd 0.0500 time 0.3790 (0.3828) data time 0.0013 (0.0030) model time 0.3778 (0.3811) loss 2.5773 (3.4319) grad_norm 1.5622 (1.3255) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][250/625] eta 0:02:24 lr 0.001641 wd 0.0500 time 0.4242 (0.3844) data time 0.0010 (0.0029) model time 0.4232 (0.3832) loss 3.8509 (3.4228) grad_norm 2.1418 (1.3351) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][260/625] eta 0:02:20 lr 0.001641 wd 0.0500 time 0.3741 (0.3842) data time 0.0011 (0.0029) model time 0.3731 (0.3829) loss 3.7062 (3.4181) grad_norm 1.0820 (1.3292) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][270/625] eta 0:02:16 lr 0.001641 wd 0.0500 time 0.3814 (0.3840) data time 0.0010 (0.0028) model time 0.3803 (0.3826) loss 3.6029 (3.4149) grad_norm 1.2025 (1.3259) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][280/625] eta 0:02:13 lr 0.001641 wd 0.0500 time 0.3837 (0.3859) data time 0.0013 (0.0027) model time 0.3824 (0.3850) loss 3.8365 (3.4134) grad_norm 1.5387 (1.3296) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][290/625] eta 0:02:09 lr 0.001641 wd 0.0500 time 0.3781 (0.3856) data time 0.0009 (0.0027) model time 0.3772 (0.3846) loss 3.6422 (3.4106) grad_norm 1.3165 (1.3258) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][300/625] eta 0:02:05 lr 0.001640 wd 0.0500 time 0.3855 (0.3854) data time 0.0011 (0.0026) model time 0.3844 (0.3843) loss 3.0497 (3.4043) grad_norm 1.0825 (1.3225) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][310/625] eta 0:02:01 lr 0.001640 wd 0.0500 time 0.3815 (0.3851) data time 0.0008 (0.0026) model time 0.3807 (0.3840) loss 2.8625 (3.4053) grad_norm 1.1620 (1.3208) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][320/625] eta 0:01:57 lr 0.001640 wd 0.0500 time 0.3796 (0.3854) data time 0.0008 (0.0026) model time 0.3788 (0.3843) loss 4.5071 (3.4148) grad_norm 1.2075 (1.3171) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][330/625] eta 0:01:53 lr 0.001640 wd 0.0500 time 0.3780 (0.3851) data time 0.0011 (0.0025) model time 0.3769 (0.3840) loss 3.7529 (3.4220) grad_norm 1.5281 (1.3179) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][340/625] eta 0:01:49 lr 0.001640 wd 0.0500 time 0.3693 (0.3849) data time 0.0009 (0.0025) model time 0.3684 (0.3837) loss 2.8069 (3.4223) grad_norm 1.2236 (1.3230) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][350/625] eta 0:01:45 lr 0.001640 wd 0.0500 time 0.3762 (0.3846) data time 0.0011 (0.0024) model time 0.3752 (0.3834) loss 3.7679 (3.4288) grad_norm 1.2305 (1.3287) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][360/625] eta 0:01:41 lr 0.001640 wd 0.0500 time 0.3731 (0.3844) data time 0.0011 (0.0024) model time 0.3720 (0.3831) loss 3.8613 (3.4336) grad_norm 1.0683 (1.3364) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][370/625] eta 0:01:38 lr 0.001639 wd 0.0500 time 0.3794 (0.3853) data time 0.0009 (0.0024) model time 0.3785 (0.3842) loss 4.4135 (3.4382) grad_norm 0.9863 (1.3338) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][380/625] eta 0:01:34 lr 0.001639 wd 0.0500 time 0.3791 (0.3851) data time 0.0011 (0.0023) model time 0.3780 (0.3840) loss 2.1998 (3.4341) grad_norm 1.3087 (1.3292) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:39:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][390/625] eta 0:01:30 lr 0.001639 wd 0.0500 time 0.3717 (0.3849) data time 0.0011 (0.0023) model time 0.3706 (0.3838) loss 1.9859 (3.4308) grad_norm 1.3108 (1.3310) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][400/625] eta 0:01:26 lr 0.001639 wd 0.0500 time 0.3762 (0.3848) data time 0.0008 (0.0023) model time 0.3753 (0.3836) loss 3.8456 (3.4299) grad_norm 1.7136 (1.3366) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][410/625] eta 0:01:22 lr 0.001639 wd 0.0500 time 0.3755 (0.3846) data time 0.0011 (0.0022) model time 0.3744 (0.3835) loss 4.0238 (3.4315) grad_norm 1.0002 (1.3353) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][420/625] eta 0:01:18 lr 0.001639 wd 0.0500 time 0.3774 (0.3844) data time 0.0011 (0.0022) model time 0.3763 (0.3832) loss 2.8155 (3.4358) grad_norm 1.3872 (1.3358) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][430/625] eta 0:01:14 lr 0.001639 wd 0.0500 time 0.3741 (0.3842) data time 0.0012 (0.0022) model time 0.3729 (0.3830) loss 3.6959 (3.4389) grad_norm 0.9251 (1.3337) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][440/625] eta 0:01:11 lr 0.001638 wd 0.0500 time 0.3750 (0.3840) data time 0.0008 (0.0022) model time 0.3742 (0.3828) loss 3.9419 (3.4412) grad_norm 1.1436 (1.3329) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][450/625] eta 0:01:07 lr 0.001638 wd 0.0500 time 0.3716 (0.3839) data time 0.0008 (0.0021) model time 0.3707 (0.3827) loss 2.4451 (3.4429) grad_norm 1.2485 (1.3313) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][460/625] eta 0:01:03 lr 0.001638 wd 0.0500 time 0.6001 (0.3847) data time 0.0011 (0.0021) model time 0.5991 (0.3835) loss 3.3704 (3.4369) grad_norm 2.2165 (1.3351) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][470/625] eta 0:00:59 lr 0.001638 wd 0.0500 time 0.3763 (0.3845) data time 0.0010 (0.0021) model time 0.3753 (0.3833) loss 2.7326 (3.4337) grad_norm 1.4298 (1.3341) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][480/625] eta 0:00:55 lr 0.001638 wd 0.0500 time 0.3855 (0.3844) data time 0.0008 (0.0021) model time 0.3846 (0.3832) loss 3.4691 (3.4362) grad_norm 1.1244 (1.3303) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][490/625] eta 0:00:51 lr 0.001638 wd 0.0500 time 0.3751 (0.3843) data time 0.0009 (0.0021) model time 0.3743 (0.3831) loss 3.1987 (3.4320) grad_norm 1.1804 (1.3292) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][500/625] eta 0:00:48 lr 0.001638 wd 0.0500 time 0.3790 (0.3845) data time 0.0011 (0.0020) model time 0.3779 (0.3833) loss 3.3381 (3.4354) grad_norm 1.0457 (1.3265) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][510/625] eta 0:00:44 lr 0.001637 wd 0.0500 time 0.3783 (0.3844) data time 0.0008 (0.0020) model time 0.3775 (0.3832) loss 2.8309 (3.4377) grad_norm 1.4399 (1.3287) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][520/625] eta 0:00:40 lr 0.001637 wd 0.0500 time 0.3787 (0.3843) data time 0.0010 (0.0020) model time 0.3777 (0.3831) loss 3.4321 (3.4384) grad_norm 1.3685 (1.3298) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][530/625] eta 0:00:36 lr 0.001637 wd 0.0500 time 0.3777 (0.3842) data time 0.0011 (0.0020) model time 0.3766 (0.3830) loss 3.7640 (3.4393) grad_norm 1.2887 (1.3302) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][540/625] eta 0:00:32 lr 0.001637 wd 0.0500 time 0.3778 (0.3841) data time 0.0008 (0.0020) model time 0.3770 (0.3829) loss 4.5208 (3.4337) grad_norm 0.8429 (1.3270) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:40:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][550/625] eta 0:00:28 lr 0.001637 wd 0.0500 time 0.3808 (0.3840) data time 0.0010 (0.0020) model time 0.3798 (0.3827) loss 3.5409 (3.4270) grad_norm 1.6327 (1.3291) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][560/625] eta 0:00:24 lr 0.001637 wd 0.0500 time 0.3774 (0.3839) data time 0.0011 (0.0020) model time 0.3763 (0.3826) loss 3.6426 (3.4268) grad_norm 1.4121 (1.3391) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][570/625] eta 0:00:21 lr 0.001637 wd 0.0500 time 0.5904 (0.3841) data time 0.0012 (0.0019) model time 0.5892 (0.3829) loss 2.7226 (3.4275) grad_norm 1.0696 (1.3461) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][580/625] eta 0:00:17 lr 0.001637 wd 0.0500 time 0.3781 (0.3847) data time 0.0011 (0.0019) model time 0.3771 (0.3835) loss 3.6940 (3.4338) grad_norm 1.4707 (1.3467) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][590/625] eta 0:00:13 lr 0.001636 wd 0.0500 time 0.3823 (0.3849) data time 0.0008 (0.0019) model time 0.3814 (0.3838) loss 3.8825 (3.4360) grad_norm 1.1360 (1.3432) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][600/625] eta 0:00:09 lr 0.001636 wd 0.0500 time 0.3680 (0.3848) data time 0.0014 (0.0019) model time 0.3666 (0.3836) loss 3.3557 (3.4351) grad_norm 1.0009 (1.3416) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][610/625] eta 0:00:05 lr 0.001636 wd 0.0500 time 0.3687 (0.3847) data time 0.0006 (0.0019) model time 0.3681 (0.3835) loss 4.0560 (3.4403) grad_norm 1.0538 (1.3442) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [98/300][620/625] eta 0:00:01 lr 0.001636 wd 0.0500 time 0.3663 (0.3847) data time 0.0006 (0.0019) model time 0.3658 (0.3836) loss 3.5471 (3.4416) grad_norm 1.1121 (1.3441) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 98 training takes 0:04:00 [2024-08-26 09:41:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:41:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:41:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.531 (0.531) Loss 0.6621 (0.6621) Acc@1 84.766 (84.766) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 09:41:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.147) Loss 1.1074 (0.8004) Acc@1 74.365 (81.969) Acc@5 92.969 (96.209) Mem 11225MB [2024-08-26 09:41:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.2314 (0.9776) Acc@1 72.217 (77.874) Acc@5 91.113 (94.103) Mem 11225MB [2024-08-26 09:41:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.639 Acc@5 94.102 [2024-08-26 09:41:31 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.6% [2024-08-26 09:41:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.803 (0.803) Loss 0.5469 (0.5469) Acc@1 88.037 (88.037) Acc@5 98.096 (98.096) Mem 11225MB [2024-08-26 09:41:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.175) Loss 0.9141 (0.6863) Acc@1 78.076 (84.344) Acc@5 95.068 (97.168) Mem 11225MB [2024-08-26 09:41:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.136) Loss 1.0781 (0.8347) Acc@1 73.242 (80.608) Acc@5 93.018 (95.508) Mem 11225MB [2024-08-26 09:41:34 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.328 Acc@5 95.489 [2024-08-26 09:41:34 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.3% [2024-08-26 09:41:34 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.33% [2024-08-26 09:41:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:41:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:41:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][0/625] eta 0:08:30 lr 0.001636 wd 0.0500 time 0.8171 (0.8171) data time 0.4838 (0.4838) model time 0.0000 (0.0000) loss 2.3386 (2.3386) grad_norm 1.1572 (1.1572) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][10/625] eta 0:04:16 lr 0.001636 wd 0.0500 time 0.3758 (0.4166) data time 0.0011 (0.0449) model time 0.0000 (0.0000) loss 3.5007 (3.4312) grad_norm 1.0472 (1.4754) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][20/625] eta 0:04:01 lr 0.001636 wd 0.0500 time 0.3839 (0.3999) data time 0.0008 (0.0241) model time 0.0000 (0.0000) loss 1.9781 (3.4091) grad_norm 1.2198 (1.4513) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][30/625] eta 0:03:53 lr 0.001635 wd 0.0500 time 0.3792 (0.3921) data time 0.0011 (0.0167) model time 0.0000 (0.0000) loss 3.0720 (3.4595) grad_norm 1.1450 (1.3877) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][40/625] eta 0:03:47 lr 0.001635 wd 0.0500 time 0.3734 (0.3881) data time 0.0011 (0.0129) model time 0.0000 (0.0000) loss 3.5628 (3.4723) grad_norm 1.4019 (1.3824) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][50/625] eta 0:03:41 lr 0.001635 wd 0.0500 time 0.3764 (0.3858) data time 0.0008 (0.0106) model time 0.0000 (0.0000) loss 3.6116 (3.4752) grad_norm 1.0803 (1.3442) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:41:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][60/625] eta 0:03:37 lr 0.001635 wd 0.0500 time 0.3719 (0.3842) data time 0.0011 (0.0090) model time 0.3708 (0.3751) loss 3.3074 (3.4327) grad_norm 1.3688 (1.3348) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][70/625] eta 0:03:32 lr 0.001635 wd 0.0500 time 0.3849 (0.3835) data time 0.0008 (0.0079) model time 0.3841 (0.3764) loss 3.2876 (3.4092) grad_norm 1.2850 (1.3092) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][80/625] eta 0:03:28 lr 0.001635 wd 0.0500 time 0.3745 (0.3829) data time 0.0011 (0.0071) model time 0.3735 (0.3766) loss 3.5681 (3.3797) grad_norm 1.2778 (1.3185) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][90/625] eta 0:03:24 lr 0.001635 wd 0.0500 time 0.3752 (0.3823) data time 0.0008 (0.0065) model time 0.3743 (0.3765) loss 2.1652 (3.3814) grad_norm 1.2057 (1.3181) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][100/625] eta 0:03:23 lr 0.001635 wd 0.0500 time 0.3754 (0.3872) data time 0.0010 (0.0060) model time 0.3743 (0.3873) loss 3.7721 (3.3909) grad_norm 1.7535 (1.3267) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][110/625] eta 0:03:19 lr 0.001634 wd 0.0500 time 0.3814 (0.3866) data time 0.0010 (0.0055) model time 0.3803 (0.3861) loss 4.0324 (3.3951) grad_norm 1.0963 (1.3492) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][120/625] eta 0:03:15 lr 0.001634 wd 0.0500 time 0.3316 (0.3868) data time 0.0009 (0.0052) model time 0.3308 (0.3864) loss 3.4485 (3.4051) grad_norm 2.1886 (1.3433) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][130/625] eta 0:03:11 lr 0.001634 wd 0.0500 time 0.3803 (0.3862) data time 0.0010 (0.0049) model time 0.3793 (0.3853) loss 3.3984 (3.4288) grad_norm 1.3377 (1.3430) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][140/625] eta 0:03:07 lr 0.001634 wd 0.0500 time 0.3809 (0.3856) data time 0.0011 (0.0046) model time 0.3799 (0.3843) loss 3.8625 (3.4288) grad_norm 1.3320 (1.3498) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][150/625] eta 0:03:03 lr 0.001634 wd 0.0500 time 0.3800 (0.3863) data time 0.0011 (0.0044) model time 0.3789 (0.3854) loss 3.9004 (3.4521) grad_norm 1.5061 (1.3411) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][160/625] eta 0:02:59 lr 0.001634 wd 0.0500 time 0.3780 (0.3870) data time 0.0008 (0.0042) model time 0.3772 (0.3863) loss 4.0905 (3.4568) grad_norm 1.4931 (1.3317) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][170/625] eta 0:02:55 lr 0.001634 wd 0.0500 time 0.3769 (0.3863) data time 0.0011 (0.0040) model time 0.3759 (0.3854) loss 3.0250 (3.4434) grad_norm 0.9992 (1.3251) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][180/625] eta 0:02:51 lr 0.001633 wd 0.0500 time 0.3744 (0.3858) data time 0.0008 (0.0038) model time 0.3736 (0.3846) loss 3.9346 (3.4480) grad_norm 1.2351 (1.3182) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][190/625] eta 0:02:47 lr 0.001633 wd 0.0500 time 0.3743 (0.3853) data time 0.0012 (0.0037) model time 0.3731 (0.3840) loss 3.3534 (3.4455) grad_norm 2.0015 (1.3205) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][200/625] eta 0:02:43 lr 0.001633 wd 0.0500 time 0.3749 (0.3849) data time 0.0008 (0.0036) model time 0.3741 (0.3834) loss 3.3544 (3.4393) grad_norm 0.9853 (1.3233) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:42:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][210/625] eta 0:02:39 lr 0.001633 wd 0.0500 time 0.3745 (0.3846) data time 0.0009 (0.0035) model time 0.3736 (0.3830) loss 3.0963 (3.4347) grad_norm 1.0609 (1.3203) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][220/625] eta 0:02:35 lr 0.001633 wd 0.0500 time 0.3796 (0.3843) data time 0.0010 (0.0034) model time 0.3786 (0.3827) loss 3.4824 (3.4218) grad_norm 1.0889 (1.3314) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][230/625] eta 0:02:31 lr 0.001633 wd 0.0500 time 0.3864 (0.3841) data time 0.0011 (0.0033) model time 0.3853 (0.3824) loss 3.2623 (3.4136) grad_norm 0.9935 (1.3324) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][240/625] eta 0:02:28 lr 0.001633 wd 0.0500 time 0.3403 (0.3845) data time 0.0011 (0.0032) model time 0.3393 (0.3830) loss 3.6540 (3.4038) grad_norm 1.0694 (1.3262) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][250/625] eta 0:02:24 lr 0.001632 wd 0.0500 time 0.3750 (0.3843) data time 0.0012 (0.0031) model time 0.3738 (0.3828) loss 3.5863 (3.3969) grad_norm 1.0986 (1.3308) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][260/625] eta 0:02:20 lr 0.001632 wd 0.0500 time 0.3783 (0.3840) data time 0.0011 (0.0030) model time 0.3772 (0.3823) loss 3.4073 (3.3959) grad_norm 1.0467 (1.3329) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][270/625] eta 0:02:16 lr 0.001632 wd 0.0500 time 0.3747 (0.3838) data time 0.0009 (0.0029) model time 0.3738 (0.3822) loss 2.6493 (3.4007) grad_norm 1.1389 (1.3296) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][280/625] eta 0:02:12 lr 0.001632 wd 0.0500 time 0.3782 (0.3836) data time 0.0011 (0.0029) model time 0.3771 (0.3819) loss 3.8531 (3.4017) grad_norm 1.4142 (1.3246) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][290/625] eta 0:02:08 lr 0.001632 wd 0.0500 time 0.3796 (0.3834) data time 0.0010 (0.0028) model time 0.3786 (0.3818) loss 3.3579 (3.4144) grad_norm 1.0086 (1.3192) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][300/625] eta 0:02:04 lr 0.001632 wd 0.0500 time 0.3657 (0.3832) data time 0.0009 (0.0028) model time 0.3647 (0.3815) loss 4.2096 (3.4136) grad_norm 0.9328 (1.3183) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][310/625] eta 0:02:01 lr 0.001632 wd 0.0500 time 0.3719 (0.3849) data time 0.0010 (0.0027) model time 0.3709 (0.3836) loss 3.5446 (3.4038) grad_norm 1.2580 (1.3161) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][320/625] eta 0:01:57 lr 0.001631 wd 0.0500 time 0.3737 (0.3847) data time 0.0009 (0.0027) model time 0.3728 (0.3833) loss 3.9341 (3.4053) grad_norm 1.2767 (1.3234) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][330/625] eta 0:01:53 lr 0.001631 wd 0.0500 time 0.3757 (0.3845) data time 0.0011 (0.0026) model time 0.3746 (0.3831) loss 3.6287 (3.4142) grad_norm 0.9733 (1.3215) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][340/625] eta 0:01:49 lr 0.001631 wd 0.0500 time 0.3869 (0.3854) data time 0.0011 (0.0026) model time 0.3857 (0.3842) loss 3.5389 (3.4106) grad_norm 1.1221 (1.3173) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][350/625] eta 0:01:45 lr 0.001631 wd 0.0500 time 0.3734 (0.3852) data time 0.0008 (0.0025) model time 0.3726 (0.3839) loss 3.7971 (3.4007) grad_norm 1.9284 (1.3253) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][360/625] eta 0:01:42 lr 0.001631 wd 0.0500 time 0.3836 (0.3850) data time 0.0008 (0.0025) model time 0.3828 (0.3837) loss 2.6530 (3.3920) grad_norm 1.4325 (1.3242) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][370/625] eta 0:01:38 lr 0.001631 wd 0.0500 time 0.3757 (0.3848) data time 0.0011 (0.0025) model time 0.3746 (0.3835) loss 3.1098 (3.4007) grad_norm 1.4777 (1.3241) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:44:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][380/625] eta 0:01:34 lr 0.001631 wd 0.0500 time 0.3779 (0.3846) data time 0.0011 (0.0024) model time 0.3768 (0.3833) loss 2.9644 (3.3991) grad_norm 1.6410 (1.3288) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:44:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][390/625] eta 0:01:30 lr 0.001631 wd 0.0500 time 0.3780 (0.3845) data time 0.0008 (0.0024) model time 0.3771 (0.3831) loss 4.3863 (3.3973) grad_norm 1.7237 (1.3305) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:44:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][400/625] eta 0:01:26 lr 0.001630 wd 0.0500 time 0.3797 (0.3843) data time 0.0011 (0.0024) model time 0.3786 (0.3829) loss 3.2742 (3.3938) grad_norm 1.5593 (1.3307) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:44:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][410/625] eta 0:01:22 lr 0.001630 wd 0.0500 time 0.3806 (0.3841) data time 0.0010 (0.0023) model time 0.3796 (0.3827) loss 3.1852 (3.3923) grad_norm 1.2598 (1.3320) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:44:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][420/625] eta 0:01:18 lr 0.001630 wd 0.0500 time 0.3817 (0.3840) data time 0.0010 (0.0023) model time 0.3808 (0.3826) loss 3.6614 (3.3870) grad_norm 1.6042 (1.3296) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 09:44:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][430/625] eta 0:01:14 lr 0.001630 wd 0.0500 time 0.3784 (0.3838) data time 0.0012 (0.0023) model time 0.3772 (0.3823) loss 3.7187 (3.3932) grad_norm 1.1551 (inf) loss_scale 1024.0000 (2038.4965) mem 11225MB [2024-08-26 09:44:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][440/625] eta 0:01:10 lr 0.001630 wd 0.0500 time 0.3781 (0.3836) data time 0.0012 (0.0023) model time 0.3769 (0.3822) loss 3.5560 (3.3917) grad_norm 1.3907 (inf) loss_scale 1024.0000 (2015.4921) mem 11225MB [2024-08-26 09:44:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][450/625] eta 0:01:07 lr 0.001630 wd 0.0500 time 0.3768 (0.3839) data time 0.0010 (0.0023) model time 0.3757 (0.3825) loss 3.5739 (3.3894) grad_norm 1.1711 (inf) loss_scale 1024.0000 (1993.5078) mem 11225MB [2024-08-26 09:44:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][460/625] eta 0:01:03 lr 0.001630 wd 0.0500 time 0.3783 (0.3846) data time 0.0011 (0.0022) model time 0.3772 (0.3833) loss 3.4673 (3.3919) grad_norm 1.2193 (inf) loss_scale 1024.0000 (1972.4772) mem 11225MB [2024-08-26 09:44:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][470/625] eta 0:00:59 lr 0.001629 wd 0.0500 time 0.3761 (0.3845) data time 0.0009 (0.0022) model time 0.3752 (0.3831) loss 3.5090 (3.3874) grad_norm 1.5571 (inf) loss_scale 1024.0000 (1952.3397) mem 11225MB [2024-08-26 09:44:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][480/625] eta 0:00:55 lr 0.001629 wd 0.0500 time 0.3703 (0.3843) data time 0.0011 (0.0022) model time 0.3692 (0.3829) loss 3.4754 (3.3807) grad_norm 1.5002 (inf) loss_scale 1024.0000 (1933.0395) mem 11225MB [2024-08-26 09:44:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][490/625] eta 0:00:52 lr 0.001629 wd 0.0500 time 0.3696 (0.3853) data time 0.0011 (0.0022) model time 0.3685 (0.3841) loss 3.1866 (3.3821) grad_norm 2.3216 (inf) loss_scale 1024.0000 (1914.5255) mem 11225MB [2024-08-26 09:44:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][500/625] eta 0:00:48 lr 0.001629 wd 0.0500 time 0.3768 (0.3852) data time 0.0012 (0.0022) model time 0.3756 (0.3839) loss 3.7202 (3.3853) grad_norm 1.5773 (inf) loss_scale 1024.0000 (1896.7505) mem 11225MB [2024-08-26 09:44:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][510/625] eta 0:00:44 lr 0.001629 wd 0.0500 time 0.3780 (0.3850) data time 0.0012 (0.0021) model time 0.3768 (0.3837) loss 3.1355 (3.3850) grad_norm 1.2048 (inf) loss_scale 1024.0000 (1879.6712) mem 11225MB [2024-08-26 09:44:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][520/625] eta 0:00:40 lr 0.001629 wd 0.0500 time 0.3734 (0.3849) data time 0.0011 (0.0021) model time 0.3722 (0.3836) loss 2.7112 (3.3856) grad_norm 1.0678 (inf) loss_scale 1024.0000 (1863.2476) mem 11225MB [2024-08-26 09:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][530/625] eta 0:00:36 lr 0.001629 wd 0.0500 time 0.3865 (0.3848) data time 0.0009 (0.0021) model time 0.3856 (0.3834) loss 3.4950 (3.3834) grad_norm 1.1017 (inf) loss_scale 1024.0000 (1847.4426) mem 11225MB [2024-08-26 09:45:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][540/625] eta 0:00:32 lr 0.001628 wd 0.0500 time 0.3763 (0.3846) data time 0.0011 (0.0021) model time 0.3752 (0.3833) loss 3.6034 (3.3809) grad_norm 1.0689 (inf) loss_scale 1024.0000 (1832.2218) mem 11225MB [2024-08-26 09:45:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][550/625] eta 0:00:28 lr 0.001628 wd 0.0500 time 0.3734 (0.3845) data time 0.0008 (0.0021) model time 0.3726 (0.3832) loss 3.9586 (3.3793) grad_norm 1.3073 (inf) loss_scale 1024.0000 (1817.5535) mem 11225MB [2024-08-26 09:45:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][560/625] eta 0:00:24 lr 0.001628 wd 0.0500 time 0.3781 (0.3844) data time 0.0011 (0.0020) model time 0.3771 (0.3830) loss 3.5856 (3.3824) grad_norm 1.0724 (inf) loss_scale 1024.0000 (1803.4082) mem 11225MB [2024-08-26 09:45:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][570/625] eta 0:00:21 lr 0.001628 wd 0.0500 time 0.3756 (0.3842) data time 0.0011 (0.0020) model time 0.3745 (0.3829) loss 3.9623 (3.3839) grad_norm 1.6647 (inf) loss_scale 1024.0000 (1789.7583) mem 11225MB [2024-08-26 09:45:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][580/625] eta 0:00:17 lr 0.001628 wd 0.0500 time 0.3743 (0.3841) data time 0.0008 (0.0020) model time 0.3735 (0.3828) loss 4.0932 (3.3855) grad_norm 1.5189 (inf) loss_scale 1024.0000 (1776.5783) mem 11225MB [2024-08-26 09:45:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][590/625] eta 0:00:13 lr 0.001628 wd 0.0500 time 0.3722 (0.3840) data time 0.0011 (0.0020) model time 0.3711 (0.3826) loss 3.5894 (3.3863) grad_norm 1.3514 (inf) loss_scale 1024.0000 (1763.8443) mem 11225MB [2024-08-26 09:45:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][600/625] eta 0:00:09 lr 0.001628 wd 0.0500 time 0.3786 (0.3838) data time 0.0011 (0.0020) model time 0.3776 (0.3825) loss 3.0627 (3.3900) grad_norm 2.1432 (inf) loss_scale 1024.0000 (1751.5341) mem 11225MB [2024-08-26 09:45:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][610/625] eta 0:00:05 lr 0.001627 wd 0.0500 time 0.3680 (0.3847) data time 0.0008 (0.0020) model time 0.3672 (0.3834) loss 3.8005 (3.3960) grad_norm 1.7037 (inf) loss_scale 1024.0000 (1739.6268) mem 11225MB [2024-08-26 09:45:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [99/300][620/625] eta 0:00:01 lr 0.001627 wd 0.0500 time 0.3683 (0.3844) data time 0.0005 (0.0020) model time 0.3677 (0.3831) loss 4.2889 (3.3945) grad_norm 1.2673 (inf) loss_scale 1024.0000 (1728.1031) mem 11225MB [2024-08-26 09:45:36 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 99 training takes 0:04:00 [2024-08-26 09:45:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:45:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:45:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.574 (0.574) Loss 0.6201 (0.6201) Acc@1 85.938 (85.938) Acc@5 97.803 (97.803) Mem 11225MB [2024-08-26 09:45:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.148) Loss 1.0898 (0.7928) Acc@1 73.828 (81.880) Acc@5 93.408 (96.453) Mem 11225MB [2024-08-26 09:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.2217 (0.9625) Acc@1 71.777 (77.983) Acc@5 90.527 (94.189) Mem 11225MB [2024-08-26 09:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.785 Acc@5 94.158 [2024-08-26 09:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.8% [2024-08-26 09:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 77.79% [2024-08-26 09:45:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 09:45:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 09:45:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.544 (0.544) Loss 0.5474 (0.5474) Acc@1 87.988 (87.988) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 09:45:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.148) Loss 0.9111 (0.6846) Acc@1 78.223 (84.393) Acc@5 95.068 (97.190) Mem 11225MB [2024-08-26 09:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.0762 (0.8331) Acc@1 73.193 (80.604) Acc@5 93.018 (95.552) Mem 11225MB [2024-08-26 09:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.330 Acc@5 95.529 [2024-08-26 09:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.3% [2024-08-26 09:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.33% [2024-08-26 09:45:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:45:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:45:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][0/625] eta 0:07:46 lr 0.001627 wd 0.0500 time 0.7462 (0.7462) data time 0.3947 (0.3947) model time 0.0000 (0.0000) loss 4.2633 (4.2633) grad_norm 0.9912 (0.9912) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:45:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][10/625] eta 0:04:11 lr 0.001627 wd 0.0500 time 0.3840 (0.4090) data time 0.0011 (0.0370) model time 0.0000 (0.0000) loss 2.5334 (3.6122) grad_norm 2.0300 (1.4120) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:45:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][20/625] eta 0:04:03 lr 0.001627 wd 0.0500 time 0.3687 (0.4020) data time 0.0011 (0.0200) model time 0.0000 (0.0000) loss 3.6853 (3.3370) grad_norm 1.1241 (1.3976) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:45:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][30/625] eta 0:03:58 lr 0.001627 wd 0.0500 time 0.3771 (0.4003) data time 0.0009 (0.0139) model time 0.0000 (0.0000) loss 4.2525 (3.4299) grad_norm 1.1842 (1.4086) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][40/625] eta 0:03:51 lr 0.001627 wd 0.0500 time 0.4115 (0.3956) data time 0.0011 (0.0108) model time 0.0000 (0.0000) loss 2.7881 (3.2998) grad_norm 1.0391 (1.3714) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][50/625] eta 0:03:45 lr 0.001627 wd 0.0500 time 0.3802 (0.3921) data time 0.0009 (0.0089) model time 0.0000 (0.0000) loss 2.4111 (3.3460) grad_norm 1.1357 (1.3833) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][60/625] eta 0:03:40 lr 0.001626 wd 0.0500 time 0.3733 (0.3895) data time 0.0011 (0.0076) model time 0.3722 (0.3749) loss 2.9884 (3.3824) grad_norm 0.9969 (1.3941) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][70/625] eta 0:03:35 lr 0.001626 wd 0.0500 time 0.3776 (0.3876) data time 0.0008 (0.0067) model time 0.3768 (0.3748) loss 2.7963 (3.3929) grad_norm 1.1757 (1.3785) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][80/625] eta 0:03:31 lr 0.001626 wd 0.0500 time 0.3765 (0.3883) data time 0.0009 (0.0060) model time 0.3756 (0.3807) loss 2.8727 (3.3731) grad_norm 1.2481 (1.3926) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][90/625] eta 0:03:27 lr 0.001626 wd 0.0500 time 0.3750 (0.3870) data time 0.0008 (0.0055) model time 0.3742 (0.3793) loss 3.3301 (3.3745) grad_norm 1.4420 (1.3619) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][100/625] eta 0:03:22 lr 0.001626 wd 0.0500 time 0.3760 (0.3858) data time 0.0011 (0.0051) model time 0.3750 (0.3783) loss 3.6795 (3.3941) grad_norm 1.4756 (1.3770) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][110/625] eta 0:03:18 lr 0.001626 wd 0.0500 time 0.3775 (0.3850) data time 0.0010 (0.0048) model time 0.3764 (0.3777) loss 3.6624 (3.4082) grad_norm 0.8594 (1.3911) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][120/625] eta 0:03:14 lr 0.001626 wd 0.0500 time 0.3740 (0.3842) data time 0.0011 (0.0045) model time 0.3728 (0.3773) loss 3.0383 (3.3793) grad_norm 1.3749 (1.3866) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][130/625] eta 0:03:09 lr 0.001625 wd 0.0500 time 0.3733 (0.3836) data time 0.0011 (0.0042) model time 0.3722 (0.3770) loss 3.7547 (3.3795) grad_norm 2.1555 (1.3881) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][140/625] eta 0:03:05 lr 0.001625 wd 0.0500 time 0.3764 (0.3830) data time 0.0009 (0.0040) model time 0.3755 (0.3766) loss 2.6753 (3.3832) grad_norm 1.3002 (1.3827) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][150/625] eta 0:03:01 lr 0.001625 wd 0.0500 time 0.3730 (0.3825) data time 0.0011 (0.0038) model time 0.3720 (0.3764) loss 3.6018 (3.3640) grad_norm 1.3790 (1.3614) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][160/625] eta 0:02:57 lr 0.001625 wd 0.0500 time 0.3772 (0.3822) data time 0.0008 (0.0036) model time 0.3764 (0.3764) loss 2.5218 (3.3383) grad_norm 2.3776 (1.3685) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][170/625] eta 0:02:55 lr 0.001625 wd 0.0500 time 0.3744 (0.3849) data time 0.0008 (0.0035) model time 0.3736 (0.3807) loss 2.8511 (3.3585) grad_norm 1.3250 (1.3626) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][180/625] eta 0:02:51 lr 0.001625 wd 0.0500 time 0.3727 (0.3845) data time 0.0011 (0.0034) model time 0.3716 (0.3804) loss 2.9674 (3.3552) grad_norm 0.9617 (1.3529) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][190/625] eta 0:02:47 lr 0.001625 wd 0.0500 time 0.3842 (0.3842) data time 0.0011 (0.0032) model time 0.3832 (0.3801) loss 3.1814 (3.3500) grad_norm 1.0083 (1.3580) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][200/625] eta 0:02:43 lr 0.001624 wd 0.0500 time 0.3741 (0.3838) data time 0.0009 (0.0031) model time 0.3732 (0.3799) loss 2.5562 (3.3375) grad_norm 1.3384 (1.3498) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][210/625] eta 0:02:39 lr 0.001624 wd 0.0500 time 0.3846 (0.3836) data time 0.0008 (0.0030) model time 0.3837 (0.3798) loss 2.4792 (3.3318) grad_norm 1.0615 (1.3542) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][220/625] eta 0:02:35 lr 0.001624 wd 0.0500 time 0.3725 (0.3834) data time 0.0008 (0.0030) model time 0.3717 (0.3797) loss 3.7026 (3.3324) grad_norm 1.3583 (1.3567) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][230/625] eta 0:02:31 lr 0.001624 wd 0.0500 time 0.5824 (0.3840) data time 0.0011 (0.0029) model time 0.5814 (0.3806) loss 3.5298 (3.3442) grad_norm 1.7606 (1.3705) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][240/625] eta 0:02:28 lr 0.001624 wd 0.0500 time 0.3761 (0.3846) data time 0.0011 (0.0028) model time 0.3751 (0.3815) loss 3.2560 (3.3359) grad_norm 0.9618 (1.3671) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][250/625] eta 0:02:24 lr 0.001624 wd 0.0500 time 0.3911 (0.3843) data time 0.0011 (0.0028) model time 0.3900 (0.3812) loss 3.0193 (3.3328) grad_norm 1.5953 (1.3707) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][260/625] eta 0:02:20 lr 0.001624 wd 0.0500 time 0.3689 (0.3841) data time 0.0011 (0.0027) model time 0.3678 (0.3809) loss 3.0031 (3.3322) grad_norm 0.9216 (1.3681) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][270/625] eta 0:02:16 lr 0.001623 wd 0.0500 time 0.3810 (0.3838) data time 0.0011 (0.0026) model time 0.3800 (0.3806) loss 3.6760 (3.3258) grad_norm 1.3821 (1.3750) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][280/625] eta 0:02:12 lr 0.001623 wd 0.0500 time 0.3843 (0.3836) data time 0.0010 (0.0026) model time 0.3832 (0.3805) loss 3.6127 (3.3203) grad_norm 1.0874 (1.3739) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][290/625] eta 0:02:09 lr 0.001623 wd 0.0500 time 0.3786 (0.3862) data time 0.0011 (0.0025) model time 0.3776 (0.3837) loss 3.5247 (3.3251) grad_norm 1.0396 (1.3667) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][300/625] eta 0:02:05 lr 0.001623 wd 0.0500 time 0.3818 (0.3859) data time 0.0009 (0.0025) model time 0.3810 (0.3834) loss 3.8646 (3.3287) grad_norm 1.1448 (1.3629) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][310/625] eta 0:02:01 lr 0.001623 wd 0.0500 time 0.3775 (0.3856) data time 0.0008 (0.0025) model time 0.3767 (0.3832) loss 2.5674 (3.3329) grad_norm 1.1142 (1.3622) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][320/625] eta 0:01:57 lr 0.001623 wd 0.0500 time 0.3758 (0.3854) data time 0.0009 (0.0024) model time 0.3750 (0.3830) loss 3.0211 (3.3367) grad_norm 1.6248 (1.3674) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][330/625] eta 0:01:53 lr 0.001623 wd 0.0500 time 0.3705 (0.3852) data time 0.0008 (0.0024) model time 0.3697 (0.3827) loss 2.6466 (3.3268) grad_norm 1.3902 (1.3619) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:47:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][340/625] eta 0:01:49 lr 0.001623 wd 0.0500 time 0.3715 (0.3849) data time 0.0008 (0.0023) model time 0.3707 (0.3825) loss 4.1208 (3.3284) grad_norm 1.3685 (1.3587) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][350/625] eta 0:01:45 lr 0.001622 wd 0.0500 time 0.3802 (0.3847) data time 0.0011 (0.0023) model time 0.3791 (0.3822) loss 3.3429 (3.3275) grad_norm 1.1461 (1.3654) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][360/625] eta 0:01:41 lr 0.001622 wd 0.0500 time 0.3698 (0.3845) data time 0.0011 (0.0023) model time 0.3687 (0.3821) loss 3.8373 (3.3293) grad_norm 1.3800 (1.3693) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][370/625] eta 0:01:37 lr 0.001622 wd 0.0500 time 0.3753 (0.3843) data time 0.0009 (0.0022) model time 0.3744 (0.3819) loss 3.2021 (3.3312) grad_norm 1.2102 (1.3667) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][380/625] eta 0:01:34 lr 0.001622 wd 0.0500 time 0.3723 (0.3841) data time 0.0011 (0.0022) model time 0.3712 (0.3817) loss 2.8188 (3.3307) grad_norm 1.4289 (1.3773) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][390/625] eta 0:01:30 lr 0.001622 wd 0.0500 time 0.3762 (0.3839) data time 0.0008 (0.0022) model time 0.3754 (0.3815) loss 3.6873 (3.3340) grad_norm 1.4544 (1.3767) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][400/625] eta 0:01:26 lr 0.001622 wd 0.0500 time 0.3716 (0.3837) data time 0.0008 (0.0022) model time 0.3707 (0.3814) loss 2.8621 (3.3317) grad_norm 1.2663 (1.3745) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][410/625] eta 0:01:22 lr 0.001622 wd 0.0500 time 0.3765 (0.3836) data time 0.0012 (0.0021) model time 0.3753 (0.3812) loss 3.7313 (3.3277) grad_norm 0.9756 (1.3710) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][420/625] eta 0:01:18 lr 0.001621 wd 0.0500 time 0.3728 (0.3849) data time 0.0012 (0.0021) model time 0.3716 (0.3827) loss 2.7887 (3.3268) grad_norm 1.1370 (1.3699) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][430/625] eta 0:01:15 lr 0.001621 wd 0.0500 time 0.3768 (0.3847) data time 0.0012 (0.0021) model time 0.3756 (0.3825) loss 2.5097 (3.3223) grad_norm 1.2589 (1.3747) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][440/625] eta 0:01:11 lr 0.001621 wd 0.0500 time 0.3756 (0.3845) data time 0.0011 (0.0021) model time 0.3745 (0.3824) loss 3.7334 (3.3311) grad_norm 1.1647 (1.3701) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][450/625] eta 0:01:07 lr 0.001621 wd 0.0500 time 0.3726 (0.3844) data time 0.0008 (0.0020) model time 0.3718 (0.3822) loss 3.3059 (3.3310) grad_norm 1.0271 (1.3636) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][460/625] eta 0:01:03 lr 0.001621 wd 0.0500 time 0.3754 (0.3842) data time 0.0009 (0.0020) model time 0.3746 (0.3820) loss 2.3034 (3.3347) grad_norm 1.3376 (1.3629) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][470/625] eta 0:00:59 lr 0.001621 wd 0.0500 time 0.6207 (0.3845) data time 0.0009 (0.0020) model time 0.6198 (0.3825) loss 3.7904 (3.3407) grad_norm 0.9421 (1.3695) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][480/625] eta 0:00:55 lr 0.001621 wd 0.0500 time 0.3727 (0.3843) data time 0.0011 (0.0020) model time 0.3716 (0.3822) loss 2.7055 (3.3435) grad_norm 1.3294 (1.3724) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][490/625] eta 0:00:51 lr 0.001620 wd 0.0500 time 0.3733 (0.3841) data time 0.0010 (0.0020) model time 0.3723 (0.3820) loss 3.5797 (3.3422) grad_norm 1.1077 (1.3714) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:48:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][500/625] eta 0:00:48 lr 0.001620 wd 0.0500 time 0.3721 (0.3851) data time 0.0008 (0.0020) model time 0.3712 (0.3832) loss 2.7028 (3.3429) grad_norm 1.2738 (1.3700) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][510/625] eta 0:00:44 lr 0.001620 wd 0.0500 time 0.3769 (0.3849) data time 0.0011 (0.0019) model time 0.3757 (0.3830) loss 3.5703 (3.3456) grad_norm 1.4115 (1.3688) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][520/625] eta 0:00:40 lr 0.001620 wd 0.0500 time 0.3765 (0.3848) data time 0.0008 (0.0019) model time 0.3757 (0.3828) loss 4.0696 (3.3460) grad_norm 1.1972 (1.3672) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][530/625] eta 0:00:36 lr 0.001620 wd 0.0500 time 0.3767 (0.3846) data time 0.0008 (0.0019) model time 0.3759 (0.3826) loss 3.9246 (3.3492) grad_norm 1.1027 (1.3658) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][540/625] eta 0:00:32 lr 0.001620 wd 0.0500 time 0.3751 (0.3856) data time 0.0008 (0.0019) model time 0.3742 (0.3837) loss 4.1407 (3.3556) grad_norm 1.3952 (1.3681) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][550/625] eta 0:00:28 lr 0.001620 wd 0.0500 time 0.3779 (0.3854) data time 0.0008 (0.0019) model time 0.3772 (0.3836) loss 4.6791 (3.3603) grad_norm 1.5607 (1.3702) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][560/625] eta 0:00:25 lr 0.001619 wd 0.0500 time 0.3744 (0.3853) data time 0.0011 (0.0019) model time 0.3733 (0.3834) loss 3.8249 (3.3608) grad_norm 1.0958 (1.3675) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][570/625] eta 0:00:21 lr 0.001619 wd 0.0500 time 0.3782 (0.3852) data time 0.0011 (0.0019) model time 0.3771 (0.3833) loss 2.6323 (3.3536) grad_norm 1.5777 (1.3664) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][580/625] eta 0:00:17 lr 0.001619 wd 0.0500 time 0.3856 (0.3850) data time 0.0009 (0.0018) model time 0.3848 (0.3832) loss 3.9399 (3.3537) grad_norm 1.2772 (1.3647) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][590/625] eta 0:00:13 lr 0.001619 wd 0.0500 time 0.3358 (0.3852) data time 0.0009 (0.0018) model time 0.3350 (0.3834) loss 3.9286 (3.3511) grad_norm 1.5103 (1.3658) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][600/625] eta 0:00:09 lr 0.001619 wd 0.0500 time 0.3775 (0.3850) data time 0.0013 (0.0018) model time 0.3762 (0.3832) loss 2.8879 (3.3491) grad_norm 1.4226 (1.3660) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][610/625] eta 0:00:05 lr 0.001619 wd 0.0500 time 0.3745 (0.3849) data time 0.0008 (0.0018) model time 0.3737 (0.3831) loss 3.0717 (3.3432) grad_norm 1.3721 (1.3652) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [100/300][620/625] eta 0:00:01 lr 0.001619 wd 0.0500 time 0.3639 (0.3847) data time 0.0006 (0.0018) model time 0.3633 (0.3828) loss 3.3739 (3.3414) grad_norm 1.2551 (1.3634) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:45 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 100 training takes 0:04:00 [2024-08-26 09:49:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:49:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.566 (0.566) Loss 0.6426 (0.6426) Acc@1 86.572 (86.572) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 09:49:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.150) Loss 1.0771 (0.8225) Acc@1 74.951 (81.530) Acc@5 93.701 (96.351) Mem 11225MB [2024-08-26 09:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.124) Loss 1.1992 (0.9882) Acc@1 72.803 (77.888) Acc@5 91.943 (94.380) Mem 11225MB [2024-08-26 09:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.581 Acc@5 94.344 [2024-08-26 09:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.6% [2024-08-26 09:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.825 (0.825) Loss 0.5469 (0.5469) Acc@1 87.939 (87.939) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 09:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.180) Loss 0.9116 (0.6836) Acc@1 78.027 (84.393) Acc@5 95.166 (97.217) Mem 11225MB [2024-08-26 09:49:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.139) Loss 1.0752 (0.8321) Acc@1 73.193 (80.629) Acc@5 92.920 (95.573) Mem 11225MB [2024-08-26 09:49:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.356 Acc@5 95.559 [2024-08-26 09:49:53 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.4% [2024-08-26 09:49:53 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.36% [2024-08-26 09:49:53 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:49:54 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:49:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][0/625] eta 0:08:10 lr 0.001619 wd 0.0500 time 0.7848 (0.7848) data time 0.4430 (0.4430) model time 0.0000 (0.0000) loss 3.0571 (3.0571) grad_norm 0.9966 (0.9966) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:49:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][10/625] eta 0:04:17 lr 0.001618 wd 0.0500 time 0.3886 (0.4179) data time 0.0008 (0.0413) model time 0.0000 (0.0000) loss 3.9478 (3.3092) grad_norm 0.8105 (1.1584) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][20/625] eta 0:04:01 lr 0.001618 wd 0.0500 time 0.3730 (0.3994) data time 0.0012 (0.0222) model time 0.0000 (0.0000) loss 3.6948 (3.3476) grad_norm 1.2908 (1.1392) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][30/625] eta 0:03:53 lr 0.001618 wd 0.0500 time 0.3768 (0.3918) data time 0.0010 (0.0154) model time 0.0000 (0.0000) loss 3.2672 (3.3588) grad_norm 1.2098 (1.1658) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][40/625] eta 0:03:47 lr 0.001618 wd 0.0500 time 0.3798 (0.3885) data time 0.0011 (0.0119) model time 0.0000 (0.0000) loss 3.4666 (3.3379) grad_norm 1.1212 (1.1747) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][50/625] eta 0:03:42 lr 0.001618 wd 0.0500 time 0.3818 (0.3866) data time 0.0008 (0.0098) model time 0.0000 (0.0000) loss 3.3805 (3.3716) grad_norm 1.3995 (1.1987) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][60/625] eta 0:03:38 lr 0.001618 wd 0.0500 time 0.3772 (0.3872) data time 0.0009 (0.0084) model time 0.3764 (0.3890) loss 3.2632 (3.3472) grad_norm 1.0194 (1.2107) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][70/625] eta 0:03:40 lr 0.001618 wd 0.0500 time 0.3791 (0.3974) data time 0.0012 (0.0074) model time 0.3779 (0.4238) loss 3.6116 (3.3584) grad_norm 1.4797 (1.2167) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][80/625] eta 0:03:36 lr 0.001617 wd 0.0500 time 0.3747 (0.3970) data time 0.0008 (0.0066) model time 0.3739 (0.4135) loss 3.6253 (3.3738) grad_norm 3.2559 (1.2521) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][90/625] eta 0:03:31 lr 0.001617 wd 0.0500 time 0.3763 (0.3947) data time 0.0011 (0.0060) model time 0.3752 (0.4039) loss 3.9139 (3.3670) grad_norm 1.1431 (1.2749) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][100/625] eta 0:03:26 lr 0.001617 wd 0.0500 time 0.3906 (0.3930) data time 0.0010 (0.0055) model time 0.3896 (0.3984) loss 3.5818 (3.3701) grad_norm 1.2961 (1.2966) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][110/625] eta 0:03:21 lr 0.001617 wd 0.0500 time 0.3735 (0.3915) data time 0.0009 (0.0051) model time 0.3727 (0.3946) loss 2.3585 (3.3639) grad_norm 1.1635 (1.2976) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][120/625] eta 0:03:17 lr 0.001617 wd 0.0500 time 0.3828 (0.3913) data time 0.0011 (0.0048) model time 0.3817 (0.3936) loss 2.5215 (3.3522) grad_norm 0.9558 (1.2896) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][130/625] eta 0:03:13 lr 0.001617 wd 0.0500 time 0.3807 (0.3903) data time 0.0012 (0.0045) model time 0.3795 (0.3915) loss 2.4124 (3.3531) grad_norm 1.3290 (1.2829) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][140/625] eta 0:03:08 lr 0.001617 wd 0.0500 time 0.3787 (0.3896) data time 0.0011 (0.0043) model time 0.3776 (0.3901) loss 3.6479 (3.3631) grad_norm 1.4481 (1.2776) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][150/625] eta 0:03:04 lr 0.001616 wd 0.0500 time 0.3754 (0.3888) data time 0.0012 (0.0041) model time 0.3742 (0.3888) loss 3.6458 (3.3688) grad_norm 1.2938 (1.2861) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:50:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][160/625] eta 0:03:00 lr 0.001616 wd 0.0500 time 0.3756 (0.3880) data time 0.0011 (0.0039) model time 0.3744 (0.3875) loss 3.3696 (3.3722) grad_norm 1.1873 (1.2766) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][170/625] eta 0:02:56 lr 0.001616 wd 0.0500 time 0.3735 (0.3874) data time 0.0009 (0.0037) model time 0.3726 (0.3866) loss 2.1656 (3.3658) grad_norm 1.1280 (1.2779) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][180/625] eta 0:02:52 lr 0.001616 wd 0.0500 time 0.3782 (0.3869) data time 0.0011 (0.0036) model time 0.3771 (0.3859) loss 3.6946 (3.3756) grad_norm 1.0347 (1.2878) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][190/625] eta 0:02:48 lr 0.001616 wd 0.0500 time 0.3800 (0.3865) data time 0.0009 (0.0035) model time 0.3791 (0.3854) loss 3.3339 (3.3730) grad_norm 1.1155 (1.2929) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][200/625] eta 0:02:44 lr 0.001616 wd 0.0500 time 0.3741 (0.3860) data time 0.0008 (0.0034) model time 0.3732 (0.3847) loss 4.2476 (3.3775) grad_norm 1.5081 (1.2948) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][210/625] eta 0:02:40 lr 0.001616 wd 0.0500 time 0.3795 (0.3856) data time 0.0009 (0.0032) model time 0.3787 (0.3841) loss 4.0840 (3.3836) grad_norm 1.4169 (1.2897) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][220/625] eta 0:02:36 lr 0.001615 wd 0.0500 time 0.3730 (0.3853) data time 0.0008 (0.0032) model time 0.3723 (0.3837) loss 4.5720 (3.3849) grad_norm 1.7301 (1.2951) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][230/625] eta 0:02:32 lr 0.001615 wd 0.0500 time 0.3728 (0.3849) data time 0.0011 (0.0031) model time 0.3716 (0.3833) loss 3.0938 (3.3844) grad_norm 1.8347 (1.3050) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][240/625] eta 0:02:28 lr 0.001615 wd 0.0500 time 0.5309 (0.3853) data time 0.0011 (0.0030) model time 0.5299 (0.3838) loss 2.4819 (3.3740) grad_norm 1.2990 (1.3060) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][250/625] eta 0:02:24 lr 0.001615 wd 0.0500 time 0.3814 (0.3865) data time 0.0011 (0.0029) model time 0.3803 (0.3854) loss 3.6053 (3.3766) grad_norm 1.6809 (1.3091) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][260/625] eta 0:02:20 lr 0.001615 wd 0.0500 time 0.3780 (0.3862) data time 0.0010 (0.0029) model time 0.3770 (0.3850) loss 3.1619 (3.3789) grad_norm 1.0952 (1.3126) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][270/625] eta 0:02:16 lr 0.001615 wd 0.0500 time 0.3778 (0.3858) data time 0.0008 (0.0028) model time 0.3770 (0.3845) loss 2.5657 (3.3827) grad_norm 1.3920 (1.3111) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][280/625] eta 0:02:13 lr 0.001615 wd 0.0500 time 0.5930 (0.3872) data time 0.0008 (0.0027) model time 0.5922 (0.3862) loss 3.1075 (3.3853) grad_norm 1.4894 (1.3150) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][290/625] eta 0:02:09 lr 0.001614 wd 0.0500 time 0.3776 (0.3875) data time 0.0008 (0.0027) model time 0.3768 (0.3866) loss 3.0890 (3.3833) grad_norm 1.1553 (1.3221) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][300/625] eta 0:02:06 lr 0.001614 wd 0.0500 time 0.3406 (0.3877) data time 0.0011 (0.0026) model time 0.3396 (0.3868) loss 2.3546 (3.3790) grad_norm 1.3870 (1.3170) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][310/625] eta 0:02:02 lr 0.001614 wd 0.0500 time 0.3795 (0.3873) data time 0.0010 (0.0026) model time 0.3785 (0.3864) loss 3.8283 (3.3800) grad_norm 1.2195 (1.3216) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:51:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][320/625] eta 0:01:58 lr 0.001614 wd 0.0500 time 0.3776 (0.3871) data time 0.0011 (0.0025) model time 0.3765 (0.3860) loss 3.7215 (3.3787) grad_norm 1.2791 (1.3271) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][330/625] eta 0:01:54 lr 0.001614 wd 0.0500 time 0.3906 (0.3868) data time 0.0011 (0.0025) model time 0.3895 (0.3857) loss 3.8627 (3.3730) grad_norm 1.1785 (1.3260) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][340/625] eta 0:01:50 lr 0.001614 wd 0.0500 time 0.3845 (0.3866) data time 0.0012 (0.0024) model time 0.3833 (0.3855) loss 2.9820 (3.3670) grad_norm 1.5620 (1.3308) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][350/625] eta 0:01:46 lr 0.001614 wd 0.0500 time 0.3770 (0.3864) data time 0.0010 (0.0024) model time 0.3761 (0.3852) loss 4.0894 (3.3758) grad_norm 1.5795 (1.3333) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][360/625] eta 0:01:42 lr 0.001613 wd 0.0500 time 0.5483 (0.3867) data time 0.0010 (0.0024) model time 0.5473 (0.3856) loss 2.7756 (3.3773) grad_norm 1.7422 (1.3332) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][370/625] eta 0:01:38 lr 0.001613 wd 0.0500 time 0.3748 (0.3876) data time 0.0012 (0.0023) model time 0.3736 (0.3866) loss 3.7621 (3.3732) grad_norm 1.1157 (1.3340) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][380/625] eta 0:01:34 lr 0.001613 wd 0.0500 time 0.3715 (0.3873) data time 0.0008 (0.0023) model time 0.3706 (0.3863) loss 3.3258 (3.3829) grad_norm 1.1695 (1.3351) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][390/625] eta 0:01:30 lr 0.001613 wd 0.0500 time 0.3810 (0.3871) data time 0.0011 (0.0023) model time 0.3800 (0.3860) loss 2.3111 (3.3840) grad_norm 1.4256 (1.3319) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][400/625] eta 0:01:27 lr 0.001613 wd 0.0500 time 0.3769 (0.3868) data time 0.0011 (0.0023) model time 0.3759 (0.3857) loss 3.2466 (3.3842) grad_norm 1.7109 (1.3326) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][410/625] eta 0:01:23 lr 0.001613 wd 0.0500 time 0.3806 (0.3867) data time 0.0009 (0.0022) model time 0.3796 (0.3856) loss 3.7357 (3.3914) grad_norm 1.6737 (1.3343) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][420/625] eta 0:01:19 lr 0.001613 wd 0.0500 time 0.3321 (0.3870) data time 0.0008 (0.0022) model time 0.3313 (0.3859) loss 2.8440 (3.3862) grad_norm 0.9597 (1.3304) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][430/625] eta 0:01:15 lr 0.001612 wd 0.0500 time 0.3750 (0.3868) data time 0.0011 (0.0022) model time 0.3738 (0.3857) loss 3.0071 (3.3798) grad_norm 1.2979 (1.3324) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][440/625] eta 0:01:11 lr 0.001612 wd 0.0500 time 0.3706 (0.3866) data time 0.0009 (0.0022) model time 0.3698 (0.3854) loss 2.6851 (3.3761) grad_norm 1.3972 (1.3323) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][450/625] eta 0:01:07 lr 0.001612 wd 0.0500 time 0.3917 (0.3864) data time 0.0011 (0.0021) model time 0.3906 (0.3852) loss 3.5314 (3.3717) grad_norm 2.0708 (1.3361) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][460/625] eta 0:01:03 lr 0.001612 wd 0.0500 time 0.3744 (0.3861) data time 0.0012 (0.0021) model time 0.3732 (0.3849) loss 3.7077 (3.3757) grad_norm 1.0187 (1.3402) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:52:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][470/625] eta 0:00:59 lr 0.001612 wd 0.0500 time 0.3771 (0.3868) data time 0.0014 (0.0021) model time 0.3758 (0.3857) loss 2.7398 (3.3720) grad_norm 1.1039 (1.3382) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][480/625] eta 0:00:56 lr 0.001612 wd 0.0500 time 0.3844 (0.3867) data time 0.0009 (0.0021) model time 0.3835 (0.3855) loss 2.5396 (3.3698) grad_norm 1.6081 (1.3373) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][490/625] eta 0:00:52 lr 0.001612 wd 0.0500 time 0.3768 (0.3865) data time 0.0010 (0.0021) model time 0.3758 (0.3853) loss 3.2907 (3.3702) grad_norm 1.7862 (1.3437) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][500/625] eta 0:00:48 lr 0.001611 wd 0.0500 time 0.3836 (0.3863) data time 0.0009 (0.0020) model time 0.3828 (0.3852) loss 3.4636 (3.3710) grad_norm 0.9439 (1.3415) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][510/625] eta 0:00:44 lr 0.001611 wd 0.0500 time 0.3830 (0.3862) data time 0.0011 (0.0020) model time 0.3819 (0.3850) loss 2.9849 (3.3740) grad_norm 1.4063 (1.3413) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][520/625] eta 0:00:40 lr 0.001611 wd 0.0500 time 0.3931 (0.3860) data time 0.0008 (0.0020) model time 0.3922 (0.3848) loss 3.9558 (3.3742) grad_norm 1.2829 (1.3422) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][530/625] eta 0:00:36 lr 0.001611 wd 0.0500 time 0.3797 (0.3860) data time 0.0008 (0.0020) model time 0.3789 (0.3847) loss 3.6511 (3.3751) grad_norm 1.9226 (1.3457) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][540/625] eta 0:00:32 lr 0.001611 wd 0.0500 time 0.3785 (0.3858) data time 0.0011 (0.0020) model time 0.3775 (0.3846) loss 3.6164 (3.3754) grad_norm 1.3202 (1.3508) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][550/625] eta 0:00:28 lr 0.001611 wd 0.0500 time 0.3804 (0.3857) data time 0.0008 (0.0020) model time 0.3796 (0.3844) loss 4.2937 (3.3751) grad_norm 1.9517 (1.3518) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][560/625] eta 0:00:25 lr 0.001611 wd 0.0500 time 0.3806 (0.3855) data time 0.0013 (0.0019) model time 0.3793 (0.3842) loss 3.6134 (3.3806) grad_norm 1.0659 (1.3505) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][570/625] eta 0:00:21 lr 0.001611 wd 0.0500 time 0.3784 (0.3857) data time 0.0010 (0.0019) model time 0.3774 (0.3844) loss 3.1462 (3.3793) grad_norm 1.4232 (1.3566) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][580/625] eta 0:00:17 lr 0.001610 wd 0.0500 time 0.5797 (0.3866) data time 0.0009 (0.0019) model time 0.5788 (0.3854) loss 2.9998 (3.3824) grad_norm 1.1657 (1.3539) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][590/625] eta 0:00:13 lr 0.001610 wd 0.0500 time 0.3747 (0.3868) data time 0.0011 (0.0019) model time 0.3736 (0.3856) loss 3.3818 (3.3853) grad_norm 1.2186 (1.3545) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][600/625] eta 0:00:09 lr 0.001610 wd 0.0500 time 0.3804 (0.3866) data time 0.0010 (0.0019) model time 0.3793 (0.3855) loss 3.8131 (3.3843) grad_norm 1.4503 (1.3541) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][610/625] eta 0:00:05 lr 0.001610 wd 0.0500 time 0.3717 (0.3865) data time 0.0008 (0.0019) model time 0.3709 (0.3853) loss 2.9700 (3.3826) grad_norm 1.3460 (1.3542) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [101/300][620/625] eta 0:00:01 lr 0.001610 wd 0.0500 time 0.3674 (0.3862) data time 0.0006 (0.0019) model time 0.3668 (0.3850) loss 2.3184 (3.3830) grad_norm 1.0147 (1.3530) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:53:55 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 101 training takes 0:04:01 [2024-08-26 09:53:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:53:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.6367 (0.6367) Acc@1 86.133 (86.133) Acc@5 97.998 (97.998) Mem 11225MB [2024-08-26 09:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.140) Loss 1.0566 (0.8058) Acc@1 75.391 (81.778) Acc@5 93.506 (96.387) Mem 11225MB [2024-08-26 09:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 1.2393 (0.9634) Acc@1 71.387 (78.095) Acc@5 90.820 (94.366) Mem 11225MB [2024-08-26 09:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.861 Acc@5 94.338 [2024-08-26 09:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.9% [2024-08-26 09:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 77.86% [2024-08-26 09:54:00 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 09:54:01 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 09:54:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 0.5464 (0.5464) Acc@1 87.988 (87.988) Acc@5 98.193 (98.193) Mem 11225MB [2024-08-26 09:54:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 0.9087 (0.6826) Acc@1 78.027 (84.397) Acc@5 95.166 (97.208) Mem 11225MB [2024-08-26 09:54:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 1.0752 (0.8310) Acc@1 73.047 (80.676) Acc@5 92.920 (95.585) Mem 11225MB [2024-08-26 09:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.396 Acc@5 95.569 [2024-08-26 09:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.4% [2024-08-26 09:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.40% [2024-08-26 09:54:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:54:05 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:54:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][0/625] eta 0:08:29 lr 0.001610 wd 0.0500 time 0.8156 (0.8156) data time 0.4800 (0.4800) model time 0.0000 (0.0000) loss 3.5789 (3.5789) grad_norm 1.2428 (1.2428) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][10/625] eta 0:04:16 lr 0.001610 wd 0.0500 time 0.3756 (0.4170) data time 0.0011 (0.0447) model time 0.0000 (0.0000) loss 3.1609 (3.3191) grad_norm 1.5839 (1.2985) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][20/625] eta 0:04:01 lr 0.001609 wd 0.0500 time 0.3787 (0.3993) data time 0.0011 (0.0240) model time 0.0000 (0.0000) loss 3.8667 (3.3757) grad_norm 1.5248 (1.3324) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][30/625] eta 0:03:53 lr 0.001609 wd 0.0500 time 0.3764 (0.3927) data time 0.0008 (0.0166) model time 0.0000 (0.0000) loss 4.0888 (3.3331) grad_norm 1.0582 (1.2979) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][40/625] eta 0:03:47 lr 0.001609 wd 0.0500 time 0.3820 (0.3886) data time 0.0010 (0.0129) model time 0.0000 (0.0000) loss 3.4247 (3.3020) grad_norm 1.5658 (1.3100) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][50/625] eta 0:03:42 lr 0.001609 wd 0.0500 time 0.3720 (0.3864) data time 0.0008 (0.0105) model time 0.0000 (0.0000) loss 2.4083 (3.2058) grad_norm 1.5216 (1.3191) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][60/625] eta 0:03:37 lr 0.001609 wd 0.0500 time 0.3797 (0.3851) data time 0.0011 (0.0090) model time 0.3786 (0.3769) loss 3.8313 (3.2442) grad_norm 1.3065 (1.3264) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][70/625] eta 0:03:33 lr 0.001609 wd 0.0500 time 0.3748 (0.3841) data time 0.0012 (0.0079) model time 0.3736 (0.3770) loss 2.8174 (3.1939) grad_norm 0.9543 (1.3114) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][80/625] eta 0:03:28 lr 0.001609 wd 0.0500 time 0.3717 (0.3832) data time 0.0009 (0.0071) model time 0.3708 (0.3765) loss 3.9117 (3.2415) grad_norm 1.4694 (1.3335) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][90/625] eta 0:03:24 lr 0.001608 wd 0.0500 time 0.3762 (0.3824) data time 0.0008 (0.0064) model time 0.3755 (0.3761) loss 3.2571 (3.2628) grad_norm 1.3015 (1.3400) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][100/625] eta 0:03:21 lr 0.001608 wd 0.0500 time 0.3744 (0.3838) data time 0.0009 (0.0059) model time 0.3736 (0.3800) loss 3.8960 (3.2545) grad_norm 1.3570 (1.3403) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][110/625] eta 0:03:17 lr 0.001608 wd 0.0500 time 0.3735 (0.3833) data time 0.0008 (0.0055) model time 0.3727 (0.3794) loss 3.6921 (3.2270) grad_norm 1.2613 (1.3632) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][120/625] eta 0:03:14 lr 0.001608 wd 0.0500 time 0.3767 (0.3844) data time 0.0011 (0.0051) model time 0.3756 (0.3818) loss 3.2549 (3.2313) grad_norm 1.1449 (1.3619) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][130/625] eta 0:03:09 lr 0.001608 wd 0.0500 time 0.3780 (0.3838) data time 0.0009 (0.0048) model time 0.3772 (0.3810) loss 3.8119 (3.2475) grad_norm 1.2095 (1.3536) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:54:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][140/625] eta 0:03:05 lr 0.001608 wd 0.0500 time 0.3787 (0.3835) data time 0.0009 (0.0045) model time 0.3778 (0.3806) loss 3.0131 (3.2487) grad_norm 1.5424 (1.3525) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][150/625] eta 0:03:02 lr 0.001608 wd 0.0500 time 0.3801 (0.3842) data time 0.0009 (0.0043) model time 0.3792 (0.3819) loss 3.8994 (3.2464) grad_norm 1.9409 (1.3740) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][160/625] eta 0:02:59 lr 0.001607 wd 0.0500 time 0.3766 (0.3862) data time 0.0011 (0.0041) model time 0.3756 (0.3849) loss 3.5097 (3.2460) grad_norm 1.3979 (1.3794) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][170/625] eta 0:02:55 lr 0.001607 wd 0.0500 time 0.3747 (0.3856) data time 0.0011 (0.0039) model time 0.3737 (0.3842) loss 3.3457 (3.2601) grad_norm 1.3136 (1.3875) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][180/625] eta 0:02:51 lr 0.001607 wd 0.0500 time 0.3785 (0.3852) data time 0.0008 (0.0038) model time 0.3777 (0.3836) loss 4.1323 (3.2778) grad_norm 0.8837 (1.3842) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][190/625] eta 0:02:47 lr 0.001607 wd 0.0500 time 0.3739 (0.3848) data time 0.0012 (0.0036) model time 0.3727 (0.3831) loss 3.5202 (3.2781) grad_norm 1.5644 (1.3712) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][200/625] eta 0:02:43 lr 0.001607 wd 0.0500 time 0.3676 (0.3844) data time 0.0009 (0.0035) model time 0.3668 (0.3826) loss 2.6001 (3.2889) grad_norm 1.3388 (1.3654) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][210/625] eta 0:02:39 lr 0.001607 wd 0.0500 time 0.3770 (0.3842) data time 0.0009 (0.0034) model time 0.3761 (0.3824) loss 4.2179 (3.2917) grad_norm 1.6065 (1.3642) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][220/625] eta 0:02:35 lr 0.001607 wd 0.0500 time 0.3826 (0.3840) data time 0.0012 (0.0033) model time 0.3813 (0.3822) loss 2.8982 (3.2941) grad_norm 1.4069 (1.3613) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][230/625] eta 0:02:31 lr 0.001606 wd 0.0500 time 0.3752 (0.3838) data time 0.0008 (0.0032) model time 0.3743 (0.3819) loss 3.7942 (3.2974) grad_norm 1.5711 (1.3802) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][240/625] eta 0:02:27 lr 0.001606 wd 0.0500 time 0.3733 (0.3843) data time 0.0008 (0.0031) model time 0.3725 (0.3826) loss 3.0188 (3.2958) grad_norm 1.3574 (1.3774) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][250/625] eta 0:02:24 lr 0.001606 wd 0.0500 time 0.3759 (0.3840) data time 0.0008 (0.0030) model time 0.3751 (0.3823) loss 3.8319 (3.2993) grad_norm 1.5719 (1.3813) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][260/625] eta 0:02:20 lr 0.001606 wd 0.0500 time 0.3850 (0.3838) data time 0.0009 (0.0030) model time 0.3841 (0.3820) loss 4.4898 (3.3173) grad_norm 1.2203 (1.3709) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][270/625] eta 0:02:16 lr 0.001606 wd 0.0500 time 0.3785 (0.3837) data time 0.0011 (0.0029) model time 0.3773 (0.3819) loss 2.7996 (3.3198) grad_norm 1.2621 (1.3632) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][280/625] eta 0:02:12 lr 0.001606 wd 0.0500 time 0.3837 (0.3835) data time 0.0009 (0.0028) model time 0.3828 (0.3817) loss 3.5165 (3.3285) grad_norm 1.0775 (1.3587) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:55:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][290/625] eta 0:02:08 lr 0.001606 wd 0.0500 time 0.3749 (0.3832) data time 0.0011 (0.0028) model time 0.3738 (0.3814) loss 2.4549 (3.3297) grad_norm 1.6896 (1.3562) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][300/625] eta 0:02:04 lr 0.001605 wd 0.0500 time 0.3860 (0.3832) data time 0.0009 (0.0027) model time 0.3851 (0.3814) loss 3.1268 (3.3358) grad_norm 1.1591 (1.3559) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][310/625] eta 0:02:00 lr 0.001605 wd 0.0500 time 0.3852 (0.3838) data time 0.0010 (0.0027) model time 0.3842 (0.3821) loss 3.4388 (3.3319) grad_norm 1.7954 (1.3653) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][320/625] eta 0:01:57 lr 0.001605 wd 0.0500 time 0.3763 (0.3837) data time 0.0008 (0.0026) model time 0.3756 (0.3821) loss 3.5057 (3.3359) grad_norm 1.3343 (1.3613) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][330/625] eta 0:01:53 lr 0.001605 wd 0.0500 time 0.3762 (0.3836) data time 0.0011 (0.0026) model time 0.3752 (0.3820) loss 2.5723 (3.3354) grad_norm 1.5067 (1.3618) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][340/625] eta 0:01:49 lr 0.001605 wd 0.0500 time 0.3722 (0.3850) data time 0.0009 (0.0025) model time 0.3713 (0.3837) loss 2.2072 (3.3295) grad_norm 1.0774 (1.3603) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][350/625] eta 0:01:45 lr 0.001605 wd 0.0500 time 0.3715 (0.3849) data time 0.0010 (0.0025) model time 0.3705 (0.3835) loss 3.7281 (3.3313) grad_norm 1.2241 (1.3595) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][360/625] eta 0:01:41 lr 0.001605 wd 0.0500 time 0.3724 (0.3847) data time 0.0011 (0.0025) model time 0.3713 (0.3833) loss 2.8472 (3.3297) grad_norm 1.4584 (1.3709) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][370/625] eta 0:01:38 lr 0.001604 wd 0.0500 time 0.3750 (0.3845) data time 0.0010 (0.0024) model time 0.3740 (0.3831) loss 3.0853 (3.3308) grad_norm 1.5134 (1.3708) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][380/625] eta 0:01:34 lr 0.001604 wd 0.0500 time 0.3844 (0.3843) data time 0.0008 (0.0024) model time 0.3836 (0.3829) loss 3.4078 (3.3334) grad_norm 1.3913 (1.3768) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][390/625] eta 0:01:30 lr 0.001604 wd 0.0500 time 0.3747 (0.3841) data time 0.0008 (0.0024) model time 0.3739 (0.3827) loss 3.1721 (3.3322) grad_norm 1.8829 (1.3859) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][400/625] eta 0:01:26 lr 0.001604 wd 0.0500 time 0.3778 (0.3839) data time 0.0010 (0.0023) model time 0.3768 (0.3824) loss 3.5978 (3.3278) grad_norm 1.6864 (1.3839) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][410/625] eta 0:01:22 lr 0.001604 wd 0.0500 time 0.3781 (0.3838) data time 0.0008 (0.0023) model time 0.3773 (0.3823) loss 4.0874 (3.3275) grad_norm 1.1982 (1.3831) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][420/625] eta 0:01:18 lr 0.001604 wd 0.0500 time 0.3768 (0.3837) data time 0.0010 (0.0023) model time 0.3759 (0.3822) loss 2.8377 (3.3288) grad_norm 1.3257 (1.3849) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][430/625] eta 0:01:14 lr 0.001604 wd 0.0500 time 0.3844 (0.3836) data time 0.0008 (0.0022) model time 0.3836 (0.3821) loss 2.7984 (3.3204) grad_norm 1.0019 (1.3807) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][440/625] eta 0:01:10 lr 0.001603 wd 0.0500 time 0.3736 (0.3834) data time 0.0010 (0.0022) model time 0.3726 (0.3819) loss 3.6690 (3.3191) grad_norm 1.1675 (1.3775) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:56:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][450/625] eta 0:01:07 lr 0.001603 wd 0.0500 time 0.5546 (0.3841) data time 0.0010 (0.0022) model time 0.5535 (0.3827) loss 2.3838 (3.3178) grad_norm 1.1673 (1.3775) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][460/625] eta 0:01:03 lr 0.001603 wd 0.0500 time 0.3845 (0.3849) data time 0.0011 (0.0022) model time 0.3834 (0.3836) loss 3.6796 (3.3174) grad_norm 1.4105 (1.3736) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][470/625] eta 0:00:59 lr 0.001603 wd 0.0500 time 0.3719 (0.3848) data time 0.0011 (0.0021) model time 0.3708 (0.3834) loss 3.0832 (3.3201) grad_norm 1.0767 (1.3714) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][480/625] eta 0:00:55 lr 0.001603 wd 0.0500 time 0.3758 (0.3846) data time 0.0011 (0.0021) model time 0.3747 (0.3833) loss 2.9284 (3.3197) grad_norm 1.2343 (1.3716) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][490/625] eta 0:00:51 lr 0.001603 wd 0.0500 time 0.3808 (0.3850) data time 0.0008 (0.0021) model time 0.3800 (0.3837) loss 3.9204 (3.3227) grad_norm 1.3142 (1.3697) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][500/625] eta 0:00:48 lr 0.001603 wd 0.0500 time 0.3791 (0.3849) data time 0.0010 (0.0021) model time 0.3781 (0.3835) loss 3.7660 (3.3238) grad_norm 1.7164 (1.3717) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][510/625] eta 0:00:44 lr 0.001602 wd 0.0500 time 0.3828 (0.3847) data time 0.0010 (0.0021) model time 0.3817 (0.3834) loss 3.7678 (3.3244) grad_norm 1.7070 (1.3735) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][520/625] eta 0:00:40 lr 0.001602 wd 0.0500 time 0.3861 (0.3846) data time 0.0008 (0.0020) model time 0.3853 (0.3833) loss 3.4146 (3.3253) grad_norm 1.3037 (1.3780) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][530/625] eta 0:00:36 lr 0.001602 wd 0.0500 time 0.3764 (0.3845) data time 0.0008 (0.0020) model time 0.3757 (0.3832) loss 2.4779 (3.3276) grad_norm 1.1520 (1.3780) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][540/625] eta 0:00:32 lr 0.001602 wd 0.0500 time 0.3915 (0.3845) data time 0.0009 (0.0020) model time 0.3907 (0.3831) loss 4.0174 (3.3294) grad_norm 0.9381 (1.3746) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][550/625] eta 0:00:28 lr 0.001602 wd 0.0500 time 0.3872 (0.3844) data time 0.0009 (0.0020) model time 0.3862 (0.3830) loss 2.3148 (3.3299) grad_norm 1.6164 (1.3718) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:57:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][560/625] eta 0:00:24 lr 0.001602 wd 0.0500 time 0.3772 (0.3843) data time 0.0009 (0.0020) model time 0.3763 (0.3830) loss 3.5756 (3.3307) grad_norm 1.5957 (1.3721) loss_scale 2048.0000 (1040.4278) mem 11225MB [2024-08-26 09:57:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][570/625] eta 0:00:21 lr 0.001602 wd 0.0500 time 0.3673 (0.3842) data time 0.0011 (0.0020) model time 0.3662 (0.3829) loss 4.0642 (3.3365) grad_norm 1.6573 (1.3737) loss_scale 2048.0000 (1058.0736) mem 11225MB [2024-08-26 09:57:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][580/625] eta 0:00:17 lr 0.001601 wd 0.0500 time 0.3708 (0.3841) data time 0.0015 (0.0020) model time 0.3693 (0.3828) loss 3.7221 (3.3389) grad_norm 1.1633 (1.3751) loss_scale 2048.0000 (1075.1119) mem 11225MB [2024-08-26 09:57:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][590/625] eta 0:00:13 lr 0.001601 wd 0.0500 time 0.3777 (0.3840) data time 0.0009 (0.0019) model time 0.3768 (0.3826) loss 3.7019 (3.3400) grad_norm 1.1739 (1.3738) loss_scale 2048.0000 (1091.5736) mem 11225MB [2024-08-26 09:57:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][600/625] eta 0:00:09 lr 0.001601 wd 0.0500 time 0.3745 (0.3839) data time 0.0011 (0.0019) model time 0.3733 (0.3825) loss 3.6132 (3.3450) grad_norm 1.2891 (nan) loss_scale 1024.0000 (1095.5607) mem 11225MB [2024-08-26 09:58:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][610/625] eta 0:00:05 lr 0.001601 wd 0.0500 time 0.3668 (0.3841) data time 0.0006 (0.0019) model time 0.3662 (0.3827) loss 3.0692 (3.3453) grad_norm 1.3879 (nan) loss_scale 1024.0000 (1094.3895) mem 11225MB [2024-08-26 09:58:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [102/300][620/625] eta 0:00:01 lr 0.001601 wd 0.0500 time 0.3627 (0.3838) data time 0.0006 (0.0019) model time 0.3621 (0.3824) loss 2.5472 (3.3421) grad_norm 1.7142 (nan) loss_scale 1024.0000 (1093.2560) mem 11225MB [2024-08-26 09:58:05 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 102 training takes 0:03:59 [2024-08-26 09:58:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 09:58:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 09:58:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.500 (0.500) Loss 0.6304 (0.6304) Acc@1 86.230 (86.230) Acc@5 97.510 (97.510) Mem 11225MB [2024-08-26 09:58:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 1.0811 (0.7777) Acc@1 73.682 (82.018) Acc@5 93.262 (96.365) Mem 11225MB [2024-08-26 09:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.2021 (0.9464) Acc@1 71.045 (78.234) Acc@5 91.895 (94.396) Mem 11225MB [2024-08-26 09:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.001 Acc@5 94.358 [2024-08-26 09:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.0% [2024-08-26 09:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.00% [2024-08-26 09:58:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 09:58:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 09:58:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.5454 (0.5454) Acc@1 88.184 (88.184) Acc@5 98.340 (98.340) Mem 11225MB [2024-08-26 09:58:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.143) Loss 0.9067 (0.6817) Acc@1 78.271 (84.450) Acc@5 94.971 (97.221) Mem 11225MB [2024-08-26 09:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.0713 (0.8295) Acc@1 73.193 (80.743) Acc@5 92.969 (95.610) Mem 11225MB [2024-08-26 09:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.466 Acc@5 95.597 [2024-08-26 09:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.5% [2024-08-26 09:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.47% [2024-08-26 09:58:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 09:58:15 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 09:58:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][0/625] eta 0:08:23 lr 0.001601 wd 0.0500 time 0.8061 (0.8061) data time 0.4546 (0.4546) model time 0.0000 (0.0000) loss 4.2230 (4.2230) grad_norm 2.0330 (2.0330) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][10/625] eta 0:04:15 lr 0.001601 wd 0.0500 time 0.3768 (0.4156) data time 0.0010 (0.0423) model time 0.0000 (0.0000) loss 3.7578 (3.5373) grad_norm 1.1723 (1.6523) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][20/625] eta 0:04:03 lr 0.001601 wd 0.0500 time 0.5079 (0.4031) data time 0.0008 (0.0227) model time 0.0000 (0.0000) loss 2.6806 (3.4110) grad_norm 1.5383 (1.4870) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][30/625] eta 0:04:02 lr 0.001600 wd 0.0500 time 0.3753 (0.4074) data time 0.0011 (0.0158) model time 0.0000 (0.0000) loss 4.2609 (3.4145) grad_norm 1.1822 (1.3964) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][40/625] eta 0:03:54 lr 0.001600 wd 0.0500 time 0.3779 (0.4008) data time 0.0010 (0.0122) model time 0.0000 (0.0000) loss 2.8299 (3.3001) grad_norm 1.7476 (1.4056) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][50/625] eta 0:03:47 lr 0.001600 wd 0.0500 time 0.3810 (0.3964) data time 0.0008 (0.0100) model time 0.0000 (0.0000) loss 3.3636 (3.2905) grad_norm 1.6586 (1.3735) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][60/625] eta 0:03:42 lr 0.001600 wd 0.0500 time 0.3768 (0.3934) data time 0.0011 (0.0086) model time 0.3757 (0.3769) loss 3.4942 (3.3068) grad_norm 1.0229 (1.3722) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][70/625] eta 0:03:37 lr 0.001600 wd 0.0500 time 0.3759 (0.3914) data time 0.0011 (0.0075) model time 0.3748 (0.3773) loss 3.7499 (3.2938) grad_norm 1.1817 (1.3642) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][80/625] eta 0:03:33 lr 0.001600 wd 0.0500 time 0.3778 (0.3919) data time 0.0009 (0.0067) model time 0.3769 (0.3831) loss 4.2228 (3.3074) grad_norm 1.1686 (1.3354) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][90/625] eta 0:03:28 lr 0.001600 wd 0.0500 time 0.3740 (0.3900) data time 0.0011 (0.0061) model time 0.3729 (0.3808) loss 2.5245 (3.3040) grad_norm 1.5343 (1.3527) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][100/625] eta 0:03:24 lr 0.001599 wd 0.0500 time 0.3867 (0.3889) data time 0.0011 (0.0056) model time 0.3856 (0.3801) loss 4.0666 (3.3222) grad_norm 1.1671 (1.3821) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:58:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][110/625] eta 0:03:19 lr 0.001599 wd 0.0500 time 0.3807 (0.3879) data time 0.0011 (0.0052) model time 0.3796 (0.3795) loss 3.5804 (3.3182) grad_norm 1.1381 (1.3698) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][120/625] eta 0:03:15 lr 0.001599 wd 0.0500 time 0.3802 (0.3872) data time 0.0012 (0.0049) model time 0.3790 (0.3793) loss 2.7292 (3.3126) grad_norm 0.9936 (1.3453) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][130/625] eta 0:03:11 lr 0.001599 wd 0.0500 time 0.3722 (0.3864) data time 0.0008 (0.0046) model time 0.3714 (0.3788) loss 3.8632 (3.3264) grad_norm 1.3560 (1.3371) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][140/625] eta 0:03:07 lr 0.001599 wd 0.0500 time 0.3718 (0.3858) data time 0.0012 (0.0044) model time 0.3707 (0.3787) loss 3.2829 (3.3362) grad_norm 1.2721 (1.3375) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][150/625] eta 0:03:03 lr 0.001599 wd 0.0500 time 0.3955 (0.3854) data time 0.0009 (0.0042) model time 0.3946 (0.3787) loss 2.5116 (3.3393) grad_norm 1.7745 (1.3379) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][160/625] eta 0:02:58 lr 0.001599 wd 0.0500 time 0.3792 (0.3848) data time 0.0011 (0.0040) model time 0.3781 (0.3783) loss 3.1646 (3.3556) grad_norm 1.2307 (1.3286) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][170/625] eta 0:02:55 lr 0.001598 wd 0.0500 time 0.3828 (0.3855) data time 0.0008 (0.0038) model time 0.3820 (0.3797) loss 2.4283 (3.3574) grad_norm 1.4199 (1.3530) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][180/625] eta 0:02:51 lr 0.001598 wd 0.0500 time 0.3702 (0.3849) data time 0.0011 (0.0037) model time 0.3691 (0.3793) loss 3.1232 (3.3613) grad_norm 1.7058 (1.3550) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][190/625] eta 0:02:47 lr 0.001598 wd 0.0500 time 0.3738 (0.3844) data time 0.0008 (0.0035) model time 0.3729 (0.3789) loss 4.2024 (3.3638) grad_norm 0.9401 (1.3524) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][200/625] eta 0:02:43 lr 0.001598 wd 0.0500 time 0.3820 (0.3841) data time 0.0008 (0.0034) model time 0.3812 (0.3787) loss 2.2094 (3.3731) grad_norm 1.1278 (1.3512) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][210/625] eta 0:02:39 lr 0.001598 wd 0.0500 time 0.3808 (0.3838) data time 0.0009 (0.0033) model time 0.3799 (0.3786) loss 3.4009 (3.3736) grad_norm 1.9115 (1.3566) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][220/625] eta 0:02:35 lr 0.001598 wd 0.0500 time 0.3714 (0.3836) data time 0.0008 (0.0032) model time 0.3706 (0.3786) loss 3.7978 (3.3808) grad_norm 1.0694 (1.3633) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][230/625] eta 0:02:31 lr 0.001598 wd 0.0500 time 0.5913 (0.3843) data time 0.0011 (0.0031) model time 0.5902 (0.3798) loss 3.9032 (3.3743) grad_norm 1.3254 (1.3617) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][240/625] eta 0:02:28 lr 0.001597 wd 0.0500 time 0.3834 (0.3849) data time 0.0011 (0.0030) model time 0.3823 (0.3807) loss 3.2699 (3.3710) grad_norm 1.1779 (1.3574) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][250/625] eta 0:02:24 lr 0.001597 wd 0.0500 time 0.3870 (0.3847) data time 0.0008 (0.0030) model time 0.3862 (0.3805) loss 3.8564 (3.3735) grad_norm 1.8155 (1.3978) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][260/625] eta 0:02:20 lr 0.001597 wd 0.0500 time 0.3801 (0.3845) data time 0.0008 (0.0029) model time 0.3793 (0.3804) loss 4.0993 (3.3770) grad_norm 1.6387 (1.4077) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 09:59:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][270/625] eta 0:02:16 lr 0.001597 wd 0.0500 time 0.3798 (0.3843) data time 0.0008 (0.0028) model time 0.3790 (0.3803) loss 2.2533 (3.3775) grad_norm 1.1686 (1.4099) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][280/625] eta 0:02:12 lr 0.001597 wd 0.0500 time 0.3773 (0.3841) data time 0.0009 (0.0028) model time 0.3764 (0.3802) loss 2.8579 (3.3838) grad_norm 1.4841 (1.4108) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][290/625] eta 0:02:09 lr 0.001597 wd 0.0500 time 0.3379 (0.3852) data time 0.0011 (0.0027) model time 0.3368 (0.3817) loss 3.4726 (3.3865) grad_norm 1.6544 (1.4071) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][300/625] eta 0:02:05 lr 0.001597 wd 0.0500 time 0.3780 (0.3849) data time 0.0008 (0.0027) model time 0.3771 (0.3814) loss 4.1424 (3.3962) grad_norm 1.5444 (1.4064) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][310/625] eta 0:02:01 lr 0.001596 wd 0.0500 time 0.3795 (0.3847) data time 0.0011 (0.0026) model time 0.3784 (0.3812) loss 3.3002 (3.3947) grad_norm 1.9010 (1.4062) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][320/625] eta 0:01:57 lr 0.001596 wd 0.0500 time 0.3655 (0.3844) data time 0.0016 (0.0026) model time 0.3639 (0.3810) loss 3.2652 (3.3903) grad_norm 1.1734 (1.4111) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][330/625] eta 0:01:53 lr 0.001596 wd 0.0500 time 0.3776 (0.3843) data time 0.0011 (0.0025) model time 0.3765 (0.3809) loss 2.9819 (3.3862) grad_norm 1.0756 (1.4083) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][340/625] eta 0:01:49 lr 0.001596 wd 0.0500 time 0.3832 (0.3841) data time 0.0008 (0.0025) model time 0.3824 (0.3807) loss 2.5053 (3.3826) grad_norm 1.0736 (1.4043) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][350/625] eta 0:01:45 lr 0.001596 wd 0.0500 time 0.3707 (0.3838) data time 0.0008 (0.0025) model time 0.3699 (0.3805) loss 2.7678 (3.3837) grad_norm 1.3289 (1.4030) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][360/625] eta 0:01:41 lr 0.001596 wd 0.0500 time 0.3754 (0.3836) data time 0.0011 (0.0024) model time 0.3744 (0.3804) loss 3.2715 (3.3813) grad_norm 1.0155 (1.3970) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][370/625] eta 0:01:37 lr 0.001596 wd 0.0500 time 0.3764 (0.3835) data time 0.0011 (0.0024) model time 0.3753 (0.3802) loss 3.5144 (3.3723) grad_norm 0.9404 (1.3904) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][380/625] eta 0:01:33 lr 0.001595 wd 0.0500 time 0.3730 (0.3833) data time 0.0011 (0.0024) model time 0.3719 (0.3801) loss 3.7621 (3.3808) grad_norm 1.4511 (1.3932) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][390/625] eta 0:01:30 lr 0.001595 wd 0.0500 time 0.3831 (0.3832) data time 0.0008 (0.0023) model time 0.3824 (0.3800) loss 4.4385 (3.3817) grad_norm 1.5446 (1.3939) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][400/625] eta 0:01:26 lr 0.001595 wd 0.0500 time 0.3791 (0.3830) data time 0.0009 (0.0023) model time 0.3782 (0.3799) loss 4.1644 (3.3812) grad_norm 1.5988 (1.3992) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][410/625] eta 0:01:22 lr 0.001595 wd 0.0500 time 0.3787 (0.3829) data time 0.0011 (0.0023) model time 0.3776 (0.3798) loss 3.2354 (3.3780) grad_norm 1.0331 (1.3975) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][420/625] eta 0:01:18 lr 0.001595 wd 0.0500 time 0.3780 (0.3837) data time 0.0011 (0.0022) model time 0.3769 (0.3808) loss 3.5702 (3.3852) grad_norm 1.2442 (1.3932) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][430/625] eta 0:01:14 lr 0.001595 wd 0.0500 time 0.3739 (0.3836) data time 0.0012 (0.0022) model time 0.3727 (0.3807) loss 2.4047 (3.3810) grad_norm 1.5007 (1.3925) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][440/625] eta 0:01:10 lr 0.001595 wd 0.0500 time 0.3745 (0.3834) data time 0.0013 (0.0022) model time 0.3732 (0.3805) loss 3.4431 (3.3871) grad_norm 0.8719 (1.3911) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][450/625] eta 0:01:07 lr 0.001594 wd 0.0500 time 0.3775 (0.3832) data time 0.0011 (0.0022) model time 0.3764 (0.3804) loss 2.5739 (3.3889) grad_norm 1.5769 (1.3887) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][460/625] eta 0:01:03 lr 0.001594 wd 0.0500 time 0.3774 (0.3831) data time 0.0011 (0.0022) model time 0.3764 (0.3802) loss 3.5804 (3.3858) grad_norm 1.0108 (1.3871) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][470/625] eta 0:00:59 lr 0.001594 wd 0.0500 time 0.3788 (0.3830) data time 0.0008 (0.0021) model time 0.3779 (0.3802) loss 3.1333 (3.3852) grad_norm 1.3960 (1.3863) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][480/625] eta 0:00:55 lr 0.001594 wd 0.0500 time 0.3746 (0.3831) data time 0.0008 (0.0021) model time 0.3738 (0.3803) loss 3.3620 (3.3861) grad_norm 0.9226 (1.3837) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][490/625] eta 0:00:51 lr 0.001594 wd 0.0500 time 0.3820 (0.3830) data time 0.0011 (0.0021) model time 0.3808 (0.3802) loss 2.6720 (3.3894) grad_norm 1.6955 (1.3852) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][500/625] eta 0:00:47 lr 0.001594 wd 0.0500 time 0.3746 (0.3833) data time 0.0011 (0.0021) model time 0.3735 (0.3807) loss 3.7041 (3.3887) grad_norm 1.5525 (1.3872) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][510/625] eta 0:00:44 lr 0.001594 wd 0.0500 time 0.3772 (0.3832) data time 0.0011 (0.0021) model time 0.3761 (0.3805) loss 3.6007 (3.3885) grad_norm 1.2617 (1.3855) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][520/625] eta 0:00:40 lr 0.001593 wd 0.0500 time 0.3747 (0.3831) data time 0.0012 (0.0020) model time 0.3735 (0.3805) loss 3.4033 (3.3901) grad_norm 1.2680 (1.3871) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][530/625] eta 0:00:36 lr 0.001593 wd 0.0500 time 0.3809 (0.3830) data time 0.0009 (0.0020) model time 0.3800 (0.3804) loss 4.2064 (3.3928) grad_norm 1.2816 (1.3860) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][540/625] eta 0:00:32 lr 0.001593 wd 0.0500 time 0.3703 (0.3837) data time 0.0008 (0.0020) model time 0.3694 (0.3811) loss 2.2626 (3.3860) grad_norm 1.1782 (1.3850) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][550/625] eta 0:00:28 lr 0.001593 wd 0.0500 time 0.3716 (0.3835) data time 0.0008 (0.0020) model time 0.3708 (0.3810) loss 4.0689 (3.3942) grad_norm 1.1577 (1.3822) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][560/625] eta 0:00:24 lr 0.001593 wd 0.0500 time 0.3806 (0.3834) data time 0.0010 (0.0020) model time 0.3796 (0.3809) loss 3.6987 (3.3983) grad_norm 1.8340 (1.3793) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][570/625] eta 0:00:21 lr 0.001593 wd 0.0500 time 0.3775 (0.3833) data time 0.0009 (0.0020) model time 0.3766 (0.3808) loss 2.9313 (3.3965) grad_norm 2.0555 (1.3836) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][580/625] eta 0:00:17 lr 0.001593 wd 0.0500 time 0.3731 (0.3832) data time 0.0011 (0.0020) model time 0.3720 (0.3807) loss 2.7187 (3.3953) grad_norm 1.4857 (1.3829) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][590/625] eta 0:00:13 lr 0.001592 wd 0.0500 time 0.3765 (0.3831) data time 0.0008 (0.0019) model time 0.3757 (0.3806) loss 4.0331 (3.3974) grad_norm 1.2223 (1.3814) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][600/625] eta 0:00:09 lr 0.001592 wd 0.0500 time 0.3828 (0.3833) data time 0.0008 (0.0019) model time 0.3820 (0.3809) loss 4.2934 (3.4005) grad_norm 0.9939 (1.3784) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][610/625] eta 0:00:05 lr 0.001592 wd 0.0500 time 0.3650 (0.3832) data time 0.0006 (0.0019) model time 0.3644 (0.3808) loss 2.3322 (3.3981) grad_norm 1.9624 (1.3805) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [103/300][620/625] eta 0:00:01 lr 0.001592 wd 0.0500 time 0.3744 (0.3830) data time 0.0006 (0.0019) model time 0.3738 (0.3806) loss 3.3742 (3.3942) grad_norm 1.4731 (1.3805) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:14 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 103 training takes 0:03:59 [2024-08-26 10:02:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:02:15 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:02:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.6460 (0.6460) Acc@1 86.279 (86.279) Acc@5 97.510 (97.510) Mem 11225MB [2024-08-26 10:02:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 1.0830 (0.7896) Acc@1 73.535 (82.053) Acc@5 93.750 (96.458) Mem 11225MB [2024-08-26 10:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.2520 (0.9590) Acc@1 72.461 (78.278) Acc@5 91.064 (94.406) Mem 11225MB [2024-08-26 10:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.051 Acc@5 94.358 [2024-08-26 10:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.1% [2024-08-26 10:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.05% [2024-08-26 10:02:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 10:02:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 10:02:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.5449 (0.5449) Acc@1 88.135 (88.135) Acc@5 98.340 (98.340) Mem 11225MB [2024-08-26 10:02:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.144) Loss 0.9038 (0.6809) Acc@1 78.564 (84.548) Acc@5 95.068 (97.217) Mem 11225MB [2024-08-26 10:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.0674 (0.8281) Acc@1 73.389 (80.857) Acc@5 92.969 (95.612) Mem 11225MB [2024-08-26 10:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.590 Acc@5 95.603 [2024-08-26 10:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.6% [2024-08-26 10:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.59% [2024-08-26 10:02:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:02:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:02:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][0/625] eta 0:08:28 lr 0.001592 wd 0.0500 time 0.8142 (0.8142) data time 0.4843 (0.4843) model time 0.0000 (0.0000) loss 3.4487 (3.4487) grad_norm 1.8981 (1.8981) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][10/625] eta 0:04:16 lr 0.001592 wd 0.0500 time 0.3762 (0.4164) data time 0.0008 (0.0450) model time 0.0000 (0.0000) loss 3.9118 (3.6641) grad_norm 2.3344 (1.5140) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][20/625] eta 0:04:00 lr 0.001592 wd 0.0500 time 0.3710 (0.3972) data time 0.0012 (0.0242) model time 0.0000 (0.0000) loss 2.7503 (3.3849) grad_norm 1.5538 (1.5018) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][30/625] eta 0:03:52 lr 0.001591 wd 0.0500 time 0.3736 (0.3902) data time 0.0009 (0.0167) model time 0.0000 (0.0000) loss 3.9523 (3.4231) grad_norm 1.6497 (1.4621) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][40/625] eta 0:03:46 lr 0.001591 wd 0.0500 time 0.3874 (0.3870) data time 0.0008 (0.0129) model time 0.0000 (0.0000) loss 3.0099 (3.4146) grad_norm 1.0177 (1.4093) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][50/625] eta 0:03:41 lr 0.001591 wd 0.0500 time 0.3996 (0.3855) data time 0.0012 (0.0107) model time 0.0000 (0.0000) loss 3.6200 (3.4298) grad_norm 0.9656 (1.3781) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][60/625] eta 0:03:36 lr 0.001591 wd 0.0500 time 0.3761 (0.3840) data time 0.0011 (0.0091) model time 0.3750 (0.3750) loss 3.5986 (3.4347) grad_norm 1.5658 (1.3923) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][70/625] eta 0:03:37 lr 0.001591 wd 0.0500 time 0.3722 (0.3914) data time 0.0011 (0.0080) model time 0.3711 (0.4052) loss 3.4245 (3.4285) grad_norm 2.0537 (1.4052) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][80/625] eta 0:03:32 lr 0.001591 wd 0.0500 time 0.3816 (0.3897) data time 0.0009 (0.0072) model time 0.3808 (0.3956) loss 4.2832 (3.4292) grad_norm 1.4189 (1.4106) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][90/625] eta 0:03:28 lr 0.001591 wd 0.0500 time 0.3915 (0.3905) data time 0.0009 (0.0065) model time 0.3906 (0.3957) loss 3.9808 (3.4146) grad_norm 1.6001 (1.4141) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][100/625] eta 0:03:24 lr 0.001590 wd 0.0500 time 0.3733 (0.3890) data time 0.0008 (0.0060) model time 0.3725 (0.3915) loss 2.1434 (3.3776) grad_norm 1.1984 (1.3949) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][110/625] eta 0:03:19 lr 0.001590 wd 0.0500 time 0.3772 (0.3880) data time 0.0009 (0.0055) model time 0.3764 (0.3889) loss 3.0803 (3.3618) grad_norm 1.4383 (1.3875) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][120/625] eta 0:03:15 lr 0.001590 wd 0.0500 time 0.3801 (0.3871) data time 0.0010 (0.0052) model time 0.3791 (0.3871) loss 3.4033 (3.3477) grad_norm 1.5947 (1.3816) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][130/625] eta 0:03:11 lr 0.001590 wd 0.0500 time 0.3777 (0.3864) data time 0.0008 (0.0049) model time 0.3769 (0.3858) loss 2.3308 (3.3467) grad_norm 1.0190 (1.3757) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][140/625] eta 0:03:07 lr 0.001590 wd 0.0500 time 0.3774 (0.3858) data time 0.0009 (0.0046) model time 0.3765 (0.3848) loss 3.8339 (3.3585) grad_norm 1.2170 (1.3634) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][150/625] eta 0:03:02 lr 0.001590 wd 0.0500 time 0.3758 (0.3852) data time 0.0011 (0.0044) model time 0.3748 (0.3839) loss 3.3486 (3.3699) grad_norm 1.2472 (1.3508) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][160/625] eta 0:02:58 lr 0.001590 wd 0.0500 time 0.3838 (0.3847) data time 0.0008 (0.0042) model time 0.3829 (0.3833) loss 2.5887 (3.3556) grad_norm 1.1646 (1.3430) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][170/625] eta 0:02:54 lr 0.001589 wd 0.0500 time 0.3763 (0.3843) data time 0.0008 (0.0040) model time 0.3755 (0.3827) loss 3.2030 (3.3497) grad_norm 0.9611 (1.3669) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][180/625] eta 0:02:50 lr 0.001589 wd 0.0500 time 0.3795 (0.3839) data time 0.0008 (0.0038) model time 0.3787 (0.3821) loss 2.4619 (3.3398) grad_norm 1.0289 (1.3705) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][190/625] eta 0:02:46 lr 0.001589 wd 0.0500 time 0.3731 (0.3835) data time 0.0009 (0.0037) model time 0.3722 (0.3817) loss 1.9526 (3.3338) grad_norm 1.3612 (1.3675) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][200/625] eta 0:02:42 lr 0.001589 wd 0.0500 time 0.3776 (0.3834) data time 0.0011 (0.0036) model time 0.3765 (0.3816) loss 3.6006 (3.3403) grad_norm 1.0364 (1.3598) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][210/625] eta 0:02:39 lr 0.001589 wd 0.0500 time 0.3863 (0.3832) data time 0.0008 (0.0035) model time 0.3855 (0.3813) loss 3.4838 (3.3299) grad_norm 1.3955 (1.3632) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][220/625] eta 0:02:35 lr 0.001589 wd 0.0500 time 0.3836 (0.3830) data time 0.0010 (0.0033) model time 0.3826 (0.3810) loss 3.6497 (3.3323) grad_norm 0.9969 (1.3541) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][230/625] eta 0:02:31 lr 0.001589 wd 0.0500 time 0.3782 (0.3828) data time 0.0009 (0.0032) model time 0.3773 (0.3809) loss 3.8803 (3.3363) grad_norm 1.6442 (1.3633) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:03:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][240/625] eta 0:02:27 lr 0.001588 wd 0.0500 time 0.3790 (0.3826) data time 0.0008 (0.0032) model time 0.3782 (0.3807) loss 2.2637 (3.3385) grad_norm 1.2693 (1.3619) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][250/625] eta 0:02:24 lr 0.001588 wd 0.0500 time 0.5629 (0.3840) data time 0.0011 (0.0031) model time 0.5618 (0.3825) loss 3.4956 (3.3342) grad_norm 1.4842 (1.3585) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][260/625] eta 0:02:20 lr 0.001588 wd 0.0500 time 0.3802 (0.3838) data time 0.0008 (0.0030) model time 0.3794 (0.3823) loss 2.9113 (3.3352) grad_norm 0.9381 (1.3674) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][270/625] eta 0:02:16 lr 0.001588 wd 0.0500 time 0.3887 (0.3843) data time 0.0008 (0.0029) model time 0.3879 (0.3830) loss 2.4410 (3.3303) grad_norm 0.9833 (1.3601) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][280/625] eta 0:02:13 lr 0.001588 wd 0.0500 time 0.5755 (0.3856) data time 0.0008 (0.0029) model time 0.5747 (0.3845) loss 3.5994 (3.3237) grad_norm 1.2718 (1.3615) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][290/625] eta 0:02:09 lr 0.001588 wd 0.0500 time 0.3693 (0.3853) data time 0.0011 (0.0028) model time 0.3682 (0.3842) loss 3.0363 (3.3268) grad_norm 0.9656 (1.3615) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][300/625] eta 0:02:05 lr 0.001588 wd 0.0500 time 0.3781 (0.3850) data time 0.0011 (0.0027) model time 0.3771 (0.3838) loss 3.4435 (3.3172) grad_norm 1.3048 (1.3586) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][310/625] eta 0:02:01 lr 0.001587 wd 0.0500 time 0.3822 (0.3848) data time 0.0009 (0.0027) model time 0.3813 (0.3835) loss 4.3005 (3.3332) grad_norm 1.2285 (1.3557) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][320/625] eta 0:01:57 lr 0.001587 wd 0.0500 time 0.3625 (0.3845) data time 0.0009 (0.0026) model time 0.3616 (0.3832) loss 4.1876 (3.3375) grad_norm 1.8777 (1.3552) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][330/625] eta 0:01:53 lr 0.001587 wd 0.0500 time 0.3703 (0.3843) data time 0.0008 (0.0026) model time 0.3695 (0.3830) loss 2.5435 (3.3319) grad_norm 1.5420 (1.3577) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][340/625] eta 0:01:49 lr 0.001587 wd 0.0500 time 0.3766 (0.3841) data time 0.0010 (0.0026) model time 0.3755 (0.3828) loss 3.3066 (3.3325) grad_norm 1.2392 (1.3548) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][350/625] eta 0:01:45 lr 0.001587 wd 0.0500 time 0.3721 (0.3839) data time 0.0008 (0.0025) model time 0.3713 (0.3825) loss 2.3524 (3.3226) grad_norm 0.8464 (1.3523) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][360/625] eta 0:01:41 lr 0.001587 wd 0.0500 time 0.3855 (0.3837) data time 0.0011 (0.0025) model time 0.3844 (0.3823) loss 3.9639 (3.3296) grad_norm 1.3699 (1.3471) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][370/625] eta 0:01:38 lr 0.001587 wd 0.0500 time 0.5837 (0.3847) data time 0.0008 (0.0024) model time 0.5829 (0.3835) loss 3.2313 (3.3333) grad_norm 2.1389 (1.3476) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][380/625] eta 0:01:34 lr 0.001586 wd 0.0500 time 0.3794 (0.3846) data time 0.0011 (0.0024) model time 0.3782 (0.3833) loss 3.8221 (3.3433) grad_norm 1.2942 (1.3453) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][390/625] eta 0:01:30 lr 0.001586 wd 0.0500 time 0.3810 (0.3849) data time 0.0011 (0.0024) model time 0.3798 (0.3837) loss 2.9973 (3.3380) grad_norm 1.4074 (1.3450) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][400/625] eta 0:01:26 lr 0.001586 wd 0.0500 time 0.3810 (0.3847) data time 0.0009 (0.0024) model time 0.3801 (0.3834) loss 3.1982 (3.3444) grad_norm 0.8583 (1.3436) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][410/625] eta 0:01:22 lr 0.001586 wd 0.0500 time 0.3783 (0.3846) data time 0.0012 (0.0023) model time 0.3771 (0.3833) loss 3.5908 (3.3489) grad_norm 1.2056 (1.3445) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][420/625] eta 0:01:18 lr 0.001586 wd 0.0500 time 0.3821 (0.3845) data time 0.0009 (0.0023) model time 0.3813 (0.3832) loss 4.3277 (3.3534) grad_norm 1.1116 (1.3441) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][430/625] eta 0:01:14 lr 0.001586 wd 0.0500 time 0.3752 (0.3844) data time 0.0012 (0.0023) model time 0.3740 (0.3831) loss 3.6776 (3.3566) grad_norm 1.4639 (1.3423) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][440/625] eta 0:01:11 lr 0.001586 wd 0.0500 time 0.3684 (0.3843) data time 0.0008 (0.0023) model time 0.3676 (0.3829) loss 3.9379 (3.3611) grad_norm 1.1076 (1.3396) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][450/625] eta 0:01:07 lr 0.001585 wd 0.0500 time 0.3777 (0.3842) data time 0.0011 (0.0022) model time 0.3766 (0.3829) loss 3.4494 (3.3592) grad_norm 1.0630 (1.3392) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][460/625] eta 0:01:03 lr 0.001585 wd 0.0500 time 0.3813 (0.3845) data time 0.0008 (0.0022) model time 0.3805 (0.3833) loss 3.9749 (3.3638) grad_norm 1.1016 (1.3355) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][470/625] eta 0:00:59 lr 0.001585 wd 0.0500 time 0.3804 (0.3848) data time 0.0011 (0.0022) model time 0.3793 (0.3836) loss 3.4647 (3.3638) grad_norm 1.1826 (1.3341) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][480/625] eta 0:00:55 lr 0.001585 wd 0.0500 time 0.3795 (0.3848) data time 0.0008 (0.0022) model time 0.3787 (0.3835) loss 3.3689 (3.3623) grad_norm 1.1143 (1.3323) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][490/625] eta 0:00:51 lr 0.001585 wd 0.0500 time 0.3803 (0.3846) data time 0.0011 (0.0021) model time 0.3792 (0.3834) loss 3.3357 (3.3628) grad_norm 1.4959 (1.3326) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][500/625] eta 0:00:48 lr 0.001585 wd 0.0500 time 0.3810 (0.3845) data time 0.0010 (0.0021) model time 0.3800 (0.3833) loss 3.6202 (3.3647) grad_norm 1.0351 (1.3342) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][510/625] eta 0:00:44 lr 0.001585 wd 0.0500 time 0.3737 (0.3844) data time 0.0009 (0.0021) model time 0.3728 (0.3831) loss 3.5690 (3.3697) grad_norm 1.0216 (1.3319) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][520/625] eta 0:00:40 lr 0.001584 wd 0.0500 time 0.3762 (0.3842) data time 0.0009 (0.0021) model time 0.3754 (0.3830) loss 2.9261 (3.3687) grad_norm 1.5383 (1.3358) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][530/625] eta 0:00:36 lr 0.001584 wd 0.0500 time 0.3776 (0.3841) data time 0.0009 (0.0021) model time 0.3767 (0.3828) loss 3.5154 (3.3651) grad_norm 0.9617 (1.3350) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][540/625] eta 0:00:32 lr 0.001584 wd 0.0500 time 0.3801 (0.3840) data time 0.0011 (0.0020) model time 0.3790 (0.3827) loss 3.1036 (3.3625) grad_norm 1.3290 (1.3352) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][550/625] eta 0:00:28 lr 0.001584 wd 0.0500 time 0.3836 (0.3839) data time 0.0008 (0.0020) model time 0.3828 (0.3826) loss 4.1518 (3.3657) grad_norm 1.6179 (1.3338) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:05:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][560/625] eta 0:00:24 lr 0.001584 wd 0.0500 time 0.3737 (0.3838) data time 0.0008 (0.0020) model time 0.3729 (0.3825) loss 4.1149 (3.3667) grad_norm 1.0806 (1.3336) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][570/625] eta 0:00:21 lr 0.001584 wd 0.0500 time 0.3739 (0.3836) data time 0.0011 (0.0020) model time 0.3728 (0.3823) loss 3.5788 (3.3628) grad_norm 1.4948 (1.3346) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][580/625] eta 0:00:17 lr 0.001584 wd 0.0500 time 0.3784 (0.3846) data time 0.0009 (0.0020) model time 0.3776 (0.3834) loss 3.7280 (3.3632) grad_norm 1.0118 (1.3311) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][590/625] eta 0:00:13 lr 0.001583 wd 0.0500 time 0.3747 (0.3848) data time 0.0008 (0.0020) model time 0.3739 (0.3836) loss 3.5684 (3.3658) grad_norm 1.7156 (1.3287) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][600/625] eta 0:00:09 lr 0.001583 wd 0.0500 time 0.3768 (0.3850) data time 0.0011 (0.0019) model time 0.3756 (0.3838) loss 3.3854 (3.3646) grad_norm 1.5523 (1.3318) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][610/625] eta 0:00:05 lr 0.001583 wd 0.0500 time 0.3631 (0.3848) data time 0.0006 (0.0019) model time 0.3625 (0.3836) loss 3.4568 (3.3626) grad_norm 4.0317 (1.3352) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [104/300][620/625] eta 0:00:01 lr 0.001583 wd 0.0500 time 0.3651 (0.3845) data time 0.0008 (0.0019) model time 0.3643 (0.3833) loss 3.4474 (3.3624) grad_norm 1.3479 (1.3391) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 104 training takes 0:04:00 [2024-08-26 10:06:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:06:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:06:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 0.6704 (0.6704) Acc@1 85.938 (85.938) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 10:06:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.1562 (0.8384) Acc@1 73.340 (81.716) Acc@5 92.676 (96.351) Mem 11225MB [2024-08-26 10:06:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.097 (0.121) Loss 1.2188 (0.9920) Acc@1 71.240 (78.127) Acc@5 92.236 (94.406) Mem 11225MB [2024-08-26 10:06:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.787 Acc@5 94.320 [2024-08-26 10:06:28 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.8% [2024-08-26 10:06:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.908 (0.908) Loss 0.5454 (0.5454) Acc@1 88.232 (88.232) Acc@5 98.389 (98.389) Mem 11225MB [2024-08-26 10:06:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.184) Loss 0.9033 (0.6802) Acc@1 78.613 (84.610) Acc@5 95.068 (97.239) Mem 11225MB [2024-08-26 10:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.096 (0.141) Loss 1.0654 (0.8268) Acc@1 73.389 (80.901) Acc@5 93.066 (95.640) Mem 11225MB [2024-08-26 10:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.626 Acc@5 95.635 [2024-08-26 10:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.6% [2024-08-26 10:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.63% [2024-08-26 10:06:31 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:06:32 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][0/625] eta 0:08:43 lr 0.001583 wd 0.0500 time 0.8369 (0.8369) data time 0.4963 (0.4963) model time 0.0000 (0.0000) loss 3.3564 (3.3564) grad_norm 1.7033 (1.7033) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][10/625] eta 0:04:16 lr 0.001583 wd 0.0500 time 0.3799 (0.4166) data time 0.0010 (0.0461) model time 0.0000 (0.0000) loss 3.2493 (3.7056) grad_norm 1.4613 (1.3661) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][20/625] eta 0:04:01 lr 0.001583 wd 0.0500 time 0.3714 (0.3989) data time 0.0011 (0.0248) model time 0.0000 (0.0000) loss 3.3866 (3.5093) grad_norm 1.1363 (1.3286) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][30/625] eta 0:03:53 lr 0.001582 wd 0.0500 time 0.3686 (0.3919) data time 0.0008 (0.0171) model time 0.0000 (0.0000) loss 3.8779 (3.4048) grad_norm 1.0628 (1.3715) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][40/625] eta 0:03:47 lr 0.001582 wd 0.0500 time 0.3779 (0.3886) data time 0.0010 (0.0132) model time 0.0000 (0.0000) loss 2.4402 (3.4399) grad_norm 1.2210 (1.3421) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][50/625] eta 0:03:42 lr 0.001582 wd 0.0500 time 0.3825 (0.3865) data time 0.0011 (0.0108) model time 0.0000 (0.0000) loss 3.3788 (3.3632) grad_norm 1.1931 (1.3315) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][60/625] eta 0:03:37 lr 0.001582 wd 0.0500 time 0.3758 (0.3853) data time 0.0012 (0.0093) model time 0.3747 (0.3780) loss 2.8246 (3.3902) grad_norm 2.7684 (1.3710) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][70/625] eta 0:03:33 lr 0.001582 wd 0.0500 time 0.3743 (0.3843) data time 0.0010 (0.0081) model time 0.3734 (0.3773) loss 3.2079 (3.4170) grad_norm 1.2041 (1.3931) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][80/625] eta 0:03:29 lr 0.001582 wd 0.0500 time 0.3784 (0.3836) data time 0.0008 (0.0072) model time 0.3776 (0.3775) loss 3.5939 (3.3838) grad_norm 1.1381 (1.3872) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][90/625] eta 0:03:24 lr 0.001582 wd 0.0500 time 0.3787 (0.3827) data time 0.0011 (0.0066) model time 0.3776 (0.3767) loss 3.5918 (3.3877) grad_norm 1.3833 (1.3905) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][100/625] eta 0:03:22 lr 0.001581 wd 0.0500 time 0.3738 (0.3857) data time 0.0009 (0.0060) model time 0.3729 (0.3836) loss 3.6085 (3.4097) grad_norm 1.3696 (1.3968) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][110/625] eta 0:03:18 lr 0.001581 wd 0.0500 time 0.3773 (0.3848) data time 0.0011 (0.0056) model time 0.3763 (0.3823) loss 3.9613 (3.3941) grad_norm 1.3055 (1.3821) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][120/625] eta 0:03:14 lr 0.001581 wd 0.0500 time 0.3782 (0.3844) data time 0.0011 (0.0052) model time 0.3771 (0.3817) loss 3.1145 (3.3941) grad_norm 1.7198 (1.3892) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][130/625] eta 0:03:10 lr 0.001581 wd 0.0500 time 0.3757 (0.3852) data time 0.0011 (0.0049) model time 0.3746 (0.3831) loss 3.2470 (3.4032) grad_norm 1.0381 (1.3790) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][140/625] eta 0:03:06 lr 0.001581 wd 0.0500 time 0.3785 (0.3847) data time 0.0009 (0.0046) model time 0.3776 (0.3825) loss 3.5801 (3.4344) grad_norm 1.2156 (1.3624) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][150/625] eta 0:03:02 lr 0.001581 wd 0.0500 time 0.3782 (0.3843) data time 0.0011 (0.0044) model time 0.3771 (0.3820) loss 3.6875 (3.4331) grad_norm 1.6616 (1.3554) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][160/625] eta 0:02:59 lr 0.001581 wd 0.0500 time 0.3742 (0.3864) data time 0.0010 (0.0042) model time 0.3732 (0.3852) loss 3.6063 (3.4436) grad_norm 1.2303 (1.3575) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][170/625] eta 0:02:55 lr 0.001580 wd 0.0500 time 0.3797 (0.3860) data time 0.0008 (0.0040) model time 0.3788 (0.3847) loss 3.5391 (3.4272) grad_norm 1.5507 (1.3492) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][180/625] eta 0:02:51 lr 0.001580 wd 0.0500 time 0.3782 (0.3857) data time 0.0008 (0.0039) model time 0.3774 (0.3842) loss 3.9190 (3.4319) grad_norm 1.7205 (1.3514) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][190/625] eta 0:02:47 lr 0.001580 wd 0.0500 time 0.3788 (0.3853) data time 0.0009 (0.0037) model time 0.3779 (0.3838) loss 4.0293 (3.4388) grad_norm 1.3377 (1.3570) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][200/625] eta 0:02:43 lr 0.001580 wd 0.0500 time 0.3804 (0.3849) data time 0.0011 (0.0036) model time 0.3794 (0.3832) loss 3.8115 (3.4374) grad_norm 0.9367 (1.3438) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][210/625] eta 0:02:39 lr 0.001580 wd 0.0500 time 0.3794 (0.3847) data time 0.0008 (0.0035) model time 0.3786 (0.3830) loss 2.4203 (3.4439) grad_norm 2.1643 (1.3464) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:07:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][220/625] eta 0:02:35 lr 0.001580 wd 0.0500 time 0.4131 (0.3846) data time 0.0011 (0.0034) model time 0.4120 (0.3829) loss 3.8570 (3.4461) grad_norm 1.4728 (1.3476) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][230/625] eta 0:02:31 lr 0.001580 wd 0.0500 time 0.3777 (0.3844) data time 0.0009 (0.0033) model time 0.3768 (0.3827) loss 3.6734 (3.4447) grad_norm 1.2636 (1.3483) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][240/625] eta 0:02:27 lr 0.001579 wd 0.0500 time 0.3771 (0.3840) data time 0.0008 (0.0032) model time 0.3763 (0.3822) loss 2.9219 (3.4471) grad_norm 1.1885 (1.3427) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][250/625] eta 0:02:23 lr 0.001579 wd 0.0500 time 0.3776 (0.3837) data time 0.0011 (0.0031) model time 0.3765 (0.3818) loss 2.5089 (3.4447) grad_norm 1.2847 (1.3429) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][260/625] eta 0:02:19 lr 0.001579 wd 0.0500 time 0.3750 (0.3834) data time 0.0010 (0.0030) model time 0.3741 (0.3815) loss 2.9141 (3.4331) grad_norm 1.1405 (1.3487) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][270/625] eta 0:02:16 lr 0.001579 wd 0.0500 time 0.3854 (0.3832) data time 0.0011 (0.0030) model time 0.3843 (0.3813) loss 3.1928 (3.4345) grad_norm 1.3183 (1.3461) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][280/625] eta 0:02:12 lr 0.001579 wd 0.0500 time 0.3763 (0.3831) data time 0.0008 (0.0029) model time 0.3755 (0.3811) loss 3.6082 (3.4229) grad_norm 1.1619 (1.3438) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][290/625] eta 0:02:08 lr 0.001579 wd 0.0500 time 0.3743 (0.3829) data time 0.0008 (0.0029) model time 0.3735 (0.3809) loss 4.1978 (3.4225) grad_norm 2.2085 (1.3455) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][300/625] eta 0:02:04 lr 0.001579 wd 0.0500 time 0.3782 (0.3827) data time 0.0008 (0.0028) model time 0.3773 (0.3808) loss 4.2030 (3.4253) grad_norm 1.1152 (1.3504) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][310/625] eta 0:02:01 lr 0.001578 wd 0.0500 time 0.3691 (0.3845) data time 0.0010 (0.0027) model time 0.3682 (0.3830) loss 3.9273 (3.4309) grad_norm 1.3191 (1.3512) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][320/625] eta 0:01:57 lr 0.001578 wd 0.0500 time 0.3843 (0.3843) data time 0.0010 (0.0027) model time 0.3833 (0.3828) loss 4.0155 (3.4301) grad_norm 1.3350 (1.3489) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][330/625] eta 0:01:53 lr 0.001578 wd 0.0500 time 0.3797 (0.3842) data time 0.0011 (0.0027) model time 0.3786 (0.3826) loss 2.8494 (3.4286) grad_norm 2.2799 (1.3516) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][340/625] eta 0:01:49 lr 0.001578 wd 0.0500 time 0.3724 (0.3845) data time 0.0012 (0.0026) model time 0.3712 (0.3830) loss 3.3585 (3.4347) grad_norm 0.9221 (1.3490) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][350/625] eta 0:01:45 lr 0.001578 wd 0.0500 time 0.3720 (0.3849) data time 0.0011 (0.0026) model time 0.3710 (0.3835) loss 2.7929 (3.4289) grad_norm 1.0507 (1.3451) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][360/625] eta 0:01:41 lr 0.001578 wd 0.0500 time 0.3754 (0.3847) data time 0.0011 (0.0025) model time 0.3743 (0.3832) loss 2.9025 (3.4211) grad_norm 1.5517 (1.3431) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][370/625] eta 0:01:38 lr 0.001578 wd 0.0500 time 0.3795 (0.3845) data time 0.0011 (0.0025) model time 0.3784 (0.3831) loss 3.3787 (3.4215) grad_norm 1.6550 (1.3439) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][380/625] eta 0:01:34 lr 0.001577 wd 0.0500 time 0.3749 (0.3844) data time 0.0009 (0.0025) model time 0.3741 (0.3829) loss 3.2657 (3.4218) grad_norm 1.4795 (1.3422) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][390/625] eta 0:01:30 lr 0.001577 wd 0.0500 time 0.3734 (0.3843) data time 0.0011 (0.0024) model time 0.3723 (0.3828) loss 3.8441 (3.4123) grad_norm 1.3579 (1.3457) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][400/625] eta 0:01:26 lr 0.001577 wd 0.0500 time 0.3753 (0.3841) data time 0.0011 (0.0024) model time 0.3742 (0.3825) loss 2.7400 (3.4086) grad_norm 1.8242 (1.3753) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][410/625] eta 0:01:22 lr 0.001577 wd 0.0500 time 0.3785 (0.3839) data time 0.0009 (0.0024) model time 0.3777 (0.3824) loss 3.7096 (3.4119) grad_norm 1.1421 (1.3722) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][420/625] eta 0:01:18 lr 0.001577 wd 0.0500 time 0.3770 (0.3838) data time 0.0012 (0.0023) model time 0.3758 (0.3822) loss 3.6705 (3.4172) grad_norm 1.3496 (1.3686) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][430/625] eta 0:01:14 lr 0.001577 wd 0.0500 time 0.3669 (0.3841) data time 0.0009 (0.0023) model time 0.3660 (0.3826) loss 2.2417 (3.4141) grad_norm 1.4482 (1.3719) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][440/625] eta 0:01:11 lr 0.001576 wd 0.0500 time 0.3819 (0.3839) data time 0.0010 (0.0023) model time 0.3809 (0.3824) loss 3.5908 (3.4182) grad_norm 1.3711 (1.3723) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][450/625] eta 0:01:07 lr 0.001576 wd 0.0500 time 0.3792 (0.3838) data time 0.0011 (0.0023) model time 0.3781 (0.3823) loss 3.7303 (3.4161) grad_norm 1.0866 (1.3686) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][460/625] eta 0:01:03 lr 0.001576 wd 0.0500 time 0.3757 (0.3841) data time 0.0010 (0.0022) model time 0.3746 (0.3826) loss 3.6804 (3.4172) grad_norm 1.4595 (1.3669) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][470/625] eta 0:00:59 lr 0.001576 wd 0.0500 time 0.3734 (0.3844) data time 0.0009 (0.0022) model time 0.3725 (0.3830) loss 4.3200 (3.4170) grad_norm 1.9875 (1.3659) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][480/625] eta 0:00:55 lr 0.001576 wd 0.0500 time 0.3768 (0.3842) data time 0.0011 (0.0022) model time 0.3757 (0.3828) loss 3.4938 (3.4160) grad_norm 1.2742 (1.3674) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][490/625] eta 0:00:51 lr 0.001576 wd 0.0500 time 0.3712 (0.3849) data time 0.0010 (0.0022) model time 0.3701 (0.3835) loss 3.2546 (3.4139) grad_norm 2.0904 (1.3692) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][500/625] eta 0:00:48 lr 0.001576 wd 0.0500 time 0.3782 (0.3847) data time 0.0010 (0.0021) model time 0.3772 (0.3834) loss 2.8718 (3.4187) grad_norm 1.0289 (1.3716) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][510/625] eta 0:00:44 lr 0.001575 wd 0.0500 time 0.3757 (0.3846) data time 0.0008 (0.0021) model time 0.3749 (0.3832) loss 2.7738 (3.4175) grad_norm 1.3823 (1.3703) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][520/625] eta 0:00:40 lr 0.001575 wd 0.0500 time 0.3809 (0.3845) data time 0.0008 (0.0021) model time 0.3801 (0.3831) loss 2.9384 (3.4197) grad_norm 1.2279 (1.3714) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][530/625] eta 0:00:36 lr 0.001575 wd 0.0500 time 0.3752 (0.3843) data time 0.0012 (0.0021) model time 0.3741 (0.3830) loss 3.7398 (3.4146) grad_norm 1.8532 (1.3757) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][540/625] eta 0:00:32 lr 0.001575 wd 0.0500 time 0.3714 (0.3842) data time 0.0009 (0.0021) model time 0.3705 (0.3828) loss 3.8625 (3.4169) grad_norm 1.7769 (1.3732) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][550/625] eta 0:00:28 lr 0.001575 wd 0.0500 time 0.3758 (0.3841) data time 0.0009 (0.0020) model time 0.3749 (0.3827) loss 1.8602 (3.4116) grad_norm 1.4045 (1.3712) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][560/625] eta 0:00:24 lr 0.001575 wd 0.0500 time 0.3825 (0.3839) data time 0.0011 (0.0020) model time 0.3814 (0.3825) loss 3.9526 (3.4109) grad_norm 1.4636 (1.3702) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][570/625] eta 0:00:21 lr 0.001575 wd 0.0500 time 0.3693 (0.3838) data time 0.0008 (0.0020) model time 0.3685 (0.3824) loss 2.3912 (3.4084) grad_norm 1.2395 (1.3720) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][580/625] eta 0:00:17 lr 0.001574 wd 0.0500 time 0.3827 (0.3837) data time 0.0008 (0.0020) model time 0.3819 (0.3823) loss 3.8139 (3.4057) grad_norm 0.9959 (1.3752) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][590/625] eta 0:00:13 lr 0.001574 wd 0.0500 time 0.3812 (0.3836) data time 0.0009 (0.0020) model time 0.3803 (0.3822) loss 3.8373 (3.4063) grad_norm 1.3331 (1.3764) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][600/625] eta 0:00:09 lr 0.001574 wd 0.0500 time 0.3784 (0.3836) data time 0.0008 (0.0020) model time 0.3775 (0.3821) loss 4.0123 (3.4085) grad_norm 1.3355 (1.3747) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][610/625] eta 0:00:05 lr 0.001574 wd 0.0500 time 0.3655 (0.3841) data time 0.0006 (0.0020) model time 0.3649 (0.3828) loss 3.7450 (3.4100) grad_norm 1.4316 (1.3739) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [105/300][620/625] eta 0:00:01 lr 0.001574 wd 0.0500 time 0.3644 (0.3839) data time 0.0008 (0.0019) model time 0.3636 (0.3825) loss 3.7463 (3.4092) grad_norm 1.1790 (1.3712) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:32 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 105 training takes 0:03:59 [2024-08-26 10:10:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:10:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:10:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.6333 (0.6333) Acc@1 86.572 (86.572) Acc@5 97.559 (97.559) Mem 11225MB [2024-08-26 10:10:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.143) Loss 1.0840 (0.7904) Acc@1 75.293 (82.160) Acc@5 93.701 (96.418) Mem 11225MB [2024-08-26 10:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.1748 (0.9512) Acc@1 72.510 (78.355) Acc@5 92.139 (94.382) Mem 11225MB [2024-08-26 10:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.121 Acc@5 94.406 [2024-08-26 10:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.1% [2024-08-26 10:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.12% [2024-08-26 10:10:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 10:10:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 10:10:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.531 (0.531) Loss 0.5444 (0.5444) Acc@1 88.135 (88.135) Acc@5 98.389 (98.389) Mem 11225MB [2024-08-26 10:10:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.146) Loss 0.9019 (0.6794) Acc@1 78.662 (84.597) Acc@5 95.068 (97.239) Mem 11225MB [2024-08-26 10:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 1.0625 (0.8256) Acc@1 73.535 (80.911) Acc@5 93.115 (95.654) Mem 11225MB [2024-08-26 10:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.636 Acc@5 95.643 [2024-08-26 10:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.6% [2024-08-26 10:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.64% [2024-08-26 10:10:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:10:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:10:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][0/625] eta 0:07:56 lr 0.001574 wd 0.0500 time 0.7629 (0.7629) data time 0.4187 (0.4187) model time 0.0000 (0.0000) loss 3.0751 (3.0751) grad_norm 1.0234 (1.0234) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][10/625] eta 0:04:14 lr 0.001574 wd 0.0500 time 0.3861 (0.4134) data time 0.0008 (0.0390) model time 0.0000 (0.0000) loss 2.3914 (3.2499) grad_norm 1.7842 (1.6075) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][20/625] eta 0:03:59 lr 0.001573 wd 0.0500 time 0.3692 (0.3957) data time 0.0009 (0.0210) model time 0.0000 (0.0000) loss 3.7026 (3.4237) grad_norm 1.6036 (1.6336) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][30/625] eta 0:03:59 lr 0.001573 wd 0.0500 time 0.3733 (0.4032) data time 0.0012 (0.0146) model time 0.0000 (0.0000) loss 3.7721 (3.4782) grad_norm 1.3999 (1.5417) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:10:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][40/625] eta 0:03:52 lr 0.001573 wd 0.0500 time 0.3761 (0.3973) data time 0.0013 (0.0113) model time 0.0000 (0.0000) loss 3.2782 (3.5035) grad_norm 1.5822 (1.4805) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:11:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][50/625] eta 0:03:46 lr 0.001573 wd 0.0500 time 0.3848 (0.3938) data time 0.0011 (0.0093) model time 0.0000 (0.0000) loss 3.4213 (3.4866) grad_norm 1.2809 (1.5040) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:11:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][60/625] eta 0:03:41 lr 0.001573 wd 0.0500 time 0.3736 (0.3916) data time 0.0009 (0.0080) model time 0.3726 (0.3791) loss 3.5271 (3.4647) grad_norm 1.9436 (1.4897) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:11:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][70/625] eta 0:03:36 lr 0.001573 wd 0.0500 time 0.3731 (0.3895) data time 0.0008 (0.0070) model time 0.3722 (0.3775) loss 2.4562 (3.3939) grad_norm 1.1902 (1.4715) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:11:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][80/625] eta 0:03:31 lr 0.001573 wd 0.0500 time 0.3773 (0.3880) data time 0.0008 (0.0063) model time 0.3765 (0.3770) loss 3.5019 (3.4076) grad_norm 1.3732 (1.4278) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:11:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][90/625] eta 0:03:27 lr 0.001572 wd 0.0500 time 0.3741 (0.3870) data time 0.0011 (0.0057) model time 0.3730 (0.3771) loss 3.0236 (3.3847) grad_norm 1.0320 (1.4135) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 10:11:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][100/625] eta 0:03:22 lr 0.001572 wd 0.0500 time 0.3757 (0.3861) data time 0.0010 (0.0052) model time 0.3747 (0.3770) loss 3.6468 (3.4083) grad_norm 1.1191 (1.4155) loss_scale 2048.0000 (1094.9703) mem 11225MB [2024-08-26 10:11:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][110/625] eta 0:03:18 lr 0.001572 wd 0.0500 time 0.3777 (0.3853) data time 0.0009 (0.0049) model time 0.3768 (0.3770) loss 2.7904 (3.4053) grad_norm 1.9639 (1.4154) loss_scale 2048.0000 (1180.8288) mem 11225MB [2024-08-26 10:11:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][120/625] eta 0:03:14 lr 0.001572 wd 0.0500 time 0.3320 (0.3858) data time 0.0011 (0.0046) model time 0.3309 (0.3789) loss 3.4456 (3.4065) grad_norm 2.1683 (1.4115) loss_scale 2048.0000 (1252.4959) mem 11225MB [2024-08-26 10:11:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][130/625] eta 0:03:10 lr 0.001572 wd 0.0500 time 0.3744 (0.3852) data time 0.0011 (0.0043) model time 0.3732 (0.3786) loss 4.0840 (3.3932) grad_norm 1.1519 (1.4029) loss_scale 2048.0000 (1313.2214) mem 11225MB [2024-08-26 10:11:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][140/625] eta 0:03:06 lr 0.001572 wd 0.0500 time 0.3727 (0.3846) data time 0.0011 (0.0041) model time 0.3716 (0.3783) loss 3.3340 (3.3978) grad_norm 1.3090 (1.3878) loss_scale 2048.0000 (1365.3333) mem 11225MB [2024-08-26 10:11:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][150/625] eta 0:03:02 lr 0.001572 wd 0.0500 time 0.3771 (0.3842) data time 0.0008 (0.0039) model time 0.3763 (0.3781) loss 3.6177 (3.3820) grad_norm 1.7161 (1.3974) loss_scale 2048.0000 (1410.5430) mem 11225MB [2024-08-26 10:11:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][160/625] eta 0:02:58 lr 0.001571 wd 0.0500 time 0.3808 (0.3839) data time 0.0008 (0.0037) model time 0.3800 (0.3781) loss 4.0580 (3.3800) grad_norm 1.2249 (1.3931) loss_scale 2048.0000 (1450.1366) mem 11225MB [2024-08-26 10:11:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][170/625] eta 0:02:55 lr 0.001571 wd 0.0500 time 0.3777 (0.3857) data time 0.0008 (0.0036) model time 0.3769 (0.3810) loss 3.9305 (3.3817) grad_norm 2.1024 (1.3954) loss_scale 2048.0000 (1485.0994) mem 11225MB [2024-08-26 10:11:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][180/625] eta 0:02:51 lr 0.001571 wd 0.0500 time 0.3743 (0.3851) data time 0.0011 (0.0034) model time 0.3732 (0.3806) loss 3.1942 (3.3881) grad_norm 1.2563 (1.4008) loss_scale 2048.0000 (1516.1989) mem 11225MB [2024-08-26 10:11:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][190/625] eta 0:02:47 lr 0.001571 wd 0.0500 time 0.3796 (0.3847) data time 0.0012 (0.0033) model time 0.3784 (0.3802) loss 3.5098 (3.3871) grad_norm 1.0841 (1.4053) loss_scale 2048.0000 (1544.0419) mem 11225MB [2024-08-26 10:11:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][200/625] eta 0:02:43 lr 0.001571 wd 0.0500 time 0.3716 (0.3843) data time 0.0011 (0.0032) model time 0.3705 (0.3799) loss 2.3211 (3.3770) grad_norm 1.7626 (1.4064) loss_scale 2048.0000 (1569.1144) mem 11225MB [2024-08-26 10:12:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][210/625] eta 0:02:39 lr 0.001571 wd 0.0500 time 0.3777 (0.3839) data time 0.0013 (0.0031) model time 0.3763 (0.3796) loss 3.7383 (3.3900) grad_norm 1.7476 (1.4016) loss_scale 2048.0000 (1591.8104) mem 11225MB [2024-08-26 10:12:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][220/625] eta 0:02:35 lr 0.001571 wd 0.0500 time 0.3825 (0.3837) data time 0.0010 (0.0030) model time 0.3814 (0.3795) loss 2.4709 (3.3793) grad_norm 1.4158 (1.4025) loss_scale 2048.0000 (1612.4525) mem 11225MB [2024-08-26 10:12:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][230/625] eta 0:02:31 lr 0.001570 wd 0.0500 time 0.3741 (0.3834) data time 0.0011 (0.0029) model time 0.3730 (0.3793) loss 2.8298 (3.3761) grad_norm 1.1523 (1.4006) loss_scale 2048.0000 (1631.3074) mem 11225MB [2024-08-26 10:12:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][240/625] eta 0:02:28 lr 0.001570 wd 0.0500 time 0.3312 (0.3857) data time 0.0011 (0.0029) model time 0.3300 (0.3824) loss 3.6861 (3.3729) grad_norm 1.2673 (1.3958) loss_scale 2048.0000 (1648.5975) mem 11225MB [2024-08-26 10:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][250/625] eta 0:02:24 lr 0.001570 wd 0.0500 time 0.3731 (0.3854) data time 0.0010 (0.0028) model time 0.3722 (0.3820) loss 3.3130 (3.3782) grad_norm 1.7501 (1.3973) loss_scale 2048.0000 (1664.5100) mem 11225MB [2024-08-26 10:12:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][260/625] eta 0:02:20 lr 0.001570 wd 0.0500 time 0.3697 (0.3850) data time 0.0011 (0.0027) model time 0.3686 (0.3817) loss 3.4846 (3.3826) grad_norm 1.1224 (1.3952) loss_scale 2048.0000 (1679.2031) mem 11225MB [2024-08-26 10:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][270/625] eta 0:02:16 lr 0.001570 wd 0.0500 time 0.3765 (0.3848) data time 0.0010 (0.0027) model time 0.3755 (0.3816) loss 2.1156 (3.3685) grad_norm 2.8291 (1.4009) loss_scale 2048.0000 (1692.8118) mem 11225MB [2024-08-26 10:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][280/625] eta 0:02:12 lr 0.001570 wd 0.0500 time 0.3764 (0.3845) data time 0.0011 (0.0026) model time 0.3753 (0.3813) loss 3.1306 (3.3636) grad_norm 1.0603 (1.4058) loss_scale 2048.0000 (1705.4520) mem 11225MB [2024-08-26 10:12:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][290/625] eta 0:02:09 lr 0.001570 wd 0.0500 time 0.3786 (0.3858) data time 0.0010 (0.0026) model time 0.3776 (0.3829) loss 3.3735 (3.3642) grad_norm 1.1444 (1.4081) loss_scale 2048.0000 (1717.2234) mem 11225MB [2024-08-26 10:12:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][300/625] eta 0:02:05 lr 0.001569 wd 0.0500 time 0.3716 (0.3855) data time 0.0011 (0.0025) model time 0.3706 (0.3827) loss 3.9463 (3.3707) grad_norm 1.1344 (1.4067) loss_scale 2048.0000 (1728.2126) mem 11225MB [2024-08-26 10:12:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][310/625] eta 0:02:01 lr 0.001569 wd 0.0500 time 0.3855 (0.3853) data time 0.0010 (0.0025) model time 0.3845 (0.3825) loss 3.7593 (3.3671) grad_norm 1.2568 (1.4048) loss_scale 2048.0000 (1738.4952) mem 11225MB [2024-08-26 10:12:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][320/625] eta 0:01:57 lr 0.001569 wd 0.0500 time 0.3788 (0.3851) data time 0.0011 (0.0024) model time 0.3777 (0.3823) loss 3.8419 (3.3702) grad_norm 1.2177 (1.3974) loss_scale 2048.0000 (1748.1371) mem 11225MB [2024-08-26 10:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][330/625] eta 0:01:53 lr 0.001569 wd 0.0500 time 0.3711 (0.3849) data time 0.0009 (0.0024) model time 0.3702 (0.3821) loss 3.8590 (3.3659) grad_norm 1.4613 (1.3931) loss_scale 2048.0000 (1757.1964) mem 11225MB [2024-08-26 10:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][340/625] eta 0:01:49 lr 0.001569 wd 0.0500 time 0.3834 (0.3847) data time 0.0011 (0.0024) model time 0.3823 (0.3819) loss 3.1637 (3.3670) grad_norm 1.6105 (1.3953) loss_scale 2048.0000 (1765.7243) mem 11225MB [2024-08-26 10:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][350/625] eta 0:01:45 lr 0.001569 wd 0.0500 time 0.3698 (0.3845) data time 0.0008 (0.0023) model time 0.3690 (0.3818) loss 4.2458 (3.3683) grad_norm 1.5264 (1.3952) loss_scale 2048.0000 (1773.7664) mem 11225MB [2024-08-26 10:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][360/625] eta 0:01:41 lr 0.001569 wd 0.0500 time 0.3776 (0.3843) data time 0.0008 (0.0023) model time 0.3768 (0.3816) loss 4.1930 (3.3690) grad_norm 0.8398 (1.3925) loss_scale 2048.0000 (1781.3629) mem 11225MB [2024-08-26 10:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][370/625] eta 0:01:37 lr 0.001568 wd 0.0500 time 0.3743 (0.3841) data time 0.0009 (0.0023) model time 0.3734 (0.3814) loss 2.4664 (3.3678) grad_norm 1.6296 (1.3911) loss_scale 2048.0000 (1788.5499) mem 11225MB [2024-08-26 10:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][380/625] eta 0:01:34 lr 0.001568 wd 0.0500 time 0.3818 (0.3839) data time 0.0011 (0.0022) model time 0.3807 (0.3813) loss 2.8832 (3.3618) grad_norm 1.4141 (1.3850) loss_scale 2048.0000 (1795.3596) mem 11225MB [2024-08-26 10:13:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][390/625] eta 0:01:30 lr 0.001568 wd 0.0500 time 0.3714 (0.3838) data time 0.0009 (0.0022) model time 0.3705 (0.3812) loss 2.7646 (3.3551) grad_norm 0.9346 (1.3851) loss_scale 2048.0000 (1801.8210) mem 11225MB [2024-08-26 10:13:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][400/625] eta 0:01:26 lr 0.001568 wd 0.0500 time 0.3818 (0.3838) data time 0.0008 (0.0022) model time 0.3810 (0.3811) loss 3.8654 (3.3584) grad_norm 0.8889 (1.3962) loss_scale 2048.0000 (1807.9601) mem 11225MB [2024-08-26 10:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][410/625] eta 0:01:22 lr 0.001568 wd 0.0500 time 0.3788 (0.3837) data time 0.0008 (0.0022) model time 0.3780 (0.3811) loss 3.6688 (3.3555) grad_norm 1.0968 (1.3925) loss_scale 2048.0000 (1813.8005) mem 11225MB [2024-08-26 10:13:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][420/625] eta 0:01:18 lr 0.001568 wd 0.0500 time 0.3792 (0.3846) data time 0.0011 (0.0021) model time 0.3782 (0.3822) loss 3.8879 (3.3548) grad_norm 1.2263 (1.3884) loss_scale 2048.0000 (1819.3634) mem 11225MB [2024-08-26 10:13:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][430/625] eta 0:01:14 lr 0.001567 wd 0.0500 time 0.3819 (0.3845) data time 0.0010 (0.0021) model time 0.3809 (0.3821) loss 3.8554 (3.3517) grad_norm 1.6056 (1.3906) loss_scale 2048.0000 (1824.6682) mem 11225MB [2024-08-26 10:13:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][440/625] eta 0:01:11 lr 0.001567 wd 0.0500 time 0.3727 (0.3843) data time 0.0012 (0.0021) model time 0.3715 (0.3819) loss 3.9799 (3.3438) grad_norm 1.5931 (1.3989) loss_scale 2048.0000 (1829.7324) mem 11225MB [2024-08-26 10:13:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][450/625] eta 0:01:07 lr 0.001567 wd 0.0500 time 0.3805 (0.3845) data time 0.0011 (0.0021) model time 0.3794 (0.3822) loss 3.6153 (3.3446) grad_norm 1.0403 (1.3989) loss_scale 2048.0000 (1834.5721) mem 11225MB [2024-08-26 10:13:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][460/625] eta 0:01:03 lr 0.001567 wd 0.0500 time 0.3760 (0.3844) data time 0.0011 (0.0021) model time 0.3749 (0.3821) loss 3.7831 (3.3512) grad_norm 0.9975 (1.3960) loss_scale 2048.0000 (1839.2017) mem 11225MB [2024-08-26 10:13:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][470/625] eta 0:00:59 lr 0.001567 wd 0.0500 time 0.3764 (0.3843) data time 0.0011 (0.0020) model time 0.3754 (0.3820) loss 3.6423 (3.3515) grad_norm 1.3963 (1.3923) loss_scale 2048.0000 (1843.6348) mem 11225MB [2024-08-26 10:13:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][480/625] eta 0:00:55 lr 0.001567 wd 0.0500 time 0.3741 (0.3841) data time 0.0008 (0.0020) model time 0.3733 (0.3818) loss 4.2778 (3.3522) grad_norm 1.2569 (1.3881) loss_scale 2048.0000 (1847.8836) mem 11225MB [2024-08-26 10:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][490/625] eta 0:00:51 lr 0.001567 wd 0.0500 time 0.3781 (0.3840) data time 0.0009 (0.0020) model time 0.3772 (0.3817) loss 4.1899 (3.3498) grad_norm 1.3949 (1.3862) loss_scale 2048.0000 (1851.9593) mem 11225MB [2024-08-26 10:13:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][500/625] eta 0:00:48 lr 0.001566 wd 0.0500 time 0.3718 (0.3847) data time 0.0011 (0.0020) model time 0.3707 (0.3826) loss 2.6183 (3.3520) grad_norm 1.3623 (1.3864) loss_scale 2048.0000 (1855.8723) mem 11225MB [2024-08-26 10:13:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][510/625] eta 0:00:44 lr 0.001566 wd 0.0500 time 0.3736 (0.3846) data time 0.0011 (0.0020) model time 0.3725 (0.3824) loss 3.6650 (3.3571) grad_norm 0.8005 (1.3830) loss_scale 2048.0000 (1859.6321) mem 11225MB [2024-08-26 10:14:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][520/625] eta 0:00:40 lr 0.001566 wd 0.0500 time 0.3909 (0.3845) data time 0.0010 (0.0019) model time 0.3898 (0.3824) loss 4.4285 (3.3630) grad_norm 1.7831 (1.3824) loss_scale 2048.0000 (1863.2476) mem 11225MB [2024-08-26 10:14:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][530/625] eta 0:00:36 lr 0.001566 wd 0.0500 time 0.3773 (0.3845) data time 0.0010 (0.0019) model time 0.3763 (0.3823) loss 3.1576 (3.3599) grad_norm 0.9472 (1.3811) loss_scale 2048.0000 (1866.7269) mem 11225MB [2024-08-26 10:14:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][540/625] eta 0:00:32 lr 0.001566 wd 0.0500 time 0.3739 (0.3851) data time 0.0009 (0.0019) model time 0.3730 (0.3831) loss 3.4258 (3.3623) grad_norm 1.0476 (1.3796) loss_scale 2048.0000 (1870.0776) mem 11225MB [2024-08-26 10:14:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][550/625] eta 0:00:28 lr 0.001566 wd 0.0500 time 0.4031 (0.3851) data time 0.0009 (0.0019) model time 0.4022 (0.3831) loss 4.1710 (3.3611) grad_norm 1.2802 (1.3841) loss_scale 2048.0000 (1873.3067) mem 11225MB [2024-08-26 10:14:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][560/625] eta 0:00:25 lr 0.001566 wd 0.0500 time 0.3672 (0.3851) data time 0.0011 (0.0019) model time 0.3661 (0.3830) loss 3.5766 (3.3604) grad_norm 1.2375 (1.3829) loss_scale 2048.0000 (1876.4207) mem 11225MB [2024-08-26 10:14:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][570/625] eta 0:00:21 lr 0.001565 wd 0.0500 time 0.3767 (0.3850) data time 0.0009 (0.0019) model time 0.3759 (0.3830) loss 2.2965 (3.3575) grad_norm 1.9846 (1.3817) loss_scale 2048.0000 (1879.4256) mem 11225MB [2024-08-26 10:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][580/625] eta 0:00:17 lr 0.001565 wd 0.0500 time 0.4022 (0.3850) data time 0.0008 (0.0019) model time 0.4014 (0.3829) loss 3.9340 (3.3592) grad_norm 1.4768 (1.3851) loss_scale 2048.0000 (1882.3270) mem 11225MB [2024-08-26 10:14:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][590/625] eta 0:00:13 lr 0.001565 wd 0.0500 time 0.3643 (0.3849) data time 0.0009 (0.0019) model time 0.3634 (0.3828) loss 2.3622 (3.3584) grad_norm 1.3226 (1.3837) loss_scale 2048.0000 (1885.1303) mem 11225MB [2024-08-26 10:14:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][600/625] eta 0:00:09 lr 0.001565 wd 0.0500 time 0.3782 (0.3848) data time 0.0011 (0.0019) model time 0.3770 (0.3827) loss 3.7389 (3.3546) grad_norm 1.3179 (1.3837) loss_scale 2048.0000 (1887.8403) mem 11225MB [2024-08-26 10:14:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][610/625] eta 0:00:05 lr 0.001565 wd 0.0500 time 0.3803 (0.3847) data time 0.0006 (0.0019) model time 0.3797 (0.3827) loss 3.9365 (3.3608) grad_norm 1.2054 (1.3818) loss_scale 2048.0000 (1890.4615) mem 11225MB [2024-08-26 10:14:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [106/300][620/625] eta 0:00:01 lr 0.001565 wd 0.0500 time 0.3646 (0.3845) data time 0.0006 (0.0019) model time 0.3640 (0.3824) loss 3.1213 (3.3581) grad_norm 1.1414 (1.3817) loss_scale 2048.0000 (1892.9984) mem 11225MB [2024-08-26 10:14:42 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 106 training takes 0:04:00 [2024-08-26 10:14:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:14:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:14:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.503 (0.503) Loss 0.6128 (0.6128) Acc@1 86.279 (86.279) Acc@5 97.852 (97.852) Mem 11225MB [2024-08-26 10:14:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 1.0771 (0.7909) Acc@1 75.146 (82.120) Acc@5 93.652 (96.471) Mem 11225MB [2024-08-26 10:14:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.2402 (0.9588) Acc@1 70.605 (78.195) Acc@5 91.064 (94.468) Mem 11225MB [2024-08-26 10:14:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.939 Acc@5 94.472 [2024-08-26 10:14:46 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 77.9% [2024-08-26 10:14:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.904 (0.904) Loss 0.5444 (0.5444) Acc@1 88.428 (88.428) Acc@5 98.340 (98.340) Mem 11225MB [2024-08-26 10:14:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.181) Loss 0.9019 (0.6796) Acc@1 78.857 (84.650) Acc@5 95.117 (97.252) Mem 11225MB [2024-08-26 10:14:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.139) Loss 1.0576 (0.8250) Acc@1 73.730 (80.969) Acc@5 93.262 (95.689) Mem 11225MB [2024-08-26 10:14:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.702 Acc@5 95.677 [2024-08-26 10:14:50 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.7% [2024-08-26 10:14:50 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.70% [2024-08-26 10:14:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:14:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:14:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][0/625] eta 0:08:23 lr 0.001565 wd 0.0500 time 0.8057 (0.8057) data time 0.4755 (0.4755) model time 0.0000 (0.0000) loss 3.1636 (3.1636) grad_norm 1.7093 (1.7093) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:14:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][10/625] eta 0:04:17 lr 0.001564 wd 0.0500 time 0.3893 (0.4190) data time 0.0010 (0.0443) model time 0.0000 (0.0000) loss 3.5064 (3.3544) grad_norm 1.2550 (1.2750) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:14:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][20/625] eta 0:04:02 lr 0.001564 wd 0.0500 time 0.3687 (0.4007) data time 0.0008 (0.0241) model time 0.0000 (0.0000) loss 3.8341 (3.3738) grad_norm 1.2996 (1.2684) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][30/625] eta 0:03:54 lr 0.001564 wd 0.0500 time 0.3750 (0.3943) data time 0.0012 (0.0168) model time 0.0000 (0.0000) loss 3.2265 (3.3474) grad_norm 1.1350 (1.2994) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][40/625] eta 0:03:49 lr 0.001564 wd 0.0500 time 0.3982 (0.3918) data time 0.0008 (0.0131) model time 0.0000 (0.0000) loss 3.7936 (3.2604) grad_norm 1.0773 (1.2950) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][50/625] eta 0:03:43 lr 0.001564 wd 0.0500 time 0.3636 (0.3894) data time 0.0009 (0.0109) model time 0.0000 (0.0000) loss 2.4499 (3.2421) grad_norm 1.2009 (1.2528) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][60/625] eta 0:03:41 lr 0.001564 wd 0.0500 time 0.3798 (0.3915) data time 0.0009 (0.0093) model time 0.3789 (0.4010) loss 3.1471 (3.2315) grad_norm 1.2394 (1.2880) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][70/625] eta 0:03:40 lr 0.001564 wd 0.0500 time 0.3774 (0.3977) data time 0.0010 (0.0081) model time 0.3764 (0.4179) loss 3.0286 (3.2365) grad_norm 0.8901 (1.2946) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][80/625] eta 0:03:35 lr 0.001563 wd 0.0500 time 0.3742 (0.3951) data time 0.0011 (0.0073) model time 0.3731 (0.4036) loss 4.0063 (3.2476) grad_norm 1.2599 (1.3321) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][90/625] eta 0:03:30 lr 0.001563 wd 0.0500 time 0.3810 (0.3930) data time 0.0008 (0.0066) model time 0.3802 (0.3965) loss 4.0671 (3.2804) grad_norm 1.2116 (1.3290) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][100/625] eta 0:03:26 lr 0.001563 wd 0.0500 time 0.3688 (0.3927) data time 0.0011 (0.0061) model time 0.3677 (0.3950) loss 3.8068 (3.2746) grad_norm 1.2836 (1.3165) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][110/625] eta 0:03:21 lr 0.001563 wd 0.0500 time 0.3746 (0.3917) data time 0.0008 (0.0056) model time 0.3738 (0.3924) loss 3.6963 (3.2785) grad_norm 1.1597 (1.3209) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][120/625] eta 0:03:17 lr 0.001563 wd 0.0500 time 0.3781 (0.3905) data time 0.0011 (0.0053) model time 0.3769 (0.3902) loss 3.2268 (3.2691) grad_norm 1.2676 (1.3099) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][130/625] eta 0:03:12 lr 0.001563 wd 0.0500 time 0.3767 (0.3896) data time 0.0011 (0.0049) model time 0.3756 (0.3885) loss 3.4279 (3.2728) grad_norm 2.0753 (1.3183) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][140/625] eta 0:03:08 lr 0.001563 wd 0.0500 time 0.3745 (0.3885) data time 0.0010 (0.0047) model time 0.3736 (0.3869) loss 3.9385 (3.2792) grad_norm 1.7621 (1.3158) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][150/625] eta 0:03:04 lr 0.001562 wd 0.0500 time 0.3793 (0.3878) data time 0.0008 (0.0044) model time 0.3785 (0.3858) loss 4.2562 (3.2846) grad_norm 2.2746 (1.3318) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][160/625] eta 0:03:00 lr 0.001562 wd 0.0500 time 0.3757 (0.3871) data time 0.0011 (0.0042) model time 0.3746 (0.3849) loss 3.6179 (3.3023) grad_norm 1.4453 (1.3287) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:15:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][170/625] eta 0:02:55 lr 0.001562 wd 0.0500 time 0.3818 (0.3868) data time 0.0008 (0.0041) model time 0.3810 (0.3845) loss 4.0150 (3.3134) grad_norm 1.6412 (1.3611) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][180/625] eta 0:02:51 lr 0.001562 wd 0.0500 time 0.3863 (0.3864) data time 0.0011 (0.0039) model time 0.3852 (0.3840) loss 3.1344 (3.3280) grad_norm 1.3160 (1.3546) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][190/625] eta 0:02:47 lr 0.001562 wd 0.0500 time 0.3770 (0.3858) data time 0.0009 (0.0038) model time 0.3761 (0.3833) loss 3.8976 (3.3492) grad_norm 1.4381 (1.3527) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][200/625] eta 0:02:43 lr 0.001562 wd 0.0500 time 0.3794 (0.3854) data time 0.0009 (0.0036) model time 0.3786 (0.3829) loss 4.3605 (3.3480) grad_norm 3.4244 (1.3678) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][210/625] eta 0:02:39 lr 0.001562 wd 0.0500 time 0.3760 (0.3850) data time 0.0011 (0.0035) model time 0.3749 (0.3825) loss 2.8212 (3.3542) grad_norm 1.5282 (1.3863) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][220/625] eta 0:02:35 lr 0.001561 wd 0.0500 time 0.3772 (0.3847) data time 0.0011 (0.0034) model time 0.3762 (0.3821) loss 3.6877 (3.3597) grad_norm 0.9963 (1.3788) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][230/625] eta 0:02:31 lr 0.001561 wd 0.0500 time 0.3799 (0.3844) data time 0.0011 (0.0033) model time 0.3788 (0.3818) loss 3.0942 (3.3571) grad_norm 1.2635 (1.3750) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][240/625] eta 0:02:27 lr 0.001561 wd 0.0500 time 0.3759 (0.3841) data time 0.0011 (0.0032) model time 0.3748 (0.3815) loss 2.8575 (3.3562) grad_norm 1.4721 (1.3708) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][250/625] eta 0:02:24 lr 0.001561 wd 0.0500 time 0.5549 (0.3860) data time 0.0012 (0.0032) model time 0.5537 (0.3839) loss 3.0995 (3.3628) grad_norm 1.1098 (1.3629) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][260/625] eta 0:02:20 lr 0.001561 wd 0.0500 time 0.3685 (0.3856) data time 0.0008 (0.0031) model time 0.3677 (0.3835) loss 3.5877 (3.3712) grad_norm 1.0034 (1.3561) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][270/625] eta 0:02:16 lr 0.001561 wd 0.0500 time 0.3804 (0.3853) data time 0.0010 (0.0030) model time 0.3793 (0.3832) loss 3.5565 (3.3748) grad_norm 1.0723 (1.3630) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][280/625] eta 0:02:13 lr 0.001560 wd 0.0500 time 0.3747 (0.3859) data time 0.0011 (0.0029) model time 0.3736 (0.3839) loss 2.8611 (3.3657) grad_norm 1.1026 (1.3589) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][290/625] eta 0:02:09 lr 0.001560 wd 0.0500 time 0.3778 (0.3856) data time 0.0011 (0.0029) model time 0.3767 (0.3836) loss 3.8433 (3.3595) grad_norm 1.5053 (1.3582) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][300/625] eta 0:02:05 lr 0.001560 wd 0.0500 time 0.3811 (0.3853) data time 0.0008 (0.0028) model time 0.3803 (0.3833) loss 3.5335 (3.3569) grad_norm 1.0495 (1.3561) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][310/625] eta 0:02:01 lr 0.001560 wd 0.0500 time 0.3762 (0.3857) data time 0.0010 (0.0028) model time 0.3752 (0.3838) loss 3.2245 (3.3485) grad_norm 1.0128 (1.3570) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][320/625] eta 0:01:57 lr 0.001560 wd 0.0500 time 0.3743 (0.3855) data time 0.0011 (0.0027) model time 0.3732 (0.3835) loss 2.5953 (3.3521) grad_norm 1.0405 (1.3549) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][330/625] eta 0:01:53 lr 0.001560 wd 0.0500 time 0.3834 (0.3853) data time 0.0009 (0.0027) model time 0.3825 (0.3833) loss 3.4355 (3.3527) grad_norm 1.3412 (1.3570) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][340/625] eta 0:01:49 lr 0.001560 wd 0.0500 time 0.3800 (0.3851) data time 0.0009 (0.0026) model time 0.3791 (0.3831) loss 4.1977 (3.3609) grad_norm 1.6714 (1.3633) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][350/625] eta 0:01:45 lr 0.001559 wd 0.0500 time 0.3808 (0.3848) data time 0.0009 (0.0026) model time 0.3799 (0.3828) loss 4.0341 (3.3605) grad_norm 1.1966 (1.3720) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][360/625] eta 0:01:41 lr 0.001559 wd 0.0500 time 0.3776 (0.3845) data time 0.0011 (0.0026) model time 0.3766 (0.3826) loss 3.6596 (3.3629) grad_norm 1.6561 (1.3689) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][370/625] eta 0:01:38 lr 0.001559 wd 0.0500 time 0.6062 (0.3860) data time 0.0008 (0.0025) model time 0.6054 (0.3843) loss 3.7410 (3.3689) grad_norm 1.1403 (1.3718) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][380/625] eta 0:01:34 lr 0.001559 wd 0.0500 time 0.3825 (0.3858) data time 0.0008 (0.0025) model time 0.3817 (0.3841) loss 2.6350 (3.3682) grad_norm 0.9975 (1.3679) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][390/625] eta 0:01:30 lr 0.001559 wd 0.0500 time 0.3763 (0.3855) data time 0.0008 (0.0024) model time 0.3755 (0.3838) loss 2.9330 (3.3656) grad_norm 1.2439 (1.3667) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][400/625] eta 0:01:26 lr 0.001559 wd 0.0500 time 0.3704 (0.3853) data time 0.0010 (0.0024) model time 0.3693 (0.3835) loss 4.2705 (3.3619) grad_norm 1.2465 (1.3639) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][410/625] eta 0:01:22 lr 0.001559 wd 0.0500 time 0.3770 (0.3851) data time 0.0012 (0.0024) model time 0.3758 (0.3833) loss 2.9221 (3.3575) grad_norm 1.5234 (1.3641) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][420/625] eta 0:01:18 lr 0.001558 wd 0.0500 time 0.3697 (0.3849) data time 0.0011 (0.0024) model time 0.3687 (0.3831) loss 3.2944 (3.3597) grad_norm 1.6021 (1.3647) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][430/625] eta 0:01:15 lr 0.001558 wd 0.0500 time 0.3734 (0.3847) data time 0.0008 (0.0024) model time 0.3726 (0.3829) loss 3.3839 (3.3549) grad_norm 1.5615 (1.3684) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][440/625] eta 0:01:11 lr 0.001558 wd 0.0500 time 0.3855 (0.3845) data time 0.0008 (0.0023) model time 0.3846 (0.3827) loss 2.6750 (3.3489) grad_norm 1.4085 (1.3665) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][450/625] eta 0:01:07 lr 0.001558 wd 0.0500 time 0.3684 (0.3843) data time 0.0008 (0.0023) model time 0.3675 (0.3825) loss 2.9927 (3.3441) grad_norm 1.4951 (1.3724) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][460/625] eta 0:01:03 lr 0.001558 wd 0.0500 time 0.3806 (0.3842) data time 0.0008 (0.0023) model time 0.3798 (0.3823) loss 3.6615 (3.3500) grad_norm 1.4621 (1.3726) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][470/625] eta 0:00:59 lr 0.001558 wd 0.0500 time 0.3726 (0.3850) data time 0.0012 (0.0022) model time 0.3715 (0.3832) loss 3.2596 (3.3460) grad_norm 1.4091 (1.3723) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:17:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][480/625] eta 0:00:55 lr 0.001558 wd 0.0500 time 0.3753 (0.3848) data time 0.0008 (0.0022) model time 0.3744 (0.3831) loss 3.6313 (3.3419) grad_norm 1.2133 (1.3721) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][490/625] eta 0:00:51 lr 0.001557 wd 0.0500 time 0.6121 (0.3851) data time 0.0008 (0.0022) model time 0.6113 (0.3834) loss 3.7311 (3.3427) grad_norm 1.6252 (1.3720) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][500/625] eta 0:00:48 lr 0.001557 wd 0.0500 time 0.3725 (0.3849) data time 0.0011 (0.0022) model time 0.3714 (0.3832) loss 3.4690 (3.3427) grad_norm 1.3219 (1.3726) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][510/625] eta 0:00:44 lr 0.001557 wd 0.0500 time 0.3740 (0.3847) data time 0.0010 (0.0022) model time 0.3729 (0.3830) loss 3.5892 (3.3446) grad_norm 1.0234 (1.3719) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][520/625] eta 0:00:40 lr 0.001557 wd 0.0500 time 0.3740 (0.3846) data time 0.0012 (0.0021) model time 0.3728 (0.3828) loss 3.5887 (3.3491) grad_norm 1.4620 (1.3694) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][530/625] eta 0:00:36 lr 0.001557 wd 0.0500 time 0.3749 (0.3844) data time 0.0008 (0.0021) model time 0.3740 (0.3827) loss 2.6230 (3.3478) grad_norm 1.4119 (1.3722) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][540/625] eta 0:00:32 lr 0.001557 wd 0.0500 time 0.3708 (0.3842) data time 0.0011 (0.0021) model time 0.3697 (0.3825) loss 3.7433 (3.3514) grad_norm 1.0441 (1.3725) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][550/625] eta 0:00:28 lr 0.001556 wd 0.0500 time 0.3812 (0.3841) data time 0.0008 (0.0021) model time 0.3803 (0.3824) loss 2.5227 (3.3505) grad_norm 1.3572 (1.3724) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][560/625] eta 0:00:24 lr 0.001556 wd 0.0500 time 0.3758 (0.3840) data time 0.0009 (0.0021) model time 0.3749 (0.3823) loss 3.3319 (3.3489) grad_norm 1.5798 (1.3763) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][570/625] eta 0:00:21 lr 0.001556 wd 0.0500 time 0.5836 (0.3843) data time 0.0011 (0.0021) model time 0.5825 (0.3826) loss 3.1381 (3.3469) grad_norm 1.1007 (1.3739) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][580/625] eta 0:00:17 lr 0.001556 wd 0.0500 time 0.5717 (0.3851) data time 0.0008 (0.0020) model time 0.5709 (0.3835) loss 3.7459 (3.3492) grad_norm 1.1323 (1.3694) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][590/625] eta 0:00:13 lr 0.001556 wd 0.0500 time 0.3705 (0.3853) data time 0.0011 (0.0020) model time 0.3694 (0.3838) loss 2.4589 (3.3475) grad_norm 1.4382 (1.3710) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][600/625] eta 0:00:09 lr 0.001556 wd 0.0500 time 0.3817 (0.3852) data time 0.0008 (0.0020) model time 0.3809 (0.3837) loss 3.0943 (3.3464) grad_norm 1.6027 (1.3770) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][610/625] eta 0:00:05 lr 0.001556 wd 0.0500 time 0.3321 (0.3854) data time 0.0008 (0.0020) model time 0.3313 (0.3839) loss 3.7054 (3.3509) grad_norm 1.3475 (1.3797) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [107/300][620/625] eta 0:00:01 lr 0.001555 wd 0.0500 time 0.3645 (0.3851) data time 0.0008 (0.0020) model time 0.3636 (0.3836) loss 3.2836 (3.3571) grad_norm 0.9091 (1.3802) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:18:51 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 107 training takes 0:04:00 [2024-08-26 10:18:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:18:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:18:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.6318 (0.6318) Acc@1 86.670 (86.670) Acc@5 97.803 (97.803) Mem 11225MB [2024-08-26 10:18:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 1.1191 (0.7990) Acc@1 73.389 (81.996) Acc@5 93.359 (96.560) Mem 11225MB [2024-08-26 10:18:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.2002 (0.9632) Acc@1 72.119 (78.334) Acc@5 92.236 (94.666) Mem 11225MB [2024-08-26 10:18:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 77.963 Acc@5 94.628 [2024-08-26 10:18:56 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.0% [2024-08-26 10:18:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.991 (0.991) Loss 0.5430 (0.5430) Acc@1 88.379 (88.379) Acc@5 98.389 (98.389) Mem 11225MB [2024-08-26 10:18:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.188) Loss 0.9019 (0.6791) Acc@1 78.711 (84.646) Acc@5 95.166 (97.261) Mem 11225MB [2024-08-26 10:18:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.142) Loss 1.0537 (0.8238) Acc@1 73.926 (80.978) Acc@5 93.555 (95.703) Mem 11225MB [2024-08-26 10:18:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.708 Acc@5 95.691 [2024-08-26 10:18:59 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.7% [2024-08-26 10:18:59 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.71% [2024-08-26 10:18:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:19:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:19:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][0/625] eta 0:07:46 lr 0.001555 wd 0.0500 time 0.7464 (0.7464) data time 0.3997 (0.3997) model time 0.0000 (0.0000) loss 3.2406 (3.2406) grad_norm 0.9721 (0.9721) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][10/625] eta 0:04:12 lr 0.001555 wd 0.0500 time 0.3839 (0.4102) data time 0.0008 (0.0375) model time 0.0000 (0.0000) loss 3.8555 (3.5840) grad_norm 1.8364 (1.2380) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][20/625] eta 0:03:58 lr 0.001555 wd 0.0500 time 0.3768 (0.3942) data time 0.0011 (0.0202) model time 0.0000 (0.0000) loss 3.4280 (3.5479) grad_norm 0.9932 (1.2541) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][30/625] eta 0:03:51 lr 0.001555 wd 0.0500 time 0.3804 (0.3886) data time 0.0011 (0.0141) model time 0.0000 (0.0000) loss 2.6849 (3.4763) grad_norm 1.1416 (1.2178) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][40/625] eta 0:03:45 lr 0.001555 wd 0.0500 time 0.3804 (0.3863) data time 0.0009 (0.0109) model time 0.0000 (0.0000) loss 3.4182 (3.4632) grad_norm 1.3953 (1.2452) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][50/625] eta 0:03:40 lr 0.001555 wd 0.0500 time 0.3693 (0.3840) data time 0.0011 (0.0090) model time 0.0000 (0.0000) loss 3.1969 (3.4841) grad_norm 2.1808 (1.3032) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][60/625] eta 0:03:36 lr 0.001554 wd 0.0500 time 0.3740 (0.3827) data time 0.0009 (0.0077) model time 0.3731 (0.3747) loss 2.6855 (3.5013) grad_norm 1.1737 (1.4274) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][70/625] eta 0:03:31 lr 0.001554 wd 0.0500 time 0.3843 (0.3819) data time 0.0010 (0.0068) model time 0.3832 (0.3753) loss 3.4159 (3.4591) grad_norm 1.0660 (1.4296) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][80/625] eta 0:03:27 lr 0.001554 wd 0.0500 time 0.3758 (0.3812) data time 0.0008 (0.0061) model time 0.3750 (0.3754) loss 2.6977 (3.4528) grad_norm 1.1546 (1.3947) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][90/625] eta 0:03:23 lr 0.001554 wd 0.0500 time 0.3702 (0.3807) data time 0.0011 (0.0055) model time 0.3691 (0.3752) loss 3.3935 (3.4424) grad_norm 1.0051 (1.3842) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][100/625] eta 0:03:21 lr 0.001554 wd 0.0500 time 0.3726 (0.3844) data time 0.0009 (0.0051) model time 0.3717 (0.3838) loss 4.0363 (3.3803) grad_norm 1.3196 (1.3997) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][110/625] eta 0:03:17 lr 0.001554 wd 0.0500 time 0.3805 (0.3839) data time 0.0008 (0.0047) model time 0.3797 (0.3828) loss 3.7874 (3.3869) grad_norm 1.3387 (1.4188) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][120/625] eta 0:03:14 lr 0.001554 wd 0.0500 time 0.3725 (0.3851) data time 0.0008 (0.0044) model time 0.3717 (0.3847) loss 3.0774 (3.3648) grad_norm 1.3732 (1.3989) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][130/625] eta 0:03:10 lr 0.001553 wd 0.0500 time 0.3719 (0.3846) data time 0.0013 (0.0042) model time 0.3707 (0.3838) loss 3.5895 (3.3751) grad_norm 1.2022 (1.3775) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][140/625] eta 0:03:06 lr 0.001553 wd 0.0500 time 0.3742 (0.3840) data time 0.0009 (0.0040) model time 0.3733 (0.3828) loss 1.8509 (3.3600) grad_norm 1.1473 (1.3690) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][150/625] eta 0:03:03 lr 0.001553 wd 0.0500 time 0.3774 (0.3860) data time 0.0010 (0.0038) model time 0.3764 (0.3858) loss 3.8751 (3.3834) grad_norm 1.1912 (1.3602) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][160/625] eta 0:02:59 lr 0.001553 wd 0.0500 time 0.3747 (0.3868) data time 0.0008 (0.0036) model time 0.3739 (0.3870) loss 3.6126 (3.3690) grad_norm 1.5760 (1.3742) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][170/625] eta 0:02:55 lr 0.001553 wd 0.0500 time 0.3789 (0.3863) data time 0.0008 (0.0035) model time 0.3781 (0.3862) loss 3.9212 (3.3695) grad_norm 0.9238 (1.3650) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][180/625] eta 0:02:51 lr 0.001553 wd 0.0500 time 0.3828 (0.3859) data time 0.0012 (0.0033) model time 0.3816 (0.3854) loss 3.7580 (3.3746) grad_norm 1.5494 (1.3627) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][190/625] eta 0:02:47 lr 0.001553 wd 0.0500 time 0.3769 (0.3854) data time 0.0011 (0.0032) model time 0.3758 (0.3847) loss 3.1182 (3.3752) grad_norm 1.4676 (1.3558) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][200/625] eta 0:02:43 lr 0.001552 wd 0.0500 time 0.3764 (0.3850) data time 0.0009 (0.0031) model time 0.3755 (0.3843) loss 3.4878 (3.3652) grad_norm 1.2764 (1.3594) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][210/625] eta 0:02:39 lr 0.001552 wd 0.0500 time 0.3834 (0.3847) data time 0.0008 (0.0030) model time 0.3826 (0.3838) loss 3.4522 (3.3734) grad_norm 1.1063 (1.3721) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][220/625] eta 0:02:35 lr 0.001552 wd 0.0500 time 0.3838 (0.3843) data time 0.0008 (0.0029) model time 0.3831 (0.3833) loss 4.0656 (3.3787) grad_norm 1.6597 (1.3729) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][230/625] eta 0:02:31 lr 0.001552 wd 0.0500 time 0.3768 (0.3840) data time 0.0011 (0.0029) model time 0.3757 (0.3829) loss 3.6992 (3.3805) grad_norm 1.1967 (1.3759) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][240/625] eta 0:02:28 lr 0.001552 wd 0.0500 time 0.3711 (0.3845) data time 0.0008 (0.0028) model time 0.3703 (0.3835) loss 2.4335 (3.3575) grad_norm 1.0251 (1.3699) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][250/625] eta 0:02:24 lr 0.001552 wd 0.0500 time 0.3890 (0.3843) data time 0.0012 (0.0027) model time 0.3878 (0.3832) loss 2.8891 (3.3656) grad_norm 1.5655 (1.3763) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][260/625] eta 0:02:20 lr 0.001552 wd 0.0500 time 0.3806 (0.3841) data time 0.0009 (0.0027) model time 0.3797 (0.3830) loss 3.6005 (3.3753) grad_norm 1.0921 (1.3777) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][270/625] eta 0:02:16 lr 0.001551 wd 0.0500 time 0.3791 (0.3839) data time 0.0008 (0.0026) model time 0.3782 (0.3827) loss 2.2137 (3.3778) grad_norm 1.1144 (1.3745) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][280/625] eta 0:02:12 lr 0.001551 wd 0.0500 time 0.3814 (0.3836) data time 0.0010 (0.0026) model time 0.3803 (0.3824) loss 3.6683 (3.3775) grad_norm 1.3033 (1.3768) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][290/625] eta 0:02:08 lr 0.001551 wd 0.0500 time 0.3752 (0.3834) data time 0.0011 (0.0025) model time 0.3741 (0.3822) loss 3.3400 (3.3771) grad_norm 1.4305 (1.3760) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][300/625] eta 0:02:04 lr 0.001551 wd 0.0500 time 0.3755 (0.3833) data time 0.0011 (0.0025) model time 0.3745 (0.3820) loss 2.5017 (3.3706) grad_norm 1.7574 (1.3703) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][310/625] eta 0:02:01 lr 0.001551 wd 0.0500 time 0.3826 (0.3845) data time 0.0011 (0.0024) model time 0.3815 (0.3835) loss 3.1660 (3.3655) grad_norm 1.0913 (1.3693) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][320/625] eta 0:01:57 lr 0.001551 wd 0.0500 time 0.3723 (0.3843) data time 0.0011 (0.0024) model time 0.3712 (0.3832) loss 3.2184 (3.3616) grad_norm 1.1459 (1.3714) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][330/625] eta 0:01:53 lr 0.001550 wd 0.0500 time 0.3772 (0.3841) data time 0.0008 (0.0023) model time 0.3764 (0.3830) loss 3.2630 (3.3621) grad_norm 1.5326 (1.3752) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][340/625] eta 0:01:49 lr 0.001550 wd 0.0500 time 0.3780 (0.3850) data time 0.0011 (0.0023) model time 0.3769 (0.3840) loss 2.5435 (3.3539) grad_norm 0.7519 (1.3774) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][350/625] eta 0:01:45 lr 0.001550 wd 0.0500 time 0.3724 (0.3853) data time 0.0008 (0.0023) model time 0.3716 (0.3843) loss 2.8183 (3.3497) grad_norm 1.5075 (1.4020) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][360/625] eta 0:01:42 lr 0.001550 wd 0.0500 time 0.3771 (0.3850) data time 0.0008 (0.0022) model time 0.3763 (0.3841) loss 2.1815 (3.3490) grad_norm 1.4187 (1.3992) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][370/625] eta 0:01:38 lr 0.001550 wd 0.0500 time 0.3786 (0.3848) data time 0.0007 (0.0022) model time 0.3778 (0.3838) loss 2.6876 (3.3404) grad_norm 1.1083 (1.3991) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][380/625] eta 0:01:34 lr 0.001550 wd 0.0500 time 0.3727 (0.3846) data time 0.0010 (0.0022) model time 0.3717 (0.3835) loss 3.6211 (3.3521) grad_norm 1.2429 (1.3969) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][390/625] eta 0:01:30 lr 0.001550 wd 0.0500 time 0.3762 (0.3844) data time 0.0009 (0.0021) model time 0.3753 (0.3834) loss 3.8100 (3.3550) grad_norm 1.2322 (1.3972) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][400/625] eta 0:01:26 lr 0.001549 wd 0.0500 time 0.3820 (0.3843) data time 0.0012 (0.0021) model time 0.3808 (0.3832) loss 3.0951 (3.3503) grad_norm 1.5978 (1.3980) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][410/625] eta 0:01:22 lr 0.001549 wd 0.0500 time 0.3747 (0.3841) data time 0.0009 (0.0021) model time 0.3737 (0.3829) loss 3.3611 (3.3574) grad_norm 1.9851 (1.4019) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][420/625] eta 0:01:18 lr 0.001549 wd 0.0500 time 0.3838 (0.3839) data time 0.0011 (0.0021) model time 0.3827 (0.3828) loss 2.9419 (3.3623) grad_norm 0.9122 (1.4001) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][430/625] eta 0:01:14 lr 0.001549 wd 0.0500 time 0.3812 (0.3839) data time 0.0011 (0.0020) model time 0.3802 (0.3827) loss 3.1822 (3.3570) grad_norm 0.9795 (1.3953) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][440/625] eta 0:01:10 lr 0.001549 wd 0.0500 time 0.3735 (0.3837) data time 0.0010 (0.0020) model time 0.3725 (0.3826) loss 4.7634 (3.3618) grad_norm 2.0660 (1.3972) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][450/625] eta 0:01:07 lr 0.001549 wd 0.0500 time 0.3847 (0.3841) data time 0.0008 (0.0020) model time 0.3838 (0.3830) loss 4.3516 (3.3654) grad_norm 1.0662 (1.3934) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:21:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][460/625] eta 0:01:03 lr 0.001549 wd 0.0500 time 0.3875 (0.3844) data time 0.0011 (0.0020) model time 0.3864 (0.3833) loss 2.3754 (3.3649) grad_norm 1.2998 (1.3980) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][470/625] eta 0:00:59 lr 0.001548 wd 0.0500 time 0.3776 (0.3846) data time 0.0011 (0.0020) model time 0.3765 (0.3836) loss 3.3895 (3.3606) grad_norm 1.1808 (1.3957) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][480/625] eta 0:00:55 lr 0.001548 wd 0.0500 time 0.3820 (0.3845) data time 0.0010 (0.0020) model time 0.3810 (0.3835) loss 3.2163 (3.3602) grad_norm 1.6366 (1.4073) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][490/625] eta 0:00:52 lr 0.001548 wd 0.0500 time 0.3802 (0.3852) data time 0.0010 (0.0019) model time 0.3792 (0.3842) loss 3.5074 (3.3608) grad_norm 1.7021 (1.4106) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][500/625] eta 0:00:48 lr 0.001548 wd 0.0500 time 0.3791 (0.3850) data time 0.0010 (0.0019) model time 0.3781 (0.3840) loss 3.5209 (3.3549) grad_norm 1.9929 (1.4131) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][510/625] eta 0:00:44 lr 0.001548 wd 0.0500 time 0.3787 (0.3848) data time 0.0008 (0.0019) model time 0.3779 (0.3838) loss 3.0623 (3.3525) grad_norm 1.1806 (1.4138) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][520/625] eta 0:00:40 lr 0.001548 wd 0.0500 time 0.3778 (0.3847) data time 0.0008 (0.0019) model time 0.3770 (0.3836) loss 3.6459 (3.3531) grad_norm 1.8913 (1.4163) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][530/625] eta 0:00:36 lr 0.001548 wd 0.0500 time 0.3747 (0.3845) data time 0.0011 (0.0019) model time 0.3736 (0.3834) loss 3.4123 (3.3536) grad_norm 1.5880 (1.4166) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][540/625] eta 0:00:32 lr 0.001547 wd 0.0500 time 0.3781 (0.3844) data time 0.0010 (0.0019) model time 0.3771 (0.3833) loss 3.3693 (3.3518) grad_norm 1.4147 (1.4193) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][550/625] eta 0:00:28 lr 0.001547 wd 0.0500 time 0.3811 (0.3844) data time 0.0009 (0.0019) model time 0.3803 (0.3833) loss 3.9557 (3.3493) grad_norm 1.1490 (1.4175) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][560/625] eta 0:00:24 lr 0.001547 wd 0.0500 time 0.3818 (0.3843) data time 0.0011 (0.0018) model time 0.3807 (0.3832) loss 2.6611 (3.3523) grad_norm 1.2801 (1.4143) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][570/625] eta 0:00:21 lr 0.001547 wd 0.0500 time 0.3795 (0.3842) data time 0.0010 (0.0018) model time 0.3785 (0.3831) loss 2.9443 (3.3438) grad_norm 1.3993 (1.4140) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][580/625] eta 0:00:17 lr 0.001547 wd 0.0500 time 0.4047 (0.3842) data time 0.0008 (0.0018) model time 0.4039 (0.3831) loss 3.6120 (3.3435) grad_norm 1.0830 (1.4141) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][590/625] eta 0:00:13 lr 0.001547 wd 0.0500 time 0.3745 (0.3841) data time 0.0008 (0.0018) model time 0.3737 (0.3830) loss 3.3500 (3.3426) grad_norm 1.4420 (1.4119) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][600/625] eta 0:00:09 lr 0.001546 wd 0.0500 time 0.3755 (0.3840) data time 0.0011 (0.0018) model time 0.3744 (0.3829) loss 3.7444 (3.3410) grad_norm 0.9027 (1.4126) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][610/625] eta 0:00:05 lr 0.001546 wd 0.0500 time 0.3698 (0.3842) data time 0.0006 (0.0018) model time 0.3693 (0.3831) loss 3.7049 (3.3412) grad_norm 1.1584 (1.4082) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:22:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [108/300][620/625] eta 0:00:01 lr 0.001546 wd 0.0500 time 0.3694 (0.3840) data time 0.0006 (0.0018) model time 0.3688 (0.3829) loss 3.6186 (3.3456) grad_norm 1.3914 (1.4072) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:00 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 108 training takes 0:03:59 [2024-08-26 10:23:00 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:23:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:23:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.493 (0.493) Loss 0.6533 (0.6533) Acc@1 86.133 (86.133) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-26 10:23:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.143) Loss 1.0566 (0.7959) Acc@1 76.025 (82.466) Acc@5 93.701 (96.418) Mem 11225MB [2024-08-26 10:23:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.2744 (0.9526) Acc@1 70.264 (78.448) Acc@5 91.602 (94.501) Mem 11225MB [2024-08-26 10:23:04 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.107 Acc@5 94.428 [2024-08-26 10:23:04 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.1% [2024-08-26 10:23:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.843 (0.843) Loss 0.5425 (0.5425) Acc@1 88.379 (88.379) Acc@5 98.535 (98.535) Mem 11225MB [2024-08-26 10:23:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.178) Loss 0.9023 (0.6783) Acc@1 78.809 (84.641) Acc@5 95.264 (97.310) Mem 11225MB [2024-08-26 10:23:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.138) Loss 1.0527 (0.8218) Acc@1 74.219 (81.001) Acc@5 93.506 (95.740) Mem 11225MB [2024-08-26 10:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.724 Acc@5 95.715 [2024-08-26 10:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.7% [2024-08-26 10:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.72% [2024-08-26 10:23:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:23:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:23:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][0/625] eta 0:08:10 lr 0.001546 wd 0.0500 time 0.7846 (0.7846) data time 0.4507 (0.4507) model time 0.0000 (0.0000) loss 3.4977 (3.4977) grad_norm 1.3182 (1.3182) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][10/625] eta 0:04:14 lr 0.001546 wd 0.0500 time 0.3778 (0.4136) data time 0.0009 (0.0429) model time 0.0000 (0.0000) loss 3.4490 (3.4325) grad_norm 1.3047 (1.2803) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][20/625] eta 0:04:05 lr 0.001546 wd 0.0500 time 0.3731 (0.4055) data time 0.0011 (0.0230) model time 0.0000 (0.0000) loss 2.6611 (3.3980) grad_norm 1.1737 (1.3420) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][30/625] eta 0:04:00 lr 0.001546 wd 0.0500 time 0.3721 (0.4038) data time 0.0008 (0.0159) model time 0.0000 (0.0000) loss 4.0171 (3.4352) grad_norm 1.3933 (1.3566) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][40/625] eta 0:03:52 lr 0.001545 wd 0.0500 time 0.3913 (0.3977) data time 0.0009 (0.0123) model time 0.0000 (0.0000) loss 3.1972 (3.3507) grad_norm 1.3272 (1.3813) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][50/625] eta 0:03:46 lr 0.001545 wd 0.0500 time 0.3789 (0.3934) data time 0.0008 (0.0101) model time 0.0000 (0.0000) loss 4.4404 (3.3532) grad_norm 1.6836 (1.3665) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][60/625] eta 0:03:40 lr 0.001545 wd 0.0500 time 0.3679 (0.3906) data time 0.0011 (0.0087) model time 0.3668 (0.3748) loss 3.4288 (3.3382) grad_norm 1.2653 (1.3814) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][70/625] eta 0:03:35 lr 0.001545 wd 0.0500 time 0.3761 (0.3888) data time 0.0009 (0.0076) model time 0.3752 (0.3758) loss 3.7230 (3.3544) grad_norm 1.1902 (1.4504) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][80/625] eta 0:03:31 lr 0.001545 wd 0.0500 time 0.3787 (0.3875) data time 0.0012 (0.0068) model time 0.3775 (0.3762) loss 3.4279 (3.3603) grad_norm 0.9219 (1.4322) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][90/625] eta 0:03:26 lr 0.001545 wd 0.0500 time 0.3776 (0.3863) data time 0.0008 (0.0062) model time 0.3768 (0.3760) loss 3.0444 (3.3706) grad_norm 0.9984 (1.4170) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][100/625] eta 0:03:23 lr 0.001545 wd 0.0500 time 0.3812 (0.3868) data time 0.0011 (0.0057) model time 0.3802 (0.3787) loss 3.7041 (3.3858) grad_norm 1.3315 (1.3987) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][110/625] eta 0:03:18 lr 0.001544 wd 0.0500 time 0.3742 (0.3858) data time 0.0008 (0.0053) model time 0.3734 (0.3781) loss 2.8053 (3.3955) grad_norm 1.2172 (1.3882) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][120/625] eta 0:03:14 lr 0.001544 wd 0.0500 time 0.3823 (0.3851) data time 0.0008 (0.0050) model time 0.3815 (0.3778) loss 2.7713 (3.4036) grad_norm 1.6814 (1.3835) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:23:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][130/625] eta 0:03:10 lr 0.001544 wd 0.0500 time 0.3964 (0.3845) data time 0.0008 (0.0047) model time 0.3956 (0.3776) loss 3.1345 (3.3994) grad_norm 2.2347 (1.4050) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][140/625] eta 0:03:06 lr 0.001544 wd 0.0500 time 0.3722 (0.3836) data time 0.0012 (0.0044) model time 0.3709 (0.3769) loss 3.2499 (3.3823) grad_norm 1.4198 (1.4141) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][150/625] eta 0:03:01 lr 0.001544 wd 0.0500 time 0.3758 (0.3830) data time 0.0010 (0.0042) model time 0.3748 (0.3765) loss 2.7625 (3.3832) grad_norm 1.4159 (1.4082) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][160/625] eta 0:02:57 lr 0.001544 wd 0.0500 time 0.3680 (0.3825) data time 0.0009 (0.0040) model time 0.3671 (0.3763) loss 3.9616 (3.3794) grad_norm 1.9875 (1.4051) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][170/625] eta 0:02:54 lr 0.001544 wd 0.0500 time 0.3746 (0.3845) data time 0.0008 (0.0038) model time 0.3738 (0.3796) loss 2.4150 (3.3734) grad_norm 1.3047 (1.3983) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][180/625] eta 0:02:50 lr 0.001543 wd 0.0500 time 0.3759 (0.3840) data time 0.0010 (0.0037) model time 0.3748 (0.3791) loss 3.5447 (3.3726) grad_norm 1.2341 (1.3895) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][190/625] eta 0:02:46 lr 0.001543 wd 0.0500 time 0.4063 (0.3837) data time 0.0011 (0.0035) model time 0.4053 (0.3790) loss 3.2687 (3.3791) grad_norm 1.3644 (1.3835) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][200/625] eta 0:02:42 lr 0.001543 wd 0.0500 time 0.3804 (0.3832) data time 0.0008 (0.0034) model time 0.3796 (0.3785) loss 4.0910 (3.3873) grad_norm 1.2331 (1.3848) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][210/625] eta 0:02:38 lr 0.001543 wd 0.0500 time 0.3783 (0.3828) data time 0.0011 (0.0033) model time 0.3771 (0.3783) loss 3.5525 (3.3872) grad_norm 1.0096 (1.3731) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][220/625] eta 0:02:34 lr 0.001543 wd 0.0500 time 0.3757 (0.3825) data time 0.0010 (0.0032) model time 0.3747 (0.3781) loss 2.6793 (3.3775) grad_norm 1.1183 (1.3676) loss_scale 4096.0000 (2066.5339) mem 11225MB [2024-08-26 10:24:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][230/625] eta 0:02:31 lr 0.001543 wd 0.0500 time 0.5864 (0.3838) data time 0.0011 (0.0031) model time 0.5853 (0.3800) loss 3.0225 (3.3728) grad_norm 1.3697 (1.3663) loss_scale 4096.0000 (2154.3896) mem 11225MB [2024-08-26 10:24:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][240/625] eta 0:02:28 lr 0.001542 wd 0.0500 time 0.3894 (0.3845) data time 0.0008 (0.0030) model time 0.3885 (0.3810) loss 3.5167 (3.3714) grad_norm 1.0411 (1.3649) loss_scale 4096.0000 (2234.9544) mem 11225MB [2024-08-26 10:24:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][250/625] eta 0:02:24 lr 0.001542 wd 0.0500 time 0.3812 (0.3843) data time 0.0008 (0.0030) model time 0.3805 (0.3808) loss 3.5102 (3.3803) grad_norm 1.7122 (1.3706) loss_scale 4096.0000 (2309.0996) mem 11225MB [2024-08-26 10:24:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][260/625] eta 0:02:20 lr 0.001542 wd 0.0500 time 0.3835 (0.3842) data time 0.0008 (0.0029) model time 0.3827 (0.3809) loss 3.3015 (3.3700) grad_norm 1.5956 (1.3655) loss_scale 4096.0000 (2377.5632) mem 11225MB [2024-08-26 10:24:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][270/625] eta 0:02:16 lr 0.001542 wd 0.0500 time 0.3761 (0.3840) data time 0.0011 (0.0028) model time 0.3749 (0.3806) loss 3.5250 (3.3674) grad_norm 1.3907 (1.3638) loss_scale 4096.0000 (2440.9742) mem 11225MB [2024-08-26 10:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][280/625] eta 0:02:12 lr 0.001542 wd 0.0500 time 0.3864 (0.3837) data time 0.0008 (0.0028) model time 0.3855 (0.3805) loss 3.8191 (3.3814) grad_norm 1.3646 (1.3638) loss_scale 4096.0000 (2499.8719) mem 11225MB [2024-08-26 10:25:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][290/625] eta 0:02:08 lr 0.001542 wd 0.0500 time 0.3727 (0.3842) data time 0.0011 (0.0027) model time 0.3716 (0.3811) loss 3.3200 (3.3868) grad_norm 1.0733 (1.3653) loss_scale 4096.0000 (2554.7216) mem 11225MB [2024-08-26 10:25:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][300/625] eta 0:02:04 lr 0.001542 wd 0.0500 time 0.3752 (0.3841) data time 0.0009 (0.0027) model time 0.3744 (0.3810) loss 3.5573 (3.3797) grad_norm 0.9635 (1.3669) loss_scale 4096.0000 (2605.9269) mem 11225MB [2024-08-26 10:25:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][310/625] eta 0:02:01 lr 0.001541 wd 0.0500 time 0.3800 (0.3846) data time 0.0009 (0.0026) model time 0.3791 (0.3817) loss 3.7688 (3.3855) grad_norm 1.1966 (1.3664) loss_scale 4096.0000 (2653.8392) mem 11225MB [2024-08-26 10:25:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][320/625] eta 0:01:57 lr 0.001541 wd 0.0500 time 0.3734 (0.3844) data time 0.0011 (0.0026) model time 0.3724 (0.3816) loss 3.2064 (3.3737) grad_norm 2.0293 (1.3836) loss_scale 4096.0000 (2698.7664) mem 11225MB [2024-08-26 10:25:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][330/625] eta 0:01:53 lr 0.001541 wd 0.0500 time 0.3742 (0.3842) data time 0.0010 (0.0025) model time 0.3732 (0.3813) loss 3.8521 (3.3727) grad_norm 1.6752 (1.3828) loss_scale 4096.0000 (2740.9789) mem 11225MB [2024-08-26 10:25:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][340/625] eta 0:01:49 lr 0.001541 wd 0.0500 time 0.3765 (0.3841) data time 0.0008 (0.0025) model time 0.3757 (0.3813) loss 2.7704 (3.3683) grad_norm 1.2487 (1.3844) loss_scale 4096.0000 (2780.7155) mem 11225MB [2024-08-26 10:25:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][350/625] eta 0:01:45 lr 0.001541 wd 0.0500 time 0.3771 (0.3839) data time 0.0011 (0.0025) model time 0.3760 (0.3811) loss 2.1312 (3.3656) grad_norm 1.8392 (1.3839) loss_scale 4096.0000 (2818.1880) mem 11225MB [2024-08-26 10:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][360/625] eta 0:01:41 lr 0.001541 wd 0.0500 time 0.3832 (0.3837) data time 0.0009 (0.0024) model time 0.3823 (0.3809) loss 4.3602 (3.3746) grad_norm 1.1532 (1.3853) loss_scale 4096.0000 (2853.5845) mem 11225MB [2024-08-26 10:25:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][370/625] eta 0:01:37 lr 0.001541 wd 0.0500 time 0.3796 (0.3835) data time 0.0008 (0.0024) model time 0.3787 (0.3808) loss 2.3177 (3.3701) grad_norm 1.3542 (1.3835) loss_scale 4096.0000 (2887.0728) mem 11225MB [2024-08-26 10:25:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][380/625] eta 0:01:33 lr 0.001540 wd 0.0500 time 0.3761 (0.3834) data time 0.0008 (0.0023) model time 0.3753 (0.3807) loss 3.3046 (3.3639) grad_norm 1.3661 (1.3819) loss_scale 4096.0000 (2918.8031) mem 11225MB [2024-08-26 10:25:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][390/625] eta 0:01:30 lr 0.001540 wd 0.0500 time 0.3760 (0.3833) data time 0.0008 (0.0023) model time 0.3752 (0.3807) loss 4.3122 (3.3712) grad_norm 1.7020 (1.3826) loss_scale 4096.0000 (2948.9105) mem 11225MB [2024-08-26 10:25:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][400/625] eta 0:01:26 lr 0.001540 wd 0.0500 time 0.3851 (0.3833) data time 0.0008 (0.0023) model time 0.3843 (0.3807) loss 3.1285 (3.3779) grad_norm 1.1464 (1.3785) loss_scale 4096.0000 (2977.5162) mem 11225MB [2024-08-26 10:25:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][410/625] eta 0:01:22 lr 0.001540 wd 0.0500 time 0.3710 (0.3831) data time 0.0011 (0.0023) model time 0.3700 (0.3805) loss 2.3365 (3.3754) grad_norm 1.3552 (1.3770) loss_scale 4096.0000 (3004.7299) mem 11225MB [2024-08-26 10:25:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][420/625] eta 0:01:18 lr 0.001540 wd 0.0500 time 0.5850 (0.3844) data time 0.0008 (0.0022) model time 0.5842 (0.3820) loss 3.2393 (3.3737) grad_norm 1.7038 (1.3762) loss_scale 4096.0000 (3030.6508) mem 11225MB [2024-08-26 10:25:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][430/625] eta 0:01:14 lr 0.001540 wd 0.0500 time 0.3785 (0.3842) data time 0.0009 (0.0022) model time 0.3776 (0.3819) loss 4.2261 (3.3745) grad_norm 1.2299 (1.3725) loss_scale 4096.0000 (3055.3689) mem 11225MB [2024-08-26 10:25:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][440/625] eta 0:01:11 lr 0.001539 wd 0.0500 time 0.3773 (0.3841) data time 0.0008 (0.0022) model time 0.3765 (0.3818) loss 4.0854 (3.3817) grad_norm 1.1843 (1.3745) loss_scale 4096.0000 (3078.9660) mem 11225MB [2024-08-26 10:26:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][450/625] eta 0:01:07 lr 0.001539 wd 0.0500 time 0.3774 (0.3840) data time 0.0011 (0.0022) model time 0.3763 (0.3816) loss 2.4496 (3.3828) grad_norm 1.4283 (1.3741) loss_scale 4096.0000 (3101.5166) mem 11225MB [2024-08-26 10:26:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][460/625] eta 0:01:03 lr 0.001539 wd 0.0500 time 0.3761 (0.3838) data time 0.0011 (0.0021) model time 0.3750 (0.3815) loss 3.6790 (3.3812) grad_norm 1.3956 (1.3726) loss_scale 4096.0000 (3123.0889) mem 11225MB [2024-08-26 10:26:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][470/625] eta 0:00:59 lr 0.001539 wd 0.0500 time 0.3845 (0.3837) data time 0.0008 (0.0021) model time 0.3836 (0.3814) loss 3.6997 (3.3778) grad_norm 1.3548 (1.3725) loss_scale 4096.0000 (3143.7452) mem 11225MB [2024-08-26 10:26:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][480/625] eta 0:00:55 lr 0.001539 wd 0.0500 time 0.3768 (0.3836) data time 0.0010 (0.0021) model time 0.3758 (0.3812) loss 3.5817 (3.3828) grad_norm 1.1252 (1.3706) loss_scale 4096.0000 (3163.5426) mem 11225MB [2024-08-26 10:26:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][490/625] eta 0:00:51 lr 0.001539 wd 0.0500 time 0.3703 (0.3838) data time 0.0010 (0.0021) model time 0.3693 (0.3816) loss 2.8803 (3.3775) grad_norm 1.5569 (1.3765) loss_scale 4096.0000 (3182.5336) mem 11225MB [2024-08-26 10:26:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][500/625] eta 0:00:48 lr 0.001539 wd 0.0500 time 0.3789 (0.3842) data time 0.0012 (0.0021) model time 0.3777 (0.3820) loss 3.0463 (3.3797) grad_norm 1.8290 (1.3821) loss_scale 4096.0000 (3200.7665) mem 11225MB [2024-08-26 10:26:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][510/625] eta 0:00:44 lr 0.001538 wd 0.0500 time 0.3799 (0.3840) data time 0.0008 (0.0020) model time 0.3790 (0.3818) loss 4.2034 (3.3760) grad_norm 1.2576 (1.3871) loss_scale 4096.0000 (3218.2857) mem 11225MB [2024-08-26 10:26:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][520/625] eta 0:00:40 lr 0.001538 wd 0.0500 time 0.3771 (0.3840) data time 0.0011 (0.0020) model time 0.3760 (0.3818) loss 3.9188 (3.3757) grad_norm 1.3614 (1.3869) loss_scale 4096.0000 (3235.1324) mem 11225MB [2024-08-26 10:26:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][530/625] eta 0:00:36 lr 0.001538 wd 0.0500 time 0.3699 (0.3838) data time 0.0009 (0.0020) model time 0.3690 (0.3817) loss 4.0332 (3.3810) grad_norm 1.0989 (1.3877) loss_scale 4096.0000 (3251.3446) mem 11225MB [2024-08-26 10:26:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][540/625] eta 0:00:32 lr 0.001538 wd 0.0500 time 0.3760 (0.3845) data time 0.0009 (0.0020) model time 0.3751 (0.3824) loss 3.6650 (3.3858) grad_norm 1.3030 (1.3884) loss_scale 4096.0000 (3266.9575) mem 11225MB [2024-08-26 10:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][550/625] eta 0:00:28 lr 0.001538 wd 0.0500 time 0.3728 (0.3843) data time 0.0012 (0.0020) model time 0.3717 (0.3823) loss 3.7924 (3.3851) grad_norm 1.0101 (1.3883) loss_scale 4096.0000 (3282.0036) mem 11225MB [2024-08-26 10:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][560/625] eta 0:00:24 lr 0.001538 wd 0.0500 time 0.3772 (0.3842) data time 0.0008 (0.0020) model time 0.3764 (0.3822) loss 3.8132 (3.3858) grad_norm 1.3038 (1.3872) loss_scale 4096.0000 (3296.5134) mem 11225MB [2024-08-26 10:26:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][570/625] eta 0:00:21 lr 0.001538 wd 0.0500 time 0.3783 (0.3841) data time 0.0009 (0.0019) model time 0.3774 (0.3820) loss 3.2371 (3.3818) grad_norm 1.7780 (inf) loss_scale 2048.0000 (3288.9947) mem 11225MB [2024-08-26 10:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][580/625] eta 0:00:17 lr 0.001537 wd 0.0500 time 0.3723 (0.3840) data time 0.0010 (0.0019) model time 0.3714 (0.3819) loss 4.1548 (3.3798) grad_norm 1.1802 (inf) loss_scale 2048.0000 (3267.6351) mem 11225MB [2024-08-26 10:26:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][590/625] eta 0:00:13 lr 0.001537 wd 0.0500 time 0.3726 (0.3839) data time 0.0011 (0.0019) model time 0.3715 (0.3818) loss 3.2877 (3.3799) grad_norm 1.3070 (inf) loss_scale 2048.0000 (3246.9983) mem 11225MB [2024-08-26 10:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][600/625] eta 0:00:09 lr 0.001537 wd 0.0500 time 0.3756 (0.3838) data time 0.0011 (0.0019) model time 0.3745 (0.3817) loss 3.6040 (3.3757) grad_norm 1.4316 (inf) loss_scale 2048.0000 (3227.0483) mem 11225MB [2024-08-26 10:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][610/625] eta 0:00:05 lr 0.001537 wd 0.0500 time 0.3699 (0.3840) data time 0.0006 (0.0019) model time 0.3693 (0.3819) loss 3.6916 (3.3707) grad_norm 1.1491 (inf) loss_scale 2048.0000 (3207.7512) mem 11225MB [2024-08-26 10:27:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [109/300][620/625] eta 0:00:01 lr 0.001537 wd 0.0500 time 0.3704 (0.3837) data time 0.0005 (0.0019) model time 0.3699 (0.3817) loss 3.4193 (3.3677) grad_norm 1.4578 (inf) loss_scale 2048.0000 (3189.0757) mem 11225MB [2024-08-26 10:27:09 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 109 training takes 0:03:59 [2024-08-26 10:27:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:27:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:27:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 0.6167 (0.6167) Acc@1 86.572 (86.572) Acc@5 98.047 (98.047) Mem 11225MB [2024-08-26 10:27:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.0332 (0.7654) Acc@1 75.830 (82.497) Acc@5 93.799 (96.551) Mem 11225MB [2024-08-26 10:27:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.102 (0.120) Loss 1.1572 (0.9276) Acc@1 72.900 (78.657) Acc@5 92.773 (94.657) Mem 11225MB [2024-08-26 10:27:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.355 Acc@5 94.646 [2024-08-26 10:27:13 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.4% [2024-08-26 10:27:13 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.35% [2024-08-26 10:27:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 10:27:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 10:27:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.497 (0.497) Loss 0.5410 (0.5410) Acc@1 88.574 (88.574) Acc@5 98.438 (98.438) Mem 11225MB [2024-08-26 10:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 0.9014 (0.6774) Acc@1 79.053 (84.686) Acc@5 95.215 (97.332) Mem 11225MB [2024-08-26 10:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.0488 (0.8201) Acc@1 74.463 (81.050) Acc@5 93.457 (95.724) Mem 11225MB [2024-08-26 10:27:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.748 Acc@5 95.709 [2024-08-26 10:27:17 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.7% [2024-08-26 10:27:17 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.75% [2024-08-26 10:27:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:27:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:27:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][0/625] eta 0:08:36 lr 0.001537 wd 0.0500 time 0.8262 (0.8262) data time 0.4920 (0.4920) model time 0.0000 (0.0000) loss 3.4887 (3.4887) grad_norm 1.1228 (1.1228) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][10/625] eta 0:04:16 lr 0.001537 wd 0.0500 time 0.3728 (0.4163) data time 0.0012 (0.0458) model time 0.0000 (0.0000) loss 3.2393 (3.3929) grad_norm 1.4222 (1.2754) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][20/625] eta 0:03:59 lr 0.001536 wd 0.0500 time 0.3723 (0.3963) data time 0.0011 (0.0246) model time 0.0000 (0.0000) loss 4.0087 (3.5051) grad_norm 1.3352 (1.2893) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][30/625] eta 0:03:51 lr 0.001536 wd 0.0500 time 0.3734 (0.3895) data time 0.0012 (0.0170) model time 0.0000 (0.0000) loss 2.4522 (3.3561) grad_norm 1.4396 (1.3107) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][40/625] eta 0:03:45 lr 0.001536 wd 0.0500 time 0.3847 (0.3861) data time 0.0011 (0.0132) model time 0.0000 (0.0000) loss 3.6048 (3.3756) grad_norm 1.2460 (1.3198) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][50/625] eta 0:03:41 lr 0.001536 wd 0.0500 time 0.3715 (0.3844) data time 0.0012 (0.0108) model time 0.0000 (0.0000) loss 3.1601 (3.4355) grad_norm 1.1506 (1.2961) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][60/625] eta 0:03:38 lr 0.001536 wd 0.0500 time 0.5977 (0.3865) data time 0.0012 (0.0092) model time 0.5964 (0.3960) loss 3.5101 (3.4417) grad_norm 1.1138 (1.2926) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][70/625] eta 0:03:37 lr 0.001536 wd 0.0500 time 0.5788 (0.3927) data time 0.0009 (0.0081) model time 0.5780 (0.4126) loss 3.6560 (3.4199) grad_norm 1.0554 (1.2973) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][80/625] eta 0:03:32 lr 0.001536 wd 0.0500 time 0.3750 (0.3905) data time 0.0008 (0.0073) model time 0.3742 (0.3998) loss 3.7289 (3.4175) grad_norm 1.0499 (1.2858) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][90/625] eta 0:03:28 lr 0.001535 wd 0.0500 time 0.3780 (0.3892) data time 0.0012 (0.0066) model time 0.3768 (0.3940) loss 3.3545 (3.4184) grad_norm 1.3253 (1.3022) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:27:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][100/625] eta 0:03:23 lr 0.001535 wd 0.0500 time 0.3774 (0.3878) data time 0.0010 (0.0061) model time 0.3763 (0.3900) loss 3.8373 (3.4065) grad_norm 1.1897 (1.3111) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][110/625] eta 0:03:19 lr 0.001535 wd 0.0500 time 0.3777 (0.3869) data time 0.0010 (0.0057) model time 0.3767 (0.3877) loss 3.8075 (3.4098) grad_norm 1.4601 (1.3041) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][120/625] eta 0:03:14 lr 0.001535 wd 0.0500 time 0.3745 (0.3860) data time 0.0011 (0.0053) model time 0.3734 (0.3859) loss 3.0870 (3.4101) grad_norm 2.0351 (1.3201) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][130/625] eta 0:03:10 lr 0.001535 wd 0.0500 time 0.3732 (0.3851) data time 0.0008 (0.0050) model time 0.3724 (0.3843) loss 3.7692 (3.4029) grad_norm 1.4077 (1.3296) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][140/625] eta 0:03:06 lr 0.001535 wd 0.0500 time 0.3727 (0.3843) data time 0.0008 (0.0047) model time 0.3719 (0.3831) loss 4.1998 (3.4214) grad_norm 1.2619 (1.3250) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][150/625] eta 0:03:02 lr 0.001534 wd 0.0500 time 0.3736 (0.3838) data time 0.0008 (0.0045) model time 0.3728 (0.3822) loss 2.5672 (3.3991) grad_norm 1.2406 (1.3351) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][160/625] eta 0:02:58 lr 0.001534 wd 0.0500 time 0.3736 (0.3833) data time 0.0011 (0.0042) model time 0.3725 (0.3816) loss 3.7128 (3.3840) grad_norm 1.1921 (1.3748) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][170/625] eta 0:02:54 lr 0.001534 wd 0.0500 time 0.3764 (0.3829) data time 0.0012 (0.0041) model time 0.3752 (0.3810) loss 3.5852 (3.3775) grad_norm 1.4840 (1.3819) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][180/625] eta 0:02:50 lr 0.001534 wd 0.0500 time 0.3849 (0.3826) data time 0.0011 (0.0039) model time 0.3839 (0.3807) loss 2.6725 (3.3810) grad_norm 1.4475 (1.3850) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][190/625] eta 0:02:46 lr 0.001534 wd 0.0500 time 0.3816 (0.3822) data time 0.0010 (0.0038) model time 0.3806 (0.3802) loss 1.9743 (3.3774) grad_norm 1.4819 (1.3891) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][200/625] eta 0:02:42 lr 0.001534 wd 0.0500 time 0.3711 (0.3819) data time 0.0008 (0.0036) model time 0.3703 (0.3798) loss 4.0029 (3.3894) grad_norm 1.0495 (1.3999) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][210/625] eta 0:02:38 lr 0.001534 wd 0.0500 time 0.3792 (0.3816) data time 0.0010 (0.0035) model time 0.3782 (0.3795) loss 3.5499 (3.3852) grad_norm 0.9085 (1.3934) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][220/625] eta 0:02:34 lr 0.001533 wd 0.0500 time 0.3765 (0.3813) data time 0.0010 (0.0034) model time 0.3755 (0.3792) loss 3.4732 (3.4042) grad_norm 1.2001 (1.3852) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][230/625] eta 0:02:30 lr 0.001533 wd 0.0500 time 0.3772 (0.3812) data time 0.0008 (0.0033) model time 0.3763 (0.3791) loss 3.4732 (3.3969) grad_norm 1.9501 (1.3854) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][240/625] eta 0:02:26 lr 0.001533 wd 0.0500 time 0.3765 (0.3811) data time 0.0011 (0.0032) model time 0.3755 (0.3790) loss 3.4701 (3.3837) grad_norm 1.4077 (1.3865) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][250/625] eta 0:02:23 lr 0.001533 wd 0.0500 time 0.3760 (0.3817) data time 0.0008 (0.0031) model time 0.3751 (0.3799) loss 3.8595 (3.3966) grad_norm 0.9477 (1.3788) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:28:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][260/625] eta 0:02:19 lr 0.001533 wd 0.0500 time 0.3756 (0.3815) data time 0.0011 (0.0031) model time 0.3745 (0.3797) loss 3.5251 (3.3958) grad_norm 1.2351 (1.3767) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][270/625] eta 0:02:15 lr 0.001533 wd 0.0500 time 0.3767 (0.3813) data time 0.0011 (0.0030) model time 0.3756 (0.3794) loss 3.7360 (3.3957) grad_norm 1.4073 (1.3701) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][280/625] eta 0:02:11 lr 0.001532 wd 0.0500 time 0.3787 (0.3825) data time 0.0008 (0.0029) model time 0.3779 (0.3810) loss 2.9788 (3.3902) grad_norm 2.3685 (1.3776) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][290/625] eta 0:02:08 lr 0.001532 wd 0.0500 time 0.3790 (0.3830) data time 0.0011 (0.0029) model time 0.3779 (0.3816) loss 3.5803 (3.3942) grad_norm 1.2046 (1.3768) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][300/625] eta 0:02:04 lr 0.001532 wd 0.0500 time 0.3807 (0.3829) data time 0.0009 (0.0028) model time 0.3798 (0.3814) loss 2.9733 (3.3945) grad_norm 1.7716 (1.3851) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][310/625] eta 0:02:00 lr 0.001532 wd 0.0500 time 0.3828 (0.3828) data time 0.0011 (0.0027) model time 0.3817 (0.3813) loss 3.6033 (3.3908) grad_norm 1.1491 (1.3868) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][320/625] eta 0:01:56 lr 0.001532 wd 0.0500 time 0.3742 (0.3827) data time 0.0011 (0.0027) model time 0.3731 (0.3812) loss 3.7681 (3.3939) grad_norm 1.3383 (1.3893) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][330/625] eta 0:01:52 lr 0.001532 wd 0.0500 time 0.3819 (0.3825) data time 0.0011 (0.0026) model time 0.3808 (0.3810) loss 2.6451 (3.3972) grad_norm 2.4712 (1.3914) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][340/625] eta 0:01:48 lr 0.001532 wd 0.0500 time 0.3855 (0.3824) data time 0.0011 (0.0026) model time 0.3845 (0.3809) loss 3.5968 (3.3976) grad_norm 1.3864 (1.3948) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][350/625] eta 0:01:45 lr 0.001531 wd 0.0500 time 0.3786 (0.3822) data time 0.0010 (0.0026) model time 0.3776 (0.3807) loss 2.4125 (3.3886) grad_norm 1.9128 (1.3984) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][360/625] eta 0:01:41 lr 0.001531 wd 0.0500 time 0.3756 (0.3820) data time 0.0010 (0.0025) model time 0.3746 (0.3805) loss 3.5833 (3.3836) grad_norm 2.2819 (1.4033) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][370/625] eta 0:01:37 lr 0.001531 wd 0.0500 time 0.3665 (0.3824) data time 0.0011 (0.0025) model time 0.3654 (0.3809) loss 3.3889 (3.3850) grad_norm 1.2280 (1.3998) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][380/625] eta 0:01:33 lr 0.001531 wd 0.0500 time 0.3756 (0.3822) data time 0.0008 (0.0024) model time 0.3748 (0.3808) loss 3.1787 (3.3771) grad_norm 1.3870 (1.3967) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][390/625] eta 0:01:29 lr 0.001531 wd 0.0500 time 0.3618 (0.3821) data time 0.0011 (0.0024) model time 0.3607 (0.3806) loss 3.4551 (3.3679) grad_norm 1.1650 (1.3958) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][400/625] eta 0:01:25 lr 0.001531 wd 0.0500 time 0.3777 (0.3820) data time 0.0008 (0.0024) model time 0.3769 (0.3805) loss 3.6139 (3.3643) grad_norm 1.4996 (1.3906) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][410/625] eta 0:01:22 lr 0.001531 wd 0.0500 time 0.3807 (0.3818) data time 0.0011 (0.0023) model time 0.3796 (0.3803) loss 3.1128 (3.3654) grad_norm 2.0098 (1.3932) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:29:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][420/625] eta 0:01:18 lr 0.001530 wd 0.0500 time 0.3719 (0.3817) data time 0.0011 (0.0023) model time 0.3708 (0.3802) loss 3.3543 (3.3631) grad_norm 1.4420 (1.3941) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][430/625] eta 0:01:14 lr 0.001530 wd 0.0500 time 0.3730 (0.3816) data time 0.0008 (0.0023) model time 0.3722 (0.3801) loss 2.4405 (3.3525) grad_norm 1.2465 (1.3922) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][440/625] eta 0:01:10 lr 0.001530 wd 0.0500 time 0.3571 (0.3816) data time 0.0009 (0.0023) model time 0.3562 (0.3801) loss 2.4166 (3.3528) grad_norm 1.7051 (1.3916) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][450/625] eta 0:01:06 lr 0.001530 wd 0.0500 time 0.3800 (0.3816) data time 0.0011 (0.0022) model time 0.3790 (0.3801) loss 3.5200 (3.3562) grad_norm 1.4830 (1.3904) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][460/625] eta 0:01:03 lr 0.001530 wd 0.0500 time 0.5829 (0.3823) data time 0.0010 (0.0022) model time 0.5819 (0.3809) loss 3.4858 (3.3510) grad_norm 1.6469 (1.3919) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][470/625] eta 0:00:59 lr 0.001530 wd 0.0500 time 0.3820 (0.3831) data time 0.0011 (0.0022) model time 0.3809 (0.3818) loss 3.2296 (3.3488) grad_norm 1.1589 (1.3964) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][480/625] eta 0:00:55 lr 0.001529 wd 0.0500 time 0.3905 (0.3830) data time 0.0010 (0.0022) model time 0.3895 (0.3817) loss 3.6499 (3.3457) grad_norm 1.1981 (1.3962) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][490/625] eta 0:00:51 lr 0.001529 wd 0.0500 time 0.3810 (0.3831) data time 0.0009 (0.0022) model time 0.3802 (0.3817) loss 3.6147 (3.3446) grad_norm 1.1918 (1.3947) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][500/625] eta 0:00:47 lr 0.001529 wd 0.0500 time 0.3624 (0.3831) data time 0.0009 (0.0022) model time 0.3615 (0.3817) loss 4.2461 (3.3505) grad_norm 1.6120 (1.3910) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][510/625] eta 0:00:44 lr 0.001529 wd 0.0500 time 0.3764 (0.3830) data time 0.0008 (0.0022) model time 0.3756 (0.3817) loss 3.0083 (3.3519) grad_norm 1.4979 (1.3881) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][520/625] eta 0:00:40 lr 0.001529 wd 0.0500 time 0.3874 (0.3830) data time 0.0009 (0.0021) model time 0.3865 (0.3816) loss 3.1600 (3.3529) grad_norm 1.0312 (1.3900) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][530/625] eta 0:00:36 lr 0.001529 wd 0.0500 time 0.3772 (0.3828) data time 0.0009 (0.0021) model time 0.3763 (0.3815) loss 2.7420 (3.3540) grad_norm 1.0752 (1.3875) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][540/625] eta 0:00:32 lr 0.001529 wd 0.0500 time 0.3730 (0.3827) data time 0.0008 (0.0021) model time 0.3721 (0.3814) loss 2.8125 (3.3524) grad_norm 1.1198 (1.3838) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][550/625] eta 0:00:28 lr 0.001528 wd 0.0500 time 0.3763 (0.3826) data time 0.0008 (0.0021) model time 0.3755 (0.3813) loss 2.4581 (3.3472) grad_norm 1.6285 (1.3817) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][560/625] eta 0:00:24 lr 0.001528 wd 0.0500 time 0.3788 (0.3826) data time 0.0011 (0.0021) model time 0.3777 (0.3812) loss 3.2611 (3.3507) grad_norm 1.0908 (1.3862) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:30:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][570/625] eta 0:00:21 lr 0.001528 wd 0.0500 time 0.3759 (0.3825) data time 0.0013 (0.0021) model time 0.3747 (0.3812) loss 3.6357 (3.3518) grad_norm 3.6682 (1.3881) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][580/625] eta 0:00:17 lr 0.001528 wd 0.0500 time 0.5881 (0.3839) data time 0.0012 (0.0020) model time 0.5869 (0.3826) loss 3.5031 (3.3480) grad_norm 1.3160 (1.3900) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][590/625] eta 0:00:13 lr 0.001528 wd 0.0500 time 0.3824 (0.3841) data time 0.0010 (0.0020) model time 0.3813 (0.3829) loss 3.6741 (3.3504) grad_norm 1.1773 (1.3883) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][600/625] eta 0:00:09 lr 0.001528 wd 0.0500 time 0.3797 (0.3840) data time 0.0009 (0.0020) model time 0.3788 (0.3828) loss 2.7124 (3.3507) grad_norm 1.2543 (1.3884) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][610/625] eta 0:00:05 lr 0.001528 wd 0.0500 time 0.3866 (0.3839) data time 0.0008 (0.0020) model time 0.3858 (0.3826) loss 3.5845 (3.3477) grad_norm 1.2513 (1.3931) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [110/300][620/625] eta 0:00:01 lr 0.001527 wd 0.0500 time 0.3669 (0.3838) data time 0.0006 (0.0020) model time 0.3663 (0.3825) loss 2.4297 (3.3465) grad_norm 1.2854 (1.3919) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:18 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 110 training takes 0:03:59 [2024-08-26 10:31:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:31:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.6763 (0.6763) Acc@1 85.840 (85.840) Acc@5 97.461 (97.461) Mem 11225MB [2024-08-26 10:31:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.146) Loss 1.0977 (0.8076) Acc@1 74.561 (82.089) Acc@5 93.604 (96.316) Mem 11225MB [2024-08-26 10:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.2871 (0.9634) Acc@1 69.678 (78.583) Acc@5 90.771 (94.457) Mem 11225MB [2024-08-26 10:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.371 Acc@5 94.474 [2024-08-26 10:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.4% [2024-08-26 10:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.37% [2024-08-26 10:31:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 10:31:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 10:31:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.498 (0.498) Loss 0.5410 (0.5410) Acc@1 88.477 (88.477) Acc@5 98.291 (98.291) Mem 11225MB [2024-08-26 10:31:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.9028 (0.6769) Acc@1 79.248 (84.695) Acc@5 95.264 (97.310) Mem 11225MB [2024-08-26 10:31:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.0469 (0.8191) Acc@1 74.365 (81.099) Acc@5 93.604 (95.731) Mem 11225MB [2024-08-26 10:31:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.806 Acc@5 95.709 [2024-08-26 10:31:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.8% [2024-08-26 10:31:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.81% [2024-08-26 10:31:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:31:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:31:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][0/625] eta 0:08:29 lr 0.001527 wd 0.0500 time 0.8159 (0.8159) data time 0.4815 (0.4815) model time 0.0000 (0.0000) loss 2.4858 (2.4858) grad_norm 1.0744 (1.0744) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][10/625] eta 0:04:18 lr 0.001527 wd 0.0500 time 0.3989 (0.4201) data time 0.0009 (0.0449) model time 0.0000 (0.0000) loss 3.3659 (3.2198) grad_norm 1.1382 (1.3222) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][20/625] eta 0:04:03 lr 0.001527 wd 0.0500 time 0.3700 (0.4017) data time 0.0010 (0.0243) model time 0.0000 (0.0000) loss 4.0377 (3.3603) grad_norm 1.2475 (1.3486) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][30/625] eta 0:03:55 lr 0.001527 wd 0.0500 time 0.3819 (0.3953) data time 0.0012 (0.0169) model time 0.0000 (0.0000) loss 3.1671 (3.3211) grad_norm 1.3916 (1.3596) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][40/625] eta 0:03:49 lr 0.001527 wd 0.0500 time 0.4154 (0.3924) data time 0.0010 (0.0131) model time 0.0000 (0.0000) loss 3.2828 (3.4070) grad_norm 1.1341 (1.3424) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][50/625] eta 0:03:44 lr 0.001527 wd 0.0500 time 0.3720 (0.3897) data time 0.0011 (0.0108) model time 0.0000 (0.0000) loss 3.8405 (3.4156) grad_norm 1.2823 (1.3209) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][60/625] eta 0:03:39 lr 0.001526 wd 0.0500 time 0.3725 (0.3878) data time 0.0011 (0.0092) model time 0.3714 (0.3767) loss 3.0951 (3.3613) grad_norm 0.8667 (1.3296) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][70/625] eta 0:03:36 lr 0.001526 wd 0.0500 time 0.3825 (0.3894) data time 0.0009 (0.0081) model time 0.3816 (0.3875) loss 3.6065 (3.3983) grad_norm 1.4335 (1.3517) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:31:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][80/625] eta 0:03:31 lr 0.001526 wd 0.0500 time 0.3774 (0.3883) data time 0.0012 (0.0072) model time 0.3762 (0.3849) loss 3.8365 (3.3697) grad_norm 1.2548 (1.3629) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][90/625] eta 0:03:27 lr 0.001526 wd 0.0500 time 0.3713 (0.3870) data time 0.0011 (0.0066) model time 0.3702 (0.3825) loss 3.6325 (3.3754) grad_norm 1.4442 (1.3622) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][100/625] eta 0:03:26 lr 0.001526 wd 0.0500 time 0.3861 (0.3925) data time 0.0010 (0.0060) model time 0.3850 (0.3942) loss 3.7085 (3.3629) grad_norm 1.6502 (1.3674) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][110/625] eta 0:03:21 lr 0.001526 wd 0.0500 time 0.3712 (0.3913) data time 0.0011 (0.0056) model time 0.3701 (0.3915) loss 3.7391 (3.3702) grad_norm 1.2372 (1.3722) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][120/625] eta 0:03:17 lr 0.001525 wd 0.0500 time 0.3887 (0.3906) data time 0.0011 (0.0052) model time 0.3876 (0.3901) loss 3.2958 (3.3816) grad_norm 1.3460 (1.3619) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][130/625] eta 0:03:13 lr 0.001525 wd 0.0500 time 0.3908 (0.3899) data time 0.0011 (0.0050) model time 0.3897 (0.3888) loss 3.7074 (3.3984) grad_norm 1.2853 (1.3545) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][140/625] eta 0:03:08 lr 0.001525 wd 0.0500 time 0.3764 (0.3894) data time 0.0009 (0.0047) model time 0.3755 (0.3880) loss 2.4506 (3.3826) grad_norm 1.4651 (1.3462) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][150/625] eta 0:03:05 lr 0.001525 wd 0.0500 time 0.5984 (0.3903) data time 0.0011 (0.0045) model time 0.5974 (0.3894) loss 4.1613 (3.3907) grad_norm 1.2602 (1.3567) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][160/625] eta 0:03:01 lr 0.001525 wd 0.0500 time 0.3817 (0.3910) data time 0.0011 (0.0043) model time 0.3806 (0.3903) loss 3.4159 (3.3948) grad_norm 1.9691 (1.3654) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][170/625] eta 0:02:57 lr 0.001525 wd 0.0500 time 0.3723 (0.3902) data time 0.0009 (0.0041) model time 0.3714 (0.3892) loss 3.4356 (3.3964) grad_norm 1.5984 (1.3603) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][180/625] eta 0:02:53 lr 0.001525 wd 0.0500 time 0.3720 (0.3897) data time 0.0012 (0.0039) model time 0.3709 (0.3885) loss 3.3457 (3.3928) grad_norm 1.3187 (1.3558) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][190/625] eta 0:02:49 lr 0.001524 wd 0.0500 time 0.3808 (0.3891) data time 0.0011 (0.0038) model time 0.3798 (0.3877) loss 2.8716 (3.3895) grad_norm 1.8121 (1.3524) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][200/625] eta 0:02:45 lr 0.001524 wd 0.0500 time 0.3830 (0.3886) data time 0.0011 (0.0037) model time 0.3819 (0.3869) loss 3.6575 (3.3920) grad_norm 1.3768 (1.3514) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][210/625] eta 0:02:41 lr 0.001524 wd 0.0500 time 0.3721 (0.3881) data time 0.0011 (0.0036) model time 0.3710 (0.3864) loss 3.2019 (3.3979) grad_norm 1.8818 (1.3590) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][220/625] eta 0:02:37 lr 0.001524 wd 0.0500 time 0.3746 (0.3877) data time 0.0011 (0.0034) model time 0.3734 (0.3859) loss 3.1337 (3.3888) grad_norm 1.0554 (1.3588) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:32:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][230/625] eta 0:02:32 lr 0.001524 wd 0.0500 time 0.3756 (0.3873) data time 0.0008 (0.0033) model time 0.3748 (0.3854) loss 3.0646 (3.3890) grad_norm 1.6194 (1.3613) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][240/625] eta 0:02:28 lr 0.001524 wd 0.0500 time 0.3719 (0.3870) data time 0.0013 (0.0033) model time 0.3706 (0.3850) loss 2.5613 (3.3795) grad_norm 1.3735 (1.3687) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][250/625] eta 0:02:24 lr 0.001523 wd 0.0500 time 0.3813 (0.3866) data time 0.0013 (0.0032) model time 0.3800 (0.3846) loss 3.0400 (3.3796) grad_norm 1.5435 (1.3663) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][260/625] eta 0:02:21 lr 0.001523 wd 0.0500 time 0.3774 (0.3870) data time 0.0011 (0.0031) model time 0.3763 (0.3852) loss 3.4471 (3.3836) grad_norm 0.9854 (1.3680) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][270/625] eta 0:02:17 lr 0.001523 wd 0.0500 time 0.3806 (0.3867) data time 0.0008 (0.0030) model time 0.3797 (0.3848) loss 3.0268 (3.3714) grad_norm 1.5304 (1.3659) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][280/625] eta 0:02:13 lr 0.001523 wd 0.0500 time 0.3761 (0.3863) data time 0.0008 (0.0030) model time 0.3753 (0.3844) loss 3.8768 (3.3847) grad_norm 1.1572 (1.3620) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][290/625] eta 0:02:09 lr 0.001523 wd 0.0500 time 0.3717 (0.3860) data time 0.0011 (0.0029) model time 0.3706 (0.3840) loss 3.8698 (3.3913) grad_norm 1.1869 (1.3682) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][300/625] eta 0:02:05 lr 0.001523 wd 0.0500 time 0.3779 (0.3857) data time 0.0009 (0.0028) model time 0.3770 (0.3837) loss 3.1911 (3.3820) grad_norm 1.4004 (1.3647) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][310/625] eta 0:02:02 lr 0.001523 wd 0.0500 time 0.3725 (0.3879) data time 0.0011 (0.0028) model time 0.3714 (0.3863) loss 3.9303 (3.3897) grad_norm 1.2297 (1.3626) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][320/625] eta 0:01:58 lr 0.001522 wd 0.0500 time 0.3733 (0.3876) data time 0.0014 (0.0027) model time 0.3719 (0.3860) loss 3.2549 (3.3879) grad_norm 1.4192 (1.3585) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][330/625] eta 0:01:54 lr 0.001522 wd 0.0500 time 0.3771 (0.3873) data time 0.0008 (0.0027) model time 0.3763 (0.3857) loss 2.1911 (3.3818) grad_norm 1.0571 (1.3605) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][340/625] eta 0:01:50 lr 0.001522 wd 0.0500 time 0.3914 (0.3877) data time 0.0012 (0.0026) model time 0.3901 (0.3861) loss 3.7096 (3.3820) grad_norm 1.0348 (1.3608) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][350/625] eta 0:01:46 lr 0.001522 wd 0.0500 time 0.3697 (0.3874) data time 0.0012 (0.0026) model time 0.3685 (0.3858) loss 2.8757 (3.3770) grad_norm 1.5661 (1.3613) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][360/625] eta 0:01:42 lr 0.001522 wd 0.0500 time 0.3879 (0.3872) data time 0.0011 (0.0026) model time 0.3868 (0.3856) loss 3.6087 (3.3729) grad_norm 1.5465 (1.3629) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][370/625] eta 0:01:38 lr 0.001522 wd 0.0500 time 0.6072 (0.3876) data time 0.0009 (0.0025) model time 0.6064 (0.3860) loss 4.3638 (3.3781) grad_norm 1.6662 (1.3697) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][380/625] eta 0:01:34 lr 0.001522 wd 0.0500 time 0.3779 (0.3872) data time 0.0011 (0.0025) model time 0.3768 (0.3856) loss 3.3997 (3.3745) grad_norm 1.4117 (1.3835) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:33:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][390/625] eta 0:01:30 lr 0.001521 wd 0.0500 time 0.3743 (0.3869) data time 0.0011 (0.0025) model time 0.3732 (0.3853) loss 3.2080 (3.3733) grad_norm 1.4231 (1.3863) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][400/625] eta 0:01:27 lr 0.001521 wd 0.0500 time 0.3707 (0.3867) data time 0.0010 (0.0024) model time 0.3697 (0.3851) loss 2.3287 (3.3692) grad_norm 1.5218 (1.3848) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][410/625] eta 0:01:23 lr 0.001521 wd 0.0500 time 0.3840 (0.3866) data time 0.0012 (0.0024) model time 0.3828 (0.3849) loss 3.7186 (3.3677) grad_norm 1.6935 (1.3937) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][420/625] eta 0:01:19 lr 0.001521 wd 0.0500 time 0.3795 (0.3864) data time 0.0011 (0.0024) model time 0.3785 (0.3847) loss 3.7450 (3.3737) grad_norm 1.2287 (1.3969) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][430/625] eta 0:01:15 lr 0.001521 wd 0.0500 time 0.3769 (0.3861) data time 0.0012 (0.0023) model time 0.3757 (0.3844) loss 3.1292 (3.3752) grad_norm 2.1494 (1.3992) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][440/625] eta 0:01:11 lr 0.001521 wd 0.0500 time 0.3710 (0.3859) data time 0.0009 (0.0023) model time 0.3702 (0.3842) loss 3.4850 (3.3792) grad_norm 1.1358 (1.3936) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][450/625] eta 0:01:07 lr 0.001520 wd 0.0500 time 0.3786 (0.3857) data time 0.0011 (0.0023) model time 0.3775 (0.3839) loss 3.5697 (3.3727) grad_norm 1.2635 (1.3940) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][460/625] eta 0:01:03 lr 0.001520 wd 0.0500 time 0.3805 (0.3860) data time 0.0011 (0.0023) model time 0.3795 (0.3843) loss 3.4323 (3.3746) grad_norm 1.6694 (1.3993) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][470/625] eta 0:00:59 lr 0.001520 wd 0.0500 time 0.3781 (0.3858) data time 0.0008 (0.0022) model time 0.3773 (0.3841) loss 2.5410 (3.3705) grad_norm 1.4780 (1.3984) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][480/625] eta 0:00:55 lr 0.001520 wd 0.0500 time 0.3772 (0.3856) data time 0.0008 (0.0022) model time 0.3765 (0.3839) loss 4.2290 (3.3742) grad_norm 1.2475 (1.3970) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][490/625] eta 0:00:52 lr 0.001520 wd 0.0500 time 0.3791 (0.3866) data time 0.0008 (0.0022) model time 0.3782 (0.3850) loss 3.0588 (3.3707) grad_norm 1.3122 (1.3984) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][500/625] eta 0:00:48 lr 0.001520 wd 0.0500 time 0.3774 (0.3864) data time 0.0008 (0.0022) model time 0.3765 (0.3848) loss 4.1152 (3.3692) grad_norm 1.3363 (1.3956) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][510/625] eta 0:00:44 lr 0.001520 wd 0.0500 time 0.3735 (0.3863) data time 0.0008 (0.0021) model time 0.3727 (0.3847) loss 3.4853 (3.3743) grad_norm 2.2870 (1.3996) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][520/625] eta 0:00:40 lr 0.001519 wd 0.0500 time 0.3766 (0.3861) data time 0.0011 (0.0021) model time 0.3756 (0.3845) loss 2.8434 (3.3713) grad_norm 1.1687 (1.3973) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][530/625] eta 0:00:36 lr 0.001519 wd 0.0500 time 0.3738 (0.3860) data time 0.0011 (0.0021) model time 0.3726 (0.3844) loss 3.4918 (3.3690) grad_norm 1.2664 (1.3956) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:34:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][540/625] eta 0:00:32 lr 0.001519 wd 0.0500 time 0.3811 (0.3859) data time 0.0012 (0.0021) model time 0.3799 (0.3843) loss 3.8001 (3.3721) grad_norm 1.3658 (1.4015) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][550/625] eta 0:00:28 lr 0.001519 wd 0.0500 time 0.3860 (0.3858) data time 0.0008 (0.0021) model time 0.3852 (0.3842) loss 3.8769 (3.3687) grad_norm 1.3262 (1.3989) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][560/625] eta 0:00:25 lr 0.001519 wd 0.0500 time 0.3851 (0.3856) data time 0.0008 (0.0021) model time 0.3843 (0.3841) loss 3.5494 (3.3674) grad_norm 1.2293 (1.3948) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][570/625] eta 0:00:21 lr 0.001519 wd 0.0500 time 0.3817 (0.3855) data time 0.0011 (0.0020) model time 0.3806 (0.3839) loss 2.7555 (3.3614) grad_norm 1.8103 (1.3939) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][580/625] eta 0:00:17 lr 0.001518 wd 0.0500 time 0.5609 (0.3857) data time 0.0009 (0.0020) model time 0.5600 (0.3841) loss 4.3268 (3.3603) grad_norm 1.7591 (1.3960) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][590/625] eta 0:00:13 lr 0.001518 wd 0.0500 time 0.3818 (0.3856) data time 0.0011 (0.0020) model time 0.3807 (0.3840) loss 3.4776 (3.3637) grad_norm 1.4530 (1.3982) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][600/625] eta 0:00:09 lr 0.001518 wd 0.0500 time 0.3745 (0.3855) data time 0.0011 (0.0020) model time 0.3734 (0.3839) loss 3.3640 (3.3664) grad_norm 1.3836 (1.3997) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][610/625] eta 0:00:05 lr 0.001518 wd 0.0500 time 0.3691 (0.3863) data time 0.0006 (0.0020) model time 0.3685 (0.3848) loss 3.7840 (3.3702) grad_norm 1.4947 (1.3995) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [111/300][620/625] eta 0:00:01 lr 0.001518 wd 0.0500 time 0.3665 (0.3860) data time 0.0008 (0.0020) model time 0.3657 (0.3845) loss 3.5446 (3.3745) grad_norm 1.1853 (1.3967) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:29 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 111 training takes 0:04:01 [2024-08-26 10:35:29 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:35:30 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:35:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.508 (0.508) Loss 0.6509 (0.6509) Acc@1 85.547 (85.547) Acc@5 97.705 (97.705) Mem 11225MB [2024-08-26 10:35:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.147) Loss 0.9980 (0.7776) Acc@1 76.562 (82.888) Acc@5 93.848 (96.529) Mem 11225MB [2024-08-26 10:35:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 1.2354 (0.9386) Acc@1 70.801 (79.025) Acc@5 91.357 (94.624) Mem 11225MB [2024-08-26 10:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.737 Acc@5 94.606 [2024-08-26 10:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.7% [2024-08-26 10:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.74% [2024-08-26 10:35:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 10:35:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 10:35:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 0.5400 (0.5400) Acc@1 88.428 (88.428) Acc@5 98.291 (98.291) Mem 11225MB [2024-08-26 10:35:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 0.9033 (0.6761) Acc@1 79.053 (84.721) Acc@5 95.361 (97.314) Mem 11225MB [2024-08-26 10:35:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.0459 (0.8175) Acc@1 74.170 (81.110) Acc@5 93.750 (95.764) Mem 11225MB [2024-08-26 10:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.836 Acc@5 95.731 [2024-08-26 10:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.8% [2024-08-26 10:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.84% [2024-08-26 10:35:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:35:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:35:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][0/625] eta 0:08:07 lr 0.001518 wd 0.0500 time 0.7797 (0.7797) data time 0.4443 (0.4443) model time 0.0000 (0.0000) loss 3.0353 (3.0353) grad_norm 2.0972 (2.0972) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][10/625] eta 0:04:18 lr 0.001518 wd 0.0500 time 0.3918 (0.4198) data time 0.0010 (0.0416) model time 0.0000 (0.0000) loss 3.8050 (3.4377) grad_norm 1.3528 (1.5138) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][20/625] eta 0:04:02 lr 0.001517 wd 0.0500 time 0.3629 (0.4014) data time 0.0008 (0.0224) model time 0.0000 (0.0000) loss 2.4711 (3.2040) grad_norm 0.9780 (1.3868) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][30/625] eta 0:04:02 lr 0.001517 wd 0.0500 time 0.3713 (0.4068) data time 0.0011 (0.0164) model time 0.0000 (0.0000) loss 3.7138 (3.3722) grad_norm 1.4437 (1.3679) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][40/625] eta 0:03:53 lr 0.001517 wd 0.0500 time 0.3898 (0.3997) data time 0.0009 (0.0127) model time 0.0000 (0.0000) loss 4.4028 (3.4509) grad_norm 1.1257 (1.4053) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:35:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][50/625] eta 0:03:47 lr 0.001517 wd 0.0500 time 0.3774 (0.3952) data time 0.0008 (0.0104) model time 0.0000 (0.0000) loss 2.2402 (3.3761) grad_norm 0.8489 (1.3848) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][60/625] eta 0:03:41 lr 0.001517 wd 0.0500 time 0.3717 (0.3923) data time 0.0009 (0.0089) model time 0.3709 (0.3769) loss 3.7332 (3.4133) grad_norm 1.4726 (1.3793) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][70/625] eta 0:03:37 lr 0.001517 wd 0.0500 time 0.3768 (0.3928) data time 0.0008 (0.0078) model time 0.3760 (0.3857) loss 3.1482 (3.3865) grad_norm 1.1328 (1.3483) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][80/625] eta 0:03:33 lr 0.001517 wd 0.0500 time 0.3856 (0.3909) data time 0.0008 (0.0069) model time 0.3847 (0.3826) loss 2.5664 (3.3947) grad_norm 1.3524 (1.3758) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][90/625] eta 0:03:28 lr 0.001516 wd 0.0500 time 0.3811 (0.3892) data time 0.0008 (0.0063) model time 0.3803 (0.3805) loss 2.3638 (3.4096) grad_norm 1.6489 (1.3829) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][100/625] eta 0:03:23 lr 0.001516 wd 0.0500 time 0.3904 (0.3882) data time 0.0011 (0.0058) model time 0.3893 (0.3801) loss 3.6204 (3.4046) grad_norm 1.8271 (1.3855) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][110/625] eta 0:03:19 lr 0.001516 wd 0.0500 time 0.3697 (0.3873) data time 0.0008 (0.0054) model time 0.3688 (0.3796) loss 3.5615 (3.4192) grad_norm 1.5674 (1.3828) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][120/625] eta 0:03:15 lr 0.001516 wd 0.0500 time 0.3749 (0.3872) data time 0.0011 (0.0052) model time 0.3739 (0.3800) loss 2.6986 (3.4040) grad_norm 1.3953 (1.3884) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][130/625] eta 0:03:11 lr 0.001516 wd 0.0500 time 0.3808 (0.3867) data time 0.0010 (0.0049) model time 0.3797 (0.3800) loss 3.3206 (3.3884) grad_norm 1.2641 (1.3891) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][140/625] eta 0:03:07 lr 0.001516 wd 0.0500 time 0.3811 (0.3860) data time 0.0011 (0.0046) model time 0.3799 (0.3795) loss 3.4420 (3.3841) grad_norm 0.9228 (1.3846) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][150/625] eta 0:03:03 lr 0.001515 wd 0.0500 time 0.3779 (0.3856) data time 0.0008 (0.0044) model time 0.3770 (0.3794) loss 2.7955 (3.3652) grad_norm 1.0293 (1.3754) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][160/625] eta 0:02:59 lr 0.001515 wd 0.0500 time 0.3735 (0.3850) data time 0.0010 (0.0042) model time 0.3724 (0.3790) loss 2.5738 (3.3714) grad_norm 1.4704 (1.3664) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][170/625] eta 0:02:55 lr 0.001515 wd 0.0500 time 0.3671 (0.3868) data time 0.0009 (0.0040) model time 0.3662 (0.3819) loss 2.5989 (3.3576) grad_norm 1.4784 (1.3686) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][180/625] eta 0:02:51 lr 0.001515 wd 0.0500 time 0.3718 (0.3863) data time 0.0014 (0.0039) model time 0.3704 (0.3815) loss 3.6833 (3.3722) grad_norm 1.3314 (1.3732) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][190/625] eta 0:02:48 lr 0.001515 wd 0.0500 time 0.3969 (0.3870) data time 0.0009 (0.0037) model time 0.3960 (0.3827) loss 4.2742 (3.3849) grad_norm 1.0811 (1.3645) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][200/625] eta 0:02:44 lr 0.001515 wd 0.0500 time 0.3696 (0.3865) data time 0.0009 (0.0036) model time 0.3688 (0.3823) loss 3.9442 (3.3881) grad_norm 2.2256 (1.3723) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:36:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][210/625] eta 0:02:40 lr 0.001515 wd 0.0500 time 0.3731 (0.3860) data time 0.0009 (0.0035) model time 0.3722 (0.3818) loss 2.9254 (3.3759) grad_norm 1.3154 (1.3831) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][220/625] eta 0:02:36 lr 0.001514 wd 0.0500 time 0.4168 (0.3858) data time 0.0008 (0.0034) model time 0.4160 (0.3817) loss 2.7354 (3.3606) grad_norm 1.7662 (1.3872) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][230/625] eta 0:02:32 lr 0.001514 wd 0.0500 time 0.3729 (0.3854) data time 0.0011 (0.0033) model time 0.3718 (0.3813) loss 3.5352 (3.3588) grad_norm 2.4461 (1.4072) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][240/625] eta 0:02:28 lr 0.001514 wd 0.0500 time 0.3745 (0.3866) data time 0.0010 (0.0032) model time 0.3735 (0.3831) loss 4.2619 (3.3615) grad_norm 1.6501 (1.4155) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][250/625] eta 0:02:24 lr 0.001514 wd 0.0500 time 0.3725 (0.3866) data time 0.0009 (0.0031) model time 0.3716 (0.3831) loss 4.2596 (3.3686) grad_norm 1.1558 (1.4068) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][260/625] eta 0:02:20 lr 0.001514 wd 0.0500 time 0.3771 (0.3863) data time 0.0011 (0.0030) model time 0.3760 (0.3829) loss 3.8117 (3.3725) grad_norm 1.9177 (1.4004) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][270/625] eta 0:02:16 lr 0.001514 wd 0.0500 time 0.3740 (0.3859) data time 0.0008 (0.0030) model time 0.3732 (0.3825) loss 3.7902 (3.3766) grad_norm 1.2596 (1.4031) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][280/625] eta 0:02:13 lr 0.001513 wd 0.0500 time 0.3754 (0.3856) data time 0.0009 (0.0029) model time 0.3745 (0.3822) loss 3.1312 (3.3824) grad_norm 0.8921 (1.4029) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][290/625] eta 0:02:09 lr 0.001513 wd 0.0500 time 0.3805 (0.3874) data time 0.0008 (0.0028) model time 0.3797 (0.3845) loss 4.3104 (3.3844) grad_norm 1.1499 (1.4004) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][300/625] eta 0:02:05 lr 0.001513 wd 0.0500 time 0.3684 (0.3871) data time 0.0008 (0.0028) model time 0.3676 (0.3842) loss 4.5373 (3.3833) grad_norm 2.0881 (1.4005) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][310/625] eta 0:02:01 lr 0.001513 wd 0.0500 time 0.3784 (0.3867) data time 0.0011 (0.0027) model time 0.3773 (0.3838) loss 2.4804 (3.3720) grad_norm 1.8024 (1.3977) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][320/625] eta 0:01:57 lr 0.001513 wd 0.0500 time 0.3805 (0.3864) data time 0.0012 (0.0027) model time 0.3793 (0.3835) loss 3.0540 (3.3711) grad_norm 1.0133 (1.3942) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][330/625] eta 0:01:53 lr 0.001513 wd 0.0500 time 0.3825 (0.3861) data time 0.0014 (0.0026) model time 0.3811 (0.3832) loss 3.0031 (3.3745) grad_norm 1.9632 (1.3967) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][340/625] eta 0:01:49 lr 0.001513 wd 0.0500 time 0.3875 (0.3859) data time 0.0009 (0.0026) model time 0.3865 (0.3831) loss 3.8633 (3.3706) grad_norm 1.1653 (1.3938) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][350/625] eta 0:01:46 lr 0.001512 wd 0.0500 time 0.3753 (0.3857) data time 0.0012 (0.0026) model time 0.3741 (0.3829) loss 3.2418 (3.3654) grad_norm 1.2191 (1.3870) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:37:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][360/625] eta 0:01:42 lr 0.001512 wd 0.0500 time 0.3733 (0.3856) data time 0.0011 (0.0025) model time 0.3722 (0.3828) loss 3.3576 (3.3641) grad_norm 1.1464 (1.3849) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][370/625] eta 0:01:38 lr 0.001512 wd 0.0500 time 0.3937 (0.3854) data time 0.0011 (0.0025) model time 0.3926 (0.3826) loss 3.4714 (3.3645) grad_norm 1.7788 (1.3864) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][380/625] eta 0:01:34 lr 0.001512 wd 0.0500 time 0.3816 (0.3853) data time 0.0012 (0.0025) model time 0.3804 (0.3825) loss 2.4325 (3.3566) grad_norm 2.2781 (1.3887) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][390/625] eta 0:01:30 lr 0.001512 wd 0.0500 time 0.3819 (0.3851) data time 0.0008 (0.0024) model time 0.3810 (0.3824) loss 3.1575 (3.3575) grad_norm 1.3126 (1.3943) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][400/625] eta 0:01:26 lr 0.001512 wd 0.0500 time 0.3807 (0.3855) data time 0.0009 (0.0024) model time 0.3798 (0.3828) loss 3.9609 (3.3578) grad_norm 1.3200 (1.3934) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][410/625] eta 0:01:22 lr 0.001512 wd 0.0500 time 0.3736 (0.3853) data time 0.0009 (0.0024) model time 0.3727 (0.3827) loss 2.9819 (3.3505) grad_norm 1.5994 (1.3934) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][420/625] eta 0:01:19 lr 0.001511 wd 0.0500 time 0.3757 (0.3861) data time 0.0008 (0.0023) model time 0.3749 (0.3837) loss 3.9591 (3.3575) grad_norm 1.9177 (1.3883) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][430/625] eta 0:01:15 lr 0.001511 wd 0.0500 time 0.3685 (0.3859) data time 0.0009 (0.0023) model time 0.3676 (0.3835) loss 3.9496 (3.3599) grad_norm 1.0409 (1.3869) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][440/625] eta 0:01:11 lr 0.001511 wd 0.0500 time 0.3812 (0.3858) data time 0.0009 (0.0023) model time 0.3803 (0.3834) loss 2.5325 (3.3614) grad_norm 2.0091 (1.3913) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][450/625] eta 0:01:07 lr 0.001511 wd 0.0500 time 0.3721 (0.3857) data time 0.0008 (0.0023) model time 0.3713 (0.3833) loss 4.0858 (3.3612) grad_norm 1.4044 (1.3979) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][460/625] eta 0:01:03 lr 0.001511 wd 0.0500 time 0.3838 (0.3856) data time 0.0009 (0.0022) model time 0.3830 (0.3832) loss 2.5523 (3.3545) grad_norm 0.9349 (1.3945) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][470/625] eta 0:00:59 lr 0.001511 wd 0.0500 time 0.3753 (0.3855) data time 0.0009 (0.0022) model time 0.3744 (0.3831) loss 3.9759 (3.3534) grad_norm 1.4643 (1.3973) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][480/625] eta 0:00:55 lr 0.001510 wd 0.0500 time 0.3764 (0.3853) data time 0.0009 (0.0022) model time 0.3755 (0.3830) loss 4.2547 (3.3608) grad_norm 1.5258 (1.3942) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][490/625] eta 0:00:52 lr 0.001510 wd 0.0500 time 0.3733 (0.3852) data time 0.0008 (0.0022) model time 0.3725 (0.3829) loss 2.7109 (3.3625) grad_norm 1.0677 (1.3926) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][500/625] eta 0:00:48 lr 0.001510 wd 0.0500 time 0.4015 (0.3867) data time 0.0011 (0.0022) model time 0.4004 (0.3845) loss 3.1633 (3.3607) grad_norm 1.1121 (1.3913) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][510/625] eta 0:00:44 lr 0.001510 wd 0.0500 time 0.3739 (0.3866) data time 0.0010 (0.0022) model time 0.3729 (0.3844) loss 3.3266 (3.3587) grad_norm 1.2551 (1.3907) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][520/625] eta 0:00:40 lr 0.001510 wd 0.0500 time 0.3805 (0.3865) data time 0.0009 (0.0022) model time 0.3796 (0.3843) loss 2.9136 (3.3573) grad_norm 1.5673 (1.3902) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][530/625] eta 0:00:36 lr 0.001510 wd 0.0500 time 0.3929 (0.3864) data time 0.0010 (0.0022) model time 0.3919 (0.3842) loss 3.5838 (3.3555) grad_norm 1.6474 (1.3891) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][540/625] eta 0:00:32 lr 0.001510 wd 0.0500 time 0.3806 (0.3870) data time 0.0011 (0.0021) model time 0.3795 (0.3849) loss 3.3414 (3.3527) grad_norm 1.6110 (1.3870) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][550/625] eta 0:00:29 lr 0.001509 wd 0.0500 time 0.3769 (0.3868) data time 0.0010 (0.0021) model time 0.3758 (0.3847) loss 3.4227 (3.3524) grad_norm 1.3803 (1.3849) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][560/625] eta 0:00:25 lr 0.001509 wd 0.0500 time 0.3779 (0.3867) data time 0.0008 (0.0021) model time 0.3771 (0.3845) loss 3.6642 (3.3538) grad_norm 1.4440 (1.3874) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][570/625] eta 0:00:21 lr 0.001509 wd 0.0500 time 0.3771 (0.3865) data time 0.0011 (0.0021) model time 0.3761 (0.3844) loss 2.7249 (3.3569) grad_norm 1.5410 (1.3883) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][580/625] eta 0:00:17 lr 0.001509 wd 0.0500 time 0.3814 (0.3867) data time 0.0010 (0.0021) model time 0.3804 (0.3846) loss 3.0748 (3.3532) grad_norm 1.2283 (1.3863) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][590/625] eta 0:00:13 lr 0.001509 wd 0.0500 time 0.3699 (0.3866) data time 0.0009 (0.0021) model time 0.3690 (0.3845) loss 3.5846 (3.3506) grad_norm 1.7342 (1.3896) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][600/625] eta 0:00:09 lr 0.001509 wd 0.0500 time 0.3805 (0.3864) data time 0.0011 (0.0020) model time 0.3794 (0.3844) loss 3.6104 (3.3519) grad_norm 1.4572 (1.3906) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][610/625] eta 0:00:05 lr 0.001508 wd 0.0500 time 0.3622 (0.3863) data time 0.0008 (0.0020) model time 0.3614 (0.3842) loss 3.0313 (3.3492) grad_norm 1.2323 (1.3867) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [112/300][620/625] eta 0:00:01 lr 0.001508 wd 0.0500 time 0.3641 (0.3859) data time 0.0008 (0.0020) model time 0.3633 (0.3838) loss 3.5855 (3.3447) grad_norm 1.1424 (1.3837) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 10:39:39 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 112 training takes 0:04:01 [2024-08-26 10:39:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:39:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:39:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.6860 (0.6860) Acc@1 85.107 (85.107) Acc@5 97.559 (97.559) Mem 11225MB [2024-08-26 10:39:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.0527 (0.7959) Acc@1 75.684 (82.457) Acc@5 94.189 (96.422) Mem 11225MB [2024-08-26 10:39:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.1719 (0.9488) Acc@1 72.754 (78.753) Acc@5 92.139 (94.638) Mem 11225MB [2024-08-26 10:39:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.423 Acc@5 94.534 [2024-08-26 10:39:43 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.4% [2024-08-26 10:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.860 (0.860) Loss 0.5396 (0.5396) Acc@1 88.477 (88.477) Acc@5 98.340 (98.340) Mem 11225MB [2024-08-26 10:39:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.179) Loss 0.9023 (0.6749) Acc@1 78.955 (84.695) Acc@5 95.312 (97.319) Mem 11225MB [2024-08-26 10:39:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.138) Loss 1.0449 (0.8162) Acc@1 74.219 (81.124) Acc@5 93.311 (95.752) Mem 11225MB [2024-08-26 10:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-26 10:41:23 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-26 10:41:35 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-26 10:41:48 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-26 10:41:48 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-26 10:41:50 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-26 10:41:52 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-26 10:41:52 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 112) [2024-08-26 10:41:52 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-26 10:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][0/625] eta 2:41:25 lr 0.001508 wd 0.0500 time 15.4972 (15.4972) data time 0.7077 (0.7077) model time 0.0000 (0.0000) loss 3.8654 (3.8654) grad_norm 2.6285 (2.6285) loss_scale 2048.0000 (2048.0000) mem 23875MB [2024-08-26 10:42:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][10/625] eta 0:18:46 lr 0.001508 wd 0.0500 time 0.3504 (1.8321) data time 0.0010 (0.0653) model time 0.0000 (0.0000) loss 2.7491 (3.8314) grad_norm 1.7174 (1.6966) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-26 10:42:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][20/625] eta 0:11:19 lr 0.001508 wd 0.0500 time 0.3425 (1.1239) data time 0.0009 (0.0347) model time 0.0000 (0.0000) loss 3.5680 (3.6364) grad_norm 0.9625 (1.4874) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-26 10:42:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][30/625] eta 0:08:43 lr 0.001508 wd 0.0500 time 0.3425 (0.8792) data time 0.0006 (0.0238) model time 0.0000 (0.0000) loss 2.4433 (3.6058) grad_norm 1.0647 (1.4035) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-26 10:42:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][40/625] eta 0:07:22 lr 0.001508 wd 0.0500 time 0.6147 (0.7556) data time 0.0009 (0.0182) model time 0.0000 (0.0000) loss 3.3072 (3.5592) grad_norm 1.4526 (1.3718) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-26 10:42:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][50/625] eta 0:06:28 lr 0.001507 wd 0.0500 time 0.3447 (0.6751) data time 0.0007 (0.0149) model time 0.0000 (0.0000) loss 4.2194 (3.5448) grad_norm 1.2932 (1.3677) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-26 10:42:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][60/625] eta 0:05:50 lr 0.001507 wd 0.0500 time 0.3438 (0.6210) data time 0.0009 (0.0126) model time 0.3429 (0.3441) loss 3.5488 (3.5002) grad_norm 1.3106 (1.3602) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-26 10:42:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][70/625] eta 0:05:22 lr 0.001507 wd 0.0500 time 0.3423 (0.5819) data time 0.0008 (0.0109) model time 0.3415 (0.3433) loss 3.1761 (3.4545) grad_norm 1.2809 (1.3641) loss_scale 4096.0000 (2221.0704) mem 11219MB [2024-08-26 10:42:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][80/625] eta 0:05:01 lr 0.001507 wd 0.0500 time 0.3392 (0.5523) data time 0.0008 (0.0097) model time 0.3384 (0.3427) loss 3.0669 (3.4426) grad_norm 1.6678 (1.3793) loss_scale 4096.0000 (2452.5432) mem 11219MB [2024-08-26 10:42:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][90/625] eta 0:04:43 lr 0.001507 wd 0.0500 time 0.3502 (0.5295) data time 0.0007 (0.0088) model time 0.3495 (0.3430) loss 3.8073 (3.4296) grad_norm 0.9453 (1.3690) loss_scale 4096.0000 (2633.1429) mem 11219MB [2024-08-26 10:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][100/625] eta 0:04:28 lr 0.001507 wd 0.0500 time 0.3486 (0.5112) data time 0.0006 (0.0080) model time 0.3480 (0.3431) loss 3.9355 (3.4425) grad_norm 2.3170 (1.3819) loss_scale 4096.0000 (2777.9802) mem 11219MB [2024-08-26 10:42:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][110/625] eta 0:04:15 lr 0.001507 wd 0.0500 time 0.3399 (0.4961) data time 0.0009 (0.0073) model time 0.3390 (0.3431) loss 3.0140 (3.4336) grad_norm 1.0691 (1.3602) loss_scale 4096.0000 (2896.7207) mem 11219MB [2024-08-26 10:42:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][120/625] eta 0:04:04 lr 0.001506 wd 0.0500 time 0.3438 (0.4838) data time 0.0006 (0.0068) model time 0.3431 (0.3435) loss 2.4604 (3.4380) grad_norm 1.2216 (1.3566) loss_scale 4096.0000 (2995.8347) mem 11219MB [2024-08-26 10:42:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][130/625] eta 0:03:54 lr 0.001506 wd 0.0500 time 0.3463 (0.4732) data time 0.0009 (0.0064) model time 0.3454 (0.3436) loss 3.4718 (3.4206) grad_norm 1.0635 (1.3503) loss_scale 4096.0000 (3079.8168) mem 11219MB [2024-08-26 10:43:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][140/625] eta 0:03:45 lr 0.001506 wd 0.0500 time 0.3398 (0.4641) data time 0.0007 (0.0060) model time 0.3391 (0.3436) loss 3.2391 (3.4010) grad_norm 1.3095 (1.3558) loss_scale 4096.0000 (3151.8865) mem 11219MB [2024-08-26 10:43:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][150/625] eta 0:03:36 lr 0.001506 wd 0.0500 time 0.3417 (0.4561) data time 0.0010 (0.0056) model time 0.3407 (0.3436) loss 2.7968 (3.3923) grad_norm 1.4184 (1.3606) loss_scale 4096.0000 (3214.4106) mem 11219MB [2024-08-26 10:43:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][160/625] eta 0:03:28 lr 0.001506 wd 0.0500 time 0.3424 (0.4493) data time 0.0009 (0.0053) model time 0.3415 (0.3437) loss 3.2982 (3.3868) grad_norm 1.2782 (1.3705) loss_scale 4096.0000 (3269.1677) mem 11219MB [2024-08-26 10:43:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][170/625] eta 0:03:21 lr 0.001506 wd 0.0500 time 0.3447 (0.4433) data time 0.0009 (0.0051) model time 0.3439 (0.3438) loss 3.1441 (3.3798) grad_norm 1.4156 (1.3727) loss_scale 4096.0000 (3317.5205) mem 11219MB [2024-08-26 10:43:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][180/625] eta 0:03:14 lr 0.001505 wd 0.0500 time 0.3423 (0.4379) data time 0.0008 (0.0049) model time 0.3415 (0.3439) loss 3.8025 (3.3712) grad_norm 2.0348 (1.3863) loss_scale 4096.0000 (3360.5304) mem 11219MB [2024-08-26 10:43:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][190/625] eta 0:03:08 lr 0.001505 wd 0.0500 time 0.3461 (0.4331) data time 0.0010 (0.0046) model time 0.3452 (0.3440) loss 2.9849 (3.3748) grad_norm 1.4509 (1.3881) loss_scale 4096.0000 (3399.0366) mem 11219MB [2024-08-26 10:43:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][200/625] eta 0:03:02 lr 0.001505 wd 0.0500 time 0.3431 (0.4289) data time 0.0009 (0.0045) model time 0.3423 (0.3442) loss 2.9774 (3.3584) grad_norm 1.6117 (1.3807) loss_scale 4096.0000 (3433.7114) mem 11219MB [2024-08-26 10:43:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][210/625] eta 0:02:56 lr 0.001505 wd 0.0500 time 0.3499 (0.4250) data time 0.0008 (0.0043) model time 0.3491 (0.3443) loss 3.6660 (3.3523) grad_norm 1.8104 (1.3847) loss_scale 4096.0000 (3465.0995) mem 11219MB [2024-08-26 10:43:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][220/625] eta 0:02:50 lr 0.001505 wd 0.0500 time 0.3449 (0.4214) data time 0.0006 (0.0042) model time 0.3443 (0.3444) loss 3.3822 (3.3495) grad_norm 1.1732 (1.3802) loss_scale 4096.0000 (3493.6471) mem 11219MB [2024-08-26 10:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][230/625] eta 0:02:45 lr 0.001505 wd 0.0500 time 0.3529 (0.4200) data time 0.0006 (0.0040) model time 0.3522 (0.3468) loss 2.1392 (3.3470) grad_norm 1.2796 (1.3834) loss_scale 4096.0000 (3519.7229) mem 11219MB [2024-08-26 10:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][240/625] eta 0:02:40 lr 0.001505 wd 0.0500 time 0.3406 (0.4177) data time 0.0008 (0.0039) model time 0.3399 (0.3476) loss 3.5992 (3.3477) grad_norm 1.2324 (1.3836) loss_scale 4096.0000 (3543.6349) mem 11219MB [2024-08-26 10:43:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][250/625] eta 0:02:35 lr 0.001504 wd 0.0500 time 0.3497 (0.4148) data time 0.0007 (0.0038) model time 0.3490 (0.3475) loss 3.5782 (3.3342) grad_norm 1.4076 (1.3754) loss_scale 4096.0000 (3565.6414) mem 11219MB [2024-08-26 10:43:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][260/625] eta 0:02:30 lr 0.001504 wd 0.0500 time 0.3414 (0.4122) data time 0.0006 (0.0037) model time 0.3407 (0.3474) loss 3.1401 (3.3265) grad_norm 2.9610 (1.3882) loss_scale 4096.0000 (3585.9617) mem 11219MB [2024-08-26 10:43:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][270/625] eta 0:02:25 lr 0.001504 wd 0.0500 time 0.3442 (0.4097) data time 0.0006 (0.0036) model time 0.3436 (0.3473) loss 4.0179 (3.3256) grad_norm 1.7983 (1.3965) loss_scale 4096.0000 (3604.7823) mem 11219MB [2024-08-26 10:43:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][280/625] eta 0:02:20 lr 0.001504 wd 0.0500 time 0.3429 (0.4074) data time 0.0009 (0.0035) model time 0.3420 (0.3471) loss 3.2387 (3.3276) grad_norm 1.5947 (1.4019) loss_scale 4096.0000 (3622.2633) mem 11219MB [2024-08-26 10:43:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][290/625] eta 0:02:15 lr 0.001504 wd 0.0500 time 0.3402 (0.4054) data time 0.0008 (0.0034) model time 0.3394 (0.3471) loss 2.5322 (3.3210) grad_norm 1.3361 (1.4059) loss_scale 4096.0000 (3638.5430) mem 11219MB [2024-08-26 10:43:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][300/625] eta 0:02:11 lr 0.001504 wd 0.0500 time 0.3403 (0.4034) data time 0.0009 (0.0033) model time 0.3394 (0.3470) loss 3.2287 (3.3093) grad_norm 1.3178 (1.4050) loss_scale 4096.0000 (3653.7409) mem 11219MB [2024-08-26 10:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][310/625] eta 0:02:06 lr 0.001503 wd 0.0500 time 0.3405 (0.4015) data time 0.0010 (0.0032) model time 0.3395 (0.3469) loss 3.4631 (3.3062) grad_norm 1.4139 (1.4001) loss_scale 4096.0000 (3667.9614) mem 11219MB [2024-08-26 10:44:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][320/625] eta 0:02:01 lr 0.001503 wd 0.0500 time 0.3405 (0.3997) data time 0.0007 (0.0032) model time 0.3399 (0.3468) loss 4.0110 (3.3176) grad_norm 1.5638 (1.3987) loss_scale 4096.0000 (3681.2960) mem 11219MB [2024-08-26 10:44:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][330/625] eta 0:01:57 lr 0.001503 wd 0.0500 time 0.3427 (0.3981) data time 0.0006 (0.0031) model time 0.3421 (0.3466) loss 2.5192 (3.3150) grad_norm 1.3709 (1.3948) loss_scale 4096.0000 (3693.8248) mem 11219MB [2024-08-26 10:44:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][340/625] eta 0:01:53 lr 0.001503 wd 0.0500 time 0.3492 (0.3965) data time 0.0009 (0.0030) model time 0.3483 (0.3466) loss 3.5822 (3.3192) grad_norm 1.3948 (1.3957) loss_scale 4096.0000 (3705.6188) mem 11219MB [2024-08-26 10:44:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][350/625] eta 0:01:48 lr 0.001503 wd 0.0500 time 0.3450 (0.3962) data time 0.0007 (0.0030) model time 0.3443 (0.3479) loss 3.4585 (3.3176) grad_norm 1.5961 (1.3941) loss_scale 4096.0000 (3716.7407) mem 11219MB [2024-08-26 10:44:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][360/625] eta 0:01:44 lr 0.001503 wd 0.0500 time 0.3531 (0.3948) data time 0.0007 (0.0029) model time 0.3523 (0.3477) loss 2.9864 (3.3174) grad_norm 1.8699 (1.3897) loss_scale 4096.0000 (3727.2465) mem 11219MB [2024-08-26 10:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][370/625] eta 0:01:40 lr 0.001503 wd 0.0500 time 0.3554 (0.3935) data time 0.0009 (0.0029) model time 0.3545 (0.3476) loss 3.0429 (3.3172) grad_norm 1.4562 (1.3872) loss_scale 4096.0000 (3737.1860) mem 11219MB [2024-08-26 10:44:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][380/625] eta 0:01:36 lr 0.001502 wd 0.0500 time 0.3492 (0.3923) data time 0.0007 (0.0028) model time 0.3485 (0.3476) loss 2.2816 (3.3178) grad_norm 1.9963 (1.3918) loss_scale 4096.0000 (3746.6037) mem 11219MB [2024-08-26 10:44:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][390/625] eta 0:01:31 lr 0.001502 wd 0.0500 time 0.3469 (0.3910) data time 0.0007 (0.0028) model time 0.3462 (0.3475) loss 4.4179 (3.3146) grad_norm 2.5548 (1.4018) loss_scale 4096.0000 (3755.5396) mem 11219MB [2024-08-26 10:44:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][400/625] eta 0:01:27 lr 0.001502 wd 0.0500 time 0.3688 (0.3900) data time 0.0008 (0.0027) model time 0.3680 (0.3474) loss 3.6285 (3.3186) grad_norm 1.4490 (1.3994) loss_scale 4096.0000 (3764.0299) mem 11219MB [2024-08-26 10:44:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][410/625] eta 0:01:23 lr 0.001502 wd 0.0500 time 0.3434 (0.3888) data time 0.0009 (0.0027) model time 0.3425 (0.3473) loss 3.7904 (3.3249) grad_norm 1.5610 (1.3969) loss_scale 4096.0000 (3772.1071) mem 11219MB [2024-08-26 10:44:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][420/625] eta 0:01:19 lr 0.001502 wd 0.0500 time 0.3449 (0.3882) data time 0.0007 (0.0027) model time 0.3442 (0.3477) loss 4.0156 (3.3235) grad_norm 1.1782 (1.3936) loss_scale 4096.0000 (3779.8005) mem 11219MB [2024-08-26 10:44:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][430/625] eta 0:01:15 lr 0.001502 wd 0.0500 time 0.3483 (0.3872) data time 0.0007 (0.0026) model time 0.3477 (0.3476) loss 3.8052 (3.3306) grad_norm 1.2302 (1.3914) loss_scale 4096.0000 (3787.1369) mem 11219MB [2024-08-26 10:44:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][440/625] eta 0:01:11 lr 0.001501 wd 0.0500 time 0.3495 (0.3863) data time 0.0006 (0.0026) model time 0.3488 (0.3476) loss 4.1608 (3.3358) grad_norm 1.1120 (1.3886) loss_scale 4096.0000 (3794.1406) mem 11219MB [2024-08-26 10:44:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][450/625] eta 0:01:07 lr 0.001501 wd 0.0500 time 0.3460 (0.3854) data time 0.0007 (0.0026) model time 0.3453 (0.3475) loss 3.2667 (3.3343) grad_norm 1.9591 (1.3906) loss_scale 4096.0000 (3800.8337) mem 11219MB [2024-08-26 10:44:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][460/625] eta 0:01:03 lr 0.001501 wd 0.0500 time 0.3677 (0.3846) data time 0.0009 (0.0025) model time 0.3668 (0.3475) loss 3.6821 (3.3293) grad_norm 1.7246 (1.3898) loss_scale 4096.0000 (3807.2364) mem 11219MB [2024-08-26 10:44:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][470/625] eta 0:00:59 lr 0.001501 wd 0.0500 time 0.3473 (0.3838) data time 0.0009 (0.0025) model time 0.3464 (0.3475) loss 3.2415 (3.3229) grad_norm 1.1593 (1.3932) loss_scale 4096.0000 (3813.3673) mem 11219MB [2024-08-26 10:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][480/625] eta 0:00:55 lr 0.001501 wd 0.0500 time 0.3451 (0.3830) data time 0.0007 (0.0025) model time 0.3444 (0.3474) loss 3.4827 (3.3218) grad_norm 1.3023 (1.3937) loss_scale 4096.0000 (3819.2432) mem 11219MB [2024-08-26 10:45:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][490/625] eta 0:00:51 lr 0.001501 wd 0.0500 time 0.3463 (0.3823) data time 0.0009 (0.0024) model time 0.3455 (0.3474) loss 3.5711 (3.3266) grad_norm 0.9531 (1.3878) loss_scale 4096.0000 (3824.8798) mem 11219MB [2024-08-26 10:45:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][500/625] eta 0:00:47 lr 0.001501 wd 0.0500 time 0.3443 (0.3815) data time 0.0008 (0.0024) model time 0.3435 (0.3473) loss 3.7811 (3.3257) grad_norm 1.8222 (1.3856) loss_scale 4096.0000 (3830.2914) mem 11219MB [2024-08-26 10:45:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][510/625] eta 0:00:43 lr 0.001500 wd 0.0500 time 0.3465 (0.3808) data time 0.0006 (0.0024) model time 0.3459 (0.3472) loss 3.8662 (3.3293) grad_norm 1.1375 (1.3851) loss_scale 4096.0000 (3835.4912) mem 11219MB [2024-08-26 10:45:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][520/625] eta 0:00:39 lr 0.001500 wd 0.0500 time 0.3442 (0.3802) data time 0.0006 (0.0023) model time 0.3436 (0.3472) loss 3.0312 (3.3292) grad_norm 1.4092 (1.3855) loss_scale 4096.0000 (3840.4914) mem 11219MB [2024-08-26 10:45:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][530/625] eta 0:00:36 lr 0.001500 wd 0.0500 time 0.3434 (0.3796) data time 0.0009 (0.0023) model time 0.3425 (0.3472) loss 3.4568 (3.3229) grad_norm 1.2849 (1.3874) loss_scale 4096.0000 (3845.3032) mem 11219MB [2024-08-26 10:45:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][540/625] eta 0:00:32 lr 0.001500 wd 0.0500 time 0.3407 (0.3794) data time 0.0009 (0.0023) model time 0.3398 (0.3476) loss 3.6240 (3.3227) grad_norm 1.2978 (1.3853) loss_scale 4096.0000 (3849.9372) mem 11219MB [2024-08-26 10:45:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][550/625] eta 0:00:28 lr 0.001500 wd 0.0500 time 0.3437 (0.3787) data time 0.0007 (0.0023) model time 0.3430 (0.3475) loss 3.6397 (3.3240) grad_norm 2.0022 (1.3893) loss_scale 4096.0000 (3854.4029) mem 11219MB [2024-08-26 10:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][560/625] eta 0:00:24 lr 0.001500 wd 0.0500 time 0.3470 (0.3789) data time 0.0008 (0.0022) model time 0.3461 (0.3483) loss 3.5326 (3.3299) grad_norm 1.0897 (1.3920) loss_scale 4096.0000 (3858.7094) mem 11219MB [2024-08-26 10:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][570/625] eta 0:00:20 lr 0.001499 wd 0.0500 time 0.3471 (0.3783) data time 0.0009 (0.0022) model time 0.3462 (0.3482) loss 3.7018 (3.3312) grad_norm 1.1972 (1.3946) loss_scale 4096.0000 (3862.8651) mem 11219MB [2024-08-26 10:45:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][580/625] eta 0:00:16 lr 0.001499 wd 0.0500 time 0.3399 (0.3777) data time 0.0007 (0.0022) model time 0.3392 (0.3481) loss 3.9327 (3.3346) grad_norm 1.7531 (1.4008) loss_scale 4096.0000 (3866.8778) mem 11219MB [2024-08-26 10:45:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][590/625] eta 0:00:13 lr 0.001499 wd 0.0500 time 0.3474 (0.3772) data time 0.0009 (0.0022) model time 0.3465 (0.3481) loss 3.7058 (3.3366) grad_norm 1.7595 (1.4005) loss_scale 4096.0000 (3870.7547) mem 11219MB [2024-08-26 10:45:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][600/625] eta 0:00:09 lr 0.001499 wd 0.0500 time 0.3426 (0.3767) data time 0.0008 (0.0022) model time 0.3417 (0.3480) loss 2.2628 (3.3360) grad_norm 1.4282 (1.3991) loss_scale 4096.0000 (3874.5025) mem 11219MB [2024-08-26 10:45:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][610/625] eta 0:00:05 lr 0.001499 wd 0.0500 time 0.3478 (0.3762) data time 0.0007 (0.0021) model time 0.3472 (0.3480) loss 3.2572 (3.3356) grad_norm 1.0712 (1.3985) loss_scale 4096.0000 (3878.1277) mem 11219MB [2024-08-26 10:45:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [113/300][620/625] eta 0:00:01 lr 0.001499 wd 0.0500 time 0.3363 (0.3756) data time 0.0005 (0.0021) model time 0.3359 (0.3479) loss 4.4023 (3.3361) grad_norm 1.0815 (1.4000) loss_scale 4096.0000 (3881.6361) mem 11219MB [2024-08-26 10:45:50 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 113 training takes 0:03:54 [2024-08-26 10:45:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:45:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:45:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.446 (0.446) Loss 0.6328 (0.6328) Acc@1 86.133 (86.133) Acc@5 97.461 (97.461) Mem 11219MB [2024-08-26 10:45:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.132) Loss 1.0811 (0.7881) Acc@1 74.951 (82.244) Acc@5 93.213 (96.294) Mem 11219MB [2024-08-26 10:45:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.1650 (0.9447) Acc@1 72.168 (78.492) Acc@5 92.529 (94.424) Mem 11219MB [2024-08-26 10:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.353 Acc@5 94.472 [2024-08-26 10:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.4% [2024-08-26 10:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.785 (0.785) Loss 0.5376 (0.5376) Acc@1 88.525 (88.525) Acc@5 98.291 (98.291) Mem 11219MB [2024-08-26 10:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.164) Loss 0.9019 (0.6741) Acc@1 78.955 (84.757) Acc@5 95.312 (97.354) Mem 11219MB [2024-08-26 10:46:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.129) Loss 1.0430 (0.8147) Acc@1 74.316 (81.176) Acc@5 93.652 (95.815) Mem 11219MB [2024-08-26 10:46:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.902 Acc@5 95.771 [2024-08-26 10:46:03 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 80.9% [2024-08-26 10:46:03 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.90% [2024-08-26 10:46:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:46:05 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:46:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][0/625] eta 0:07:40 lr 0.001499 wd 0.0500 time 0.7371 (0.7371) data time 0.3860 (0.3860) model time 0.0000 (0.0000) loss 2.7174 (2.7174) grad_norm 1.5966 (1.5966) loss_scale 4096.0000 (4096.0000) mem 11228MB [2024-08-26 10:46:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][10/625] eta 0:03:54 lr 0.001498 wd 0.0500 time 0.3476 (0.3820) data time 0.0006 (0.0359) model time 0.0000 (0.0000) loss 3.2922 (3.4245) grad_norm 1.5386 (1.3667) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][20/625] eta 0:03:41 lr 0.001498 wd 0.0500 time 0.3474 (0.3654) data time 0.0009 (0.0192) model time 0.0000 (0.0000) loss 3.8985 (3.2792) grad_norm 1.8063 (1.2998) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][30/625] eta 0:03:33 lr 0.001498 wd 0.0500 time 0.3503 (0.3585) data time 0.0009 (0.0133) model time 0.0000 (0.0000) loss 2.8989 (3.3162) grad_norm 1.3918 (1.4216) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][40/625] eta 0:03:28 lr 0.001498 wd 0.0500 time 0.3616 (0.3560) data time 0.0006 (0.0103) model time 0.0000 (0.0000) loss 3.7202 (3.2763) grad_norm 1.2919 (1.3938) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][50/625] eta 0:03:23 lr 0.001498 wd 0.0500 time 0.3445 (0.3548) data time 0.0006 (0.0085) model time 0.0000 (0.0000) loss 2.8520 (3.3488) grad_norm 1.2402 (1.3876) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][60/625] eta 0:03:19 lr 0.001498 wd 0.0500 time 0.3477 (0.3533) data time 0.0007 (0.0072) model time 0.3471 (0.3450) loss 3.2050 (3.3592) grad_norm 1.5360 (1.3683) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][70/625] eta 0:03:16 lr 0.001497 wd 0.0500 time 0.3506 (0.3548) data time 0.0007 (0.0064) model time 0.3499 (0.3538) loss 2.6473 (3.3304) grad_norm 1.1901 (1.3931) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][80/625] eta 0:03:12 lr 0.001497 wd 0.0500 time 0.3406 (0.3537) data time 0.0006 (0.0057) model time 0.3400 (0.3508) loss 3.9269 (3.3407) grad_norm 1.2696 (1.4169) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][90/625] eta 0:03:08 lr 0.001497 wd 0.0500 time 0.3465 (0.3527) data time 0.0008 (0.0052) model time 0.3456 (0.3492) loss 3.2648 (3.3000) grad_norm 1.2361 (1.3928) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][100/625] eta 0:03:04 lr 0.001497 wd 0.0500 time 0.3540 (0.3521) data time 0.0007 (0.0047) model time 0.3533 (0.3484) loss 3.7433 (3.3005) grad_norm 1.3545 (1.4043) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][110/625] eta 0:03:01 lr 0.001497 wd 0.0500 time 0.3466 (0.3517) data time 0.0007 (0.0044) model time 0.3459 (0.3481) loss 3.9753 (3.3157) grad_norm 1.1686 (1.4007) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][120/625] eta 0:02:57 lr 0.001497 wd 0.0500 time 0.3409 (0.3514) data time 0.0007 (0.0041) model time 0.3403 (0.3479) loss 3.6855 (3.3233) grad_norm 1.1101 (1.3996) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][130/625] eta 0:02:56 lr 0.001497 wd 0.0500 time 0.3472 (0.3567) data time 0.0006 (0.0039) model time 0.3466 (0.3570) loss 3.8541 (3.3114) grad_norm 1.0645 (1.3919) loss_scale 4096.0000 (4096.0000) mem 11224MB [2024-08-26 10:46:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][140/625] eta 0:02:52 lr 0.001496 wd 0.0500 time 0.3508 (0.3563) data time 0.0009 (0.0037) model time 0.3499 (0.3563) loss 3.3470 (3.3252) grad_norm 0.9339 (inf) loss_scale 2048.0000 (4066.9504) mem 11224MB [2024-08-26 10:46:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][150/625] eta 0:02:48 lr 0.001496 wd 0.0500 time 0.3475 (0.3556) data time 0.0009 (0.0035) model time 0.3466 (0.3552) loss 3.8455 (3.3339) grad_norm 1.7423 (inf) loss_scale 2048.0000 (3933.2450) mem 11224MB [2024-08-26 10:47:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][160/625] eta 0:02:45 lr 0.001496 wd 0.0500 time 0.3449 (0.3552) data time 0.0008 (0.0033) model time 0.3441 (0.3545) loss 2.9833 (3.3249) grad_norm 1.1302 (inf) loss_scale 2048.0000 (3816.1491) mem 11224MB [2024-08-26 10:47:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][170/625] eta 0:02:41 lr 0.001496 wd 0.0500 time 0.3434 (0.3547) data time 0.0007 (0.0032) model time 0.3427 (0.3537) loss 2.2129 (3.3268) grad_norm 1.4556 (inf) loss_scale 2048.0000 (3712.7485) mem 11224MB [2024-08-26 10:47:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][180/625] eta 0:02:37 lr 0.001496 wd 0.0500 time 0.3462 (0.3543) data time 0.0007 (0.0030) model time 0.3456 (0.3531) loss 1.9853 (3.3161) grad_norm 1.0800 (inf) loss_scale 2048.0000 (3620.7735) mem 11224MB [2024-08-26 10:47:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][190/625] eta 0:02:33 lr 0.001496 wd 0.0500 time 0.3458 (0.3539) data time 0.0007 (0.0029) model time 0.3451 (0.3527) loss 3.8191 (3.3061) grad_norm 1.2507 (inf) loss_scale 2048.0000 (3538.4293) mem 11224MB [2024-08-26 10:47:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][200/625] eta 0:02:30 lr 0.001495 wd 0.0500 time 0.3460 (0.3535) data time 0.0008 (0.0028) model time 0.3451 (0.3521) loss 2.1051 (3.2952) grad_norm 1.6304 (inf) loss_scale 2048.0000 (3464.2786) mem 11224MB [2024-08-26 10:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][210/625] eta 0:02:26 lr 0.001495 wd 0.0500 time 0.3465 (0.3531) data time 0.0007 (0.0027) model time 0.3458 (0.3517) loss 4.4153 (3.2957) grad_norm 1.2286 (inf) loss_scale 2048.0000 (3397.1564) mem 11224MB [2024-08-26 10:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][220/625] eta 0:02:22 lr 0.001495 wd 0.0500 time 0.3496 (0.3528) data time 0.0007 (0.0027) model time 0.3489 (0.3512) loss 4.2713 (3.2997) grad_norm 1.0681 (inf) loss_scale 2048.0000 (3336.1086) mem 11224MB [2024-08-26 10:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][230/625] eta 0:02:19 lr 0.001495 wd 0.0500 time 0.3500 (0.3525) data time 0.0009 (0.0026) model time 0.3491 (0.3509) loss 3.5351 (3.3054) grad_norm 1.3692 (inf) loss_scale 2048.0000 (3280.3463) mem 11224MB [2024-08-26 10:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][240/625] eta 0:02:15 lr 0.001495 wd 0.0500 time 0.3411 (0.3523) data time 0.0008 (0.0025) model time 0.3403 (0.3507) loss 3.9548 (3.3045) grad_norm 1.3942 (inf) loss_scale 2048.0000 (3229.2116) mem 11224MB [2024-08-26 10:47:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][250/625] eta 0:02:12 lr 0.001495 wd 0.0500 time 0.3451 (0.3528) data time 0.0007 (0.0025) model time 0.3444 (0.3513) loss 3.2109 (3.3116) grad_norm 1.3906 (inf) loss_scale 2048.0000 (3182.1514) mem 11224MB [2024-08-26 10:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][260/625] eta 0:02:08 lr 0.001495 wd 0.0500 time 0.3429 (0.3525) data time 0.0007 (0.0024) model time 0.3423 (0.3510) loss 3.9592 (3.3085) grad_norm 1.0604 (inf) loss_scale 2048.0000 (3138.6973) mem 11224MB [2024-08-26 10:47:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][270/625] eta 0:02:05 lr 0.001494 wd 0.0500 time 0.3505 (0.3523) data time 0.0009 (0.0023) model time 0.3496 (0.3508) loss 3.4547 (3.3012) grad_norm 1.2382 (inf) loss_scale 2048.0000 (3098.4502) mem 11224MB [2024-08-26 10:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][280/625] eta 0:02:01 lr 0.001494 wd 0.0500 time 0.3447 (0.3521) data time 0.0009 (0.0023) model time 0.3438 (0.3505) loss 3.6328 (3.3060) grad_norm 1.8579 (inf) loss_scale 2048.0000 (3061.0676) mem 11224MB [2024-08-26 10:47:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][290/625] eta 0:01:57 lr 0.001494 wd 0.0500 time 0.3484 (0.3520) data time 0.0007 (0.0022) model time 0.3478 (0.3505) loss 4.2120 (3.3070) grad_norm 1.1089 (inf) loss_scale 2048.0000 (3026.2543) mem 11224MB [2024-08-26 10:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][300/625] eta 0:01:54 lr 0.001494 wd 0.0500 time 0.3507 (0.3519) data time 0.0009 (0.0022) model time 0.3498 (0.3503) loss 3.2195 (3.3136) grad_norm 1.7600 (inf) loss_scale 2048.0000 (2993.7542) mem 11224MB [2024-08-26 10:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][310/625] eta 0:01:51 lr 0.001494 wd 0.0500 time 0.5502 (0.3536) data time 0.0007 (0.0022) model time 0.5495 (0.3524) loss 2.6521 (3.3138) grad_norm 1.4499 (inf) loss_scale 2048.0000 (2963.3441) mem 11224MB [2024-08-26 10:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][320/625] eta 0:01:48 lr 0.001494 wd 0.0500 time 0.3582 (0.3545) data time 0.0008 (0.0021) model time 0.3575 (0.3535) loss 3.8399 (3.3120) grad_norm 1.6392 (inf) loss_scale 2048.0000 (2934.8287) mem 11224MB [2024-08-26 10:48:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][330/625] eta 0:01:44 lr 0.001493 wd 0.0500 time 0.3437 (0.3542) data time 0.0007 (0.0021) model time 0.3430 (0.3532) loss 3.7509 (3.3014) grad_norm 1.1343 (inf) loss_scale 2048.0000 (2908.0363) mem 11224MB [2024-08-26 10:48:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][340/625] eta 0:01:40 lr 0.001493 wd 0.0500 time 0.3473 (0.3540) data time 0.0009 (0.0021) model time 0.3464 (0.3530) loss 3.9367 (3.3059) grad_norm 1.1758 (inf) loss_scale 2048.0000 (2882.8152) mem 11224MB [2024-08-26 10:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][350/625] eta 0:01:37 lr 0.001493 wd 0.0500 time 0.3449 (0.3538) data time 0.0008 (0.0020) model time 0.3441 (0.3527) loss 3.2320 (3.3136) grad_norm 1.0445 (inf) loss_scale 2048.0000 (2859.0313) mem 11224MB [2024-08-26 10:48:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][360/625] eta 0:01:33 lr 0.001493 wd 0.0500 time 0.3506 (0.3536) data time 0.0008 (0.0020) model time 0.3498 (0.3525) loss 2.1551 (3.3050) grad_norm 2.1374 (inf) loss_scale 2048.0000 (2836.5651) mem 11224MB [2024-08-26 10:48:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][370/625] eta 0:01:30 lr 0.001493 wd 0.0500 time 0.3458 (0.3538) data time 0.0008 (0.0020) model time 0.3449 (0.3527) loss 3.4636 (3.3076) grad_norm 1.7708 (inf) loss_scale 2048.0000 (2815.3100) mem 11224MB [2024-08-26 10:48:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][380/625] eta 0:01:26 lr 0.001493 wd 0.0500 time 0.3720 (0.3537) data time 0.0009 (0.0019) model time 0.3711 (0.3526) loss 3.3045 (3.3013) grad_norm 1.1986 (inf) loss_scale 2048.0000 (2795.1706) mem 11224MB [2024-08-26 10:48:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][390/625] eta 0:01:23 lr 0.001493 wd 0.0500 time 0.3455 (0.3535) data time 0.0006 (0.0019) model time 0.3449 (0.3524) loss 2.5573 (3.3115) grad_norm 1.6607 (inf) loss_scale 2048.0000 (2776.0614) mem 11224MB [2024-08-26 10:48:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][400/625] eta 0:01:19 lr 0.001492 wd 0.0500 time 0.3471 (0.3533) data time 0.0009 (0.0019) model time 0.3462 (0.3522) loss 3.3869 (3.3035) grad_norm 1.3210 (inf) loss_scale 2048.0000 (2757.9052) mem 11224MB [2024-08-26 10:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][410/625] eta 0:01:15 lr 0.001492 wd 0.0500 time 0.3508 (0.3532) data time 0.0007 (0.0019) model time 0.3501 (0.3521) loss 3.9933 (3.3061) grad_norm 1.5837 (inf) loss_scale 2048.0000 (2740.6326) mem 11224MB [2024-08-26 10:48:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][420/625] eta 0:01:12 lr 0.001492 wd 0.0500 time 0.3469 (0.3532) data time 0.0008 (0.0018) model time 0.3461 (0.3520) loss 3.6379 (3.3116) grad_norm 1.9733 (inf) loss_scale 2048.0000 (2724.1805) mem 11224MB [2024-08-26 10:48:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][430/625] eta 0:01:09 lr 0.001492 wd 0.0500 time 0.3514 (0.3541) data time 0.0009 (0.0018) model time 0.3505 (0.3530) loss 3.4797 (3.3144) grad_norm 1.3146 (inf) loss_scale 2048.0000 (2708.4919) mem 11224MB [2024-08-26 10:48:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][440/625] eta 0:01:05 lr 0.001492 wd 0.0500 time 0.3567 (0.3544) data time 0.0006 (0.0018) model time 0.3560 (0.3534) loss 2.7643 (3.3163) grad_norm 1.0518 (inf) loss_scale 2048.0000 (2693.5147) mem 11224MB [2024-08-26 10:48:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][450/625] eta 0:01:01 lr 0.001492 wd 0.0500 time 0.3465 (0.3542) data time 0.0009 (0.0018) model time 0.3457 (0.3532) loss 2.8416 (3.3101) grad_norm 1.3092 (inf) loss_scale 2048.0000 (2679.2018) mem 11224MB [2024-08-26 10:48:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][460/625] eta 0:00:58 lr 0.001491 wd 0.0500 time 0.3400 (0.3540) data time 0.0007 (0.0018) model time 0.3393 (0.3530) loss 3.6049 (3.3106) grad_norm 1.2670 (inf) loss_scale 2048.0000 (2665.5098) mem 11224MB [2024-08-26 10:48:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][470/625] eta 0:00:54 lr 0.001491 wd 0.0500 time 0.3463 (0.3539) data time 0.0006 (0.0018) model time 0.3456 (0.3528) loss 4.4225 (3.3109) grad_norm 1.1894 (inf) loss_scale 2048.0000 (2652.3992) mem 11224MB [2024-08-26 10:48:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][480/625] eta 0:00:51 lr 0.001491 wd 0.0500 time 0.3470 (0.3538) data time 0.0008 (0.0018) model time 0.3462 (0.3527) loss 3.5496 (3.3139) grad_norm 1.4487 (inf) loss_scale 2048.0000 (2639.8337) mem 11224MB [2024-08-26 10:48:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][490/625] eta 0:00:47 lr 0.001491 wd 0.0500 time 0.3468 (0.3537) data time 0.0009 (0.0017) model time 0.3459 (0.3526) loss 3.8046 (3.3201) grad_norm 1.2984 (inf) loss_scale 2048.0000 (2627.7800) mem 11224MB [2024-08-26 10:49:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][500/625] eta 0:00:44 lr 0.001491 wd 0.0500 time 0.3454 (0.3536) data time 0.0008 (0.0017) model time 0.3446 (0.3525) loss 3.2653 (3.3213) grad_norm 1.6229 (inf) loss_scale 2048.0000 (2616.2076) mem 11224MB [2024-08-26 10:49:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][510/625] eta 0:00:40 lr 0.001491 wd 0.0500 time 0.3369 (0.3534) data time 0.0009 (0.0017) model time 0.3360 (0.3523) loss 3.6962 (3.3215) grad_norm 2.0553 (inf) loss_scale 2048.0000 (2605.0881) mem 11224MB [2024-08-26 10:49:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][520/625] eta 0:00:37 lr 0.001491 wd 0.0500 time 0.3413 (0.3533) data time 0.0006 (0.0017) model time 0.3407 (0.3522) loss 2.9020 (3.3220) grad_norm 1.0643 (inf) loss_scale 2048.0000 (2594.3954) mem 11224MB [2024-08-26 10:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][530/625] eta 0:00:33 lr 0.001490 wd 0.0500 time 0.3442 (0.3532) data time 0.0007 (0.0017) model time 0.3435 (0.3521) loss 3.6469 (3.3225) grad_norm 1.1644 (inf) loss_scale 2048.0000 (2584.1055) mem 11224MB [2024-08-26 10:49:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][540/625] eta 0:00:30 lr 0.001490 wd 0.0500 time 0.3540 (0.3532) data time 0.0007 (0.0017) model time 0.3533 (0.3520) loss 4.3522 (3.3192) grad_norm 1.3199 (inf) loss_scale 2048.0000 (2574.1959) mem 11224MB [2024-08-26 10:49:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][550/625] eta 0:00:26 lr 0.001490 wd 0.0500 time 0.3463 (0.3530) data time 0.0009 (0.0016) model time 0.3453 (0.3519) loss 4.1663 (3.3268) grad_norm 1.1356 (inf) loss_scale 2048.0000 (2564.6461) mem 11224MB [2024-08-26 10:49:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][560/625] eta 0:00:22 lr 0.001490 wd 0.0500 time 0.3473 (0.3530) data time 0.0009 (0.0016) model time 0.3464 (0.3518) loss 2.9270 (3.3295) grad_norm 1.2660 (inf) loss_scale 2048.0000 (2555.4367) mem 11224MB [2024-08-26 10:49:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][570/625] eta 0:00:19 lr 0.001490 wd 0.0500 time 0.3507 (0.3529) data time 0.0008 (0.0016) model time 0.3499 (0.3517) loss 3.2889 (3.3285) grad_norm 1.6871 (inf) loss_scale 2048.0000 (2546.5499) mem 11224MB [2024-08-26 10:49:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][580/625] eta 0:00:15 lr 0.001490 wd 0.0500 time 0.3467 (0.3531) data time 0.0006 (0.0016) model time 0.3461 (0.3520) loss 3.3231 (3.3333) grad_norm 0.9885 (inf) loss_scale 2048.0000 (2537.9690) mem 11224MB [2024-08-26 10:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][590/625] eta 0:00:12 lr 0.001489 wd 0.0500 time 0.3475 (0.3530) data time 0.0006 (0.0016) model time 0.3469 (0.3519) loss 2.5636 (3.3341) grad_norm 1.3574 (inf) loss_scale 2048.0000 (2529.6785) mem 11224MB [2024-08-26 10:49:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][600/625] eta 0:00:08 lr 0.001489 wd 0.0500 time 0.3490 (0.3529) data time 0.0006 (0.0016) model time 0.3484 (0.3518) loss 2.8290 (3.3323) grad_norm 1.3982 (inf) loss_scale 2048.0000 (2521.6639) mem 11224MB [2024-08-26 10:49:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][610/625] eta 0:00:05 lr 0.001489 wd 0.0500 time 0.3369 (0.3528) data time 0.0004 (0.0016) model time 0.3365 (0.3516) loss 2.8561 (3.3297) grad_norm 1.3541 (inf) loss_scale 2048.0000 (2513.9116) mem 11224MB [2024-08-26 10:49:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [114/300][620/625] eta 0:00:01 lr 0.001489 wd 0.0500 time 0.3430 (0.3526) data time 0.0006 (0.0016) model time 0.3423 (0.3514) loss 3.3888 (3.3321) grad_norm 1.2206 (inf) loss_scale 2048.0000 (2506.4090) mem 11224MB [2024-08-26 10:49:46 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 114 training takes 0:03:40 [2024-08-26 10:49:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:49:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.448 (0.448) Loss 0.6245 (0.6245) Acc@1 86.035 (86.035) Acc@5 97.803 (97.803) Mem 11224MB [2024-08-26 10:49:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.129) Loss 1.0273 (0.7573) Acc@1 76.611 (82.750) Acc@5 93.848 (96.400) Mem 11224MB [2024-08-26 10:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 1.1250 (0.9169) Acc@1 72.949 (78.813) Acc@5 92.432 (94.675) Mem 11224MB [2024-08-26 10:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.561 Acc@5 94.644 [2024-08-26 10:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.6% [2024-08-26 10:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.817 (0.817) Loss 0.5361 (0.5361) Acc@1 88.477 (88.477) Acc@5 98.291 (98.291) Mem 11224MB [2024-08-26 10:49:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.165) Loss 0.9004 (0.6727) Acc@1 78.857 (84.770) Acc@5 95.459 (97.350) Mem 11224MB [2024-08-26 10:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.130) Loss 1.0410 (0.8130) Acc@1 74.609 (81.257) Acc@5 93.555 (95.801) Mem 11224MB [2024-08-26 10:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.974 Acc@5 95.771 [2024-08-26 10:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.0% [2024-08-26 10:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.97% [2024-08-26 10:49:52 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:49:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][0/625] eta 0:06:49 lr 0.001489 wd 0.0500 time 0.6544 (0.6544) data time 0.3353 (0.3353) model time 0.0000 (0.0000) loss 3.5817 (3.5817) grad_norm 1.1518 (1.1518) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:49:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][10/625] eta 0:04:23 lr 0.001489 wd 0.0500 time 0.5198 (0.4280) data time 0.0010 (0.0314) model time 0.0000 (0.0000) loss 3.1588 (3.4356) grad_norm 1.3795 (1.4057) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][20/625] eta 0:03:55 lr 0.001489 wd 0.0500 time 0.3522 (0.3886) data time 0.0006 (0.0169) model time 0.0000 (0.0000) loss 2.7443 (3.4476) grad_norm 1.4506 (1.3558) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][30/625] eta 0:03:42 lr 0.001488 wd 0.0500 time 0.3461 (0.3745) data time 0.0006 (0.0117) model time 0.0000 (0.0000) loss 4.1061 (3.5092) grad_norm 1.5181 (1.3487) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][40/625] eta 0:03:35 lr 0.001488 wd 0.0500 time 0.3471 (0.3679) data time 0.0006 (0.0091) model time 0.0000 (0.0000) loss 3.7089 (3.4772) grad_norm 1.0195 (1.3330) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][50/625] eta 0:03:29 lr 0.001488 wd 0.0500 time 0.3478 (0.3635) data time 0.0006 (0.0075) model time 0.0000 (0.0000) loss 3.7016 (3.4325) grad_norm 1.4749 (1.3705) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][60/625] eta 0:03:23 lr 0.001488 wd 0.0500 time 0.3412 (0.3607) data time 0.0006 (0.0064) model time 0.3406 (0.3453) loss 2.2703 (3.3906) grad_norm 1.1584 (1.3482) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][70/625] eta 0:03:19 lr 0.001488 wd 0.0500 time 0.3469 (0.3587) data time 0.0009 (0.0056) model time 0.3460 (0.3455) loss 2.2909 (3.3412) grad_norm 1.1965 (1.3327) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][80/625] eta 0:03:16 lr 0.001488 wd 0.0500 time 0.3171 (0.3597) data time 0.0007 (0.0050) model time 0.3164 (0.3523) loss 3.8344 (3.3501) grad_norm 2.0430 (1.3256) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][90/625] eta 0:03:11 lr 0.001487 wd 0.0500 time 0.3502 (0.3583) data time 0.0006 (0.0046) model time 0.3496 (0.3508) loss 4.3726 (3.3658) grad_norm 1.2244 (1.3353) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][100/625] eta 0:03:07 lr 0.001487 wd 0.0500 time 0.3488 (0.3573) data time 0.0009 (0.0042) model time 0.3480 (0.3500) loss 3.2049 (3.3670) grad_norm 1.2529 (1.3414) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][110/625] eta 0:03:03 lr 0.001487 wd 0.0500 time 0.3464 (0.3571) data time 0.0010 (0.0039) model time 0.3454 (0.3508) loss 3.3398 (3.3827) grad_norm 1.0270 (1.3431) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][120/625] eta 0:02:59 lr 0.001487 wd 0.0500 time 0.3443 (0.3562) data time 0.0009 (0.0037) model time 0.3435 (0.3500) loss 3.4768 (3.3867) grad_norm 1.7664 (1.3573) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][130/625] eta 0:02:56 lr 0.001487 wd 0.0500 time 0.3476 (0.3556) data time 0.0009 (0.0035) model time 0.3467 (0.3497) loss 3.6296 (3.3924) grad_norm 1.3340 (1.3717) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][140/625] eta 0:02:52 lr 0.001487 wd 0.0500 time 0.3605 (0.3553) data time 0.0007 (0.0033) model time 0.3599 (0.3496) loss 3.4456 (3.3989) grad_norm 1.4128 (1.3687) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][150/625] eta 0:02:48 lr 0.001487 wd 0.0500 time 0.3483 (0.3548) data time 0.0009 (0.0032) model time 0.3473 (0.3494) loss 3.2471 (3.3946) grad_norm 1.2859 (1.3588) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][160/625] eta 0:02:44 lr 0.001486 wd 0.0500 time 0.3466 (0.3545) data time 0.0007 (0.0030) model time 0.3459 (0.3493) loss 3.5636 (3.4106) grad_norm 1.4816 (1.3880) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][170/625] eta 0:02:41 lr 0.001486 wd 0.0500 time 0.3534 (0.3541) data time 0.0009 (0.0029) model time 0.3525 (0.3492) loss 3.2781 (3.4092) grad_norm 1.8995 (1.3970) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:50:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][180/625] eta 0:02:37 lr 0.001486 wd 0.0500 time 0.3487 (0.3538) data time 0.0007 (0.0028) model time 0.3480 (0.3491) loss 4.0233 (3.4050) grad_norm 1.0345 (1.3978) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][190/625] eta 0:02:33 lr 0.001486 wd 0.0500 time 0.3500 (0.3535) data time 0.0007 (0.0027) model time 0.3493 (0.3489) loss 3.1002 (3.3981) grad_norm 1.3964 (1.3873) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][200/625] eta 0:02:30 lr 0.001486 wd 0.0500 time 0.3438 (0.3541) data time 0.0006 (0.0026) model time 0.3432 (0.3500) loss 3.5493 (3.3991) grad_norm 2.5157 (1.3964) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][210/625] eta 0:02:26 lr 0.001486 wd 0.0500 time 0.3472 (0.3538) data time 0.0009 (0.0025) model time 0.3462 (0.3498) loss 3.3537 (3.4048) grad_norm 1.7130 (1.4055) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][220/625] eta 0:02:24 lr 0.001485 wd 0.0500 time 0.5669 (0.3565) data time 0.0007 (0.0025) model time 0.5662 (0.3534) loss 2.4224 (3.3926) grad_norm 1.5138 (1.3995) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][230/625] eta 0:02:20 lr 0.001485 wd 0.0500 time 0.3484 (0.3561) data time 0.0007 (0.0024) model time 0.3478 (0.3531) loss 3.7606 (3.3799) grad_norm 1.3249 (1.3925) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][240/625] eta 0:02:16 lr 0.001485 wd 0.0500 time 0.3461 (0.3557) data time 0.0009 (0.0023) model time 0.3452 (0.3527) loss 2.9212 (3.3580) grad_norm 1.0251 (1.3995) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][250/625] eta 0:02:13 lr 0.001485 wd 0.0500 time 0.3494 (0.3553) data time 0.0009 (0.0023) model time 0.3485 (0.3523) loss 3.0468 (3.3596) grad_norm 1.1251 (1.3973) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][260/625] eta 0:02:09 lr 0.001485 wd 0.0500 time 0.3521 (0.3551) data time 0.0009 (0.0022) model time 0.3513 (0.3521) loss 3.4800 (3.3604) grad_norm 1.1510 (1.3895) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][270/625] eta 0:02:05 lr 0.001485 wd 0.0500 time 0.3482 (0.3548) data time 0.0007 (0.0022) model time 0.3475 (0.3519) loss 3.7160 (3.3646) grad_norm 1.0783 (1.3822) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][280/625] eta 0:02:02 lr 0.001485 wd 0.0500 time 0.3626 (0.3547) data time 0.0007 (0.0021) model time 0.3620 (0.3518) loss 2.0871 (3.3619) grad_norm 1.3411 (1.3754) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][290/625] eta 0:01:58 lr 0.001484 wd 0.0500 time 0.3491 (0.3546) data time 0.0006 (0.0021) model time 0.3485 (0.3517) loss 2.5071 (3.3617) grad_norm 1.9807 (1.3802) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][300/625] eta 0:01:55 lr 0.001484 wd 0.0500 time 0.3553 (0.3544) data time 0.0009 (0.0021) model time 0.3544 (0.3515) loss 3.1154 (3.3546) grad_norm 1.6069 (1.3955) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][310/625] eta 0:01:51 lr 0.001484 wd 0.0500 time 0.3637 (0.3543) data time 0.0007 (0.0021) model time 0.3630 (0.3515) loss 2.0012 (3.3515) grad_norm 3.2208 (1.3989) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][320/625] eta 0:01:47 lr 0.001484 wd 0.0500 time 0.3436 (0.3541) data time 0.0006 (0.0020) model time 0.3430 (0.3513) loss 3.3923 (3.3400) grad_norm 1.5407 (1.3967) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][330/625] eta 0:01:44 lr 0.001484 wd 0.0500 time 0.3464 (0.3539) data time 0.0006 (0.0020) model time 0.3458 (0.3511) loss 2.4538 (3.3375) grad_norm 1.6461 (1.3931) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][340/625] eta 0:01:40 lr 0.001484 wd 0.0500 time 0.3431 (0.3537) data time 0.0008 (0.0020) model time 0.3423 (0.3510) loss 3.3565 (3.3434) grad_norm 1.2880 (1.3891) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:51:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][350/625] eta 0:01:37 lr 0.001483 wd 0.0500 time 0.3476 (0.3537) data time 0.0006 (0.0020) model time 0.3470 (0.3510) loss 4.1439 (3.3512) grad_norm 1.1404 (1.3924) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][360/625] eta 0:01:33 lr 0.001483 wd 0.0500 time 0.3501 (0.3536) data time 0.0007 (0.0019) model time 0.3493 (0.3509) loss 2.9438 (3.3515) grad_norm 1.2845 (1.3917) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][370/625] eta 0:01:30 lr 0.001483 wd 0.0500 time 0.3452 (0.3535) data time 0.0008 (0.0019) model time 0.3444 (0.3508) loss 3.0585 (3.3527) grad_norm 0.8586 (1.3860) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][380/625] eta 0:01:26 lr 0.001483 wd 0.0500 time 0.3472 (0.3533) data time 0.0009 (0.0019) model time 0.3464 (0.3507) loss 3.6247 (3.3526) grad_norm 1.8483 (1.3878) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][390/625] eta 0:01:23 lr 0.001483 wd 0.0500 time 0.3495 (0.3532) data time 0.0007 (0.0019) model time 0.3488 (0.3506) loss 3.5557 (3.3544) grad_norm 1.3245 (1.3861) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][400/625] eta 0:01:19 lr 0.001483 wd 0.0500 time 0.5592 (0.3541) data time 0.0008 (0.0018) model time 0.5584 (0.3517) loss 2.2566 (3.3437) grad_norm 1.7858 (1.3814) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][410/625] eta 0:01:16 lr 0.001482 wd 0.0500 time 0.3425 (0.3557) data time 0.0009 (0.0018) model time 0.3416 (0.3536) loss 2.8096 (3.3340) grad_norm 1.4581 (1.3900) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][420/625] eta 0:01:12 lr 0.001482 wd 0.0500 time 0.3469 (0.3555) data time 0.0008 (0.0018) model time 0.3460 (0.3534) loss 3.5163 (3.3356) grad_norm 1.1734 (1.3869) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][430/625] eta 0:01:09 lr 0.001482 wd 0.0500 time 0.3440 (0.3553) data time 0.0009 (0.0018) model time 0.3431 (0.3532) loss 3.7270 (3.3356) grad_norm 3.1697 (1.3868) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][440/625] eta 0:01:05 lr 0.001482 wd 0.0500 time 0.3533 (0.3551) data time 0.0007 (0.0017) model time 0.3525 (0.3530) loss 3.5165 (3.3364) grad_norm 1.1733 (1.3869) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][450/625] eta 0:01:02 lr 0.001482 wd 0.0500 time 0.3518 (0.3550) data time 0.0006 (0.0017) model time 0.3512 (0.3530) loss 4.3254 (3.3458) grad_norm 1.0858 (1.3854) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][460/625] eta 0:00:58 lr 0.001482 wd 0.0500 time 0.3480 (0.3549) data time 0.0007 (0.0017) model time 0.3473 (0.3529) loss 3.0833 (3.3466) grad_norm 1.1945 (1.3952) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][470/625] eta 0:00:54 lr 0.001482 wd 0.0500 time 0.3486 (0.3548) data time 0.0007 (0.0017) model time 0.3479 (0.3527) loss 2.4819 (3.3451) grad_norm 1.8138 (1.3921) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][480/625] eta 0:00:51 lr 0.001481 wd 0.0500 time 0.3490 (0.3546) data time 0.0009 (0.0017) model time 0.3481 (0.3526) loss 3.6293 (3.3476) grad_norm 1.6159 (1.3912) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][490/625] eta 0:00:47 lr 0.001481 wd 0.0500 time 0.3505 (0.3544) data time 0.0009 (0.0017) model time 0.3496 (0.3524) loss 2.5875 (3.3469) grad_norm 1.2139 (1.3858) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][500/625] eta 0:00:44 lr 0.001481 wd 0.0500 time 0.3498 (0.3543) data time 0.0009 (0.0016) model time 0.3489 (0.3523) loss 2.6681 (3.3426) grad_norm 1.2073 (1.3865) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][510/625] eta 0:00:40 lr 0.001481 wd 0.0500 time 0.3482 (0.3543) data time 0.0007 (0.0016) model time 0.3475 (0.3523) loss 3.8737 (3.3429) grad_norm 2.3797 (1.3975) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:52:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][520/625] eta 0:00:37 lr 0.001481 wd 0.0500 time 0.3497 (0.3549) data time 0.0009 (0.0016) model time 0.3488 (0.3530) loss 3.3548 (3.3467) grad_norm 1.4677 (1.4016) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][530/625] eta 0:00:33 lr 0.001481 wd 0.0500 time 0.3443 (0.3555) data time 0.0007 (0.0016) model time 0.3436 (0.3537) loss 3.9114 (3.3484) grad_norm 1.4701 (1.3968) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][540/625] eta 0:00:30 lr 0.001480 wd 0.0500 time 0.3507 (0.3554) data time 0.0007 (0.0016) model time 0.3500 (0.3535) loss 3.6794 (3.3464) grad_norm 1.4545 (1.3960) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][550/625] eta 0:00:26 lr 0.001480 wd 0.0500 time 0.3414 (0.3552) data time 0.0008 (0.0016) model time 0.3406 (0.3533) loss 3.1481 (3.3506) grad_norm 0.9120 (1.3937) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][560/625] eta 0:00:23 lr 0.001480 wd 0.0500 time 0.3472 (0.3550) data time 0.0006 (0.0016) model time 0.3465 (0.3531) loss 2.5911 (3.3467) grad_norm 1.4226 (1.3906) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][570/625] eta 0:00:19 lr 0.001480 wd 0.0500 time 0.3497 (0.3549) data time 0.0008 (0.0016) model time 0.3490 (0.3530) loss 3.6133 (3.3494) grad_norm 1.0480 (1.3866) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][580/625] eta 0:00:15 lr 0.001480 wd 0.0500 time 0.3486 (0.3547) data time 0.0009 (0.0016) model time 0.3477 (0.3529) loss 3.0891 (3.3508) grad_norm 1.4428 (1.3890) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][590/625] eta 0:00:12 lr 0.001480 wd 0.0500 time 0.5455 (0.3550) data time 0.0006 (0.0016) model time 0.5449 (0.3532) loss 2.1557 (3.3491) grad_norm 2.3967 (1.3919) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][600/625] eta 0:00:08 lr 0.001480 wd 0.0500 time 0.3539 (0.3548) data time 0.0007 (0.0016) model time 0.3532 (0.3530) loss 2.6141 (3.3441) grad_norm 0.9980 (1.3959) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][610/625] eta 0:00:05 lr 0.001479 wd 0.0500 time 0.3422 (0.3547) data time 0.0005 (0.0016) model time 0.3417 (0.3529) loss 3.5402 (3.3441) grad_norm 1.1134 (1.3967) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [115/300][620/625] eta 0:00:01 lr 0.001479 wd 0.0500 time 0.3359 (0.3545) data time 0.0006 (0.0015) model time 0.3353 (0.3527) loss 3.4026 (3.3397) grad_norm 1.8191 (1.3980) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:35 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 115 training takes 0:03:41 [2024-08-26 10:53:35 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:53:36 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.451 (0.451) Loss 0.6743 (0.6743) Acc@1 86.230 (86.230) Acc@5 97.559 (97.559) Mem 11224MB [2024-08-26 10:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.130) Loss 1.0859 (0.8154) Acc@1 74.951 (82.435) Acc@5 93.994 (96.555) Mem 11224MB [2024-08-26 10:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 1.2148 (0.9734) Acc@1 72.754 (78.688) Acc@5 91.357 (94.466) Mem 11224MB [2024-08-26 10:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.423 Acc@5 94.520 [2024-08-26 10:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.4% [2024-08-26 10:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.792 (0.792) Loss 0.5356 (0.5356) Acc@1 88.379 (88.379) Acc@5 98.340 (98.340) Mem 11224MB [2024-08-26 10:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.164) Loss 0.8994 (0.6720) Acc@1 78.955 (84.775) Acc@5 95.410 (97.332) Mem 11224MB [2024-08-26 10:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.130) Loss 1.0400 (0.8118) Acc@1 74.365 (81.248) Acc@5 93.652 (95.810) Mem 11224MB [2024-08-26 10:53:42 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.964 Acc@5 95.783 [2024-08-26 10:53:42 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.0% [2024-08-26 10:53:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][0/625] eta 0:12:46 lr 0.001479 wd 0.0500 time 1.2271 (1.2271) data time 0.5641 (0.5641) model time 0.0000 (0.0000) loss 3.9459 (3.9459) grad_norm 1.3234 (1.3234) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][10/625] eta 0:04:23 lr 0.001479 wd 0.0500 time 0.3498 (0.4287) data time 0.0007 (0.0521) model time 0.0000 (0.0000) loss 3.6183 (3.5227) grad_norm 1.4130 (1.3191) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][20/625] eta 0:03:56 lr 0.001479 wd 0.0500 time 0.3455 (0.3908) data time 0.0009 (0.0277) model time 0.0000 (0.0000) loss 3.6688 (3.3038) grad_norm 1.1140 (1.3171) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][30/625] eta 0:03:44 lr 0.001479 wd 0.0500 time 0.3548 (0.3776) data time 0.0008 (0.0191) model time 0.0000 (0.0000) loss 3.7540 (3.3250) grad_norm 1.3309 (1.3187) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][40/625] eta 0:03:37 lr 0.001479 wd 0.0500 time 0.3481 (0.3711) data time 0.0006 (0.0146) model time 0.0000 (0.0000) loss 3.1468 (3.2250) grad_norm 1.4542 (1.3356) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][50/625] eta 0:03:30 lr 0.001478 wd 0.0500 time 0.3452 (0.3667) data time 0.0007 (0.0119) model time 0.0000 (0.0000) loss 2.3078 (3.2021) grad_norm 1.3890 (1.3707) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][60/625] eta 0:03:29 lr 0.001478 wd 0.0500 time 0.5564 (0.3710) data time 0.0007 (0.0101) model time 0.5557 (0.3924) loss 2.6267 (3.2060) grad_norm 1.6357 (1.3906) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][70/625] eta 0:03:26 lr 0.001478 wd 0.0500 time 0.3441 (0.3727) data time 0.0006 (0.0088) model time 0.3436 (0.3871) loss 2.7639 (3.2126) grad_norm 0.8967 (1.3620) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][80/625] eta 0:03:21 lr 0.001478 wd 0.0500 time 0.3437 (0.3694) data time 0.0006 (0.0078) model time 0.3431 (0.3731) loss 3.9701 (3.1880) grad_norm 1.1593 (1.3522) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][90/625] eta 0:03:16 lr 0.001478 wd 0.0500 time 0.3434 (0.3672) data time 0.0009 (0.0071) model time 0.3425 (0.3670) loss 2.2449 (3.1872) grad_norm 1.1783 (1.3481) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][100/625] eta 0:03:11 lr 0.001478 wd 0.0500 time 0.3476 (0.3649) data time 0.0009 (0.0065) model time 0.3467 (0.3622) loss 3.6653 (3.1935) grad_norm 1.3470 (1.3326) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][110/625] eta 0:03:07 lr 0.001477 wd 0.0500 time 0.3396 (0.3633) data time 0.0010 (0.0060) model time 0.3386 (0.3596) loss 3.0634 (3.1818) grad_norm 2.1693 (1.3344) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][120/625] eta 0:03:02 lr 0.001477 wd 0.0500 time 0.3472 (0.3620) data time 0.0006 (0.0056) model time 0.3466 (0.3578) loss 3.8180 (3.1868) grad_norm 1.1640 (1.3507) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][130/625] eta 0:02:58 lr 0.001477 wd 0.0500 time 0.3486 (0.3608) data time 0.0007 (0.0052) model time 0.3479 (0.3562) loss 4.0499 (3.2098) grad_norm 3.7243 (1.3748) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][140/625] eta 0:02:54 lr 0.001477 wd 0.0500 time 0.3463 (0.3596) data time 0.0007 (0.0049) model time 0.3456 (0.3547) loss 3.2678 (3.2216) grad_norm 1.2174 (1.3999) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][150/625] eta 0:02:50 lr 0.001477 wd 0.0500 time 0.3457 (0.3588) data time 0.0009 (0.0046) model time 0.3447 (0.3539) loss 3.6943 (3.2281) grad_norm 2.1093 (1.3902) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][160/625] eta 0:02:46 lr 0.001477 wd 0.0500 time 0.3468 (0.3581) data time 0.0009 (0.0044) model time 0.3458 (0.3532) loss 3.5317 (3.2409) grad_norm 1.3970 (1.3914) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][170/625] eta 0:02:42 lr 0.001476 wd 0.0500 time 0.3390 (0.3575) data time 0.0007 (0.0042) model time 0.3383 (0.3527) loss 3.1813 (3.2518) grad_norm 1.0989 (1.3860) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][180/625] eta 0:02:38 lr 0.001476 wd 0.0500 time 0.3458 (0.3568) data time 0.0006 (0.0040) model time 0.3452 (0.3520) loss 3.0288 (3.2569) grad_norm 1.2460 (1.3804) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][190/625] eta 0:02:34 lr 0.001476 wd 0.0500 time 0.3449 (0.3562) data time 0.0007 (0.0039) model time 0.3442 (0.3514) loss 3.2262 (3.2529) grad_norm 1.4568 (1.3777) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][200/625] eta 0:02:31 lr 0.001476 wd 0.0500 time 0.3441 (0.3557) data time 0.0006 (0.0037) model time 0.3435 (0.3510) loss 3.8088 (3.2648) grad_norm 1.3713 (1.3722) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][210/625] eta 0:02:27 lr 0.001476 wd 0.0500 time 0.3430 (0.3552) data time 0.0007 (0.0036) model time 0.3422 (0.3507) loss 3.8147 (3.2719) grad_norm 1.2557 (1.3711) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][220/625] eta 0:02:23 lr 0.001476 wd 0.0500 time 0.3458 (0.3547) data time 0.0006 (0.0034) model time 0.3452 (0.3502) loss 4.2689 (3.2822) grad_norm 1.7169 (1.3781) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][230/625] eta 0:02:19 lr 0.001476 wd 0.0500 time 0.3488 (0.3543) data time 0.0007 (0.0033) model time 0.3481 (0.3498) loss 3.8168 (3.2827) grad_norm 1.1009 (1.3693) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][240/625] eta 0:02:16 lr 0.001475 wd 0.0500 time 0.3437 (0.3538) data time 0.0007 (0.0032) model time 0.3430 (0.3495) loss 4.1112 (3.2840) grad_norm 1.2556 (1.3697) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][250/625] eta 0:02:12 lr 0.001475 wd 0.0500 time 0.3422 (0.3534) data time 0.0009 (0.0031) model time 0.3413 (0.3491) loss 3.5222 (3.2838) grad_norm 1.1486 (1.3640) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][260/625] eta 0:02:08 lr 0.001475 wd 0.0500 time 0.3448 (0.3531) data time 0.0009 (0.0031) model time 0.3439 (0.3489) loss 3.5180 (3.2947) grad_norm 1.2360 (1.3585) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][270/625] eta 0:02:05 lr 0.001475 wd 0.0500 time 0.5848 (0.3547) data time 0.0009 (0.0033) model time 0.5839 (0.3507) loss 3.4655 (3.3012) grad_norm 1.1026 (1.3635) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][280/625] eta 0:02:02 lr 0.001475 wd 0.0500 time 0.3455 (0.3552) data time 0.0008 (0.0033) model time 0.3446 (0.3512) loss 3.4262 (3.3025) grad_norm 1.6705 (1.3673) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][290/625] eta 0:01:58 lr 0.001475 wd 0.0500 time 0.3437 (0.3548) data time 0.0006 (0.0032) model time 0.3431 (0.3509) loss 3.8838 (3.3079) grad_norm 1.1936 (1.3645) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][300/625] eta 0:01:55 lr 0.001474 wd 0.0500 time 0.3464 (0.3545) data time 0.0009 (0.0032) model time 0.3455 (0.3507) loss 3.6290 (3.3061) grad_norm 1.1773 (1.3749) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][310/625] eta 0:01:51 lr 0.001474 wd 0.0500 time 0.3518 (0.3543) data time 0.0007 (0.0031) model time 0.3510 (0.3505) loss 4.4652 (3.3076) grad_norm 1.8687 (1.3873) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][320/625] eta 0:01:47 lr 0.001474 wd 0.0500 time 0.3426 (0.3540) data time 0.0006 (0.0030) model time 0.3420 (0.3503) loss 1.9354 (3.3076) grad_norm 1.3054 (1.3858) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][330/625] eta 0:01:44 lr 0.001474 wd 0.0500 time 0.3459 (0.3537) data time 0.0009 (0.0030) model time 0.3450 (0.3500) loss 2.8539 (3.3045) grad_norm 1.0416 (1.3845) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][340/625] eta 0:01:40 lr 0.001474 wd 0.0500 time 0.3438 (0.3535) data time 0.0008 (0.0029) model time 0.3430 (0.3499) loss 2.9188 (3.3051) grad_norm 1.5577 (1.3834) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][350/625] eta 0:01:37 lr 0.001474 wd 0.0500 time 0.3416 (0.3533) data time 0.0007 (0.0028) model time 0.3409 (0.3497) loss 4.2760 (3.3002) grad_norm 1.0360 (1.3847) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][360/625] eta 0:01:33 lr 0.001473 wd 0.0500 time 0.3461 (0.3530) data time 0.0008 (0.0028) model time 0.3453 (0.3495) loss 3.6978 (3.2983) grad_norm 2.7762 (1.3878) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][370/625] eta 0:01:29 lr 0.001473 wd 0.0500 time 0.3462 (0.3528) data time 0.0007 (0.0027) model time 0.3455 (0.3493) loss 3.4793 (3.2968) grad_norm 1.0683 (1.3965) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][380/625] eta 0:01:26 lr 0.001473 wd 0.0500 time 0.3480 (0.3527) data time 0.0007 (0.0027) model time 0.3473 (0.3492) loss 3.5877 (3.2950) grad_norm 0.9702 (1.3972) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:55:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][390/625] eta 0:01:22 lr 0.001473 wd 0.0500 time 0.3505 (0.3525) data time 0.0008 (0.0026) model time 0.3496 (0.3491) loss 3.2026 (3.2956) grad_norm 0.9448 (1.3921) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][400/625] eta 0:01:19 lr 0.001473 wd 0.0500 time 0.3462 (0.3524) data time 0.0007 (0.0026) model time 0.3455 (0.3490) loss 3.0408 (3.2966) grad_norm 2.0793 (1.4131) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][410/625] eta 0:01:15 lr 0.001473 wd 0.0500 time 0.3481 (0.3522) data time 0.0013 (0.0026) model time 0.3468 (0.3490) loss 3.7182 (3.2952) grad_norm 1.2558 (1.4080) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][420/625] eta 0:01:12 lr 0.001473 wd 0.0500 time 0.3452 (0.3521) data time 0.0009 (0.0025) model time 0.3443 (0.3489) loss 3.0811 (3.2941) grad_norm 1.6869 (1.4021) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][430/625] eta 0:01:08 lr 0.001472 wd 0.0500 time 0.3428 (0.3520) data time 0.0008 (0.0025) model time 0.3419 (0.3488) loss 3.4119 (3.2936) grad_norm 1.0684 (1.4078) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][440/625] eta 0:01:05 lr 0.001472 wd 0.0500 time 0.3408 (0.3518) data time 0.0007 (0.0025) model time 0.3401 (0.3487) loss 3.4774 (3.2974) grad_norm 0.9979 (1.4078) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][450/625] eta 0:01:01 lr 0.001472 wd 0.0500 time 0.5287 (0.3525) data time 0.0009 (0.0024) model time 0.5278 (0.3495) loss 3.1641 (3.3040) grad_norm 1.2174 (1.4036) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][460/625] eta 0:00:58 lr 0.001472 wd 0.0500 time 0.5486 (0.3538) data time 0.0006 (0.0024) model time 0.5480 (0.3510) loss 4.1769 (3.3109) grad_norm 1.2090 (1.4074) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][470/625] eta 0:00:54 lr 0.001472 wd 0.0500 time 0.3471 (0.3535) data time 0.0009 (0.0024) model time 0.3461 (0.3507) loss 2.6478 (3.3077) grad_norm 1.2819 (1.4056) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][480/625] eta 0:00:51 lr 0.001472 wd 0.0500 time 0.3588 (0.3534) data time 0.0009 (0.0023) model time 0.3579 (0.3506) loss 3.1231 (3.3073) grad_norm 1.6027 (1.4078) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][490/625] eta 0:00:47 lr 0.001471 wd 0.0500 time 0.3427 (0.3533) data time 0.0006 (0.0024) model time 0.3421 (0.3504) loss 4.1522 (3.3081) grad_norm 1.0775 (1.4091) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][500/625] eta 0:00:44 lr 0.001471 wd 0.0500 time 0.3395 (0.3531) data time 0.0007 (0.0024) model time 0.3388 (0.3503) loss 3.6191 (3.3055) grad_norm 1.1410 (1.4108) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][510/625] eta 0:00:40 lr 0.001471 wd 0.0500 time 0.3470 (0.3530) data time 0.0006 (0.0023) model time 0.3463 (0.3503) loss 3.7471 (3.3098) grad_norm 1.1844 (1.4073) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][520/625] eta 0:00:37 lr 0.001471 wd 0.0500 time 0.3405 (0.3529) data time 0.0009 (0.0023) model time 0.3397 (0.3502) loss 3.6838 (3.3074) grad_norm 1.4426 (1.4039) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][530/625] eta 0:00:33 lr 0.001471 wd 0.0500 time 0.3513 (0.3528) data time 0.0006 (0.0023) model time 0.3506 (0.3500) loss 4.0693 (3.3101) grad_norm 1.7735 (1.4055) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][540/625] eta 0:00:29 lr 0.001471 wd 0.0500 time 0.3618 (0.3527) data time 0.0007 (0.0023) model time 0.3612 (0.3500) loss 2.0892 (3.3044) grad_norm 2.4331 (1.4062) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][550/625] eta 0:00:26 lr 0.001470 wd 0.0500 time 0.3391 (0.3526) data time 0.0006 (0.0022) model time 0.3385 (0.3499) loss 2.0991 (3.3017) grad_norm 1.2691 (1.4050) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][560/625] eta 0:00:22 lr 0.001470 wd 0.0500 time 0.3492 (0.3525) data time 0.0009 (0.0022) model time 0.3483 (0.3499) loss 3.6638 (3.3058) grad_norm 1.6642 (1.4036) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][570/625] eta 0:00:19 lr 0.001470 wd 0.0500 time 0.3403 (0.3533) data time 0.0006 (0.0022) model time 0.3397 (0.3507) loss 3.3550 (3.3073) grad_norm 1.3106 (1.4052) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][580/625] eta 0:00:15 lr 0.001470 wd 0.0500 time 0.3103 (0.3542) data time 0.0007 (0.0022) model time 0.3096 (0.3518) loss 4.0752 (3.3138) grad_norm 1.6940 (1.4058) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][590/625] eta 0:00:12 lr 0.001470 wd 0.0500 time 0.3557 (0.3541) data time 0.0006 (0.0022) model time 0.3550 (0.3517) loss 2.1310 (3.3077) grad_norm 1.4937 (1.4069) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][600/625] eta 0:00:08 lr 0.001470 wd 0.0500 time 0.3465 (0.3540) data time 0.0006 (0.0022) model time 0.3459 (0.3516) loss 2.7973 (3.3086) grad_norm 1.0231 (1.4042) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][610/625] eta 0:00:05 lr 0.001470 wd 0.0500 time 0.3384 (0.3539) data time 0.0004 (0.0021) model time 0.3380 (0.3514) loss 4.2787 (3.3115) grad_norm 1.1450 (1.4043) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [116/300][620/625] eta 0:00:01 lr 0.001469 wd 0.0500 time 0.3509 (0.3537) data time 0.0006 (0.0021) model time 0.3503 (0.3513) loss 2.8038 (3.3113) grad_norm 1.2484 (1.4045) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 116 training takes 0:03:41 [2024-08-26 10:57:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 10:57:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 10:57:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.456 (0.456) Loss 0.6543 (0.6543) Acc@1 85.889 (85.889) Acc@5 97.510 (97.510) Mem 11224MB [2024-08-26 10:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.132) Loss 1.0762 (0.7816) Acc@1 73.877 (82.497) Acc@5 93.555 (96.391) Mem 11224MB [2024-08-26 10:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.113) Loss 1.1660 (0.9360) Acc@1 72.070 (78.925) Acc@5 92.090 (94.564) Mem 11224MB [2024-08-26 10:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.743 Acc@5 94.582 [2024-08-26 10:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.7% [2024-08-26 10:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.74% [2024-08-26 10:57:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 10:57:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 10:57:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.432 (0.432) Loss 0.5352 (0.5352) Acc@1 88.477 (88.477) Acc@5 98.242 (98.242) Mem 11224MB [2024-08-26 10:57:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.128) Loss 0.9009 (0.6710) Acc@1 78.857 (84.788) Acc@5 95.215 (97.288) Mem 11224MB [2024-08-26 10:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.110) Loss 1.0371 (0.8104) Acc@1 74.268 (81.269) Acc@5 93.701 (95.808) Mem 11224MB [2024-08-26 10:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.990 Acc@5 95.783 [2024-08-26 10:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.0% [2024-08-26 10:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 80.99% [2024-08-26 10:57:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 10:57:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 10:57:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][0/625] eta 0:07:08 lr 0.001469 wd 0.0500 time 0.6863 (0.6863) data time 0.3550 (0.3550) model time 0.0000 (0.0000) loss 4.1867 (4.1867) grad_norm 0.9038 (0.9038) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][10/625] eta 0:03:52 lr 0.001469 wd 0.0500 time 0.3446 (0.3776) data time 0.0007 (0.0331) model time 0.0000 (0.0000) loss 4.1126 (3.2658) grad_norm 1.6393 (1.2854) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][20/625] eta 0:03:39 lr 0.001469 wd 0.0500 time 0.3471 (0.3633) data time 0.0008 (0.0178) model time 0.0000 (0.0000) loss 3.3987 (3.3632) grad_norm 1.4187 (1.2549) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][30/625] eta 0:03:33 lr 0.001469 wd 0.0500 time 0.3473 (0.3590) data time 0.0009 (0.0124) model time 0.0000 (0.0000) loss 3.7422 (3.4071) grad_norm 1.2944 (1.2345) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][40/625] eta 0:03:28 lr 0.001469 wd 0.0500 time 0.3470 (0.3560) data time 0.0006 (0.0097) model time 0.0000 (0.0000) loss 2.8846 (3.3967) grad_norm 1.2056 (1.2819) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:57:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][50/625] eta 0:03:23 lr 0.001469 wd 0.0500 time 0.3432 (0.3539) data time 0.0008 (0.0080) model time 0.0000 (0.0000) loss 3.4444 (3.4069) grad_norm 1.5624 (1.3211) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][60/625] eta 0:03:19 lr 0.001468 wd 0.0500 time 0.3676 (0.3532) data time 0.0006 (0.0068) model time 0.3670 (0.3489) loss 3.8029 (3.4497) grad_norm 1.0686 (1.3264) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][70/625] eta 0:03:17 lr 0.001468 wd 0.0500 time 0.3439 (0.3554) data time 0.0006 (0.0060) model time 0.3433 (0.3582) loss 3.8378 (3.4632) grad_norm 1.2985 (1.3429) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][80/625] eta 0:03:13 lr 0.001468 wd 0.0500 time 0.3433 (0.3545) data time 0.0009 (0.0055) model time 0.3424 (0.3542) loss 3.8986 (3.4456) grad_norm 1.3922 (1.3494) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][90/625] eta 0:03:13 lr 0.001468 wd 0.0500 time 0.5294 (0.3619) data time 0.0007 (0.0050) model time 0.5287 (0.3707) loss 3.8836 (3.4141) grad_norm 1.7968 (1.3973) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][100/625] eta 0:03:09 lr 0.001468 wd 0.0500 time 0.3476 (0.3607) data time 0.0006 (0.0046) model time 0.3470 (0.3665) loss 3.6785 (3.4081) grad_norm 1.1017 (1.4179) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][110/625] eta 0:03:05 lr 0.001468 wd 0.0500 time 0.3447 (0.3599) data time 0.0007 (0.0043) model time 0.3441 (0.3638) loss 3.3701 (3.3799) grad_norm 1.2110 (1.4011) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][120/625] eta 0:03:01 lr 0.001467 wd 0.0500 time 0.3576 (0.3593) data time 0.0007 (0.0041) model time 0.3570 (0.3619) loss 3.2230 (3.3873) grad_norm 1.1309 (1.3896) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][130/625] eta 0:02:57 lr 0.001467 wd 0.0500 time 0.3427 (0.3583) data time 0.0008 (0.0039) model time 0.3420 (0.3599) loss 3.3436 (3.3828) grad_norm 1.0910 (1.3897) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][140/625] eta 0:02:53 lr 0.001467 wd 0.0500 time 0.3606 (0.3578) data time 0.0007 (0.0037) model time 0.3599 (0.3588) loss 3.8228 (3.3986) grad_norm 1.0552 (1.3842) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][150/625] eta 0:02:49 lr 0.001467 wd 0.0500 time 0.3421 (0.3571) data time 0.0008 (0.0035) model time 0.3412 (0.3576) loss 2.3576 (3.3993) grad_norm 1.5354 (1.3879) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][160/625] eta 0:02:45 lr 0.001467 wd 0.0500 time 0.3435 (0.3569) data time 0.0008 (0.0033) model time 0.3427 (0.3570) loss 3.2936 (3.3963) grad_norm 0.9835 (1.3907) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][170/625] eta 0:02:42 lr 0.001467 wd 0.0500 time 0.3573 (0.3565) data time 0.0009 (0.0032) model time 0.3565 (0.3564) loss 3.7832 (3.3917) grad_norm 1.9256 (1.3876) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][180/625] eta 0:02:38 lr 0.001466 wd 0.0500 time 0.3382 (0.3560) data time 0.0008 (0.0031) model time 0.3374 (0.3556) loss 3.6217 (3.3996) grad_norm 0.8837 (1.3751) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][190/625] eta 0:02:34 lr 0.001466 wd 0.0500 time 0.3471 (0.3556) data time 0.0009 (0.0030) model time 0.3462 (0.3550) loss 3.2852 (3.4031) grad_norm 1.1079 (1.3781) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][200/625] eta 0:02:30 lr 0.001466 wd 0.0500 time 0.3406 (0.3551) data time 0.0008 (0.0030) model time 0.3398 (0.3542) loss 3.6095 (3.4070) grad_norm 1.4797 (1.3791) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][210/625] eta 0:02:27 lr 0.001466 wd 0.0500 time 0.3457 (0.3548) data time 0.0009 (0.0029) model time 0.3448 (0.3538) loss 2.9235 (3.4131) grad_norm 1.0854 (1.3734) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:58:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][220/625] eta 0:02:23 lr 0.001466 wd 0.0500 time 0.3420 (0.3546) data time 0.0008 (0.0029) model time 0.3413 (0.3534) loss 3.2936 (3.4079) grad_norm 1.9530 (1.3778) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:59:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][230/625] eta 0:02:19 lr 0.001466 wd 0.0500 time 0.3520 (0.3544) data time 0.0009 (0.0028) model time 0.3511 (0.3532) loss 3.1556 (3.4131) grad_norm 1.2959 (1.3792) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:59:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][240/625] eta 0:02:16 lr 0.001466 wd 0.0500 time 0.3491 (0.3542) data time 0.0008 (0.0027) model time 0.3483 (0.3530) loss 2.4010 (3.3950) grad_norm 1.1866 (1.3879) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][250/625] eta 0:02:12 lr 0.001465 wd 0.0500 time 0.3450 (0.3546) data time 0.0007 (0.0027) model time 0.3443 (0.3535) loss 2.9029 (3.3951) grad_norm 1.2698 (1.3967) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][260/625] eta 0:02:09 lr 0.001465 wd 0.0500 time 0.3589 (0.3544) data time 0.0008 (0.0026) model time 0.3581 (0.3532) loss 2.5389 (3.3821) grad_norm 1.4974 (1.3992) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 10:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][270/625] eta 0:02:05 lr 0.001465 wd 0.0500 time 0.3437 (0.3540) data time 0.0009 (0.0026) model time 0.3427 (0.3528) loss 3.8754 (3.3919) grad_norm 1.3086 (1.3956) loss_scale 4096.0000 (2100.9004) mem 11224MB [2024-08-26 10:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][280/625] eta 0:02:02 lr 0.001465 wd 0.0500 time 0.3447 (0.3538) data time 0.0009 (0.0025) model time 0.3438 (0.3525) loss 3.4519 (3.3864) grad_norm 1.5302 (1.3966) loss_scale 4096.0000 (2171.9004) mem 11224MB [2024-08-26 10:59:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][290/625] eta 0:01:58 lr 0.001465 wd 0.0500 time 0.3459 (0.3537) data time 0.0008 (0.0025) model time 0.3451 (0.3523) loss 3.6337 (3.3794) grad_norm 1.8115 (1.3971) loss_scale 4096.0000 (2238.0206) mem 11224MB [2024-08-26 10:59:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][300/625] eta 0:01:55 lr 0.001465 wd 0.0500 time 0.5626 (0.3556) data time 0.0008 (0.0025) model time 0.5618 (0.3546) loss 3.3346 (3.3739) grad_norm 2.0930 (1.4062) loss_scale 4096.0000 (2299.7475) mem 11224MB [2024-08-26 10:59:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][310/625] eta 0:01:51 lr 0.001464 wd 0.0500 time 0.3476 (0.3553) data time 0.0007 (0.0024) model time 0.3470 (0.3543) loss 3.5770 (3.3722) grad_norm 1.0006 (1.4086) loss_scale 4096.0000 (2357.5048) mem 11224MB [2024-08-26 10:59:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][320/625] eta 0:01:48 lr 0.001464 wd 0.0500 time 0.3627 (0.3550) data time 0.0006 (0.0024) model time 0.3621 (0.3539) loss 3.3646 (3.3720) grad_norm 1.2176 (1.4034) loss_scale 4096.0000 (2411.6636) mem 11224MB [2024-08-26 10:59:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][330/625] eta 0:01:44 lr 0.001464 wd 0.0500 time 0.3418 (0.3547) data time 0.0007 (0.0023) model time 0.3411 (0.3536) loss 3.1428 (3.3707) grad_norm 1.1186 (1.3991) loss_scale 4096.0000 (2462.5498) mem 11224MB [2024-08-26 10:59:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][340/625] eta 0:01:41 lr 0.001464 wd 0.0500 time 0.3424 (0.3545) data time 0.0008 (0.0023) model time 0.3415 (0.3533) loss 2.8115 (3.3720) grad_norm 1.3838 (1.3989) loss_scale 4096.0000 (2510.4516) mem 11224MB [2024-08-26 10:59:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][350/625] eta 0:01:37 lr 0.001464 wd 0.0500 time 0.3523 (0.3542) data time 0.0008 (0.0022) model time 0.3515 (0.3530) loss 3.1165 (3.3678) grad_norm 0.9965 (1.3929) loss_scale 4096.0000 (2555.6239) mem 11224MB [2024-08-26 10:59:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][360/625] eta 0:01:33 lr 0.001464 wd 0.0500 time 0.3402 (0.3540) data time 0.0007 (0.0022) model time 0.3395 (0.3527) loss 3.0585 (3.3562) grad_norm 2.6531 (1.4067) loss_scale 4096.0000 (2598.2936) mem 11224MB [2024-08-26 10:59:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][370/625] eta 0:01:30 lr 0.001463 wd 0.0500 time 0.3460 (0.3542) data time 0.0006 (0.0022) model time 0.3455 (0.3531) loss 3.1368 (3.3506) grad_norm 1.2206 (inf) loss_scale 2048.0000 (2622.1024) mem 11224MB [2024-08-26 10:59:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][380/625] eta 0:01:26 lr 0.001463 wd 0.0500 time 0.3486 (0.3540) data time 0.0008 (0.0021) model time 0.3478 (0.3528) loss 3.3552 (3.3433) grad_norm 1.1582 (inf) loss_scale 2048.0000 (2607.0341) mem 11224MB [2024-08-26 10:59:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][390/625] eta 0:01:23 lr 0.001463 wd 0.0500 time 0.3392 (0.3539) data time 0.0007 (0.0021) model time 0.3385 (0.3526) loss 3.4953 (3.3367) grad_norm 1.5432 (inf) loss_scale 2048.0000 (2592.7366) mem 11224MB [2024-08-26 11:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][400/625] eta 0:01:19 lr 0.001463 wd 0.0500 time 0.3436 (0.3537) data time 0.0009 (0.0021) model time 0.3427 (0.3524) loss 3.1283 (3.3413) grad_norm 1.1718 (inf) loss_scale 2048.0000 (2579.1521) mem 11224MB [2024-08-26 11:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][410/625] eta 0:01:15 lr 0.001463 wd 0.0500 time 0.3471 (0.3535) data time 0.0008 (0.0020) model time 0.3463 (0.3522) loss 3.6114 (3.3316) grad_norm 1.1261 (inf) loss_scale 2048.0000 (2566.2287) mem 11224MB [2024-08-26 11:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][420/625] eta 0:01:12 lr 0.001463 wd 0.0500 time 0.3445 (0.3533) data time 0.0006 (0.0020) model time 0.3439 (0.3520) loss 3.0202 (3.3319) grad_norm 2.1277 (inf) loss_scale 2048.0000 (2553.9192) mem 11224MB [2024-08-26 11:00:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][430/625] eta 0:01:08 lr 0.001463 wd 0.0500 time 0.3439 (0.3531) data time 0.0006 (0.0020) model time 0.3432 (0.3518) loss 3.5028 (3.3377) grad_norm 1.2204 (inf) loss_scale 2048.0000 (2542.1810) mem 11224MB [2024-08-26 11:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][440/625] eta 0:01:05 lr 0.001462 wd 0.0500 time 0.3464 (0.3529) data time 0.0006 (0.0020) model time 0.3458 (0.3516) loss 3.8813 (3.3423) grad_norm 1.5396 (inf) loss_scale 2048.0000 (2530.9751) mem 11224MB [2024-08-26 11:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][450/625] eta 0:01:01 lr 0.001462 wd 0.0500 time 0.3387 (0.3527) data time 0.0007 (0.0019) model time 0.3380 (0.3514) loss 3.1388 (3.3367) grad_norm 1.7323 (inf) loss_scale 2048.0000 (2520.2661) mem 11224MB [2024-08-26 11:00:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][460/625] eta 0:00:58 lr 0.001462 wd 0.0500 time 0.3473 (0.3526) data time 0.0009 (0.0019) model time 0.3465 (0.3512) loss 2.9790 (3.3350) grad_norm 1.9074 (inf) loss_scale 2048.0000 (2510.0217) mem 11224MB [2024-08-26 11:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][470/625] eta 0:00:54 lr 0.001462 wd 0.0500 time 0.3435 (0.3525) data time 0.0007 (0.0019) model time 0.3429 (0.3511) loss 3.6612 (3.3342) grad_norm 1.9403 (inf) loss_scale 2048.0000 (2500.2123) mem 11224MB [2024-08-26 11:00:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][480/625] eta 0:00:51 lr 0.001462 wd 0.0500 time 0.5437 (0.3531) data time 0.0007 (0.0019) model time 0.5430 (0.3519) loss 1.9714 (3.3337) grad_norm 1.4222 (inf) loss_scale 2048.0000 (2490.8108) mem 11224MB [2024-08-26 11:00:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][490/625] eta 0:00:47 lr 0.001462 wd 0.0500 time 0.3535 (0.3542) data time 0.0009 (0.0019) model time 0.3526 (0.3531) loss 3.6081 (3.3339) grad_norm 0.8303 (inf) loss_scale 2048.0000 (2481.7923) mem 11224MB [2024-08-26 11:00:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][500/625] eta 0:00:44 lr 0.001461 wd 0.0500 time 0.3428 (0.3540) data time 0.0006 (0.0018) model time 0.3422 (0.3529) loss 3.0453 (3.3322) grad_norm 1.2444 (inf) loss_scale 2048.0000 (2473.1337) mem 11224MB [2024-08-26 11:00:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][510/625] eta 0:00:40 lr 0.001461 wd 0.0500 time 0.3453 (0.3538) data time 0.0006 (0.0018) model time 0.3446 (0.3527) loss 4.1266 (3.3323) grad_norm 1.1050 (inf) loss_scale 2048.0000 (2464.8141) mem 11224MB [2024-08-26 11:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][520/625] eta 0:00:37 lr 0.001461 wd 0.0500 time 0.3444 (0.3537) data time 0.0006 (0.0018) model time 0.3438 (0.3525) loss 3.4612 (3.3338) grad_norm 1.1489 (inf) loss_scale 2048.0000 (2456.8138) mem 11224MB [2024-08-26 11:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][530/625] eta 0:00:33 lr 0.001461 wd 0.0500 time 0.3498 (0.3535) data time 0.0007 (0.0018) model time 0.3492 (0.3523) loss 3.0503 (3.3286) grad_norm 1.4821 (inf) loss_scale 2048.0000 (2449.1149) mem 11224MB [2024-08-26 11:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][540/625] eta 0:00:30 lr 0.001461 wd 0.0500 time 0.3409 (0.3533) data time 0.0007 (0.0018) model time 0.3402 (0.3521) loss 1.9087 (3.3328) grad_norm 1.3359 (inf) loss_scale 2048.0000 (2441.7006) mem 11224MB [2024-08-26 11:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][550/625] eta 0:00:26 lr 0.001461 wd 0.0500 time 0.3426 (0.3532) data time 0.0009 (0.0018) model time 0.3417 (0.3520) loss 3.9237 (3.3348) grad_norm 1.7438 (inf) loss_scale 2048.0000 (2434.5554) mem 11224MB [2024-08-26 11:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][560/625] eta 0:00:22 lr 0.001460 wd 0.0500 time 0.3440 (0.3532) data time 0.0006 (0.0017) model time 0.3434 (0.3520) loss 3.8030 (3.3346) grad_norm 1.2581 (inf) loss_scale 2048.0000 (2427.6649) mem 11224MB [2024-08-26 11:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][570/625] eta 0:00:19 lr 0.001460 wd 0.0500 time 0.3408 (0.3531) data time 0.0009 (0.0017) model time 0.3399 (0.3519) loss 3.1973 (3.3332) grad_norm 1.3948 (inf) loss_scale 2048.0000 (2421.0158) mem 11224MB [2024-08-26 11:01:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][580/625] eta 0:00:15 lr 0.001460 wd 0.0500 time 0.3406 (0.3532) data time 0.0007 (0.0017) model time 0.3399 (0.3521) loss 2.7926 (3.3284) grad_norm 1.2773 (inf) loss_scale 2048.0000 (2414.5955) mem 11224MB [2024-08-26 11:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][590/625] eta 0:00:12 lr 0.001460 wd 0.0500 time 0.3437 (0.3531) data time 0.0008 (0.0017) model time 0.3428 (0.3519) loss 3.3086 (3.3255) grad_norm 1.0931 (inf) loss_scale 2048.0000 (2408.3926) mem 11224MB [2024-08-26 11:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][600/625] eta 0:00:08 lr 0.001460 wd 0.0500 time 0.3439 (0.3537) data time 0.0006 (0.0017) model time 0.3433 (0.3525) loss 3.9528 (3.3300) grad_norm 1.2670 (inf) loss_scale 2048.0000 (2402.3960) mem 11224MB [2024-08-26 11:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][610/625] eta 0:00:05 lr 0.001460 wd 0.0500 time 0.3388 (0.3542) data time 0.0006 (0.0017) model time 0.3381 (0.3531) loss 3.0675 (3.3283) grad_norm 1.2781 (inf) loss_scale 2048.0000 (2396.5957) mem 11224MB [2024-08-26 11:01:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [117/300][620/625] eta 0:00:01 lr 0.001460 wd 0.0500 time 0.3372 (0.3539) data time 0.0004 (0.0017) model time 0.3369 (0.3528) loss 3.0725 (3.3286) grad_norm 1.0309 (inf) loss_scale 2048.0000 (2390.9823) mem 11224MB [2024-08-26 11:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 117 training takes 0:03:41 [2024-08-26 11:01:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:01:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:01:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.455 (0.455) Loss 0.6387 (0.6387) Acc@1 85.986 (85.986) Acc@5 97.852 (97.852) Mem 11224MB [2024-08-26 11:01:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.131) Loss 1.0557 (0.7860) Acc@1 76.611 (82.848) Acc@5 93.359 (96.489) Mem 11224MB [2024-08-26 11:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.1572 (0.9313) Acc@1 72.852 (79.248) Acc@5 92.480 (94.654) Mem 11224MB [2024-08-26 11:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.881 Acc@5 94.646 [2024-08-26 11:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.9% [2024-08-26 11:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.88% [2024-08-26 11:01:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 11:01:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 11:01:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.443 (0.443) Loss 0.5342 (0.5342) Acc@1 88.184 (88.184) Acc@5 98.340 (98.340) Mem 11224MB [2024-08-26 11:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.129) Loss 0.8965 (0.6697) Acc@1 78.955 (84.792) Acc@5 95.215 (97.323) Mem 11224MB [2024-08-26 11:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 1.0352 (0.8089) Acc@1 74.609 (81.299) Acc@5 93.555 (95.824) Mem 11224MB [2024-08-26 11:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.018 Acc@5 95.803 [2024-08-26 11:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.0% [2024-08-26 11:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.02% [2024-08-26 11:01:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:01:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:01:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][0/625] eta 0:07:26 lr 0.001459 wd 0.0500 time 0.7143 (0.7143) data time 0.3988 (0.3988) model time 0.0000 (0.0000) loss 2.7651 (2.7651) grad_norm 1.0969 (1.0969) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][10/625] eta 0:03:53 lr 0.001459 wd 0.0500 time 0.3450 (0.3802) data time 0.0010 (0.0370) model time 0.0000 (0.0000) loss 3.6641 (3.3515) grad_norm 1.0075 (1.2598) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:01:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][20/625] eta 0:03:40 lr 0.001459 wd 0.0500 time 0.3430 (0.3637) data time 0.0007 (0.0198) model time 0.0000 (0.0000) loss 2.3319 (3.1056) grad_norm 1.0731 (1.3338) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:01:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][30/625] eta 0:03:33 lr 0.001459 wd 0.0500 time 0.3445 (0.3580) data time 0.0006 (0.0137) model time 0.0000 (0.0000) loss 3.6132 (3.2043) grad_norm 1.4876 (1.3401) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][40/625] eta 0:03:27 lr 0.001459 wd 0.0500 time 0.3514 (0.3555) data time 0.0007 (0.0106) model time 0.0000 (0.0000) loss 2.8610 (3.1952) grad_norm 1.3861 (1.3262) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][50/625] eta 0:03:23 lr 0.001459 wd 0.0500 time 0.3536 (0.3537) data time 0.0007 (0.0087) model time 0.0000 (0.0000) loss 3.6796 (3.1832) grad_norm 1.4035 (1.3392) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][60/625] eta 0:03:20 lr 0.001459 wd 0.0500 time 0.3398 (0.3545) data time 0.0006 (0.0074) model time 0.3391 (0.3578) loss 4.0813 (3.2069) grad_norm 0.9742 (1.3409) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][70/625] eta 0:03:16 lr 0.001458 wd 0.0500 time 0.3535 (0.3534) data time 0.0006 (0.0065) model time 0.3529 (0.3516) loss 2.8710 (3.1907) grad_norm 1.6005 (1.3608) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:01:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][80/625] eta 0:03:11 lr 0.001458 wd 0.0500 time 0.3408 (0.3521) data time 0.0007 (0.0058) model time 0.3402 (0.3486) loss 4.0790 (3.2201) grad_norm 1.1632 (1.3336) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][90/625] eta 0:03:08 lr 0.001458 wd 0.0500 time 0.3520 (0.3515) data time 0.0010 (0.0053) model time 0.3510 (0.3477) loss 2.4882 (3.2251) grad_norm 1.1951 (1.3147) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][100/625] eta 0:03:04 lr 0.001458 wd 0.0500 time 0.3413 (0.3515) data time 0.0007 (0.0053) model time 0.3406 (0.3476) loss 4.0152 (3.2467) grad_norm 2.3475 (1.3481) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][110/625] eta 0:03:02 lr 0.001458 wd 0.0500 time 0.3422 (0.3540) data time 0.0007 (0.0049) model time 0.3416 (0.3526) loss 3.6801 (3.2828) grad_norm 1.2677 (1.3961) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][120/625] eta 0:02:58 lr 0.001458 wd 0.0500 time 0.3422 (0.3533) data time 0.0006 (0.0046) model time 0.3416 (0.3514) loss 2.8908 (3.2916) grad_norm 1.4439 (1.3979) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][130/625] eta 0:02:54 lr 0.001457 wd 0.0500 time 0.3403 (0.3527) data time 0.0009 (0.0043) model time 0.3394 (0.3505) loss 3.2580 (3.3058) grad_norm 1.4256 (1.3828) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][140/625] eta 0:02:50 lr 0.001457 wd 0.0500 time 0.3472 (0.3521) data time 0.0007 (0.0041) model time 0.3466 (0.3498) loss 3.7585 (3.3140) grad_norm 1.4822 (1.3972) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][150/625] eta 0:02:47 lr 0.001457 wd 0.0500 time 0.3479 (0.3518) data time 0.0006 (0.0038) model time 0.3474 (0.3494) loss 3.2609 (3.3145) grad_norm 1.4157 (1.3934) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][160/625] eta 0:02:43 lr 0.001457 wd 0.0500 time 0.3461 (0.3513) data time 0.0006 (0.0037) model time 0.3454 (0.3489) loss 3.4462 (3.2988) grad_norm 1.5901 (1.3931) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][170/625] eta 0:02:41 lr 0.001457 wd 0.0500 time 0.3419 (0.3541) data time 0.0006 (0.0035) model time 0.3413 (0.3530) loss 4.4813 (3.3230) grad_norm 1.0797 (1.3853) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][180/625] eta 0:02:37 lr 0.001457 wd 0.0500 time 0.3582 (0.3548) data time 0.0008 (0.0034) model time 0.3574 (0.3539) loss 3.9572 (3.3262) grad_norm 1.1742 (1.3800) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][190/625] eta 0:02:34 lr 0.001456 wd 0.0500 time 0.3476 (0.3542) data time 0.0005 (0.0032) model time 0.3470 (0.3531) loss 2.6029 (3.3254) grad_norm 1.8024 (1.3994) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][200/625] eta 0:02:30 lr 0.001456 wd 0.0500 time 0.3440 (0.3537) data time 0.0009 (0.0031) model time 0.3431 (0.3525) loss 2.3700 (3.3127) grad_norm 1.3261 (1.3927) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][210/625] eta 0:02:26 lr 0.001456 wd 0.0500 time 0.3585 (0.3535) data time 0.0009 (0.0030) model time 0.3576 (0.3522) loss 3.2786 (3.3071) grad_norm 1.0095 (1.3925) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][220/625] eta 0:02:23 lr 0.001456 wd 0.0500 time 0.3442 (0.3531) data time 0.0006 (0.0029) model time 0.3435 (0.3517) loss 2.5656 (3.3062) grad_norm 1.5805 (1.3962) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][230/625] eta 0:02:19 lr 0.001456 wd 0.0500 time 0.3491 (0.3528) data time 0.0005 (0.0028) model time 0.3486 (0.3514) loss 2.9668 (3.3043) grad_norm 1.9018 (1.3957) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][240/625] eta 0:02:15 lr 0.001456 wd 0.0500 time 0.3414 (0.3525) data time 0.0006 (0.0028) model time 0.3408 (0.3510) loss 3.9047 (3.3070) grad_norm 1.4112 (1.4130) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:02:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][250/625] eta 0:02:12 lr 0.001455 wd 0.0500 time 0.3401 (0.3522) data time 0.0009 (0.0027) model time 0.3393 (0.3506) loss 3.3092 (3.3085) grad_norm 1.3231 (1.4125) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:03:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][260/625] eta 0:02:08 lr 0.001455 wd 0.0500 time 0.3490 (0.3519) data time 0.0009 (0.0026) model time 0.3481 (0.3503) loss 3.3732 (3.3160) grad_norm 1.0513 (1.4056) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:03:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][270/625] eta 0:02:04 lr 0.001455 wd 0.0500 time 0.3460 (0.3518) data time 0.0009 (0.0026) model time 0.3452 (0.3503) loss 3.3859 (3.3184) grad_norm 1.9394 (1.4062) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-26 11:03:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][280/625] eta 0:02:01 lr 0.001455 wd 0.0500 time 0.3619 (0.3516) data time 0.0009 (0.0025) model time 0.3611 (0.3500) loss 3.4676 (3.3170) grad_norm 1.1953 (inf) loss_scale 1024.0000 (2022.4911) mem 11224MB [2024-08-26 11:03:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][290/625] eta 0:01:58 lr 0.001455 wd 0.0500 time 0.3445 (0.3529) data time 0.0009 (0.0024) model time 0.3437 (0.3517) loss 3.6211 (3.3202) grad_norm 1.3472 (inf) loss_scale 1024.0000 (1988.1787) mem 11224MB [2024-08-26 11:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][300/625] eta 0:01:54 lr 0.001455 wd 0.0500 time 0.3457 (0.3527) data time 0.0006 (0.0024) model time 0.3451 (0.3514) loss 3.0087 (3.3215) grad_norm 1.8548 (inf) loss_scale 1024.0000 (1956.1462) mem 11224MB [2024-08-26 11:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][310/625] eta 0:01:51 lr 0.001455 wd 0.0500 time 0.3464 (0.3524) data time 0.0007 (0.0023) model time 0.3457 (0.3511) loss 2.9305 (3.3130) grad_norm 1.6111 (inf) loss_scale 1024.0000 (1926.1736) mem 11224MB [2024-08-26 11:03:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][320/625] eta 0:01:47 lr 0.001454 wd 0.0500 time 0.3637 (0.3536) data time 0.0009 (0.0023) model time 0.3628 (0.3525) loss 3.3877 (3.3146) grad_norm 1.1425 (inf) loss_scale 1024.0000 (1898.0685) mem 11224MB [2024-08-26 11:03:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][330/625] eta 0:01:44 lr 0.001454 wd 0.0500 time 0.3375 (0.3534) data time 0.0008 (0.0023) model time 0.3367 (0.3523) loss 3.9791 (3.3120) grad_norm 1.4928 (inf) loss_scale 1024.0000 (1871.6616) mem 11224MB [2024-08-26 11:03:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][340/625] eta 0:01:40 lr 0.001454 wd 0.0500 time 0.3437 (0.3533) data time 0.0008 (0.0022) model time 0.3428 (0.3522) loss 2.5966 (3.3037) grad_norm 1.2706 (inf) loss_scale 1024.0000 (1846.8035) mem 11224MB [2024-08-26 11:03:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][350/625] eta 0:01:37 lr 0.001454 wd 0.0500 time 0.3758 (0.3534) data time 0.0008 (0.0022) model time 0.3749 (0.3522) loss 3.5024 (3.3056) grad_norm 1.9091 (inf) loss_scale 1024.0000 (1823.3618) mem 11224MB [2024-08-26 11:03:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][360/625] eta 0:01:33 lr 0.001454 wd 0.0500 time 0.3412 (0.3533) data time 0.0008 (0.0022) model time 0.3404 (0.3521) loss 2.8423 (3.3013) grad_norm 1.1497 (inf) loss_scale 1024.0000 (1801.2188) mem 11224MB [2024-08-26 11:03:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][370/625] eta 0:01:30 lr 0.001454 wd 0.0500 time 0.3413 (0.3534) data time 0.0009 (0.0022) model time 0.3404 (0.3521) loss 3.5805 (3.3049) grad_norm 1.7863 (inf) loss_scale 1024.0000 (1780.2695) mem 11224MB [2024-08-26 11:03:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][380/625] eta 0:01:26 lr 0.001453 wd 0.0500 time 0.3709 (0.3534) data time 0.0006 (0.0022) model time 0.3703 (0.3522) loss 3.8648 (3.3081) grad_norm 1.3981 (inf) loss_scale 1024.0000 (1760.4199) mem 11224MB [2024-08-26 11:03:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][390/625] eta 0:01:23 lr 0.001453 wd 0.0500 time 0.3450 (0.3538) data time 0.0006 (0.0022) model time 0.3444 (0.3526) loss 3.3272 (3.3060) grad_norm 1.2982 (inf) loss_scale 1024.0000 (1741.5857) mem 11224MB [2024-08-26 11:03:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][400/625] eta 0:01:19 lr 0.001453 wd 0.0500 time 0.3489 (0.3537) data time 0.0007 (0.0022) model time 0.3482 (0.3525) loss 2.3821 (3.3017) grad_norm 3.1940 (inf) loss_scale 1024.0000 (1723.6908) mem 11224MB [2024-08-26 11:03:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][410/625] eta 0:01:16 lr 0.001453 wd 0.0500 time 0.3477 (0.3535) data time 0.0007 (0.0021) model time 0.3470 (0.3523) loss 3.0037 (3.3044) grad_norm 1.2418 (inf) loss_scale 1024.0000 (1706.6667) mem 11224MB [2024-08-26 11:03:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][420/625] eta 0:01:12 lr 0.001453 wd 0.0500 time 0.3428 (0.3533) data time 0.0010 (0.0021) model time 0.3418 (0.3520) loss 3.1725 (3.3010) grad_norm 0.9860 (inf) loss_scale 1024.0000 (1690.4513) mem 11224MB [2024-08-26 11:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][430/625] eta 0:01:08 lr 0.001453 wd 0.0500 time 0.3476 (0.3531) data time 0.0008 (0.0021) model time 0.3468 (0.3518) loss 3.7458 (3.3009) grad_norm 3.2462 (inf) loss_scale 1024.0000 (1674.9884) mem 11224MB [2024-08-26 11:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][440/625] eta 0:01:05 lr 0.001452 wd 0.0500 time 0.3420 (0.3530) data time 0.0006 (0.0021) model time 0.3414 (0.3517) loss 2.4898 (3.3085) grad_norm 1.3463 (inf) loss_scale 1024.0000 (1660.2268) mem 11224MB [2024-08-26 11:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][450/625] eta 0:01:01 lr 0.001452 wd 0.0500 time 0.3434 (0.3529) data time 0.0009 (0.0020) model time 0.3425 (0.3516) loss 2.9858 (3.3100) grad_norm 1.0084 (inf) loss_scale 1024.0000 (1646.1197) mem 11224MB [2024-08-26 11:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][460/625] eta 0:00:58 lr 0.001452 wd 0.0500 time 0.3432 (0.3527) data time 0.0008 (0.0020) model time 0.3424 (0.3514) loss 3.3866 (3.3159) grad_norm 1.0495 (inf) loss_scale 1024.0000 (1632.6247) mem 11224MB [2024-08-26 11:04:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][470/625] eta 0:00:54 lr 0.001452 wd 0.0500 time 0.3478 (0.3526) data time 0.0006 (0.0020) model time 0.3472 (0.3513) loss 2.9006 (3.3154) grad_norm 1.2104 (inf) loss_scale 1024.0000 (1619.7028) mem 11224MB [2024-08-26 11:04:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][480/625] eta 0:00:51 lr 0.001452 wd 0.0500 time 0.3487 (0.3524) data time 0.0006 (0.0020) model time 0.3481 (0.3511) loss 3.5705 (3.3119) grad_norm 1.4022 (inf) loss_scale 1024.0000 (1607.3181) mem 11224MB [2024-08-26 11:04:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][490/625] eta 0:00:47 lr 0.001452 wd 0.0500 time 0.5651 (0.3527) data time 0.0008 (0.0019) model time 0.5642 (0.3515) loss 3.2802 (3.3136) grad_norm 2.0898 (inf) loss_scale 1024.0000 (1595.4379) mem 11224MB [2024-08-26 11:04:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][500/625] eta 0:00:44 lr 0.001452 wd 0.0500 time 0.3444 (0.3537) data time 0.0007 (0.0019) model time 0.3438 (0.3526) loss 4.0453 (3.3162) grad_norm 1.4741 (inf) loss_scale 1024.0000 (1584.0319) mem 11224MB [2024-08-26 11:04:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][510/625] eta 0:00:40 lr 0.001451 wd 0.0500 time 0.3396 (0.3536) data time 0.0009 (0.0019) model time 0.3386 (0.3524) loss 3.4075 (3.3209) grad_norm 1.5543 (inf) loss_scale 1024.0000 (1573.0724) mem 11224MB [2024-08-26 11:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][520/625] eta 0:00:37 lr 0.001451 wd 0.0500 time 0.3400 (0.3534) data time 0.0008 (0.0019) model time 0.3392 (0.3522) loss 3.4310 (3.3235) grad_norm 1.1554 (inf) loss_scale 1024.0000 (1562.5336) mem 11224MB [2024-08-26 11:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][530/625] eta 0:00:33 lr 0.001451 wd 0.0500 time 0.3425 (0.3532) data time 0.0008 (0.0019) model time 0.3418 (0.3520) loss 3.8105 (3.3243) grad_norm 0.9246 (inf) loss_scale 1024.0000 (1552.3917) mem 11224MB [2024-08-26 11:04:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][540/625] eta 0:00:30 lr 0.001451 wd 0.0500 time 0.3413 (0.3530) data time 0.0007 (0.0018) model time 0.3406 (0.3519) loss 4.1054 (3.3246) grad_norm 0.9252 (inf) loss_scale 1024.0000 (1542.6248) mem 11224MB [2024-08-26 11:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][550/625] eta 0:00:26 lr 0.001451 wd 0.0500 time 0.3454 (0.3530) data time 0.0007 (0.0018) model time 0.3447 (0.3517) loss 2.8931 (3.3265) grad_norm 1.2735 (inf) loss_scale 1024.0000 (1533.2123) mem 11224MB [2024-08-26 11:04:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][560/625] eta 0:00:22 lr 0.001451 wd 0.0500 time 0.3519 (0.3528) data time 0.0006 (0.0018) model time 0.3512 (0.3516) loss 3.0660 (3.3247) grad_norm 2.0059 (inf) loss_scale 1024.0000 (1524.1355) mem 11224MB [2024-08-26 11:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][570/625] eta 0:00:19 lr 0.001450 wd 0.0500 time 0.3473 (0.3530) data time 0.0006 (0.0018) model time 0.3467 (0.3518) loss 2.3777 (3.3228) grad_norm 1.3087 (inf) loss_scale 1024.0000 (1515.3765) mem 11224MB [2024-08-26 11:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][580/625] eta 0:00:15 lr 0.001450 wd 0.0500 time 0.3400 (0.3529) data time 0.0006 (0.0018) model time 0.3394 (0.3517) loss 2.8583 (3.3209) grad_norm 1.3333 (inf) loss_scale 1024.0000 (1506.9191) mem 11224MB [2024-08-26 11:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][590/625] eta 0:00:12 lr 0.001450 wd 0.0500 time 0.3678 (0.3529) data time 0.0009 (0.0018) model time 0.3669 (0.3517) loss 3.6209 (3.3226) grad_norm 1.2615 (inf) loss_scale 1024.0000 (1498.7479) mem 11224MB [2024-08-26 11:05:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][600/625] eta 0:00:08 lr 0.001450 wd 0.0500 time 0.3458 (0.3528) data time 0.0009 (0.0018) model time 0.3449 (0.3516) loss 3.2300 (3.3237) grad_norm 1.9588 (inf) loss_scale 1024.0000 (1490.8486) mem 11224MB [2024-08-26 11:05:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][610/625] eta 0:00:05 lr 0.001450 wd 0.0500 time 0.3383 (0.3528) data time 0.0006 (0.0018) model time 0.3378 (0.3516) loss 2.9909 (3.3250) grad_norm 1.1477 (inf) loss_scale 1024.0000 (1483.2079) mem 11224MB [2024-08-26 11:05:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [118/300][620/625] eta 0:00:01 lr 0.001450 wd 0.0500 time 0.3415 (0.3533) data time 0.0005 (0.0018) model time 0.3409 (0.3521) loss 2.5482 (3.3288) grad_norm 0.9345 (inf) loss_scale 1024.0000 (1475.8132) mem 11224MB [2024-08-26 11:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 118 training takes 0:03:40 [2024-08-26 11:05:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:05:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:05:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.446 (0.446) Loss 0.6509 (0.6509) Acc@1 86.572 (86.572) Acc@5 97.607 (97.607) Mem 11224MB [2024-08-26 11:05:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.130) Loss 1.0264 (0.7866) Acc@1 75.342 (82.520) Acc@5 94.043 (96.507) Mem 11224MB [2024-08-26 11:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.2246 (0.9464) Acc@1 71.680 (78.644) Acc@5 92.041 (94.629) Mem 11224MB [2024-08-26 11:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.399 Acc@5 94.622 [2024-08-26 11:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.4% [2024-08-26 11:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.746 (0.746) Loss 0.5332 (0.5332) Acc@1 88.086 (88.086) Acc@5 98.340 (98.340) Mem 11224MB [2024-08-26 11:05:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.161) Loss 0.8960 (0.6689) Acc@1 78.809 (84.801) Acc@5 95.410 (97.346) Mem 11224MB [2024-08-26 11:05:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.128) Loss 1.0322 (0.8072) Acc@1 74.561 (81.308) Acc@5 93.604 (95.836) Mem 11224MB [2024-08-26 11:05:15 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.028 Acc@5 95.815 [2024-08-26 11:05:15 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.0% [2024-08-26 11:05:15 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.03% [2024-08-26 11:05:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:05:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:05:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][0/625] eta 0:07:08 lr 0.001450 wd 0.0500 time 0.6854 (0.6854) data time 0.3646 (0.3646) model time 0.0000 (0.0000) loss 2.7642 (2.7642) grad_norm 1.7624 (1.7624) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][10/625] eta 0:03:58 lr 0.001449 wd 0.0500 time 0.3648 (0.3870) data time 0.0006 (0.0348) model time 0.0000 (0.0000) loss 3.9068 (3.3234) grad_norm 1.3312 (1.2363) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][20/625] eta 0:03:44 lr 0.001449 wd 0.0500 time 0.3452 (0.3715) data time 0.0006 (0.0187) model time 0.0000 (0.0000) loss 3.7100 (3.3823) grad_norm 1.0947 (1.3242) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][30/625] eta 0:03:37 lr 0.001449 wd 0.0500 time 0.3451 (0.3657) data time 0.0008 (0.0135) model time 0.0000 (0.0000) loss 2.9695 (3.2556) grad_norm 1.0401 (1.3164) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][40/625] eta 0:03:32 lr 0.001449 wd 0.0500 time 0.3541 (0.3638) data time 0.0006 (0.0104) model time 0.0000 (0.0000) loss 2.2875 (3.2860) grad_norm 1.6295 (1.2963) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][50/625] eta 0:03:27 lr 0.001449 wd 0.0500 time 0.3397 (0.3610) data time 0.0008 (0.0086) model time 0.0000 (0.0000) loss 3.2270 (3.3142) grad_norm 2.0792 (1.3823) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][60/625] eta 0:03:26 lr 0.001449 wd 0.0500 time 0.5486 (0.3654) data time 0.0006 (0.0073) model time 0.5481 (0.3870) loss 3.4473 (3.3006) grad_norm 1.0484 (1.3999) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][70/625] eta 0:03:21 lr 0.001448 wd 0.0500 time 0.3456 (0.3629) data time 0.0008 (0.0064) model time 0.3447 (0.3669) loss 3.5130 (3.2549) grad_norm 1.4746 (1.3957) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][80/625] eta 0:03:16 lr 0.001448 wd 0.0500 time 0.3394 (0.3609) data time 0.0007 (0.0059) model time 0.3387 (0.3595) loss 2.3150 (3.2782) grad_norm 1.3977 (1.3845) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][90/625] eta 0:03:12 lr 0.001448 wd 0.0500 time 0.3475 (0.3593) data time 0.0006 (0.0054) model time 0.3468 (0.3560) loss 2.9397 (3.2468) grad_norm 1.4575 (1.3692) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][100/625] eta 0:03:08 lr 0.001448 wd 0.0500 time 0.3574 (0.3583) data time 0.0006 (0.0050) model time 0.3568 (0.3543) loss 4.0327 (3.2704) grad_norm 1.1512 (1.3775) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][110/625] eta 0:03:03 lr 0.001448 wd 0.0500 time 0.3460 (0.3570) data time 0.0007 (0.0046) model time 0.3453 (0.3524) loss 2.9414 (3.2474) grad_norm 1.2106 (1.3962) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][120/625] eta 0:02:59 lr 0.001448 wd 0.0500 time 0.3413 (0.3559) data time 0.0008 (0.0043) model time 0.3405 (0.3511) loss 3.6845 (3.2549) grad_norm 1.7942 (1.4247) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][130/625] eta 0:02:55 lr 0.001447 wd 0.0500 time 0.3683 (0.3555) data time 0.0006 (0.0041) model time 0.3677 (0.3508) loss 3.9987 (3.2414) grad_norm 1.4957 (1.4480) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][140/625] eta 0:02:52 lr 0.001447 wd 0.0500 time 0.3396 (0.3547) data time 0.0007 (0.0039) model time 0.3389 (0.3499) loss 3.7328 (3.2449) grad_norm 1.1934 (1.4317) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][150/625] eta 0:02:48 lr 0.001447 wd 0.0500 time 0.3433 (0.3547) data time 0.0007 (0.0037) model time 0.3427 (0.3503) loss 3.2966 (3.2395) grad_norm 1.0367 (1.4174) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][160/625] eta 0:02:45 lr 0.001447 wd 0.0500 time 0.3977 (0.3552) data time 0.0006 (0.0035) model time 0.3970 (0.3514) loss 4.0570 (3.2452) grad_norm 1.1431 (1.4008) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][170/625] eta 0:02:41 lr 0.001447 wd 0.0500 time 0.3370 (0.3550) data time 0.0007 (0.0034) model time 0.3363 (0.3512) loss 3.5659 (3.2574) grad_norm 1.2528 (1.3887) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][180/625] eta 0:02:37 lr 0.001447 wd 0.0500 time 0.3438 (0.3548) data time 0.0009 (0.0033) model time 0.3429 (0.3511) loss 2.8959 (3.2472) grad_norm 1.7841 (1.3832) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][190/625] eta 0:02:34 lr 0.001446 wd 0.0500 time 0.3723 (0.3545) data time 0.0007 (0.0032) model time 0.3716 (0.3509) loss 3.4173 (3.2436) grad_norm 1.0310 (1.3831) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][200/625] eta 0:02:30 lr 0.001446 wd 0.0500 time 0.3426 (0.3551) data time 0.0007 (0.0031) model time 0.3419 (0.3519) loss 3.5214 (3.2367) grad_norm 1.5419 (1.3938) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][210/625] eta 0:02:27 lr 0.001446 wd 0.0500 time 0.3412 (0.3560) data time 0.0010 (0.0030) model time 0.3402 (0.3532) loss 3.4759 (3.2456) grad_norm 1.1670 (1.3847) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][220/625] eta 0:02:24 lr 0.001446 wd 0.0500 time 0.4091 (0.3563) data time 0.0007 (0.0030) model time 0.4084 (0.3536) loss 3.8081 (3.2594) grad_norm 1.2832 (1.3793) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][230/625] eta 0:02:20 lr 0.001446 wd 0.0500 time 0.3414 (0.3562) data time 0.0006 (0.0030) model time 0.3408 (0.3535) loss 2.4440 (3.2625) grad_norm 1.4575 (1.3741) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][240/625] eta 0:02:17 lr 0.001446 wd 0.0500 time 0.3446 (0.3569) data time 0.0009 (0.0030) model time 0.3438 (0.3544) loss 4.0635 (3.2651) grad_norm 1.2548 (1.3784) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][250/625] eta 0:02:14 lr 0.001446 wd 0.0500 time 0.3651 (0.3583) data time 0.0007 (0.0029) model time 0.3644 (0.3562) loss 3.6870 (3.2611) grad_norm 0.9697 (1.3747) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][260/625] eta 0:02:10 lr 0.001445 wd 0.0500 time 0.3393 (0.3580) data time 0.0007 (0.0028) model time 0.3386 (0.3558) loss 3.7937 (3.2585) grad_norm 2.0125 (1.4088) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][270/625] eta 0:02:06 lr 0.001445 wd 0.0500 time 0.3427 (0.3576) data time 0.0009 (0.0028) model time 0.3418 (0.3554) loss 3.3109 (3.2620) grad_norm 1.1858 (1.4117) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][280/625] eta 0:02:03 lr 0.001445 wd 0.0500 time 0.3553 (0.3575) data time 0.0006 (0.0028) model time 0.3547 (0.3552) loss 3.6061 (3.2636) grad_norm 1.4184 (1.4099) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][290/625] eta 0:01:59 lr 0.001445 wd 0.0500 time 0.3323 (0.3571) data time 0.0009 (0.0027) model time 0.3315 (0.3548) loss 3.4764 (3.2713) grad_norm 1.2867 (1.4076) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][300/625] eta 0:01:55 lr 0.001445 wd 0.0500 time 0.3442 (0.3567) data time 0.0008 (0.0026) model time 0.3434 (0.3544) loss 3.0675 (3.2840) grad_norm 1.1577 (1.4022) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][310/625] eta 0:01:52 lr 0.001445 wd 0.0500 time 0.3625 (0.3564) data time 0.0009 (0.0026) model time 0.3616 (0.3541) loss 3.2187 (3.2829) grad_norm 1.1023 (1.4001) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][320/625] eta 0:01:48 lr 0.001444 wd 0.0500 time 0.3422 (0.3561) data time 0.0008 (0.0025) model time 0.3414 (0.3538) loss 3.3843 (3.2709) grad_norm 1.5647 (1.4051) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][330/625] eta 0:01:45 lr 0.001444 wd 0.0500 time 0.3473 (0.3560) data time 0.0008 (0.0025) model time 0.3465 (0.3537) loss 3.9555 (3.2718) grad_norm 1.3306 (1.4063) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][340/625] eta 0:01:41 lr 0.001444 wd 0.0500 time 0.3429 (0.3558) data time 0.0007 (0.0024) model time 0.3422 (0.3536) loss 3.2845 (3.2657) grad_norm 1.7715 (1.4090) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][350/625] eta 0:01:37 lr 0.001444 wd 0.0500 time 0.3480 (0.3556) data time 0.0006 (0.0024) model time 0.3473 (0.3534) loss 3.3382 (3.2634) grad_norm 1.3412 (1.4056) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][360/625] eta 0:01:34 lr 0.001444 wd 0.0500 time 0.3462 (0.3560) data time 0.0006 (0.0024) model time 0.3456 (0.3538) loss 3.5985 (3.2590) grad_norm 1.5322 (1.4022) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][370/625] eta 0:01:30 lr 0.001444 wd 0.0500 time 0.3747 (0.3566) data time 0.0007 (0.0023) model time 0.3741 (0.3546) loss 2.8588 (3.2590) grad_norm 1.6788 (1.4087) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][380/625] eta 0:01:27 lr 0.001443 wd 0.0500 time 0.3512 (0.3565) data time 0.0007 (0.0024) model time 0.3505 (0.3544) loss 3.1513 (3.2645) grad_norm 1.2933 (1.4079) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][390/625] eta 0:01:23 lr 0.001443 wd 0.0500 time 0.3636 (0.3573) data time 0.0009 (0.0024) model time 0.3627 (0.3554) loss 2.7755 (3.2658) grad_norm 1.3761 (1.4032) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][400/625] eta 0:01:20 lr 0.001443 wd 0.0500 time 0.3411 (0.3572) data time 0.0006 (0.0024) model time 0.3405 (0.3552) loss 3.7732 (3.2649) grad_norm 1.2885 (1.4010) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][410/625] eta 0:01:16 lr 0.001443 wd 0.0500 time 0.3432 (0.3571) data time 0.0010 (0.0024) model time 0.3422 (0.3551) loss 2.1632 (3.2658) grad_norm 2.5732 (1.4050) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][420/625] eta 0:01:13 lr 0.001443 wd 0.0500 time 0.3802 (0.3572) data time 0.0006 (0.0024) model time 0.3796 (0.3551) loss 3.9188 (3.2731) grad_norm 1.9194 (1.4103) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][430/625] eta 0:01:09 lr 0.001443 wd 0.0500 time 0.3416 (0.3575) data time 0.0008 (0.0023) model time 0.3408 (0.3556) loss 3.7883 (3.2765) grad_norm 1.2728 (1.4068) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][440/625] eta 0:01:06 lr 0.001443 wd 0.0500 time 0.3458 (0.3574) data time 0.0007 (0.0023) model time 0.3452 (0.3554) loss 3.9005 (3.2754) grad_norm 2.3816 (1.4126) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:07:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][450/625] eta 0:01:02 lr 0.001442 wd 0.0500 time 0.3633 (0.3574) data time 0.0007 (0.0023) model time 0.3626 (0.3554) loss 3.3318 (3.2741) grad_norm 1.5922 (1.4145) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][460/625] eta 0:00:58 lr 0.001442 wd 0.0500 time 0.3419 (0.3573) data time 0.0009 (0.0023) model time 0.3410 (0.3553) loss 3.8089 (3.2795) grad_norm 1.2843 (1.4174) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][470/625] eta 0:00:55 lr 0.001442 wd 0.0500 time 0.3541 (0.3572) data time 0.0007 (0.0023) model time 0.3535 (0.3552) loss 3.8049 (3.2821) grad_norm 1.2493 (1.4168) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][480/625] eta 0:00:51 lr 0.001442 wd 0.0500 time 0.3675 (0.3570) data time 0.0007 (0.0023) model time 0.3668 (0.3550) loss 3.2601 (3.2775) grad_norm 1.6614 (1.4167) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][490/625] eta 0:00:48 lr 0.001442 wd 0.0500 time 0.3442 (0.3567) data time 0.0009 (0.0023) model time 0.3433 (0.3547) loss 3.3413 (3.2778) grad_norm 1.3663 (1.4158) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][500/625] eta 0:00:44 lr 0.001442 wd 0.0500 time 0.3552 (0.3566) data time 0.0007 (0.0022) model time 0.3546 (0.3546) loss 3.6158 (3.2821) grad_norm 1.3688 (1.4144) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][510/625] eta 0:00:41 lr 0.001441 wd 0.0500 time 0.3471 (0.3568) data time 0.0010 (0.0022) model time 0.3461 (0.3548) loss 2.4911 (3.2841) grad_norm 0.8736 (1.4143) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][520/625] eta 0:00:37 lr 0.001441 wd 0.0500 time 0.3444 (0.3566) data time 0.0007 (0.0022) model time 0.3437 (0.3546) loss 2.6509 (3.2803) grad_norm 1.3898 (1.4149) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][530/625] eta 0:00:33 lr 0.001441 wd 0.0500 time 0.3489 (0.3563) data time 0.0008 (0.0022) model time 0.3481 (0.3544) loss 3.1789 (3.2805) grad_norm 1.2039 (1.4182) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][540/625] eta 0:00:30 lr 0.001441 wd 0.0500 time 0.3486 (0.3562) data time 0.0008 (0.0021) model time 0.3477 (0.3542) loss 3.4687 (3.2808) grad_norm 1.0837 (1.4165) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][550/625] eta 0:00:26 lr 0.001441 wd 0.0500 time 0.3425 (0.3564) data time 0.0007 (0.0021) model time 0.3418 (0.3544) loss 2.3805 (3.2792) grad_norm 1.2843 (1.4185) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][560/625] eta 0:00:23 lr 0.001441 wd 0.0500 time 0.3489 (0.3562) data time 0.0008 (0.0021) model time 0.3481 (0.3543) loss 3.6874 (3.2792) grad_norm 2.0345 (1.4208) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][570/625] eta 0:00:19 lr 0.001440 wd 0.0500 time 0.3610 (0.3565) data time 0.0006 (0.0021) model time 0.3604 (0.3546) loss 3.8898 (3.2833) grad_norm 0.8000 (1.4201) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][580/625] eta 0:00:16 lr 0.001440 wd 0.0500 time 0.3458 (0.3568) data time 0.0007 (0.0021) model time 0.3451 (0.3549) loss 3.1626 (3.2823) grad_norm 1.6374 (1.4206) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][590/625] eta 0:00:12 lr 0.001440 wd 0.0500 time 0.3397 (0.3566) data time 0.0008 (0.0020) model time 0.3390 (0.3548) loss 3.8628 (3.2816) grad_norm 1.0270 (1.4180) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][600/625] eta 0:00:08 lr 0.001440 wd 0.0500 time 0.3714 (0.3566) data time 0.0009 (0.0020) model time 0.3705 (0.3547) loss 3.2917 (3.2792) grad_norm 2.0769 (1.4240) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][610/625] eta 0:00:05 lr 0.001440 wd 0.0500 time 0.3332 (0.3565) data time 0.0004 (0.0020) model time 0.3328 (0.3546) loss 2.3223 (3.2804) grad_norm 1.6335 (1.4248) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [119/300][620/625] eta 0:00:01 lr 0.001440 wd 0.0500 time 0.3336 (0.3563) data time 0.0006 (0.0020) model time 0.3330 (0.3544) loss 3.8164 (3.2828) grad_norm 1.4081 (1.4249) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 119 training takes 0:03:42 [2024-08-26 11:08:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:09:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:09:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.464 (0.464) Loss 0.6411 (0.6411) Acc@1 86.475 (86.475) Acc@5 97.656 (97.656) Mem 11224MB [2024-08-26 11:09:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.134) Loss 1.0537 (0.7791) Acc@1 75.098 (82.684) Acc@5 93.604 (96.533) Mem 11224MB [2024-08-26 11:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.114) Loss 1.1084 (0.9363) Acc@1 74.268 (78.967) Acc@5 92.920 (94.661) Mem 11224MB [2024-08-26 11:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.683 Acc@5 94.664 [2024-08-26 11:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.7% [2024-08-26 11:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.856 (0.856) Loss 0.5332 (0.5332) Acc@1 88.232 (88.232) Acc@5 98.340 (98.340) Mem 11224MB [2024-08-26 11:09:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.170) Loss 0.8945 (0.6684) Acc@1 79.150 (84.908) Acc@5 95.459 (97.363) Mem 11224MB [2024-08-26 11:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.134) Loss 1.0293 (0.8061) Acc@1 74.609 (81.383) Acc@5 93.652 (95.871) Mem 11224MB [2024-08-26 11:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.096 Acc@5 95.843 [2024-08-26 11:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.1% [2024-08-26 11:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.10% [2024-08-26 11:09:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:09:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:09:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][0/625] eta 0:07:01 lr 0.001440 wd 0.0500 time 0.6742 (0.6742) data time 0.3556 (0.3556) model time 0.0000 (0.0000) loss 3.0876 (3.0876) grad_norm 1.3291 (1.3291) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][10/625] eta 0:03:59 lr 0.001439 wd 0.0500 time 0.4005 (0.3896) data time 0.0007 (0.0338) model time 0.0000 (0.0000) loss 2.9965 (3.0400) grad_norm 1.2397 (1.6041) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][20/625] eta 0:03:43 lr 0.001439 wd 0.0500 time 0.3381 (0.3688) data time 0.0009 (0.0187) model time 0.0000 (0.0000) loss 3.9668 (3.2097) grad_norm 1.2678 (1.4422) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][30/625] eta 0:03:34 lr 0.001439 wd 0.0500 time 0.3397 (0.3612) data time 0.0007 (0.0130) model time 0.0000 (0.0000) loss 3.1742 (3.3067) grad_norm 1.4340 (1.4007) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][40/625] eta 0:03:28 lr 0.001439 wd 0.0500 time 0.3447 (0.3572) data time 0.0009 (0.0101) model time 0.0000 (0.0000) loss 3.2494 (3.3075) grad_norm 1.2520 (1.4104) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][50/625] eta 0:03:25 lr 0.001439 wd 0.0500 time 0.3449 (0.3578) data time 0.0007 (0.0083) model time 0.0000 (0.0000) loss 2.4799 (3.3006) grad_norm 1.6901 (1.3907) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][60/625] eta 0:03:21 lr 0.001439 wd 0.0500 time 0.3451 (0.3560) data time 0.0006 (0.0071) model time 0.3445 (0.3459) loss 3.0301 (3.2659) grad_norm 1.1220 (1.4172) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][70/625] eta 0:03:18 lr 0.001438 wd 0.0500 time 0.3456 (0.3574) data time 0.0009 (0.0062) model time 0.3448 (0.3554) loss 3.4098 (3.3123) grad_norm 1.3009 (1.4568) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][80/625] eta 0:03:14 lr 0.001438 wd 0.0500 time 0.3419 (0.3563) data time 0.0006 (0.0056) model time 0.3413 (0.3528) loss 2.7521 (3.2974) grad_norm 3.0025 (1.4555) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][90/625] eta 0:03:10 lr 0.001438 wd 0.0500 time 0.3453 (0.3552) data time 0.0006 (0.0051) model time 0.3447 (0.3511) loss 3.9964 (3.3144) grad_norm 1.5165 (1.4529) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][100/625] eta 0:03:06 lr 0.001438 wd 0.0500 time 0.3565 (0.3546) data time 0.0008 (0.0046) model time 0.3557 (0.3504) loss 2.7194 (3.3339) grad_norm 1.1589 (1.4451) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][110/625] eta 0:03:02 lr 0.001438 wd 0.0500 time 0.3455 (0.3537) data time 0.0006 (0.0043) model time 0.3449 (0.3494) loss 2.5772 (3.3437) grad_norm 1.5222 (1.4473) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][120/625] eta 0:02:58 lr 0.001438 wd 0.0500 time 0.3388 (0.3530) data time 0.0009 (0.0040) model time 0.3378 (0.3486) loss 3.3346 (3.3174) grad_norm 1.3186 (1.4454) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][130/625] eta 0:02:54 lr 0.001437 wd 0.0500 time 0.3539 (0.3524) data time 0.0009 (0.0038) model time 0.3530 (0.3481) loss 3.3974 (3.3121) grad_norm 0.9162 (1.4347) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:09:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][140/625] eta 0:02:50 lr 0.001437 wd 0.0500 time 0.3442 (0.3519) data time 0.0008 (0.0036) model time 0.3434 (0.3476) loss 2.7211 (3.2995) grad_norm 1.3066 (1.4119) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][150/625] eta 0:02:48 lr 0.001437 wd 0.0500 time 0.5701 (0.3553) data time 0.0007 (0.0035) model time 0.5694 (0.3531) loss 3.8220 (3.3046) grad_norm 1.7051 (1.4036) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][160/625] eta 0:02:45 lr 0.001437 wd 0.0500 time 0.3451 (0.3560) data time 0.0007 (0.0033) model time 0.3445 (0.3542) loss 2.9925 (3.3067) grad_norm 2.3329 (1.4108) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][170/625] eta 0:02:41 lr 0.001437 wd 0.0500 time 0.3413 (0.3553) data time 0.0009 (0.0032) model time 0.3403 (0.3532) loss 3.6343 (3.3002) grad_norm 1.2745 (1.4186) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][180/625] eta 0:02:37 lr 0.001437 wd 0.0500 time 0.3418 (0.3547) data time 0.0009 (0.0031) model time 0.3409 (0.3525) loss 3.2233 (3.3022) grad_norm 1.1460 (1.4164) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][190/625] eta 0:02:34 lr 0.001437 wd 0.0500 time 0.3415 (0.3542) data time 0.0009 (0.0029) model time 0.3407 (0.3518) loss 3.8287 (3.3117) grad_norm 1.2905 (1.4102) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][200/625] eta 0:02:30 lr 0.001436 wd 0.0500 time 0.3441 (0.3537) data time 0.0006 (0.0028) model time 0.3435 (0.3513) loss 3.9334 (3.3062) grad_norm 1.0796 (1.4017) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][210/625] eta 0:02:26 lr 0.001436 wd 0.0500 time 0.3451 (0.3533) data time 0.0007 (0.0027) model time 0.3444 (0.3509) loss 3.4603 (3.3161) grad_norm 2.7671 (1.4369) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][220/625] eta 0:02:22 lr 0.001436 wd 0.0500 time 0.3393 (0.3530) data time 0.0009 (0.0027) model time 0.3384 (0.3506) loss 3.1297 (3.3167) grad_norm 1.5577 (1.4472) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][230/625] eta 0:02:19 lr 0.001436 wd 0.0500 time 0.3428 (0.3526) data time 0.0008 (0.0026) model time 0.3421 (0.3502) loss 2.2169 (3.3258) grad_norm 1.3627 (1.4380) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][240/625] eta 0:02:15 lr 0.001436 wd 0.0500 time 0.3431 (0.3524) data time 0.0007 (0.0025) model time 0.3424 (0.3500) loss 2.2017 (3.3233) grad_norm 1.0628 (1.4290) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][250/625] eta 0:02:12 lr 0.001436 wd 0.0500 time 0.3518 (0.3522) data time 0.0008 (0.0025) model time 0.3509 (0.3499) loss 3.4735 (3.3272) grad_norm 1.6075 (1.4317) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][260/625] eta 0:02:08 lr 0.001435 wd 0.0500 time 0.3422 (0.3534) data time 0.0006 (0.0024) model time 0.3415 (0.3513) loss 3.7455 (3.3353) grad_norm 1.2506 (1.4277) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][270/625] eta 0:02:05 lr 0.001435 wd 0.0500 time 0.3461 (0.3531) data time 0.0006 (0.0023) model time 0.3455 (0.3511) loss 3.9752 (3.3395) grad_norm 1.2654 (1.4214) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][280/625] eta 0:02:01 lr 0.001435 wd 0.0500 time 0.3478 (0.3528) data time 0.0009 (0.0023) model time 0.3468 (0.3507) loss 3.5716 (3.3389) grad_norm 1.0889 (1.4149) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][290/625] eta 0:01:58 lr 0.001435 wd 0.0500 time 0.3485 (0.3526) data time 0.0007 (0.0023) model time 0.3478 (0.3505) loss 3.9293 (3.3419) grad_norm 2.0327 (1.4100) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][300/625] eta 0:01:54 lr 0.001435 wd 0.0500 time 0.3435 (0.3523) data time 0.0007 (0.0022) model time 0.3428 (0.3503) loss 2.4561 (3.3424) grad_norm 1.4803 (1.4124) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:10:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][310/625] eta 0:01:50 lr 0.001435 wd 0.0500 time 0.3484 (0.3523) data time 0.0008 (0.0022) model time 0.3475 (0.3503) loss 3.3473 (3.3421) grad_norm 1.1020 (1.4135) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][320/625] eta 0:01:47 lr 0.001434 wd 0.0500 time 0.3399 (0.3522) data time 0.0009 (0.0022) model time 0.3390 (0.3501) loss 3.8128 (3.3517) grad_norm 1.3472 (1.4144) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][330/625] eta 0:01:44 lr 0.001434 wd 0.0500 time 0.5541 (0.3531) data time 0.0006 (0.0021) model time 0.5535 (0.3513) loss 3.4266 (3.3611) grad_norm 2.0529 (1.4152) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][340/625] eta 0:01:40 lr 0.001434 wd 0.0500 time 0.3460 (0.3535) data time 0.0009 (0.0021) model time 0.3452 (0.3517) loss 3.3921 (3.3607) grad_norm 4.1302 (1.4232) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][350/625] eta 0:01:37 lr 0.001434 wd 0.0500 time 0.3447 (0.3532) data time 0.0006 (0.0021) model time 0.3440 (0.3514) loss 3.9487 (3.3657) grad_norm 1.4928 (1.4357) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][360/625] eta 0:01:33 lr 0.001434 wd 0.0500 time 0.3440 (0.3530) data time 0.0009 (0.0020) model time 0.3431 (0.3512) loss 3.5628 (3.3660) grad_norm 1.3500 (1.4363) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][370/625] eta 0:01:30 lr 0.001434 wd 0.0500 time 0.5319 (0.3533) data time 0.0009 (0.0020) model time 0.5310 (0.3516) loss 3.6228 (3.3605) grad_norm 1.7586 (1.4324) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][380/625] eta 0:01:26 lr 0.001433 wd 0.0500 time 0.3464 (0.3530) data time 0.0007 (0.0020) model time 0.3457 (0.3512) loss 2.1697 (3.3566) grad_norm 0.8180 (1.4259) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][390/625] eta 0:01:22 lr 0.001433 wd 0.0500 time 0.3410 (0.3528) data time 0.0008 (0.0020) model time 0.3402 (0.3511) loss 3.1622 (3.3560) grad_norm 1.1860 (1.4285) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][400/625] eta 0:01:19 lr 0.001433 wd 0.0500 time 0.3438 (0.3526) data time 0.0007 (0.0019) model time 0.3431 (0.3508) loss 4.3473 (3.3512) grad_norm 1.6090 (1.4280) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][410/625] eta 0:01:15 lr 0.001433 wd 0.0500 time 0.3436 (0.3524) data time 0.0009 (0.0019) model time 0.3427 (0.3506) loss 3.2043 (3.3496) grad_norm 0.9001 (1.4269) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][420/625] eta 0:01:12 lr 0.001433 wd 0.0500 time 0.3499 (0.3522) data time 0.0007 (0.0019) model time 0.3493 (0.3504) loss 2.5492 (3.3500) grad_norm 1.7368 (1.4307) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][430/625] eta 0:01:08 lr 0.001433 wd 0.0500 time 0.3384 (0.3520) data time 0.0007 (0.0019) model time 0.3377 (0.3503) loss 3.4132 (3.3524) grad_norm 1.7862 (1.4277) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][440/625] eta 0:01:05 lr 0.001432 wd 0.0500 time 0.3406 (0.3519) data time 0.0008 (0.0018) model time 0.3398 (0.3501) loss 3.2798 (3.3528) grad_norm 1.2314 (1.4254) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][450/625] eta 0:01:01 lr 0.001432 wd 0.0500 time 0.3449 (0.3531) data time 0.0010 (0.0018) model time 0.3439 (0.3515) loss 3.6610 (3.3441) grad_norm 1.0562 (1.4219) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][460/625] eta 0:00:58 lr 0.001432 wd 0.0500 time 0.3441 (0.3534) data time 0.0009 (0.0018) model time 0.3432 (0.3518) loss 3.6205 (3.3461) grad_norm 1.2647 (1.4198) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][470/625] eta 0:00:54 lr 0.001432 wd 0.0500 time 0.3442 (0.3532) data time 0.0006 (0.0018) model time 0.3436 (0.3516) loss 2.3404 (3.3443) grad_norm 1.3270 (1.4162) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:11:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][480/625] eta 0:00:51 lr 0.001432 wd 0.0500 time 0.3447 (0.3530) data time 0.0007 (0.0018) model time 0.3440 (0.3514) loss 2.7203 (3.3420) grad_norm 1.3615 (1.4170) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][490/625] eta 0:00:47 lr 0.001432 wd 0.0500 time 0.3467 (0.3528) data time 0.0008 (0.0017) model time 0.3460 (0.3512) loss 2.7204 (3.3337) grad_norm 1.4090 (1.4144) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][500/625] eta 0:00:44 lr 0.001432 wd 0.0500 time 0.3454 (0.3527) data time 0.0008 (0.0018) model time 0.3446 (0.3511) loss 3.4689 (3.3292) grad_norm 1.2945 (1.4122) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][510/625] eta 0:00:40 lr 0.001431 wd 0.0500 time 0.3418 (0.3525) data time 0.0009 (0.0017) model time 0.3409 (0.3510) loss 3.7675 (3.3270) grad_norm 1.7138 (1.4124) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][520/625] eta 0:00:37 lr 0.001431 wd 0.0500 time 0.3446 (0.3525) data time 0.0007 (0.0017) model time 0.3439 (0.3509) loss 2.8801 (3.3246) grad_norm 1.5508 (1.4124) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][530/625] eta 0:00:33 lr 0.001431 wd 0.0500 time 0.3497 (0.3523) data time 0.0007 (0.0017) model time 0.3490 (0.3507) loss 3.7164 (3.3262) grad_norm 1.1331 (1.4125) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][540/625] eta 0:00:29 lr 0.001431 wd 0.0500 time 0.3431 (0.3522) data time 0.0008 (0.0017) model time 0.3423 (0.3506) loss 3.6810 (3.3266) grad_norm 1.1747 (1.4111) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][550/625] eta 0:00:26 lr 0.001431 wd 0.0500 time 0.3448 (0.3521) data time 0.0006 (0.0017) model time 0.3442 (0.3505) loss 3.1719 (3.3254) grad_norm 1.3242 (1.4128) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][560/625] eta 0:00:22 lr 0.001431 wd 0.0500 time 0.3567 (0.3520) data time 0.0007 (0.0017) model time 0.3561 (0.3505) loss 3.8551 (3.3303) grad_norm 1.4734 (1.4116) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][570/625] eta 0:00:19 lr 0.001430 wd 0.0500 time 0.3447 (0.3526) data time 0.0006 (0.0017) model time 0.3441 (0.3510) loss 3.8152 (3.3332) grad_norm 1.9904 (1.4132) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][580/625] eta 0:00:15 lr 0.001430 wd 0.0500 time 0.5020 (0.3528) data time 0.0007 (0.0017) model time 0.5013 (0.3512) loss 3.1288 (3.3320) grad_norm 1.4118 (1.4136) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][590/625] eta 0:00:12 lr 0.001430 wd 0.0500 time 0.3425 (0.3526) data time 0.0008 (0.0017) model time 0.3417 (0.3511) loss 3.6787 (3.3309) grad_norm 0.9562 (1.4154) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][600/625] eta 0:00:08 lr 0.001430 wd 0.0500 time 0.3421 (0.3525) data time 0.0008 (0.0016) model time 0.3413 (0.3510) loss 3.2375 (3.3328) grad_norm 2.3556 (1.4166) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][610/625] eta 0:00:05 lr 0.001430 wd 0.0500 time 0.3392 (0.3524) data time 0.0004 (0.0016) model time 0.3388 (0.3509) loss 3.1305 (3.3390) grad_norm 1.0163 (1.4189) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [120/300][620/625] eta 0:00:01 lr 0.001430 wd 0.0500 time 0.3372 (0.3522) data time 0.0004 (0.0016) model time 0.3367 (0.3506) loss 4.0947 (3.3392) grad_norm 1.1224 (1.4180) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:12:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 120 training takes 0:03:40 [2024-08-26 11:12:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:12:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.439 (0.439) Loss 0.6431 (0.6431) Acc@1 86.475 (86.475) Acc@5 97.705 (97.705) Mem 11224MB [2024-08-26 11:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 1.0547 (0.7866) Acc@1 75.098 (82.635) Acc@5 93.994 (96.631) Mem 11224MB [2024-08-26 11:12:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.112) Loss 1.1631 (0.9393) Acc@1 72.217 (78.936) Acc@5 92.871 (94.729) Mem 11224MB [2024-08-26 11:12:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.707 Acc@5 94.756 [2024-08-26 11:12:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.7% [2024-08-26 11:12:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.822 (0.822) Loss 0.5332 (0.5332) Acc@1 88.135 (88.135) Acc@5 98.291 (98.291) Mem 11224MB [2024-08-26 11:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.167) Loss 0.8945 (0.6675) Acc@1 79.053 (84.903) Acc@5 95.361 (97.363) Mem 11224MB [2024-08-26 11:12:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.131) Loss 1.0254 (0.8048) Acc@1 74.658 (81.399) Acc@5 93.701 (95.873) Mem 11224MB [2024-08-26 11:12:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.096 Acc@5 95.857 [2024-08-26 11:12:54 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.1% [2024-08-26 11:12:54 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.10% [2024-08-26 11:12:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:12:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:12:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][0/625] eta 0:07:33 lr 0.001430 wd 0.0500 time 0.7259 (0.7259) data time 0.4116 (0.4116) model time 0.0000 (0.0000) loss 3.3209 (3.3209) grad_norm 0.9110 (0.9110) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][10/625] eta 0:03:54 lr 0.001429 wd 0.0500 time 0.3436 (0.3819) data time 0.0008 (0.0382) model time 0.0000 (0.0000) loss 3.1262 (3.2306) grad_norm 1.2225 (1.3445) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][20/625] eta 0:03:46 lr 0.001429 wd 0.0500 time 0.5634 (0.3749) data time 0.0006 (0.0204) model time 0.0000 (0.0000) loss 2.3741 (3.3046) grad_norm 1.3753 (1.4012) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][30/625] eta 0:03:45 lr 0.001429 wd 0.0500 time 0.3597 (0.3798) data time 0.0007 (0.0141) model time 0.0000 (0.0000) loss 2.7790 (3.2512) grad_norm 1.4948 (1.3644) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][40/625] eta 0:03:37 lr 0.001429 wd 0.0500 time 0.3429 (0.3723) data time 0.0006 (0.0109) model time 0.0000 (0.0000) loss 3.6844 (3.2603) grad_norm 1.8418 (1.4090) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][50/625] eta 0:03:31 lr 0.001429 wd 0.0500 time 0.3479 (0.3671) data time 0.0006 (0.0090) model time 0.0000 (0.0000) loss 2.3234 (3.2701) grad_norm 1.6974 (1.4477) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][60/625] eta 0:03:25 lr 0.001429 wd 0.0500 time 0.3448 (0.3634) data time 0.0008 (0.0076) model time 0.3441 (0.3435) loss 3.9869 (3.2977) grad_norm 0.9987 (1.4427) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][70/625] eta 0:03:20 lr 0.001428 wd 0.0500 time 0.3439 (0.3612) data time 0.0007 (0.0067) model time 0.3431 (0.3451) loss 2.3531 (3.2622) grad_norm 1.3727 (1.4371) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][80/625] eta 0:03:15 lr 0.001428 wd 0.0500 time 0.3409 (0.3591) data time 0.0008 (0.0060) model time 0.3401 (0.3446) loss 3.6983 (3.2563) grad_norm 1.5116 (1.4200) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][90/625] eta 0:03:14 lr 0.001428 wd 0.0500 time 0.3138 (0.3635) data time 0.0009 (0.0054) model time 0.3129 (0.3579) loss 3.3576 (3.2585) grad_norm 1.1016 (1.4033) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][100/625] eta 0:03:10 lr 0.001428 wd 0.0500 time 0.3499 (0.3623) data time 0.0010 (0.0050) model time 0.3489 (0.3564) loss 3.0572 (3.2437) grad_norm 1.1896 (1.4186) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][110/625] eta 0:03:05 lr 0.001428 wd 0.0500 time 0.3494 (0.3611) data time 0.0006 (0.0046) model time 0.3488 (0.3550) loss 2.3540 (3.2532) grad_norm 1.0963 (1.4112) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][120/625] eta 0:03:01 lr 0.001428 wd 0.0500 time 0.3431 (0.3599) data time 0.0007 (0.0043) model time 0.3424 (0.3536) loss 3.2342 (3.2573) grad_norm 1.2147 (1.3943) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][130/625] eta 0:02:57 lr 0.001427 wd 0.0500 time 0.3397 (0.3588) data time 0.0009 (0.0041) model time 0.3389 (0.3525) loss 3.3435 (3.2705) grad_norm 2.0202 (1.3951) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][140/625] eta 0:02:53 lr 0.001427 wd 0.0500 time 0.3508 (0.3580) data time 0.0010 (0.0038) model time 0.3498 (0.3518) loss 3.4657 (3.2783) grad_norm 2.3772 (1.4082) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][150/625] eta 0:02:49 lr 0.001427 wd 0.0500 time 0.3477 (0.3572) data time 0.0009 (0.0037) model time 0.3468 (0.3511) loss 3.3558 (3.2796) grad_norm 1.1199 (1.4156) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][160/625] eta 0:02:45 lr 0.001427 wd 0.0500 time 0.3452 (0.3565) data time 0.0009 (0.0035) model time 0.3443 (0.3506) loss 3.6362 (3.2849) grad_norm 1.4898 (1.4064) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:13:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][170/625] eta 0:02:41 lr 0.001427 wd 0.0500 time 0.3459 (0.3558) data time 0.0006 (0.0033) model time 0.3454 (0.3501) loss 2.8938 (3.2896) grad_norm 1.5612 (1.4017) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][180/625] eta 0:02:38 lr 0.001427 wd 0.0500 time 0.3468 (0.3553) data time 0.0006 (0.0032) model time 0.3463 (0.3497) loss 3.8139 (3.3036) grad_norm 1.2879 (1.3968) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][190/625] eta 0:02:34 lr 0.001426 wd 0.0500 time 0.3476 (0.3547) data time 0.0008 (0.0031) model time 0.3468 (0.3493) loss 3.6376 (3.2919) grad_norm 1.1322 (1.3942) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][200/625] eta 0:02:30 lr 0.001426 wd 0.0500 time 0.3477 (0.3544) data time 0.0006 (0.0030) model time 0.3471 (0.3491) loss 3.8746 (3.2931) grad_norm 1.3743 (1.3939) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][210/625] eta 0:02:27 lr 0.001426 wd 0.0500 time 0.3382 (0.3547) data time 0.0010 (0.0029) model time 0.3372 (0.3498) loss 3.3050 (3.3021) grad_norm 1.4661 (1.3878) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][220/625] eta 0:02:23 lr 0.001426 wd 0.0500 time 0.3420 (0.3543) data time 0.0007 (0.0028) model time 0.3413 (0.3494) loss 3.3768 (3.3074) grad_norm 1.3625 (1.3885) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][230/625] eta 0:02:20 lr 0.001426 wd 0.0500 time 0.3472 (0.3548) data time 0.0009 (0.0027) model time 0.3464 (0.3503) loss 3.7264 (3.3001) grad_norm 1.4321 (1.3902) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][240/625] eta 0:02:17 lr 0.001426 wd 0.0500 time 0.3437 (0.3560) data time 0.0008 (0.0027) model time 0.3429 (0.3521) loss 3.6076 (3.3136) grad_norm 1.5172 (1.3969) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][250/625] eta 0:02:13 lr 0.001425 wd 0.0500 time 0.3398 (0.3556) data time 0.0007 (0.0026) model time 0.3391 (0.3517) loss 3.5946 (3.3080) grad_norm 1.0202 (1.3864) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][260/625] eta 0:02:09 lr 0.001425 wd 0.0500 time 0.3566 (0.3552) data time 0.0006 (0.0025) model time 0.3560 (0.3514) loss 4.0417 (3.3128) grad_norm 1.4803 (1.3896) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][270/625] eta 0:02:05 lr 0.001425 wd 0.0500 time 0.3422 (0.3548) data time 0.0009 (0.0025) model time 0.3413 (0.3510) loss 3.7425 (3.3128) grad_norm 1.1742 (1.3873) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][280/625] eta 0:02:02 lr 0.001425 wd 0.0500 time 0.3409 (0.3545) data time 0.0009 (0.0024) model time 0.3400 (0.3507) loss 3.5810 (3.3143) grad_norm 1.3267 (1.3860) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][290/625] eta 0:01:58 lr 0.001425 wd 0.0500 time 0.3952 (0.3544) data time 0.0008 (0.0024) model time 0.3944 (0.3507) loss 3.4391 (3.3210) grad_norm 1.1138 (1.3881) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][300/625] eta 0:01:55 lr 0.001425 wd 0.0500 time 0.3414 (0.3548) data time 0.0009 (0.0023) model time 0.3404 (0.3513) loss 3.5253 (3.3241) grad_norm 1.1202 (1.3880) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][310/625] eta 0:01:51 lr 0.001425 wd 0.0500 time 0.3470 (0.3547) data time 0.0008 (0.0023) model time 0.3462 (0.3512) loss 2.8459 (3.3246) grad_norm 1.0491 (1.3861) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][320/625] eta 0:01:48 lr 0.001424 wd 0.0500 time 0.3468 (0.3545) data time 0.0008 (0.0023) model time 0.3460 (0.3511) loss 2.7409 (3.3110) grad_norm 1.6462 (1.3825) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][330/625] eta 0:01:44 lr 0.001424 wd 0.0500 time 0.3470 (0.3542) data time 0.0007 (0.0022) model time 0.3463 (0.3509) loss 3.3125 (3.3151) grad_norm 1.4203 (1.3864) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:14:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][340/625] eta 0:01:40 lr 0.001424 wd 0.0500 time 0.3437 (0.3540) data time 0.0009 (0.0022) model time 0.3428 (0.3507) loss 3.9436 (3.3152) grad_norm 1.1807 (1.3862) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:15:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][350/625] eta 0:01:37 lr 0.001424 wd 0.0500 time 0.3471 (0.3537) data time 0.0008 (0.0022) model time 0.3463 (0.3505) loss 3.6922 (3.3170) grad_norm 1.0326 (1.3811) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:15:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][360/625] eta 0:01:33 lr 0.001424 wd 0.0500 time 0.3439 (0.3535) data time 0.0006 (0.0021) model time 0.3433 (0.3503) loss 2.8676 (3.3121) grad_norm 1.5641 (1.3797) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:15:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][370/625] eta 0:01:30 lr 0.001424 wd 0.0500 time 0.3435 (0.3533) data time 0.0009 (0.0021) model time 0.3427 (0.3501) loss 3.4555 (3.3136) grad_norm 1.2289 (1.3815) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:15:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][380/625] eta 0:01:26 lr 0.001423 wd 0.0500 time 0.3498 (0.3531) data time 0.0008 (0.0021) model time 0.3489 (0.3500) loss 2.8020 (3.3128) grad_norm 2.5662 (1.3795) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:15:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][390/625] eta 0:01:22 lr 0.001423 wd 0.0500 time 0.3445 (0.3530) data time 0.0009 (0.0020) model time 0.3436 (0.3498) loss 2.4643 (3.3070) grad_norm 1.3103 (1.3816) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:15:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][400/625] eta 0:01:19 lr 0.001423 wd 0.0500 time 0.3470 (0.3528) data time 0.0006 (0.0020) model time 0.3464 (0.3497) loss 3.5905 (3.3180) grad_norm 1.2491 (1.3809) loss_scale 2048.0000 (1029.1072) mem 11224MB [2024-08-26 11:15:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][410/625] eta 0:01:15 lr 0.001423 wd 0.0500 time 0.3458 (0.3526) data time 0.0009 (0.0020) model time 0.3449 (0.3496) loss 2.8193 (3.3168) grad_norm 2.7729 (1.3825) loss_scale 2048.0000 (1053.8978) mem 11224MB [2024-08-26 11:15:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][420/625] eta 0:01:12 lr 0.001423 wd 0.0500 time 0.3480 (0.3548) data time 0.0007 (0.0020) model time 0.3474 (0.3522) loss 2.7025 (3.3155) grad_norm 1.2134 (1.3809) loss_scale 2048.0000 (1077.5107) mem 11224MB [2024-08-26 11:15:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][430/625] eta 0:01:09 lr 0.001423 wd 0.0500 time 0.3432 (0.3546) data time 0.0007 (0.0019) model time 0.3426 (0.3520) loss 3.8206 (3.3105) grad_norm 1.4090 (1.3789) loss_scale 2048.0000 (1100.0278) mem 11224MB [2024-08-26 11:15:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][440/625] eta 0:01:05 lr 0.001422 wd 0.0500 time 0.3446 (0.3545) data time 0.0008 (0.0019) model time 0.3438 (0.3518) loss 4.0175 (3.3093) grad_norm 1.1407 (1.3787) loss_scale 2048.0000 (1121.5238) mem 11224MB [2024-08-26 11:15:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][450/625] eta 0:01:01 lr 0.001422 wd 0.0500 time 0.3416 (0.3543) data time 0.0007 (0.0019) model time 0.3410 (0.3516) loss 3.2039 (3.3127) grad_norm 1.6191 (1.3854) loss_scale 2048.0000 (1142.0665) mem 11224MB [2024-08-26 11:15:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][460/625] eta 0:00:58 lr 0.001422 wd 0.0500 time 0.3419 (0.3541) data time 0.0007 (0.0019) model time 0.3413 (0.3515) loss 3.8424 (3.3110) grad_norm 1.2576 (1.3883) loss_scale 2048.0000 (1161.7180) mem 11224MB [2024-08-26 11:15:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][470/625] eta 0:00:54 lr 0.001422 wd 0.0500 time 0.3401 (0.3539) data time 0.0008 (0.0018) model time 0.3393 (0.3513) loss 2.8725 (3.3134) grad_norm 1.1424 (1.3891) loss_scale 2048.0000 (1180.5350) mem 11224MB [2024-08-26 11:15:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][480/625] eta 0:00:51 lr 0.001422 wd 0.0500 time 0.3428 (0.3542) data time 0.0007 (0.0018) model time 0.3422 (0.3517) loss 3.4114 (3.3118) grad_norm 1.1424 (1.3884) loss_scale 2048.0000 (1198.5696) mem 11224MB [2024-08-26 11:15:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][490/625] eta 0:00:47 lr 0.001422 wd 0.0500 time 0.3427 (0.3540) data time 0.0009 (0.0018) model time 0.3419 (0.3515) loss 3.4497 (3.3097) grad_norm 1.0932 (1.3878) loss_scale 2048.0000 (1215.8697) mem 11224MB [2024-08-26 11:15:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][500/625] eta 0:00:44 lr 0.001421 wd 0.0500 time 0.3440 (0.3539) data time 0.0007 (0.0018) model time 0.3434 (0.3514) loss 2.8440 (3.3093) grad_norm 1.9885 (1.3939) loss_scale 2048.0000 (1232.4790) mem 11224MB [2024-08-26 11:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][510/625] eta 0:00:40 lr 0.001421 wd 0.0500 time 0.3420 (0.3537) data time 0.0006 (0.0018) model time 0.3414 (0.3512) loss 3.3896 (3.3130) grad_norm 1.0779 (1.3932) loss_scale 2048.0000 (1248.4384) mem 11224MB [2024-08-26 11:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][520/625] eta 0:00:37 lr 0.001421 wd 0.0500 time 0.3440 (0.3535) data time 0.0008 (0.0018) model time 0.3432 (0.3511) loss 3.5047 (3.3125) grad_norm 1.1299 (1.3909) loss_scale 2048.0000 (1263.7850) mem 11224MB [2024-08-26 11:16:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][530/625] eta 0:00:33 lr 0.001421 wd 0.0500 time 0.3436 (0.3534) data time 0.0009 (0.0017) model time 0.3428 (0.3509) loss 3.1548 (3.3111) grad_norm 1.5298 (1.3896) loss_scale 2048.0000 (1278.5537) mem 11224MB [2024-08-26 11:16:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][540/625] eta 0:00:30 lr 0.001421 wd 0.0500 time 0.3447 (0.3547) data time 0.0007 (0.0017) model time 0.3440 (0.3525) loss 3.7927 (3.3095) grad_norm 1.4441 (1.4018) loss_scale 2048.0000 (1292.7763) mem 11224MB [2024-08-26 11:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][550/625] eta 0:00:26 lr 0.001421 wd 0.0500 time 0.3561 (0.3547) data time 0.0008 (0.0017) model time 0.3552 (0.3524) loss 3.3884 (3.3044) grad_norm 1.1664 (1.4032) loss_scale 2048.0000 (1306.4828) mem 11224MB [2024-08-26 11:16:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][560/625] eta 0:00:23 lr 0.001420 wd 0.0500 time 0.3478 (0.3545) data time 0.0006 (0.0017) model time 0.3472 (0.3523) loss 2.7173 (3.3049) grad_norm 1.6177 (1.4022) loss_scale 2048.0000 (1319.7005) mem 11224MB [2024-08-26 11:16:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][570/625] eta 0:00:19 lr 0.001420 wd 0.0500 time 0.3390 (0.3543) data time 0.0007 (0.0017) model time 0.3383 (0.3521) loss 2.8846 (3.3088) grad_norm 1.6388 (1.3985) loss_scale 2048.0000 (1332.4553) mem 11224MB [2024-08-26 11:16:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][580/625] eta 0:00:15 lr 0.001420 wd 0.0500 time 0.3444 (0.3542) data time 0.0007 (0.0017) model time 0.3438 (0.3520) loss 4.0273 (3.3053) grad_norm 1.1727 (1.3990) loss_scale 2048.0000 (1344.7711) mem 11224MB [2024-08-26 11:16:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][590/625] eta 0:00:12 lr 0.001420 wd 0.0500 time 0.3406 (0.3540) data time 0.0006 (0.0017) model time 0.3400 (0.3518) loss 3.7869 (3.3092) grad_norm 1.3476 (inf) loss_scale 1024.0000 (1342.8088) mem 11224MB [2024-08-26 11:16:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][600/625] eta 0:00:08 lr 0.001420 wd 0.0500 time 0.5480 (0.3545) data time 0.0007 (0.0016) model time 0.5474 (0.3524) loss 2.2583 (3.3073) grad_norm 1.4256 (inf) loss_scale 1024.0000 (1337.5042) mem 11224MB [2024-08-26 11:16:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][610/625] eta 0:00:05 lr 0.001420 wd 0.0500 time 0.3377 (0.3543) data time 0.0006 (0.0016) model time 0.3371 (0.3522) loss 2.6802 (3.3044) grad_norm 1.4474 (inf) loss_scale 1024.0000 (1332.3732) mem 11224MB [2024-08-26 11:16:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [121/300][620/625] eta 0:00:01 lr 0.001420 wd 0.0500 time 0.3387 (0.3541) data time 0.0004 (0.0016) model time 0.3383 (0.3519) loss 2.8290 (3.3055) grad_norm 1.0382 (inf) loss_scale 1024.0000 (1327.4074) mem 11224MB [2024-08-26 11:16:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 121 training takes 0:03:41 [2024-08-26 11:16:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:16:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:16:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.463 (0.463) Loss 0.6411 (0.6411) Acc@1 86.035 (86.035) Acc@5 97.266 (97.266) Mem 11224MB [2024-08-26 11:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.133) Loss 0.9946 (0.7666) Acc@1 77.393 (82.959) Acc@5 93.457 (96.413) Mem 11224MB [2024-08-26 11:16:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.115) Loss 1.1836 (0.9297) Acc@1 72.900 (79.157) Acc@5 91.943 (94.545) Mem 11224MB [2024-08-26 11:16:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.771 Acc@5 94.556 [2024-08-26 11:16:40 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.8% [2024-08-26 11:16:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.815 (0.815) Loss 0.5327 (0.5327) Acc@1 88.184 (88.184) Acc@5 98.242 (98.242) Mem 11224MB [2024-08-26 11:16:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.167) Loss 0.8911 (0.6669) Acc@1 79.346 (84.899) Acc@5 95.361 (97.328) Mem 11224MB [2024-08-26 11:16:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.131) Loss 1.0234 (0.8032) Acc@1 74.463 (81.443) Acc@5 93.799 (95.884) Mem 11224MB [2024-08-26 11:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.148 Acc@5 95.869 [2024-08-26 11:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.1% [2024-08-26 11:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.15% [2024-08-26 11:16:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:16:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:16:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][0/625] eta 0:07:08 lr 0.001419 wd 0.0500 time 0.6854 (0.6854) data time 0.3731 (0.3731) model time 0.0000 (0.0000) loss 3.5742 (3.5742) grad_norm 1.3950 (1.3950) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:16:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][10/625] eta 0:03:51 lr 0.001419 wd 0.0500 time 0.3490 (0.3761) data time 0.0006 (0.0347) model time 0.0000 (0.0000) loss 3.7299 (3.6331) grad_norm 1.2694 (1.4107) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:16:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][20/625] eta 0:03:38 lr 0.001419 wd 0.0500 time 0.3449 (0.3615) data time 0.0009 (0.0186) model time 0.0000 (0.0000) loss 3.3190 (3.4167) grad_norm 1.5725 (1.4329) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][30/625] eta 0:03:32 lr 0.001419 wd 0.0500 time 0.3479 (0.3564) data time 0.0006 (0.0129) model time 0.0000 (0.0000) loss 3.4182 (3.3309) grad_norm 1.1928 (1.4118) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:16:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][40/625] eta 0:03:26 lr 0.001419 wd 0.0500 time 0.3493 (0.3537) data time 0.0006 (0.0100) model time 0.0000 (0.0000) loss 3.7516 (3.3193) grad_norm 1.3853 (1.4017) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][50/625] eta 0:03:22 lr 0.001419 wd 0.0500 time 0.3428 (0.3523) data time 0.0006 (0.0082) model time 0.0000 (0.0000) loss 3.2710 (3.2885) grad_norm 1.5275 (1.3764) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][60/625] eta 0:03:18 lr 0.001418 wd 0.0500 time 0.3429 (0.3514) data time 0.0009 (0.0070) model time 0.3420 (0.3459) loss 3.6077 (3.3004) grad_norm 1.5267 (1.4017) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][70/625] eta 0:03:20 lr 0.001418 wd 0.0500 time 0.3389 (0.3616) data time 0.0006 (0.0061) model time 0.3383 (0.3843) loss 3.4269 (3.3142) grad_norm 0.9661 (1.3899) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][80/625] eta 0:03:16 lr 0.001418 wd 0.0500 time 0.3467 (0.3599) data time 0.0008 (0.0055) model time 0.3458 (0.3719) loss 3.8012 (3.3257) grad_norm 1.2240 (1.4030) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][90/625] eta 0:03:12 lr 0.001418 wd 0.0500 time 0.4923 (0.3601) data time 0.0008 (0.0050) model time 0.4916 (0.3691) loss 3.8586 (3.3074) grad_norm 1.0009 (1.3984) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][100/625] eta 0:03:08 lr 0.001418 wd 0.0500 time 0.3464 (0.3584) data time 0.0007 (0.0046) model time 0.3457 (0.3638) loss 3.5605 (3.2882) grad_norm 1.9266 (1.3886) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][110/625] eta 0:03:05 lr 0.001418 wd 0.0500 time 0.3475 (0.3592) data time 0.0007 (0.0043) model time 0.3468 (0.3642) loss 3.4590 (3.2622) grad_norm 1.8368 (1.3883) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][120/625] eta 0:03:00 lr 0.001417 wd 0.0500 time 0.3481 (0.3583) data time 0.0007 (0.0040) model time 0.3474 (0.3617) loss 3.9796 (3.2553) grad_norm 1.9303 (1.4285) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][130/625] eta 0:02:56 lr 0.001417 wd 0.0500 time 0.3462 (0.3576) data time 0.0008 (0.0037) model time 0.3453 (0.3600) loss 3.2514 (3.2623) grad_norm 1.2037 (1.4425) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][140/625] eta 0:02:53 lr 0.001417 wd 0.0500 time 0.3441 (0.3569) data time 0.0009 (0.0035) model time 0.3432 (0.3585) loss 3.1398 (3.2759) grad_norm 1.2359 (1.4371) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][150/625] eta 0:02:49 lr 0.001417 wd 0.0500 time 0.3464 (0.3562) data time 0.0009 (0.0034) model time 0.3455 (0.3573) loss 3.4077 (3.2649) grad_norm 1.6826 (1.4256) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][160/625] eta 0:02:45 lr 0.001417 wd 0.0500 time 0.3575 (0.3559) data time 0.0008 (0.0033) model time 0.3566 (0.3565) loss 3.6938 (3.2686) grad_norm 1.1420 (1.4463) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][170/625] eta 0:02:41 lr 0.001417 wd 0.0500 time 0.3520 (0.3553) data time 0.0008 (0.0031) model time 0.3513 (0.3556) loss 3.3967 (3.2501) grad_norm 1.4095 (1.4392) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][180/625] eta 0:02:37 lr 0.001417 wd 0.0500 time 0.3779 (0.3550) data time 0.0008 (0.0030) model time 0.3771 (0.3551) loss 3.7726 (3.2582) grad_norm 1.3911 (1.4261) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][190/625] eta 0:02:34 lr 0.001416 wd 0.0500 time 0.3398 (0.3545) data time 0.0009 (0.0029) model time 0.3390 (0.3544) loss 2.4740 (3.2555) grad_norm 0.9987 (1.4166) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:17:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][200/625] eta 0:02:30 lr 0.001416 wd 0.0500 time 0.3466 (0.3541) data time 0.0008 (0.0028) model time 0.3458 (0.3537) loss 3.7618 (3.2712) grad_norm 1.2361 (1.4180) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][210/625] eta 0:02:27 lr 0.001416 wd 0.0500 time 0.5408 (0.3547) data time 0.0006 (0.0028) model time 0.5402 (0.3545) loss 3.6855 (3.2779) grad_norm 1.6192 (1.4214) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][220/625] eta 0:02:23 lr 0.001416 wd 0.0500 time 0.3427 (0.3542) data time 0.0009 (0.0027) model time 0.3418 (0.3538) loss 3.9290 (3.2756) grad_norm 1.1111 (1.4206) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][230/625] eta 0:02:19 lr 0.001416 wd 0.0500 time 0.3483 (0.3538) data time 0.0009 (0.0026) model time 0.3473 (0.3533) loss 3.7880 (3.2740) grad_norm 1.5675 (1.4244) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][240/625] eta 0:02:16 lr 0.001416 wd 0.0500 time 0.3469 (0.3535) data time 0.0006 (0.0025) model time 0.3463 (0.3528) loss 3.5007 (3.2866) grad_norm 1.0613 (1.4280) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][250/625] eta 0:02:12 lr 0.001415 wd 0.0500 time 0.3449 (0.3532) data time 0.0008 (0.0025) model time 0.3441 (0.3524) loss 3.4976 (3.3011) grad_norm 1.3232 (1.4205) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][260/625] eta 0:02:08 lr 0.001415 wd 0.0500 time 0.3497 (0.3529) data time 0.0009 (0.0024) model time 0.3489 (0.3521) loss 3.6397 (3.3026) grad_norm 1.1581 (1.4229) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][270/625] eta 0:02:05 lr 0.001415 wd 0.0500 time 0.3513 (0.3527) data time 0.0007 (0.0024) model time 0.3506 (0.3518) loss 2.2458 (3.2940) grad_norm 1.5412 (1.4294) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][280/625] eta 0:02:02 lr 0.001415 wd 0.0500 time 0.3354 (0.3549) data time 0.0007 (0.0023) model time 0.3347 (0.3545) loss 3.1026 (3.2873) grad_norm 1.8021 (1.4270) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][290/625] eta 0:01:58 lr 0.001415 wd 0.0500 time 0.3483 (0.3546) data time 0.0008 (0.0023) model time 0.3474 (0.3541) loss 3.3868 (3.2946) grad_norm 1.6406 (1.4276) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][300/625] eta 0:01:55 lr 0.001415 wd 0.0500 time 0.3445 (0.3543) data time 0.0006 (0.0022) model time 0.3439 (0.3537) loss 4.3325 (3.3006) grad_norm 1.5502 (1.4252) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][310/625] eta 0:01:51 lr 0.001414 wd 0.0500 time 0.3432 (0.3540) data time 0.0007 (0.0022) model time 0.3425 (0.3534) loss 4.1271 (3.3060) grad_norm 0.9717 (1.4261) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][320/625] eta 0:01:48 lr 0.001414 wd 0.0500 time 0.3444 (0.3544) data time 0.0009 (0.0021) model time 0.3434 (0.3538) loss 2.7065 (3.3045) grad_norm 1.7413 (1.4314) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][330/625] eta 0:01:44 lr 0.001414 wd 0.0500 time 0.3735 (0.3542) data time 0.0008 (0.0021) model time 0.3727 (0.3536) loss 2.9650 (3.3000) grad_norm 1.4516 (1.4305) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][340/625] eta 0:01:40 lr 0.001414 wd 0.0500 time 0.3443 (0.3540) data time 0.0008 (0.0021) model time 0.3436 (0.3533) loss 4.0633 (3.2955) grad_norm 1.0326 (1.4311) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][350/625] eta 0:01:37 lr 0.001414 wd 0.0500 time 0.3503 (0.3538) data time 0.0007 (0.0020) model time 0.3495 (0.3531) loss 3.0151 (3.2981) grad_norm 1.4252 (1.4244) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][360/625] eta 0:01:33 lr 0.001414 wd 0.0500 time 0.3517 (0.3537) data time 0.0007 (0.0020) model time 0.3510 (0.3529) loss 3.4598 (3.3021) grad_norm 1.0405 (1.4216) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][370/625] eta 0:01:30 lr 0.001413 wd 0.0500 time 0.3439 (0.3535) data time 0.0008 (0.0020) model time 0.3431 (0.3527) loss 2.9901 (3.3001) grad_norm 1.1781 (1.4254) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:18:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][380/625] eta 0:01:26 lr 0.001413 wd 0.0500 time 0.3446 (0.3533) data time 0.0008 (0.0020) model time 0.3437 (0.3525) loss 2.7920 (3.2999) grad_norm 1.5842 (1.4294) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][390/625] eta 0:01:22 lr 0.001413 wd 0.0500 time 0.3478 (0.3531) data time 0.0009 (0.0019) model time 0.3469 (0.3523) loss 2.5104 (3.3023) grad_norm 1.1827 (1.4273) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][400/625] eta 0:01:19 lr 0.001413 wd 0.0500 time 0.3470 (0.3531) data time 0.0007 (0.0019) model time 0.3463 (0.3522) loss 4.1753 (3.3013) grad_norm 1.0754 (1.4236) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][410/625] eta 0:01:15 lr 0.001413 wd 0.0500 time 0.3442 (0.3529) data time 0.0009 (0.0019) model time 0.3433 (0.3520) loss 2.9783 (3.2908) grad_norm 1.1422 (1.4266) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][420/625] eta 0:01:12 lr 0.001413 wd 0.0500 time 0.5716 (0.3533) data time 0.0006 (0.0019) model time 0.5710 (0.3524) loss 3.7308 (3.2938) grad_norm 1.6307 (1.4356) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][430/625] eta 0:01:08 lr 0.001412 wd 0.0500 time 0.3446 (0.3532) data time 0.0008 (0.0019) model time 0.3438 (0.3523) loss 3.4378 (3.2939) grad_norm 1.5112 (1.4398) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][440/625] eta 0:01:05 lr 0.001412 wd 0.0500 time 0.3391 (0.3531) data time 0.0007 (0.0019) model time 0.3383 (0.3522) loss 3.6479 (3.2954) grad_norm 1.7802 (1.4346) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][450/625] eta 0:01:01 lr 0.001412 wd 0.0500 time 0.3761 (0.3531) data time 0.0008 (0.0018) model time 0.3753 (0.3522) loss 3.5574 (3.2977) grad_norm 1.2166 (1.4318) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][460/625] eta 0:00:58 lr 0.001412 wd 0.0500 time 0.3377 (0.3540) data time 0.0009 (0.0018) model time 0.3368 (0.3531) loss 3.6767 (3.2997) grad_norm 1.5216 (1.4265) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][470/625] eta 0:00:54 lr 0.001412 wd 0.0500 time 0.3551 (0.3547) data time 0.0008 (0.0018) model time 0.3543 (0.3540) loss 3.2229 (3.2989) grad_norm 1.7621 (1.4281) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][480/625] eta 0:00:51 lr 0.001412 wd 0.0500 time 0.3577 (0.3546) data time 0.0008 (0.0018) model time 0.3569 (0.3538) loss 3.0216 (3.2985) grad_norm 1.4685 (1.4293) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][490/625] eta 0:00:47 lr 0.001411 wd 0.0500 time 0.3399 (0.3544) data time 0.0009 (0.0018) model time 0.3390 (0.3537) loss 3.6325 (3.2961) grad_norm 1.1685 (1.4285) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][500/625] eta 0:00:44 lr 0.001411 wd 0.0500 time 0.3489 (0.3551) data time 0.0007 (0.0018) model time 0.3482 (0.3544) loss 2.9087 (3.2956) grad_norm 1.3527 (1.4261) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][510/625] eta 0:00:40 lr 0.001411 wd 0.0500 time 0.3570 (0.3550) data time 0.0007 (0.0017) model time 0.3563 (0.3542) loss 2.6214 (3.2960) grad_norm 1.8984 (1.4281) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][520/625] eta 0:00:37 lr 0.001411 wd 0.0500 time 0.3422 (0.3548) data time 0.0008 (0.0017) model time 0.3413 (0.3541) loss 3.4985 (3.2942) grad_norm 1.2419 (1.4253) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][530/625] eta 0:00:33 lr 0.001411 wd 0.0500 time 0.3445 (0.3546) data time 0.0008 (0.0017) model time 0.3437 (0.3538) loss 3.2877 (3.2968) grad_norm 1.1001 (1.4241) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:19:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][540/625] eta 0:00:30 lr 0.001411 wd 0.0500 time 0.3476 (0.3545) data time 0.0007 (0.0017) model time 0.3470 (0.3537) loss 4.1224 (3.2982) grad_norm 1.0130 (1.4191) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][550/625] eta 0:00:26 lr 0.001411 wd 0.0500 time 0.3505 (0.3544) data time 0.0008 (0.0017) model time 0.3497 (0.3535) loss 2.6879 (3.2995) grad_norm 1.3279 (1.4175) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][560/625] eta 0:00:23 lr 0.001410 wd 0.0500 time 0.3422 (0.3542) data time 0.0008 (0.0017) model time 0.3413 (0.3533) loss 3.4114 (3.2996) grad_norm 1.2970 (1.4234) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][570/625] eta 0:00:19 lr 0.001410 wd 0.0500 time 0.3437 (0.3540) data time 0.0006 (0.0017) model time 0.3431 (0.3531) loss 4.2228 (3.3040) grad_norm 1.5007 (1.4285) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][580/625] eta 0:00:15 lr 0.001410 wd 0.0500 time 0.3441 (0.3546) data time 0.0008 (0.0016) model time 0.3432 (0.3538) loss 2.9838 (3.3063) grad_norm 1.1381 (1.4259) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][590/625] eta 0:00:12 lr 0.001410 wd 0.0500 time 0.3496 (0.3551) data time 0.0009 (0.0016) model time 0.3487 (0.3544) loss 3.1518 (3.2974) grad_norm 1.2055 (1.4229) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][600/625] eta 0:00:08 lr 0.001410 wd 0.0500 time 0.3485 (0.3550) data time 0.0006 (0.0016) model time 0.3479 (0.3542) loss 2.7075 (3.2983) grad_norm 2.3898 (1.4258) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][610/625] eta 0:00:05 lr 0.001410 wd 0.0500 time 0.3385 (0.3551) data time 0.0004 (0.0016) model time 0.3381 (0.3544) loss 4.0321 (3.2982) grad_norm 1.7772 (1.4307) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [122/300][620/625] eta 0:00:01 lr 0.001409 wd 0.0500 time 0.3453 (0.3554) data time 0.0004 (0.0016) model time 0.3449 (0.3547) loss 4.0547 (3.3008) grad_norm 1.6080 (1.4328) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 122 training takes 0:03:42 [2024-08-26 11:20:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:20:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:20:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.459 (0.459) Loss 0.6255 (0.6255) Acc@1 86.670 (86.670) Acc@5 97.803 (97.803) Mem 11224MB [2024-08-26 11:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.132) Loss 1.0869 (0.7778) Acc@1 73.779 (82.515) Acc@5 92.871 (96.515) Mem 11224MB [2024-08-26 11:20:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.113) Loss 1.1426 (0.9334) Acc@1 72.998 (79.013) Acc@5 92.529 (94.643) Mem 11224MB [2024-08-26 11:20:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.775 Acc@5 94.668 [2024-08-26 11:20:30 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.8% [2024-08-26 11:20:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.794 (0.794) Loss 0.5312 (0.5312) Acc@1 88.184 (88.184) Acc@5 98.291 (98.291) Mem 11224MB [2024-08-26 11:20:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.165) Loss 0.8892 (0.6658) Acc@1 79.199 (84.934) Acc@5 95.459 (97.346) Mem 11224MB [2024-08-26 11:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.130) Loss 1.0254 (0.8018) Acc@1 74.707 (81.476) Acc@5 93.896 (95.898) Mem 11224MB [2024-08-26 11:20:34 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.180 Acc@5 95.883 [2024-08-26 11:20:34 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.2% [2024-08-26 11:20:34 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.18% [2024-08-26 11:20:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:20:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:20:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][0/625] eta 0:07:05 lr 0.001409 wd 0.0500 time 0.6813 (0.6813) data time 0.3649 (0.3649) model time 0.0000 (0.0000) loss 4.0731 (4.0731) grad_norm 1.6764 (1.6764) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][10/625] eta 0:03:55 lr 0.001409 wd 0.0500 time 0.3771 (0.3821) data time 0.0006 (0.0346) model time 0.0000 (0.0000) loss 4.4855 (3.3987) grad_norm 1.2108 (1.5064) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][20/625] eta 0:03:41 lr 0.001409 wd 0.0500 time 0.3454 (0.3669) data time 0.0007 (0.0186) model time 0.0000 (0.0000) loss 3.8875 (3.5304) grad_norm 1.5519 (1.8378) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][30/625] eta 0:03:35 lr 0.001409 wd 0.0500 time 0.3425 (0.3614) data time 0.0007 (0.0129) model time 0.0000 (0.0000) loss 3.8558 (3.4697) grad_norm 1.1846 (1.7025) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][40/625] eta 0:03:29 lr 0.001409 wd 0.0500 time 0.3428 (0.3583) data time 0.0007 (0.0100) model time 0.0000 (0.0000) loss 4.3105 (3.5062) grad_norm 0.9792 (1.6672) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][50/625] eta 0:03:24 lr 0.001408 wd 0.0500 time 0.3495 (0.3563) data time 0.0009 (0.0082) model time 0.0000 (0.0000) loss 3.8930 (3.5277) grad_norm 1.0463 (1.6983) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:20:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][60/625] eta 0:03:20 lr 0.001408 wd 0.0500 time 0.3476 (0.3546) data time 0.0006 (0.0070) model time 0.3470 (0.3452) loss 4.1242 (3.5646) grad_norm 1.0611 (1.6286) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][70/625] eta 0:03:16 lr 0.001408 wd 0.0500 time 0.3451 (0.3533) data time 0.0007 (0.0062) model time 0.3444 (0.3445) loss 2.9840 (3.4980) grad_norm 2.0859 (1.6166) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][80/625] eta 0:03:12 lr 0.001408 wd 0.0500 time 0.3430 (0.3526) data time 0.0009 (0.0055) model time 0.3421 (0.3453) loss 3.6481 (3.4835) grad_norm 1.7952 (1.5921) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][90/625] eta 0:03:08 lr 0.001408 wd 0.0500 time 0.3448 (0.3518) data time 0.0008 (0.0050) model time 0.3440 (0.3451) loss 2.1572 (3.4634) grad_norm 1.2849 (1.5792) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][100/625] eta 0:03:09 lr 0.001408 wd 0.0500 time 0.5058 (0.3608) data time 0.0006 (0.0048) model time 0.5052 (0.3642) loss 2.5193 (3.4460) grad_norm 1.5220 (1.5510) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][110/625] eta 0:03:05 lr 0.001407 wd 0.0500 time 0.3425 (0.3597) data time 0.0007 (0.0044) model time 0.3418 (0.3614) loss 4.3369 (3.4305) grad_norm 1.4896 (1.5498) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][120/625] eta 0:03:01 lr 0.001407 wd 0.0500 time 0.3637 (0.3590) data time 0.0009 (0.0042) model time 0.3627 (0.3597) loss 3.8772 (3.4330) grad_norm 1.0942 (1.5504) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][130/625] eta 0:02:57 lr 0.001407 wd 0.0500 time 0.3437 (0.3582) data time 0.0007 (0.0040) model time 0.3429 (0.3582) loss 3.9643 (3.4341) grad_norm 1.0635 (1.5758) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][140/625] eta 0:02:53 lr 0.001407 wd 0.0500 time 0.3456 (0.3577) data time 0.0007 (0.0037) model time 0.3449 (0.3573) loss 4.2239 (3.4244) grad_norm 1.0199 (1.5771) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][150/625] eta 0:02:49 lr 0.001407 wd 0.0500 time 0.3596 (0.3573) data time 0.0008 (0.0036) model time 0.3588 (0.3567) loss 2.2541 (3.4169) grad_norm 1.3093 (1.5599) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][160/625] eta 0:02:45 lr 0.001407 wd 0.0500 time 0.3443 (0.3568) data time 0.0008 (0.0034) model time 0.3434 (0.3558) loss 2.7945 (3.4081) grad_norm 1.3988 (1.5389) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][170/625] eta 0:02:42 lr 0.001407 wd 0.0500 time 0.3436 (0.3562) data time 0.0007 (0.0033) model time 0.3429 (0.3549) loss 3.7671 (3.4219) grad_norm 1.7674 (1.5321) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][180/625] eta 0:02:38 lr 0.001406 wd 0.0500 time 0.3419 (0.3555) data time 0.0009 (0.0032) model time 0.3410 (0.3540) loss 3.2230 (3.4052) grad_norm 1.4935 (1.5296) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][190/625] eta 0:02:34 lr 0.001406 wd 0.0500 time 0.3455 (0.3551) data time 0.0006 (0.0031) model time 0.3449 (0.3535) loss 3.4596 (3.3979) grad_norm 1.1906 (1.5137) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][200/625] eta 0:02:31 lr 0.001406 wd 0.0500 time 0.4802 (0.3566) data time 0.0006 (0.0029) model time 0.4796 (0.3555) loss 4.1054 (3.3918) grad_norm 1.3106 (1.5185) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][210/625] eta 0:02:27 lr 0.001406 wd 0.0500 time 0.3511 (0.3561) data time 0.0006 (0.0029) model time 0.3505 (0.3549) loss 2.8192 (3.3766) grad_norm 1.2716 (1.5192) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][220/625] eta 0:02:24 lr 0.001406 wd 0.0500 time 0.5595 (0.3566) data time 0.0007 (0.0028) model time 0.5589 (0.3555) loss 2.5003 (3.3638) grad_norm 1.6606 (1.5131) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:21:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][230/625] eta 0:02:20 lr 0.001406 wd 0.0500 time 0.3382 (0.3559) data time 0.0007 (0.0027) model time 0.3376 (0.3547) loss 3.4203 (3.3533) grad_norm 1.5898 (1.5141) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][240/625] eta 0:02:16 lr 0.001405 wd 0.0500 time 0.3883 (0.3556) data time 0.0006 (0.0026) model time 0.3877 (0.3543) loss 4.4589 (3.3510) grad_norm 1.1381 (1.5052) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][250/625] eta 0:02:13 lr 0.001405 wd 0.0500 time 0.3439 (0.3552) data time 0.0008 (0.0025) model time 0.3430 (0.3539) loss 3.6622 (3.3458) grad_norm 2.0180 (1.5000) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][260/625] eta 0:02:09 lr 0.001405 wd 0.0500 time 0.3492 (0.3549) data time 0.0009 (0.0025) model time 0.3483 (0.3534) loss 3.7089 (3.3551) grad_norm 1.1691 (1.5012) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][270/625] eta 0:02:05 lr 0.001405 wd 0.0500 time 0.3441 (0.3545) data time 0.0007 (0.0024) model time 0.3434 (0.3530) loss 2.7403 (3.3517) grad_norm 1.9091 (1.5117) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][280/625] eta 0:02:02 lr 0.001405 wd 0.0500 time 0.3422 (0.3544) data time 0.0008 (0.0024) model time 0.3414 (0.3529) loss 2.9608 (3.3530) grad_norm 1.2467 (1.4996) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][290/625] eta 0:01:58 lr 0.001405 wd 0.0500 time 0.3422 (0.3541) data time 0.0006 (0.0023) model time 0.3415 (0.3526) loss 3.6472 (3.3554) grad_norm 1.1585 (1.4891) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][300/625] eta 0:01:55 lr 0.001404 wd 0.0500 time 0.3437 (0.3539) data time 0.0007 (0.0023) model time 0.3430 (0.3523) loss 3.5159 (3.3571) grad_norm 1.4195 (1.4899) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][310/625] eta 0:01:52 lr 0.001404 wd 0.0500 time 0.3437 (0.3557) data time 0.0007 (0.0022) model time 0.3430 (0.3545) loss 4.0095 (3.3515) grad_norm 1.4738 (1.4964) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][320/625] eta 0:01:48 lr 0.001404 wd 0.0500 time 0.3573 (0.3554) data time 0.0008 (0.0022) model time 0.3565 (0.3542) loss 3.7579 (3.3563) grad_norm 1.6668 (1.4962) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][330/625] eta 0:01:44 lr 0.001404 wd 0.0500 time 0.3519 (0.3551) data time 0.0006 (0.0022) model time 0.3513 (0.3538) loss 2.4180 (3.3507) grad_norm 1.3361 (1.4930) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][340/625] eta 0:01:41 lr 0.001404 wd 0.0500 time 0.3461 (0.3549) data time 0.0007 (0.0021) model time 0.3454 (0.3535) loss 3.4865 (3.3535) grad_norm 1.1870 (1.4940) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][350/625] eta 0:01:37 lr 0.001404 wd 0.0500 time 0.3633 (0.3547) data time 0.0006 (0.0021) model time 0.3627 (0.3534) loss 2.9360 (3.3550) grad_norm 1.3958 (1.4895) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][360/625] eta 0:01:33 lr 0.001403 wd 0.0500 time 0.3476 (0.3546) data time 0.0009 (0.0021) model time 0.3467 (0.3532) loss 3.4747 (3.3462) grad_norm 1.6373 (1.4830) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][370/625] eta 0:01:30 lr 0.001403 wd 0.0500 time 0.3442 (0.3543) data time 0.0006 (0.0020) model time 0.3435 (0.3529) loss 3.9632 (3.3474) grad_norm 1.6189 (1.4819) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][380/625] eta 0:01:26 lr 0.001403 wd 0.0500 time 0.3492 (0.3541) data time 0.0007 (0.0020) model time 0.3485 (0.3527) loss 2.3045 (3.3411) grad_norm 1.2830 (1.4774) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][390/625] eta 0:01:23 lr 0.001403 wd 0.0500 time 0.3452 (0.3551) data time 0.0006 (0.0020) model time 0.3446 (0.3539) loss 2.2833 (3.3413) grad_norm 1.4984 (1.4720) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:22:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][400/625] eta 0:01:19 lr 0.001403 wd 0.0500 time 0.3448 (0.3549) data time 0.0006 (0.0019) model time 0.3442 (0.3537) loss 2.9092 (3.3379) grad_norm 2.2716 (1.4745) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][410/625] eta 0:01:16 lr 0.001403 wd 0.0500 time 0.3442 (0.3547) data time 0.0006 (0.0019) model time 0.3436 (0.3534) loss 3.8622 (3.3419) grad_norm 1.4640 (1.4773) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][420/625] eta 0:01:12 lr 0.001402 wd 0.0500 time 0.3460 (0.3545) data time 0.0006 (0.0019) model time 0.3454 (0.3532) loss 2.3592 (3.3356) grad_norm 1.1273 (1.4726) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][430/625] eta 0:01:09 lr 0.001402 wd 0.0500 time 0.5338 (0.3547) data time 0.0008 (0.0019) model time 0.5330 (0.3535) loss 3.4390 (3.3276) grad_norm 1.2520 (1.4748) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][440/625] eta 0:01:05 lr 0.001402 wd 0.0500 time 0.3485 (0.3545) data time 0.0006 (0.0018) model time 0.3479 (0.3533) loss 2.0335 (3.3257) grad_norm 1.8365 (1.4737) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][450/625] eta 0:01:02 lr 0.001402 wd 0.0500 time 0.3440 (0.3543) data time 0.0008 (0.0018) model time 0.3431 (0.3530) loss 2.4642 (3.3285) grad_norm 1.3817 (1.4740) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][460/625] eta 0:00:58 lr 0.001402 wd 0.0500 time 0.3483 (0.3541) data time 0.0007 (0.0018) model time 0.3476 (0.3528) loss 3.3759 (3.3311) grad_norm 1.8306 (1.4726) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][470/625] eta 0:00:54 lr 0.001402 wd 0.0500 time 0.3497 (0.3539) data time 0.0009 (0.0018) model time 0.3488 (0.3526) loss 3.8229 (3.3378) grad_norm 1.4644 (1.4721) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][480/625] eta 0:00:51 lr 0.001401 wd 0.0500 time 0.3445 (0.3537) data time 0.0008 (0.0018) model time 0.3437 (0.3524) loss 3.7469 (3.3418) grad_norm 1.6525 (1.4841) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][490/625] eta 0:00:47 lr 0.001401 wd 0.0500 time 0.3477 (0.3555) data time 0.0006 (0.0018) model time 0.3470 (0.3544) loss 3.8595 (3.3415) grad_norm 1.0289 (1.4824) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][500/625] eta 0:00:44 lr 0.001401 wd 0.0500 time 0.5438 (0.3557) data time 0.0007 (0.0017) model time 0.5431 (0.3546) loss 3.8397 (3.3319) grad_norm 1.3717 (1.4843) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][510/625] eta 0:00:40 lr 0.001401 wd 0.0500 time 0.3471 (0.3562) data time 0.0009 (0.0017) model time 0.3462 (0.3552) loss 3.5819 (3.3310) grad_norm 1.8611 (1.4824) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][520/625] eta 0:00:37 lr 0.001401 wd 0.0500 time 0.3465 (0.3560) data time 0.0007 (0.0017) model time 0.3458 (0.3550) loss 2.8409 (3.3259) grad_norm 1.9215 (1.4923) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][530/625] eta 0:00:33 lr 0.001401 wd 0.0500 time 0.3381 (0.3558) data time 0.0009 (0.0017) model time 0.3372 (0.3548) loss 3.5399 (3.3310) grad_norm 2.1773 (1.4975) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][540/625] eta 0:00:30 lr 0.001400 wd 0.0500 time 0.3486 (0.3557) data time 0.0007 (0.0017) model time 0.3480 (0.3546) loss 3.9015 (3.3329) grad_norm 0.9851 (1.4928) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][550/625] eta 0:00:26 lr 0.001400 wd 0.0500 time 0.3456 (0.3555) data time 0.0006 (0.0017) model time 0.3450 (0.3544) loss 3.6694 (3.3329) grad_norm 1.0719 (1.4899) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][560/625] eta 0:00:23 lr 0.001400 wd 0.0500 time 0.3451 (0.3554) data time 0.0006 (0.0017) model time 0.3446 (0.3543) loss 3.8489 (3.3351) grad_norm 1.2947 (1.4869) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:23:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][570/625] eta 0:00:19 lr 0.001400 wd 0.0500 time 0.3423 (0.3552) data time 0.0007 (0.0016) model time 0.3416 (0.3541) loss 3.0652 (3.3340) grad_norm 1.7475 (1.4846) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][580/625] eta 0:00:15 lr 0.001400 wd 0.0500 time 0.3499 (0.3550) data time 0.0009 (0.0016) model time 0.3491 (0.3539) loss 3.4480 (3.3340) grad_norm 0.9496 (1.4863) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][590/625] eta 0:00:12 lr 0.001400 wd 0.0500 time 0.3549 (0.3549) data time 0.0006 (0.0016) model time 0.3543 (0.3538) loss 2.6157 (3.3298) grad_norm 2.1482 (1.4930) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][600/625] eta 0:00:08 lr 0.001400 wd 0.0500 time 0.3484 (0.3547) data time 0.0006 (0.0016) model time 0.3478 (0.3536) loss 2.7859 (3.3270) grad_norm 1.2612 (1.4922) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][610/625] eta 0:00:05 lr 0.001399 wd 0.0500 time 0.3370 (0.3559) data time 0.0004 (0.0016) model time 0.3366 (0.3548) loss 4.1789 (3.3260) grad_norm 1.1457 (1.4872) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [123/300][620/625] eta 0:00:01 lr 0.001399 wd 0.0500 time 0.3355 (0.3559) data time 0.0004 (0.0016) model time 0.3351 (0.3548) loss 2.8444 (3.3223) grad_norm 1.6979 (1.4839) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:17 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 123 training takes 0:03:42 [2024-08-26 11:24:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:24:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:24:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.6611 (0.6611) Acc@1 86.182 (86.182) Acc@5 97.168 (97.168) Mem 11224MB [2024-08-26 11:24:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.138) Loss 1.0283 (0.7832) Acc@1 76.367 (82.662) Acc@5 94.141 (96.524) Mem 11224MB [2024-08-26 11:24:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.116) Loss 1.1885 (0.9333) Acc@1 72.559 (79.076) Acc@5 92.334 (94.708) Mem 11224MB [2024-08-26 11:24:21 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.827 Acc@5 94.710 [2024-08-26 11:24:21 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.8% [2024-08-26 11:24:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.802 (0.802) Loss 0.5312 (0.5312) Acc@1 88.281 (88.281) Acc@5 98.291 (98.291) Mem 11224MB [2024-08-26 11:24:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.166) Loss 0.8877 (0.6653) Acc@1 79.102 (85.005) Acc@5 95.410 (97.354) Mem 11224MB [2024-08-26 11:24:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.131) Loss 1.0244 (0.8011) Acc@1 74.902 (81.548) Acc@5 93.799 (95.896) Mem 11224MB [2024-08-26 11:24:24 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.256 Acc@5 95.891 [2024-08-26 11:24:24 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.3% [2024-08-26 11:24:24 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.26% [2024-08-26 11:24:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:24:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:24:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][0/625] eta 0:07:02 lr 0.001399 wd 0.0500 time 0.6753 (0.6753) data time 0.3603 (0.3603) model time 0.0000 (0.0000) loss 3.0856 (3.0856) grad_norm 1.2273 (1.2273) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][10/625] eta 0:03:52 lr 0.001399 wd 0.0500 time 0.3508 (0.3785) data time 0.0006 (0.0336) model time 0.0000 (0.0000) loss 4.3723 (3.2299) grad_norm 1.1868 (1.1932) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][20/625] eta 0:03:41 lr 0.001399 wd 0.0500 time 0.3586 (0.3654) data time 0.0009 (0.0180) model time 0.0000 (0.0000) loss 3.8499 (3.2316) grad_norm 1.0369 (1.2634) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][30/625] eta 0:03:34 lr 0.001399 wd 0.0500 time 0.3401 (0.3599) data time 0.0006 (0.0125) model time 0.0000 (0.0000) loss 2.4725 (3.1987) grad_norm 1.2476 (1.2454) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][40/625] eta 0:03:29 lr 0.001398 wd 0.0500 time 0.3451 (0.3583) data time 0.0007 (0.0097) model time 0.0000 (0.0000) loss 2.7184 (3.2003) grad_norm 1.1277 (1.3091) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][50/625] eta 0:03:28 lr 0.001398 wd 0.0500 time 0.4969 (0.3634) data time 0.0009 (0.0079) model time 0.0000 (0.0000) loss 2.9069 (3.2038) grad_norm 1.1751 (1.3431) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][60/625] eta 0:03:23 lr 0.001398 wd 0.0500 time 0.3478 (0.3608) data time 0.0006 (0.0069) model time 0.3472 (0.3458) loss 3.9376 (3.2600) grad_norm 1.1131 (1.3228) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][70/625] eta 0:03:19 lr 0.001398 wd 0.0500 time 0.3447 (0.3588) data time 0.0009 (0.0061) model time 0.3438 (0.3458) loss 3.1108 (3.2304) grad_norm 1.5404 (1.3111) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][80/625] eta 0:03:14 lr 0.001398 wd 0.0500 time 0.3485 (0.3573) data time 0.0006 (0.0054) model time 0.3479 (0.3458) loss 3.2852 (3.2028) grad_norm 1.4806 (1.3237) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][90/625] eta 0:03:10 lr 0.001398 wd 0.0500 time 0.3422 (0.3560) data time 0.0009 (0.0049) model time 0.3413 (0.3453) loss 2.8273 (3.2216) grad_norm 1.7089 (1.3716) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][100/625] eta 0:03:06 lr 0.001397 wd 0.0500 time 0.3474 (0.3552) data time 0.0008 (0.0045) model time 0.3466 (0.3457) loss 3.1483 (3.2126) grad_norm 1.7592 (1.3708) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][110/625] eta 0:03:02 lr 0.001397 wd 0.0500 time 0.3442 (0.3542) data time 0.0008 (0.0042) model time 0.3433 (0.3453) loss 2.8323 (3.2158) grad_norm 1.1284 (1.3647) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][120/625] eta 0:02:58 lr 0.001397 wd 0.0500 time 0.3412 (0.3535) data time 0.0006 (0.0039) model time 0.3406 (0.3452) loss 2.8624 (3.2183) grad_norm 1.8062 (1.4088) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][130/625] eta 0:02:55 lr 0.001397 wd 0.0500 time 0.3448 (0.3538) data time 0.0009 (0.0037) model time 0.3439 (0.3467) loss 2.3335 (3.1920) grad_norm 1.3599 (1.4095) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][140/625] eta 0:02:51 lr 0.001397 wd 0.0500 time 0.3428 (0.3533) data time 0.0009 (0.0035) model time 0.3419 (0.3466) loss 3.2981 (3.2042) grad_norm 1.2961 (1.3954) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][150/625] eta 0:02:47 lr 0.001397 wd 0.0500 time 0.3429 (0.3529) data time 0.0008 (0.0033) model time 0.3421 (0.3465) loss 4.0224 (3.2080) grad_norm 1.2805 (1.4012) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][160/625] eta 0:02:43 lr 0.001396 wd 0.0500 time 0.3498 (0.3524) data time 0.0006 (0.0032) model time 0.3492 (0.3463) loss 3.8119 (3.2163) grad_norm 1.3263 (1.3999) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][170/625] eta 0:02:42 lr 0.001396 wd 0.0500 time 0.5426 (0.3566) data time 0.0009 (0.0031) model time 0.5417 (0.3527) loss 3.4780 (3.2105) grad_norm 1.1663 (1.3898) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][180/625] eta 0:02:38 lr 0.001396 wd 0.0500 time 0.3427 (0.3560) data time 0.0009 (0.0029) model time 0.3418 (0.3521) loss 3.1099 (3.2026) grad_norm 1.4970 (1.3835) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][190/625] eta 0:02:34 lr 0.001396 wd 0.0500 time 0.3445 (0.3554) data time 0.0006 (0.0028) model time 0.3439 (0.3515) loss 3.3052 (3.2165) grad_norm 2.5204 (1.3959) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][200/625] eta 0:02:30 lr 0.001396 wd 0.0500 time 0.3460 (0.3551) data time 0.0006 (0.0028) model time 0.3454 (0.3511) loss 2.4323 (3.2080) grad_norm 1.4828 (1.3987) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][210/625] eta 0:02:27 lr 0.001396 wd 0.0500 time 0.3450 (0.3547) data time 0.0008 (0.0028) model time 0.3442 (0.3508) loss 2.9033 (3.1996) grad_norm 1.7159 (1.3995) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][220/625] eta 0:02:23 lr 0.001395 wd 0.0500 time 0.3428 (0.3542) data time 0.0007 (0.0027) model time 0.3422 (0.3504) loss 3.2725 (3.2089) grad_norm 1.7769 (1.4021) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][230/625] eta 0:02:19 lr 0.001395 wd 0.0500 time 0.3440 (0.3539) data time 0.0008 (0.0026) model time 0.3432 (0.3501) loss 2.7923 (3.2162) grad_norm 1.0998 (1.3938) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][240/625] eta 0:02:16 lr 0.001395 wd 0.0500 time 0.3439 (0.3536) data time 0.0008 (0.0025) model time 0.3431 (0.3499) loss 3.1358 (3.2153) grad_norm 1.3759 (1.3929) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][250/625] eta 0:02:12 lr 0.001395 wd 0.0500 time 0.3451 (0.3534) data time 0.0007 (0.0025) model time 0.3445 (0.3498) loss 3.9074 (3.2244) grad_norm 2.2991 (1.3954) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:25:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][260/625] eta 0:02:09 lr 0.001395 wd 0.0500 time 0.5475 (0.3549) data time 0.0009 (0.0024) model time 0.5466 (0.3517) loss 3.1987 (3.2295) grad_norm 1.4175 (1.3958) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][270/625] eta 0:02:05 lr 0.001395 wd 0.0500 time 0.3450 (0.3546) data time 0.0006 (0.0024) model time 0.3444 (0.3514) loss 3.6860 (3.2301) grad_norm 1.3019 (1.4001) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][280/625] eta 0:02:02 lr 0.001395 wd 0.0500 time 0.3429 (0.3544) data time 0.0006 (0.0024) model time 0.3423 (0.3512) loss 3.9521 (3.2524) grad_norm 1.3013 (1.3987) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][290/625] eta 0:01:59 lr 0.001394 wd 0.0500 time 0.5321 (0.3563) data time 0.0009 (0.0023) model time 0.5313 (0.3537) loss 3.0847 (3.2621) grad_norm 1.4881 (1.4003) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][300/625] eta 0:01:55 lr 0.001394 wd 0.0500 time 0.3444 (0.3560) data time 0.0006 (0.0023) model time 0.3438 (0.3534) loss 3.7741 (3.2715) grad_norm 1.1592 (1.3953) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][310/625] eta 0:01:52 lr 0.001394 wd 0.0500 time 0.4889 (0.3563) data time 0.0007 (0.0022) model time 0.4883 (0.3538) loss 3.0950 (3.2635) grad_norm 1.7998 (1.3938) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][320/625] eta 0:01:48 lr 0.001394 wd 0.0500 time 0.3647 (0.3561) data time 0.0006 (0.0022) model time 0.3641 (0.3536) loss 4.3917 (3.2675) grad_norm 1.3426 (1.3928) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][330/625] eta 0:01:44 lr 0.001394 wd 0.0500 time 0.3431 (0.3558) data time 0.0007 (0.0022) model time 0.3424 (0.3533) loss 2.6166 (3.2700) grad_norm 1.2914 (1.3874) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][340/625] eta 0:01:41 lr 0.001394 wd 0.0500 time 0.3459 (0.3556) data time 0.0006 (0.0021) model time 0.3453 (0.3531) loss 3.7196 (3.2688) grad_norm 1.4622 (1.3814) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][350/625] eta 0:01:37 lr 0.001393 wd 0.0500 time 0.3587 (0.3555) data time 0.0006 (0.0021) model time 0.3581 (0.3530) loss 3.4085 (3.2706) grad_norm 1.6012 (1.3876) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][360/625] eta 0:01:34 lr 0.001393 wd 0.0500 time 0.3507 (0.3553) data time 0.0007 (0.0021) model time 0.3500 (0.3529) loss 3.3941 (3.2678) grad_norm 1.4342 (1.3894) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][370/625] eta 0:01:30 lr 0.001393 wd 0.0500 time 0.3430 (0.3551) data time 0.0007 (0.0021) model time 0.3423 (0.3527) loss 2.8600 (3.2714) grad_norm 1.3215 (1.3920) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][380/625] eta 0:01:26 lr 0.001393 wd 0.0500 time 0.3503 (0.3550) data time 0.0008 (0.0020) model time 0.3495 (0.3525) loss 2.9208 (3.2697) grad_norm 1.2403 (1.3953) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][390/625] eta 0:01:23 lr 0.001393 wd 0.0500 time 0.3486 (0.3548) data time 0.0009 (0.0020) model time 0.3477 (0.3524) loss 3.6206 (3.2822) grad_norm 2.1247 (1.3998) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][400/625] eta 0:01:19 lr 0.001393 wd 0.0500 time 0.3443 (0.3546) data time 0.0008 (0.0020) model time 0.3435 (0.3522) loss 2.2893 (3.2828) grad_norm 1.6322 (1.3979) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][410/625] eta 0:01:16 lr 0.001392 wd 0.0500 time 0.3662 (0.3545) data time 0.0010 (0.0020) model time 0.3652 (0.3521) loss 3.5938 (3.2791) grad_norm 1.3778 (1.3950) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][420/625] eta 0:01:12 lr 0.001392 wd 0.0500 time 0.3443 (0.3544) data time 0.0006 (0.0020) model time 0.3436 (0.3520) loss 3.6532 (3.2809) grad_norm 1.1568 (1.3997) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:26:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][430/625] eta 0:01:09 lr 0.001392 wd 0.0500 time 0.5676 (0.3548) data time 0.0008 (0.0019) model time 0.5668 (0.3525) loss 2.6988 (3.2852) grad_norm 2.0705 (1.4001) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][440/625] eta 0:01:05 lr 0.001392 wd 0.0500 time 0.3597 (0.3551) data time 0.0009 (0.0019) model time 0.3588 (0.3529) loss 3.2887 (3.2884) grad_norm 1.3887 (1.3998) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][450/625] eta 0:01:02 lr 0.001392 wd 0.0500 time 0.3467 (0.3554) data time 0.0006 (0.0019) model time 0.3461 (0.3533) loss 2.4017 (3.2874) grad_norm 1.1518 (1.4036) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][460/625] eta 0:00:58 lr 0.001392 wd 0.0500 time 0.3482 (0.3553) data time 0.0010 (0.0019) model time 0.3473 (0.3532) loss 3.3415 (3.2823) grad_norm 1.2506 (1.4056) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][470/625] eta 0:00:55 lr 0.001391 wd 0.0500 time 0.3677 (0.3553) data time 0.0006 (0.0019) model time 0.3671 (0.3531) loss 3.9490 (3.2863) grad_norm 1.9650 (1.4092) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][480/625] eta 0:00:51 lr 0.001391 wd 0.0500 time 0.3413 (0.3552) data time 0.0009 (0.0019) model time 0.3404 (0.3531) loss 3.6252 (3.2856) grad_norm 2.4822 (1.4219) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][490/625] eta 0:00:47 lr 0.001391 wd 0.0500 time 0.3470 (0.3551) data time 0.0006 (0.0019) model time 0.3463 (0.3530) loss 3.5509 (3.2893) grad_norm 1.2489 (1.4223) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][500/625] eta 0:00:44 lr 0.001391 wd 0.0500 time 0.3477 (0.3563) data time 0.0006 (0.0018) model time 0.3470 (0.3543) loss 3.9685 (3.2869) grad_norm 1.2661 (1.4192) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][510/625] eta 0:00:40 lr 0.001391 wd 0.0500 time 0.3416 (0.3560) data time 0.0006 (0.0018) model time 0.3409 (0.3541) loss 3.7486 (3.2869) grad_norm 1.4691 (1.4166) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][520/625] eta 0:00:37 lr 0.001391 wd 0.0500 time 0.3431 (0.3558) data time 0.0007 (0.0018) model time 0.3424 (0.3539) loss 2.3388 (3.2884) grad_norm 1.2392 (1.4142) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][530/625] eta 0:00:33 lr 0.001390 wd 0.0500 time 0.3467 (0.3557) data time 0.0008 (0.0018) model time 0.3459 (0.3537) loss 2.9704 (3.2825) grad_norm 1.8233 (1.4110) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][540/625] eta 0:00:30 lr 0.001390 wd 0.0500 time 0.3377 (0.3555) data time 0.0009 (0.0018) model time 0.3368 (0.3535) loss 3.9591 (3.2835) grad_norm 1.9840 (1.4123) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][550/625] eta 0:00:26 lr 0.001390 wd 0.0500 time 0.3391 (0.3554) data time 0.0009 (0.0018) model time 0.3382 (0.3535) loss 4.1512 (3.2897) grad_norm 1.1092 (1.4301) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][560/625] eta 0:00:23 lr 0.001390 wd 0.0500 time 0.3482 (0.3557) data time 0.0007 (0.0018) model time 0.3475 (0.3538) loss 2.5838 (3.2898) grad_norm 1.4566 (1.4282) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][570/625] eta 0:00:19 lr 0.001390 wd 0.0500 time 0.3436 (0.3560) data time 0.0007 (0.0018) model time 0.3429 (0.3541) loss 3.1817 (3.2846) grad_norm 1.4780 (1.4256) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][580/625] eta 0:00:16 lr 0.001390 wd 0.0500 time 0.3430 (0.3559) data time 0.0008 (0.0018) model time 0.3423 (0.3540) loss 3.4725 (3.2868) grad_norm 1.4107 (1.4250) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][590/625] eta 0:00:12 lr 0.001389 wd 0.0500 time 0.3471 (0.3558) data time 0.0009 (0.0018) model time 0.3462 (0.3538) loss 3.2056 (3.2883) grad_norm 1.3015 (1.4289) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:27:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][600/625] eta 0:00:08 lr 0.001389 wd 0.0500 time 0.3492 (0.3556) data time 0.0007 (0.0018) model time 0.3485 (0.3537) loss 3.0710 (3.2908) grad_norm 1.1488 (1.4295) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][610/625] eta 0:00:05 lr 0.001389 wd 0.0500 time 0.3390 (0.3555) data time 0.0004 (0.0018) model time 0.3386 (0.3536) loss 3.2130 (3.2932) grad_norm 1.7010 (1.4302) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [124/300][620/625] eta 0:00:01 lr 0.001389 wd 0.0500 time 0.3391 (0.3552) data time 0.0004 (0.0018) model time 0.3387 (0.3533) loss 2.6045 (3.2906) grad_norm 1.5388 (1.4312) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:07 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 124 training takes 0:03:41 [2024-08-26 11:28:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:28:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:28:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.452 (0.452) Loss 0.6382 (0.6382) Acc@1 85.400 (85.400) Acc@5 97.754 (97.754) Mem 11224MB [2024-08-26 11:28:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 1.0488 (0.7721) Acc@1 75.244 (82.777) Acc@5 93.262 (96.675) Mem 11224MB [2024-08-26 11:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.1309 (0.9238) Acc@1 73.730 (79.192) Acc@5 92.627 (94.868) Mem 11224MB [2024-08-26 11:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.951 Acc@5 94.794 [2024-08-26 11:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.0% [2024-08-26 11:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 78.95% [2024-08-26 11:28:11 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 11:28:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 11:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.472 (0.472) Loss 0.5298 (0.5298) Acc@1 88.184 (88.184) Acc@5 98.389 (98.389) Mem 11224MB [2024-08-26 11:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.132) Loss 0.8853 (0.6644) Acc@1 79.346 (85.019) Acc@5 95.312 (97.328) Mem 11224MB [2024-08-26 11:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.114) Loss 1.0244 (0.7996) Acc@1 74.805 (81.596) Acc@5 93.896 (95.912) Mem 11224MB [2024-08-26 11:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.314 Acc@5 95.903 [2024-08-26 11:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.3% [2024-08-26 11:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.31% [2024-08-26 11:28:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:28:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][0/625] eta 0:07:37 lr 0.001389 wd 0.0500 time 0.7323 (0.7323) data time 0.4081 (0.4081) model time 0.0000 (0.0000) loss 3.1136 (3.1136) grad_norm 1.4501 (1.4501) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][10/625] eta 0:03:54 lr 0.001389 wd 0.0500 time 0.3432 (0.3813) data time 0.0008 (0.0380) model time 0.0000 (0.0000) loss 3.6148 (3.2061) grad_norm 1.3140 (1.3515) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][20/625] eta 0:03:40 lr 0.001389 wd 0.0500 time 0.3453 (0.3643) data time 0.0009 (0.0204) model time 0.0000 (0.0000) loss 2.8940 (3.1722) grad_norm 1.4722 (1.4578) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][30/625] eta 0:03:32 lr 0.001388 wd 0.0500 time 0.3462 (0.3579) data time 0.0006 (0.0141) model time 0.0000 (0.0000) loss 3.7991 (3.2507) grad_norm 1.0417 (1.4377) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][40/625] eta 0:03:28 lr 0.001388 wd 0.0500 time 0.3442 (0.3562) data time 0.0007 (0.0109) model time 0.0000 (0.0000) loss 3.8678 (3.2074) grad_norm 0.9984 (1.3901) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][50/625] eta 0:03:23 lr 0.001388 wd 0.0500 time 0.3421 (0.3540) data time 0.0009 (0.0089) model time 0.0000 (0.0000) loss 3.1783 (3.2425) grad_norm 1.5293 (1.3998) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][60/625] eta 0:03:21 lr 0.001388 wd 0.0500 time 0.3442 (0.3562) data time 0.0006 (0.0076) model time 0.3435 (0.3664) loss 3.1236 (3.1655) grad_norm 1.1384 (1.4000) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][70/625] eta 0:03:21 lr 0.001388 wd 0.0500 time 0.3517 (0.3635) data time 0.0006 (0.0067) model time 0.3511 (0.3869) loss 2.3361 (3.1573) grad_norm 1.3638 (1.3933) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][80/625] eta 0:03:17 lr 0.001388 wd 0.0500 time 0.4951 (0.3631) data time 0.0007 (0.0059) model time 0.4944 (0.3776) loss 2.9955 (3.1877) grad_norm 1.3860 (1.3987) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][90/625] eta 0:03:14 lr 0.001387 wd 0.0500 time 0.3553 (0.3633) data time 0.0009 (0.0054) model time 0.3544 (0.3742) loss 3.3148 (3.2079) grad_norm 1.4557 (1.3878) loss_scale 2048.0000 (1114.0220) mem 11224MB [2024-08-26 11:28:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][100/625] eta 0:03:09 lr 0.001387 wd 0.0500 time 0.3376 (0.3615) data time 0.0009 (0.0049) model time 0.3368 (0.3683) loss 3.6681 (3.2180) grad_norm 0.9934 (1.3707) loss_scale 2048.0000 (1206.4950) mem 11224MB [2024-08-26 11:28:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][110/625] eta 0:03:05 lr 0.001387 wd 0.0500 time 0.3473 (0.3600) data time 0.0009 (0.0046) model time 0.3464 (0.3641) loss 3.3655 (3.2248) grad_norm 1.4012 (1.4041) loss_scale 2048.0000 (1282.3063) mem 11224MB [2024-08-26 11:28:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][120/625] eta 0:03:01 lr 0.001387 wd 0.0500 time 0.3439 (0.3587) data time 0.0009 (0.0043) model time 0.3431 (0.3611) loss 3.6568 (3.2086) grad_norm 1.1388 (1.3867) loss_scale 2048.0000 (1345.5868) mem 11224MB [2024-08-26 11:29:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][130/625] eta 0:02:57 lr 0.001387 wd 0.0500 time 0.3589 (0.3580) data time 0.0007 (0.0041) model time 0.3582 (0.3595) loss 2.7900 (3.2220) grad_norm 1.1327 (1.3874) loss_scale 2048.0000 (1399.2061) mem 11224MB [2024-08-26 11:29:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][140/625] eta 0:02:53 lr 0.001387 wd 0.0500 time 0.3424 (0.3579) data time 0.0009 (0.0039) model time 0.3414 (0.3591) loss 3.3447 (3.2307) grad_norm 1.5932 (1.3896) loss_scale 2048.0000 (1445.2199) mem 11224MB [2024-08-26 11:29:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][150/625] eta 0:02:49 lr 0.001386 wd 0.0500 time 0.3445 (0.3572) data time 0.0007 (0.0037) model time 0.3438 (0.3578) loss 3.4842 (3.2371) grad_norm 1.0800 (1.3862) loss_scale 2048.0000 (1485.1391) mem 11224MB [2024-08-26 11:29:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][160/625] eta 0:02:45 lr 0.001386 wd 0.0500 time 0.3477 (0.3564) data time 0.0009 (0.0035) model time 0.3469 (0.3565) loss 2.6882 (3.2366) grad_norm 1.6643 (inf) loss_scale 1024.0000 (1501.0186) mem 11224MB [2024-08-26 11:29:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][170/625] eta 0:02:41 lr 0.001386 wd 0.0500 time 0.3480 (0.3559) data time 0.0008 (0.0034) model time 0.3472 (0.3557) loss 1.9538 (3.2550) grad_norm 1.1158 (inf) loss_scale 1024.0000 (1473.1228) mem 11224MB [2024-08-26 11:29:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][180/625] eta 0:02:38 lr 0.001386 wd 0.0500 time 0.3451 (0.3555) data time 0.0008 (0.0033) model time 0.3443 (0.3550) loss 3.1040 (3.2500) grad_norm 1.2799 (inf) loss_scale 1024.0000 (1448.3094) mem 11224MB [2024-08-26 11:29:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][190/625] eta 0:02:34 lr 0.001386 wd 0.0500 time 0.3452 (0.3552) data time 0.0008 (0.0032) model time 0.3444 (0.3545) loss 3.4055 (3.2584) grad_norm 1.5349 (inf) loss_scale 1024.0000 (1426.0942) mem 11224MB [2024-08-26 11:29:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][200/625] eta 0:02:30 lr 0.001386 wd 0.0500 time 0.3421 (0.3546) data time 0.0008 (0.0031) model time 0.3413 (0.3538) loss 3.7153 (3.2698) grad_norm 1.6028 (inf) loss_scale 1024.0000 (1406.0896) mem 11224MB [2024-08-26 11:29:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][210/625] eta 0:02:27 lr 0.001385 wd 0.0500 time 0.3472 (0.3543) data time 0.0009 (0.0030) model time 0.3463 (0.3533) loss 2.8133 (3.2785) grad_norm 2.1707 (inf) loss_scale 1024.0000 (1387.9810) mem 11224MB [2024-08-26 11:29:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][220/625] eta 0:02:23 lr 0.001385 wd 0.0500 time 0.3456 (0.3540) data time 0.0009 (0.0029) model time 0.3447 (0.3529) loss 2.0992 (3.2546) grad_norm 1.4743 (inf) loss_scale 1024.0000 (1371.5113) mem 11224MB [2024-08-26 11:29:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][230/625] eta 0:02:19 lr 0.001385 wd 0.0500 time 0.3425 (0.3538) data time 0.0009 (0.0028) model time 0.3416 (0.3526) loss 3.3829 (3.2616) grad_norm 1.6798 (inf) loss_scale 1024.0000 (1356.4675) mem 11224MB [2024-08-26 11:29:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][240/625] eta 0:02:16 lr 0.001385 wd 0.0500 time 0.5011 (0.3541) data time 0.0009 (0.0028) model time 0.5002 (0.3530) loss 2.6540 (3.2572) grad_norm 1.4903 (inf) loss_scale 1024.0000 (1342.6722) mem 11224MB [2024-08-26 11:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][250/625] eta 0:02:13 lr 0.001385 wd 0.0500 time 0.5516 (0.3570) data time 0.0008 (0.0027) model time 0.5508 (0.3566) loss 2.6345 (3.2592) grad_norm 1.4518 (inf) loss_scale 1024.0000 (1329.9761) mem 11224MB [2024-08-26 11:29:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][260/625] eta 0:02:10 lr 0.001385 wd 0.0500 time 0.3411 (0.3571) data time 0.0006 (0.0026) model time 0.3404 (0.3568) loss 2.9551 (3.2633) grad_norm 1.5821 (inf) loss_scale 1024.0000 (1318.2529) mem 11224MB [2024-08-26 11:29:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][270/625] eta 0:02:06 lr 0.001384 wd 0.0500 time 0.3505 (0.3567) data time 0.0007 (0.0026) model time 0.3498 (0.3563) loss 3.3089 (3.2628) grad_norm 1.1381 (inf) loss_scale 1024.0000 (1307.3948) mem 11224MB [2024-08-26 11:29:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][280/625] eta 0:02:02 lr 0.001384 wd 0.0500 time 0.3390 (0.3565) data time 0.0008 (0.0025) model time 0.3382 (0.3559) loss 3.7322 (3.2632) grad_norm 1.2561 (inf) loss_scale 1024.0000 (1297.3096) mem 11224MB [2024-08-26 11:30:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][290/625] eta 0:01:59 lr 0.001384 wd 0.0500 time 0.5641 (0.3568) data time 0.0006 (0.0024) model time 0.5634 (0.3563) loss 2.9078 (3.2672) grad_norm 1.1386 (inf) loss_scale 1024.0000 (1287.9175) mem 11224MB [2024-08-26 11:30:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][300/625] eta 0:01:56 lr 0.001384 wd 0.0500 time 0.3456 (0.3572) data time 0.0006 (0.0024) model time 0.3450 (0.3567) loss 4.5978 (3.2648) grad_norm 1.3083 (inf) loss_scale 1024.0000 (1279.1495) mem 11224MB [2024-08-26 11:30:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][310/625] eta 0:01:52 lr 0.001384 wd 0.0500 time 0.3484 (0.3570) data time 0.0009 (0.0024) model time 0.3475 (0.3565) loss 3.1762 (3.2646) grad_norm 1.1142 (inf) loss_scale 1024.0000 (1270.9453) mem 11224MB [2024-08-26 11:30:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][320/625] eta 0:01:48 lr 0.001384 wd 0.0500 time 0.3475 (0.3568) data time 0.0009 (0.0023) model time 0.3466 (0.3562) loss 3.1974 (3.2687) grad_norm 1.2464 (inf) loss_scale 1024.0000 (1263.2523) mem 11224MB [2024-08-26 11:30:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][330/625] eta 0:01:45 lr 0.001383 wd 0.0500 time 0.3736 (0.3565) data time 0.0007 (0.0023) model time 0.3730 (0.3559) loss 2.5823 (3.2668) grad_norm 1.1687 (inf) loss_scale 1024.0000 (1256.0242) mem 11224MB [2024-08-26 11:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][340/625] eta 0:01:41 lr 0.001383 wd 0.0500 time 0.3421 (0.3562) data time 0.0006 (0.0022) model time 0.3414 (0.3555) loss 3.9492 (3.2657) grad_norm 3.5518 (inf) loss_scale 1024.0000 (1249.2199) mem 11224MB [2024-08-26 11:30:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][350/625] eta 0:01:37 lr 0.001383 wd 0.0500 time 0.3476 (0.3560) data time 0.0009 (0.0022) model time 0.3468 (0.3553) loss 2.7783 (3.2698) grad_norm 1.4618 (inf) loss_scale 1024.0000 (1242.8034) mem 11224MB [2024-08-26 11:30:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][360/625] eta 0:01:34 lr 0.001383 wd 0.0500 time 0.6233 (0.3565) data time 0.0006 (0.0022) model time 0.6227 (0.3559) loss 2.8391 (3.2679) grad_norm 1.4284 (inf) loss_scale 1024.0000 (1236.7424) mem 11224MB [2024-08-26 11:30:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][370/625] eta 0:01:31 lr 0.001383 wd 0.0500 time 0.3414 (0.3580) data time 0.0008 (0.0022) model time 0.3406 (0.3575) loss 3.6641 (3.2753) grad_norm 1.7628 (inf) loss_scale 1024.0000 (1231.0081) mem 11224MB [2024-08-26 11:30:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][380/625] eta 0:01:27 lr 0.001383 wd 0.0500 time 0.3531 (0.3577) data time 0.0005 (0.0022) model time 0.3525 (0.3571) loss 3.8854 (3.2798) grad_norm 1.0112 (inf) loss_scale 1024.0000 (1225.5748) mem 11224MB [2024-08-26 11:30:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][390/625] eta 0:01:23 lr 0.001382 wd 0.0500 time 0.3398 (0.3574) data time 0.0008 (0.0021) model time 0.3389 (0.3568) loss 3.5190 (3.2795) grad_norm 2.0336 (inf) loss_scale 1024.0000 (1220.4194) mem 11224MB [2024-08-26 11:30:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][400/625] eta 0:01:20 lr 0.001382 wd 0.0500 time 0.3434 (0.3572) data time 0.0008 (0.0021) model time 0.3425 (0.3566) loss 3.5181 (3.2833) grad_norm 1.7182 (inf) loss_scale 1024.0000 (1215.5212) mem 11224MB [2024-08-26 11:30:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][410/625] eta 0:01:16 lr 0.001382 wd 0.0500 time 0.3735 (0.3571) data time 0.0009 (0.0021) model time 0.3726 (0.3564) loss 3.3541 (3.2893) grad_norm 1.3192 (inf) loss_scale 1024.0000 (1210.8613) mem 11224MB [2024-08-26 11:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][420/625] eta 0:01:13 lr 0.001382 wd 0.0500 time 0.3407 (0.3568) data time 0.0009 (0.0021) model time 0.3398 (0.3561) loss 3.0862 (3.2865) grad_norm 1.5283 (inf) loss_scale 1024.0000 (1206.4228) mem 11224MB [2024-08-26 11:30:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][430/625] eta 0:01:09 lr 0.001382 wd 0.0500 time 0.3455 (0.3566) data time 0.0009 (0.0021) model time 0.3446 (0.3557) loss 3.0329 (3.2848) grad_norm 1.3598 (inf) loss_scale 1024.0000 (1202.1903) mem 11224MB [2024-08-26 11:30:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][440/625] eta 0:01:05 lr 0.001382 wd 0.0500 time 0.3494 (0.3563) data time 0.0007 (0.0021) model time 0.3487 (0.3555) loss 4.2023 (3.2886) grad_norm 1.1918 (inf) loss_scale 1024.0000 (1198.1497) mem 11224MB [2024-08-26 11:30:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][450/625] eta 0:01:02 lr 0.001381 wd 0.0500 time 0.3490 (0.3563) data time 0.0008 (0.0020) model time 0.3482 (0.3554) loss 2.8214 (3.2902) grad_norm 1.5472 (inf) loss_scale 1024.0000 (1194.2882) mem 11224MB [2024-08-26 11:31:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][460/625] eta 0:00:58 lr 0.001381 wd 0.0500 time 0.3553 (0.3561) data time 0.0008 (0.0020) model time 0.3544 (0.3552) loss 3.0056 (3.2880) grad_norm 2.3850 (inf) loss_scale 1024.0000 (1190.5944) mem 11224MB [2024-08-26 11:31:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][470/625] eta 0:00:55 lr 0.001381 wd 0.0500 time 0.3531 (0.3562) data time 0.0006 (0.0020) model time 0.3525 (0.3553) loss 2.4147 (3.2796) grad_norm 1.1088 (inf) loss_scale 1024.0000 (1187.0573) mem 11224MB [2024-08-26 11:31:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][480/625] eta 0:00:51 lr 0.001381 wd 0.0500 time 0.3472 (0.3568) data time 0.0008 (0.0020) model time 0.3464 (0.3559) loss 3.8054 (3.2834) grad_norm 1.2725 (inf) loss_scale 1024.0000 (1183.6674) mem 11224MB [2024-08-26 11:31:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][490/625] eta 0:00:48 lr 0.001381 wd 0.0500 time 0.3511 (0.3566) data time 0.0006 (0.0020) model time 0.3505 (0.3557) loss 2.1483 (3.2818) grad_norm 1.0931 (inf) loss_scale 1024.0000 (1180.4155) mem 11224MB [2024-08-26 11:31:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][500/625] eta 0:00:44 lr 0.001381 wd 0.0500 time 0.3449 (0.3564) data time 0.0006 (0.0019) model time 0.3443 (0.3555) loss 3.7721 (3.2808) grad_norm 1.5562 (inf) loss_scale 1024.0000 (1177.2934) mem 11224MB [2024-08-26 11:31:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][510/625] eta 0:00:40 lr 0.001380 wd 0.0500 time 0.3580 (0.3563) data time 0.0009 (0.0019) model time 0.3571 (0.3553) loss 3.0942 (3.2791) grad_norm 1.5889 (inf) loss_scale 1024.0000 (1174.2935) mem 11224MB [2024-08-26 11:31:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][520/625] eta 0:00:37 lr 0.001380 wd 0.0500 time 0.3423 (0.3560) data time 0.0007 (0.0019) model time 0.3416 (0.3551) loss 2.9932 (3.2824) grad_norm 1.1900 (inf) loss_scale 1024.0000 (1171.4088) mem 11224MB [2024-08-26 11:31:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][530/625] eta 0:00:33 lr 0.001380 wd 0.0500 time 0.3434 (0.3558) data time 0.0008 (0.0019) model time 0.3426 (0.3549) loss 2.4936 (3.2779) grad_norm 1.8205 (inf) loss_scale 1024.0000 (1168.6328) mem 11224MB [2024-08-26 11:31:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][540/625] eta 0:00:30 lr 0.001380 wd 0.0500 time 0.3447 (0.3556) data time 0.0008 (0.0019) model time 0.3439 (0.3547) loss 3.0716 (3.2798) grad_norm 1.4025 (inf) loss_scale 1024.0000 (1165.9593) mem 11224MB [2024-08-26 11:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][550/625] eta 0:00:26 lr 0.001380 wd 0.0500 time 0.3439 (0.3555) data time 0.0006 (0.0019) model time 0.3433 (0.3545) loss 3.0998 (3.2725) grad_norm 1.3400 (inf) loss_scale 1024.0000 (1163.3829) mem 11224MB [2024-08-26 11:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][560/625] eta 0:00:23 lr 0.001380 wd 0.0500 time 0.3573 (0.3554) data time 0.0007 (0.0018) model time 0.3566 (0.3544) loss 3.0168 (3.2692) grad_norm 1.2443 (inf) loss_scale 1024.0000 (1160.8984) mem 11224MB [2024-08-26 11:31:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][570/625] eta 0:00:19 lr 0.001379 wd 0.0500 time 0.3423 (0.3555) data time 0.0008 (0.0018) model time 0.3415 (0.3546) loss 3.8698 (3.2727) grad_norm 1.3621 (inf) loss_scale 1024.0000 (1158.5009) mem 11224MB [2024-08-26 11:31:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][580/625] eta 0:00:16 lr 0.001379 wd 0.0500 time 0.5603 (0.3565) data time 0.0008 (0.0018) model time 0.5595 (0.3556) loss 3.3593 (3.2746) grad_norm 1.5209 (inf) loss_scale 1024.0000 (1156.1859) mem 11224MB [2024-08-26 11:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][590/625] eta 0:00:12 lr 0.001379 wd 0.0500 time 0.3518 (0.3563) data time 0.0006 (0.0018) model time 0.3511 (0.3554) loss 3.4028 (3.2763) grad_norm 1.5679 (inf) loss_scale 1024.0000 (1153.9492) mem 11224MB [2024-08-26 11:31:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][600/625] eta 0:00:08 lr 0.001379 wd 0.0500 time 0.3350 (0.3569) data time 0.0009 (0.0018) model time 0.3341 (0.3560) loss 3.8601 (3.2790) grad_norm 1.3939 (inf) loss_scale 1024.0000 (1151.7870) mem 11224MB [2024-08-26 11:31:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][610/625] eta 0:00:05 lr 0.001379 wd 0.0500 time 0.3405 (0.3567) data time 0.0004 (0.0018) model time 0.3401 (0.3558) loss 2.8794 (3.2769) grad_norm 1.3474 (inf) loss_scale 1024.0000 (1149.6956) mem 11224MB [2024-08-26 11:31:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [125/300][620/625] eta 0:00:01 lr 0.001379 wd 0.0500 time 0.3480 (0.3565) data time 0.0006 (0.0018) model time 0.3474 (0.3556) loss 3.5450 (3.2762) grad_norm 1.3466 (inf) loss_scale 1024.0000 (1147.6715) mem 11224MB [2024-08-26 11:31:59 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 125 training takes 0:03:42 [2024-08-26 11:31:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:32:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:32:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.469 (0.469) Loss 0.5850 (0.5850) Acc@1 87.793 (87.793) Acc@5 97.900 (97.900) Mem 11224MB [2024-08-26 11:32:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 1.0049 (0.7415) Acc@1 75.781 (82.693) Acc@5 94.141 (96.555) Mem 11224MB [2024-08-26 11:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.1484 (0.9018) Acc@1 72.949 (78.962) Acc@5 92.334 (94.659) Mem 11224MB [2024-08-26 11:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.807 Acc@5 94.678 [2024-08-26 11:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.8% [2024-08-26 11:32:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.783 (0.783) Loss 0.5293 (0.5293) Acc@1 88.184 (88.184) Acc@5 98.438 (98.438) Mem 11224MB [2024-08-26 11:32:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.163) Loss 0.8828 (0.6634) Acc@1 79.395 (85.010) Acc@5 95.312 (97.328) Mem 11224MB [2024-08-26 11:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.129) Loss 1.0225 (0.7982) Acc@1 74.951 (81.629) Acc@5 93.896 (95.896) Mem 11224MB [2024-08-26 11:32:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.334 Acc@5 95.899 [2024-08-26 11:32:06 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.3% [2024-08-26 11:32:06 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.33% [2024-08-26 11:32:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:32:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:32:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][0/625] eta 0:07:27 lr 0.001379 wd 0.0500 time 0.7159 (0.7159) data time 0.4065 (0.4065) model time 0.0000 (0.0000) loss 3.5305 (3.5305) grad_norm 1.2958 (1.2958) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][10/625] eta 0:03:53 lr 0.001378 wd 0.0500 time 0.3460 (0.3797) data time 0.0007 (0.0378) model time 0.0000 (0.0000) loss 3.3368 (3.2548) grad_norm 1.4487 (1.8355) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][20/625] eta 0:03:40 lr 0.001378 wd 0.0500 time 0.3458 (0.3638) data time 0.0008 (0.0202) model time 0.0000 (0.0000) loss 3.2721 (3.3203) grad_norm 1.0709 (1.5632) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][30/625] eta 0:03:33 lr 0.001378 wd 0.0500 time 0.3418 (0.3583) data time 0.0006 (0.0140) model time 0.0000 (0.0000) loss 3.9946 (3.3174) grad_norm 1.2014 (1.4734) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][40/625] eta 0:03:28 lr 0.001378 wd 0.0500 time 0.3552 (0.3563) data time 0.0006 (0.0110) model time 0.0000 (0.0000) loss 3.2567 (3.2823) grad_norm 1.0433 (1.4361) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][50/625] eta 0:03:23 lr 0.001378 wd 0.0500 time 0.3454 (0.3541) data time 0.0007 (0.0090) model time 0.0000 (0.0000) loss 3.1840 (3.2824) grad_norm 1.3565 (1.4272) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][60/625] eta 0:03:19 lr 0.001378 wd 0.0500 time 0.3481 (0.3529) data time 0.0008 (0.0077) model time 0.3473 (0.3456) loss 3.4836 (3.2979) grad_norm 1.1207 (1.5165) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][70/625] eta 0:03:15 lr 0.001377 wd 0.0500 time 0.3533 (0.3522) data time 0.0006 (0.0067) model time 0.3527 (0.3465) loss 3.4839 (3.2903) grad_norm 1.0420 (1.4817) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][80/625] eta 0:03:11 lr 0.001377 wd 0.0500 time 0.3440 (0.3517) data time 0.0006 (0.0060) model time 0.3434 (0.3467) loss 2.5271 (3.3132) grad_norm 1.2521 (1.4596) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][90/625] eta 0:03:07 lr 0.001377 wd 0.0500 time 0.3444 (0.3512) data time 0.0006 (0.0054) model time 0.3437 (0.3466) loss 3.7562 (3.3108) grad_norm 1.4182 (1.4393) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][100/625] eta 0:03:05 lr 0.001377 wd 0.0500 time 0.3491 (0.3529) data time 0.0007 (0.0050) model time 0.3484 (0.3507) loss 2.6077 (3.3127) grad_norm 1.2734 (1.4649) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][110/625] eta 0:03:03 lr 0.001377 wd 0.0500 time 0.3378 (0.3555) data time 0.0010 (0.0046) model time 0.3369 (0.3557) loss 2.9198 (3.2923) grad_norm 1.0567 (1.4529) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][120/625] eta 0:02:59 lr 0.001377 wd 0.0500 time 0.3464 (0.3546) data time 0.0009 (0.0043) model time 0.3456 (0.3540) loss 3.5842 (3.2918) grad_norm 1.0830 (1.4422) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][130/625] eta 0:02:55 lr 0.001376 wd 0.0500 time 0.3427 (0.3539) data time 0.0006 (0.0041) model time 0.3421 (0.3529) loss 3.6839 (3.2917) grad_norm 1.5204 (1.4486) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][140/625] eta 0:02:51 lr 0.001376 wd 0.0500 time 0.3393 (0.3533) data time 0.0009 (0.0039) model time 0.3384 (0.3519) loss 3.3721 (3.3125) grad_norm 1.0801 (1.4383) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][150/625] eta 0:02:48 lr 0.001376 wd 0.0500 time 0.3487 (0.3546) data time 0.0008 (0.0037) model time 0.3479 (0.3539) loss 3.1250 (3.3134) grad_norm 1.2570 (1.4383) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][160/625] eta 0:02:46 lr 0.001376 wd 0.0500 time 0.5726 (0.3584) data time 0.0006 (0.0035) model time 0.5719 (0.3594) loss 3.2018 (3.3213) grad_norm 1.1148 (1.4639) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][170/625] eta 0:02:42 lr 0.001376 wd 0.0500 time 0.3487 (0.3577) data time 0.0006 (0.0033) model time 0.3481 (0.3583) loss 3.5262 (3.3193) grad_norm 0.9612 (1.4512) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][180/625] eta 0:02:38 lr 0.001376 wd 0.0500 time 0.3379 (0.3570) data time 0.0008 (0.0032) model time 0.3371 (0.3572) loss 2.7279 (3.3197) grad_norm 1.5722 (1.4553) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][190/625] eta 0:02:35 lr 0.001375 wd 0.0500 time 0.3422 (0.3565) data time 0.0009 (0.0031) model time 0.3413 (0.3564) loss 2.7103 (3.3067) grad_norm 1.2496 (1.4570) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][200/625] eta 0:02:31 lr 0.001375 wd 0.0500 time 0.3395 (0.3561) data time 0.0007 (0.0030) model time 0.3388 (0.3558) loss 4.2625 (3.3143) grad_norm 1.2671 (1.4400) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][210/625] eta 0:02:27 lr 0.001375 wd 0.0500 time 0.3425 (0.3556) data time 0.0006 (0.0029) model time 0.3420 (0.3552) loss 3.2095 (3.3155) grad_norm 1.0944 (1.4403) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][220/625] eta 0:02:23 lr 0.001375 wd 0.0500 time 0.3515 (0.3552) data time 0.0007 (0.0028) model time 0.3508 (0.3546) loss 1.9598 (3.3029) grad_norm 0.9406 (1.4364) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][230/625] eta 0:02:20 lr 0.001375 wd 0.0500 time 0.3398 (0.3548) data time 0.0009 (0.0028) model time 0.3389 (0.3539) loss 3.2296 (3.3043) grad_norm 1.8001 (1.4375) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][240/625] eta 0:02:16 lr 0.001375 wd 0.0500 time 0.3482 (0.3544) data time 0.0008 (0.0027) model time 0.3474 (0.3534) loss 3.8508 (3.3124) grad_norm 1.1582 (1.4304) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][250/625] eta 0:02:12 lr 0.001374 wd 0.0500 time 0.3472 (0.3542) data time 0.0006 (0.0026) model time 0.3466 (0.3532) loss 2.9712 (3.3136) grad_norm 1.3315 (1.4276) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][260/625] eta 0:02:09 lr 0.001374 wd 0.0500 time 0.3431 (0.3539) data time 0.0006 (0.0026) model time 0.3425 (0.3528) loss 3.4555 (3.3170) grad_norm 1.1330 (1.4231) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][270/625] eta 0:02:05 lr 0.001374 wd 0.0500 time 0.3470 (0.3536) data time 0.0006 (0.0025) model time 0.3464 (0.3525) loss 3.9704 (3.3226) grad_norm 1.4597 (1.4183) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][280/625] eta 0:02:01 lr 0.001374 wd 0.0500 time 0.3440 (0.3533) data time 0.0007 (0.0024) model time 0.3434 (0.3521) loss 3.7112 (3.3247) grad_norm 1.5700 (1.4169) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][290/625] eta 0:01:58 lr 0.001374 wd 0.0500 time 0.3440 (0.3531) data time 0.0006 (0.0024) model time 0.3434 (0.3519) loss 2.4643 (3.3150) grad_norm 1.3382 (1.4228) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][300/625] eta 0:01:54 lr 0.001374 wd 0.0500 time 0.3449 (0.3531) data time 0.0009 (0.0023) model time 0.3441 (0.3519) loss 2.2004 (3.3123) grad_norm 1.1511 (1.4248) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:33:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][310/625] eta 0:01:51 lr 0.001373 wd 0.0500 time 0.3508 (0.3535) data time 0.0008 (0.0023) model time 0.3500 (0.3524) loss 3.4469 (3.3066) grad_norm 1.1719 (1.4272) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][320/625] eta 0:01:48 lr 0.001373 wd 0.0500 time 0.3422 (0.3544) data time 0.0009 (0.0023) model time 0.3413 (0.3535) loss 3.7028 (3.3151) grad_norm 1.0268 (1.4210) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][330/625] eta 0:01:44 lr 0.001373 wd 0.0500 time 0.3454 (0.3542) data time 0.0008 (0.0022) model time 0.3445 (0.3533) loss 3.6177 (3.3191) grad_norm 1.5514 (1.4139) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][340/625] eta 0:01:41 lr 0.001373 wd 0.0500 time 0.5473 (0.3576) data time 0.0010 (0.0022) model time 0.5463 (0.3572) loss 3.6367 (3.3182) grad_norm 1.2244 (1.4254) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][350/625] eta 0:01:38 lr 0.001373 wd 0.0500 time 0.3538 (0.3582) data time 0.0008 (0.0021) model time 0.3530 (0.3579) loss 3.4003 (3.3121) grad_norm 2.0647 (1.4288) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][360/625] eta 0:01:34 lr 0.001373 wd 0.0500 time 0.3452 (0.3578) data time 0.0009 (0.0021) model time 0.3444 (0.3575) loss 2.3907 (3.3036) grad_norm 1.4131 (1.4268) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][370/625] eta 0:01:31 lr 0.001372 wd 0.0500 time 0.3392 (0.3575) data time 0.0008 (0.0021) model time 0.3383 (0.3571) loss 3.2476 (3.3137) grad_norm 1.2678 (1.4245) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][380/625] eta 0:01:27 lr 0.001372 wd 0.0500 time 0.3453 (0.3572) data time 0.0007 (0.0020) model time 0.3446 (0.3567) loss 3.7260 (3.3156) grad_norm 1.2999 (1.4188) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][390/625] eta 0:01:23 lr 0.001372 wd 0.0500 time 0.3457 (0.3570) data time 0.0006 (0.0020) model time 0.3451 (0.3565) loss 3.3674 (3.3208) grad_norm 1.2956 (1.4177) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][400/625] eta 0:01:20 lr 0.001372 wd 0.0500 time 0.3409 (0.3567) data time 0.0009 (0.0020) model time 0.3401 (0.3562) loss 3.7657 (3.3171) grad_norm 1.1515 (1.4179) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][410/625] eta 0:01:16 lr 0.001372 wd 0.0500 time 0.3443 (0.3564) data time 0.0009 (0.0020) model time 0.3434 (0.3558) loss 3.6481 (3.3192) grad_norm 1.2245 (1.4194) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][420/625] eta 0:01:13 lr 0.001372 wd 0.0500 time 0.3466 (0.3562) data time 0.0007 (0.0019) model time 0.3459 (0.3556) loss 4.3065 (3.3249) grad_norm 1.5769 (1.4218) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][430/625] eta 0:01:09 lr 0.001371 wd 0.0500 time 0.3428 (0.3560) data time 0.0006 (0.0019) model time 0.3422 (0.3553) loss 3.6032 (3.3278) grad_norm 1.2185 (1.4313) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][440/625] eta 0:01:05 lr 0.001371 wd 0.0500 time 0.3446 (0.3558) data time 0.0009 (0.0019) model time 0.3437 (0.3551) loss 3.0410 (3.3251) grad_norm 1.5021 (1.4296) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][450/625] eta 0:01:02 lr 0.001371 wd 0.0500 time 0.5274 (0.3560) data time 0.0006 (0.0019) model time 0.5268 (0.3553) loss 2.4615 (3.3234) grad_norm 1.2940 (1.4342) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][460/625] eta 0:00:59 lr 0.001371 wd 0.0500 time 0.3450 (0.3579) data time 0.0008 (0.0018) model time 0.3442 (0.3574) loss 2.6258 (3.3175) grad_norm 1.2486 (1.4370) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][470/625] eta 0:00:55 lr 0.001371 wd 0.0500 time 0.3455 (0.3581) data time 0.0008 (0.0018) model time 0.3447 (0.3576) loss 3.4677 (3.3235) grad_norm 1.2582 (1.4358) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][480/625] eta 0:00:51 lr 0.001371 wd 0.0500 time 0.3425 (0.3579) data time 0.0007 (0.0018) model time 0.3418 (0.3574) loss 2.6128 (3.3234) grad_norm 1.0779 (1.4297) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][490/625] eta 0:00:48 lr 0.001370 wd 0.0500 time 0.3473 (0.3576) data time 0.0009 (0.0018) model time 0.3465 (0.3571) loss 3.4033 (3.3260) grad_norm 1.4401 (1.4278) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][500/625] eta 0:00:44 lr 0.001370 wd 0.0500 time 0.3261 (0.3589) data time 0.0008 (0.0018) model time 0.3253 (0.3585) loss 3.0671 (3.3282) grad_norm 1.6981 (1.4355) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][510/625] eta 0:00:41 lr 0.001370 wd 0.0500 time 0.3415 (0.3586) data time 0.0006 (0.0018) model time 0.3409 (0.3581) loss 3.4741 (3.3251) grad_norm 1.1566 (1.4331) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][520/625] eta 0:00:37 lr 0.001370 wd 0.0500 time 0.3441 (0.3584) data time 0.0007 (0.0018) model time 0.3434 (0.3579) loss 4.1610 (3.3308) grad_norm 1.2365 (1.4303) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][530/625] eta 0:00:34 lr 0.001370 wd 0.0500 time 0.3446 (0.3583) data time 0.0008 (0.0018) model time 0.3438 (0.3577) loss 1.7687 (3.3243) grad_norm 1.9587 (1.4305) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][540/625] eta 0:00:30 lr 0.001370 wd 0.0500 time 0.3376 (0.3581) data time 0.0006 (0.0018) model time 0.3369 (0.3574) loss 4.0903 (3.3221) grad_norm 1.2233 (1.4305) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][550/625] eta 0:00:26 lr 0.001369 wd 0.0500 time 0.3426 (0.3578) data time 0.0008 (0.0018) model time 0.3417 (0.3572) loss 3.5120 (3.3213) grad_norm 1.6080 (1.4292) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][560/625] eta 0:00:23 lr 0.001369 wd 0.0500 time 0.3476 (0.3577) data time 0.0007 (0.0017) model time 0.3469 (0.3570) loss 3.5790 (3.3157) grad_norm 1.8678 (1.4313) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][570/625] eta 0:00:19 lr 0.001369 wd 0.0500 time 0.3464 (0.3574) data time 0.0009 (0.0017) model time 0.3455 (0.3567) loss 3.0724 (3.3141) grad_norm 1.8280 (1.4368) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][580/625] eta 0:00:16 lr 0.001369 wd 0.0500 time 0.3458 (0.3572) data time 0.0008 (0.0017) model time 0.3450 (0.3565) loss 3.0121 (3.3176) grad_norm 1.3954 (1.4445) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][590/625] eta 0:00:12 lr 0.001369 wd 0.0500 time 0.3443 (0.3571) data time 0.0009 (0.0017) model time 0.3434 (0.3564) loss 3.4113 (3.3171) grad_norm 1.1192 (1.4405) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][600/625] eta 0:00:08 lr 0.001369 wd 0.0500 time 0.3459 (0.3569) data time 0.0008 (0.0017) model time 0.3451 (0.3562) loss 3.5084 (3.3197) grad_norm 1.5950 (1.4361) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][610/625] eta 0:00:05 lr 0.001368 wd 0.0500 time 0.5213 (0.3570) data time 0.0004 (0.0017) model time 0.5209 (0.3563) loss 3.4454 (3.3203) grad_norm 1.3137 (1.4335) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [126/300][620/625] eta 0:00:01 lr 0.001368 wd 0.0500 time 0.3117 (0.3576) data time 0.0007 (0.0017) model time 0.3110 (0.3569) loss 3.1059 (3.3227) grad_norm 1.1972 (1.4375) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:35:50 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 126 training takes 0:03:43 [2024-08-26 11:35:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:35:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:35:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.480 (0.480) Loss 0.6060 (0.6060) Acc@1 86.523 (86.523) Acc@5 97.705 (97.705) Mem 11224MB [2024-08-26 11:35:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.135) Loss 1.0430 (0.7798) Acc@1 75.439 (82.320) Acc@5 94.336 (96.498) Mem 11224MB [2024-08-26 11:35:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.114) Loss 1.1621 (0.9267) Acc@1 73.242 (79.013) Acc@5 91.846 (94.620) Mem 11224MB [2024-08-26 11:35:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.729 Acc@5 94.624 [2024-08-26 11:35:54 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.7% [2024-08-26 11:35:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.818 (0.818) Loss 0.5273 (0.5273) Acc@1 88.379 (88.379) Acc@5 98.438 (98.438) Mem 11224MB [2024-08-26 11:35:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.168) Loss 0.8818 (0.6628) Acc@1 79.443 (85.041) Acc@5 95.264 (97.346) Mem 11224MB [2024-08-26 11:35:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.132) Loss 1.0215 (0.7973) Acc@1 74.805 (81.641) Acc@5 93.652 (95.922) Mem 11224MB [2024-08-26 11:35:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.356 Acc@5 95.927 [2024-08-26 11:35:57 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.4% [2024-08-26 11:35:57 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.36% [2024-08-26 11:35:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:35:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:35:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][0/625] eta 0:07:34 lr 0.001368 wd 0.0500 time 0.7276 (0.7276) data time 0.4116 (0.4116) model time 0.0000 (0.0000) loss 3.3220 (3.3220) grad_norm 1.5051 (1.5051) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][10/625] eta 0:03:56 lr 0.001368 wd 0.0500 time 0.3427 (0.3851) data time 0.0008 (0.0386) model time 0.0000 (0.0000) loss 3.0175 (3.5036) grad_norm 1.4868 (1.2776) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][20/625] eta 0:03:46 lr 0.001368 wd 0.0500 time 0.5024 (0.3736) data time 0.0006 (0.0206) model time 0.0000 (0.0000) loss 3.4034 (3.3787) grad_norm 2.0396 (1.4360) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][30/625] eta 0:03:54 lr 0.001368 wd 0.0500 time 0.3446 (0.3938) data time 0.0011 (0.0143) model time 0.0000 (0.0000) loss 3.2488 (3.3465) grad_norm 1.3465 (1.4882) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][40/625] eta 0:03:43 lr 0.001368 wd 0.0500 time 0.3456 (0.3825) data time 0.0008 (0.0110) model time 0.0000 (0.0000) loss 3.2660 (3.3097) grad_norm 1.1480 (1.4559) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][50/625] eta 0:03:36 lr 0.001367 wd 0.0500 time 0.3446 (0.3763) data time 0.0006 (0.0092) model time 0.0000 (0.0000) loss 3.2910 (3.2996) grad_norm 0.9726 (1.4334) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][60/625] eta 0:03:29 lr 0.001367 wd 0.0500 time 0.3483 (0.3716) data time 0.0009 (0.0078) model time 0.3474 (0.3467) loss 3.2403 (3.2874) grad_norm 1.0010 (1.4084) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][70/625] eta 0:03:24 lr 0.001367 wd 0.0500 time 0.3478 (0.3683) data time 0.0008 (0.0069) model time 0.3470 (0.3471) loss 2.9014 (3.2730) grad_norm 1.4120 (1.4202) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][80/625] eta 0:03:19 lr 0.001367 wd 0.0500 time 0.3449 (0.3658) data time 0.0006 (0.0061) model time 0.3443 (0.3470) loss 3.3452 (3.2853) grad_norm 1.5712 (1.4198) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][90/625] eta 0:03:14 lr 0.001367 wd 0.0500 time 0.3542 (0.3636) data time 0.0008 (0.0056) model time 0.3534 (0.3465) loss 3.4184 (3.2955) grad_norm 0.9880 (1.4089) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][100/625] eta 0:03:10 lr 0.001367 wd 0.0500 time 0.3454 (0.3619) data time 0.0008 (0.0051) model time 0.3446 (0.3464) loss 3.9777 (3.3034) grad_norm 1.1404 (1.4102) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][110/625] eta 0:03:05 lr 0.001366 wd 0.0500 time 0.3451 (0.3604) data time 0.0009 (0.0047) model time 0.3443 (0.3459) loss 3.4766 (3.3152) grad_norm 1.2664 (1.4200) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][120/625] eta 0:03:01 lr 0.001366 wd 0.0500 time 0.3501 (0.3593) data time 0.0009 (0.0044) model time 0.3492 (0.3460) loss 3.6117 (3.3095) grad_norm 1.2723 (1.4178) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][130/625] eta 0:02:57 lr 0.001366 wd 0.0500 time 0.3446 (0.3585) data time 0.0006 (0.0041) model time 0.3440 (0.3463) loss 2.4597 (3.2964) grad_norm 1.5234 (1.4135) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][140/625] eta 0:02:53 lr 0.001366 wd 0.0500 time 0.3396 (0.3583) data time 0.0007 (0.0039) model time 0.3389 (0.3472) loss 2.2829 (3.2769) grad_norm 2.8299 (1.4318) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][150/625] eta 0:02:49 lr 0.001366 wd 0.0500 time 0.3517 (0.3576) data time 0.0008 (0.0037) model time 0.3508 (0.3471) loss 3.5237 (3.2737) grad_norm 1.3539 (1.4223) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][160/625] eta 0:02:45 lr 0.001366 wd 0.0500 time 0.3394 (0.3568) data time 0.0008 (0.0035) model time 0.3385 (0.3468) loss 3.5512 (3.2835) grad_norm 1.4024 (1.4219) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:36:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][170/625] eta 0:02:42 lr 0.001365 wd 0.0500 time 0.3530 (0.3570) data time 0.0007 (0.0034) model time 0.3523 (0.3479) loss 3.3545 (3.2870) grad_norm 2.3014 (1.4448) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][180/625] eta 0:02:39 lr 0.001365 wd 0.0500 time 0.3458 (0.3574) data time 0.0007 (0.0033) model time 0.3451 (0.3490) loss 4.0236 (3.2985) grad_norm 1.1114 (1.4437) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][190/625] eta 0:02:35 lr 0.001365 wd 0.0500 time 0.3437 (0.3568) data time 0.0009 (0.0031) model time 0.3428 (0.3488) loss 2.8644 (3.3057) grad_norm 1.1436 (1.4343) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][200/625] eta 0:02:31 lr 0.001365 wd 0.0500 time 0.3423 (0.3573) data time 0.0006 (0.0030) model time 0.3416 (0.3498) loss 3.5226 (3.2999) grad_norm 2.7715 (1.4309) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][210/625] eta 0:02:28 lr 0.001365 wd 0.0500 time 0.3518 (0.3567) data time 0.0007 (0.0029) model time 0.3511 (0.3496) loss 3.7731 (3.3070) grad_norm 1.3916 (1.4331) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][220/625] eta 0:02:24 lr 0.001365 wd 0.0500 time 0.3402 (0.3563) data time 0.0010 (0.0029) model time 0.3392 (0.3493) loss 4.1060 (3.3249) grad_norm 1.4900 (1.4322) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][230/625] eta 0:02:20 lr 0.001364 wd 0.0500 time 0.5769 (0.3568) data time 0.0006 (0.0028) model time 0.5762 (0.3503) loss 4.0528 (3.3099) grad_norm 1.3271 (1.4270) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][240/625] eta 0:02:18 lr 0.001364 wd 0.0500 time 0.3540 (0.3589) data time 0.0009 (0.0027) model time 0.3531 (0.3532) loss 3.5698 (3.3266) grad_norm 1.6901 (1.4283) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][250/625] eta 0:02:14 lr 0.001364 wd 0.0500 time 0.3462 (0.3583) data time 0.0007 (0.0027) model time 0.3455 (0.3528) loss 2.1762 (3.3113) grad_norm 1.6516 (1.4333) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][260/625] eta 0:02:10 lr 0.001364 wd 0.0500 time 0.3443 (0.3579) data time 0.0006 (0.0026) model time 0.3436 (0.3525) loss 3.3192 (3.3014) grad_norm 1.6346 (1.4397) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][270/625] eta 0:02:06 lr 0.001364 wd 0.0500 time 0.3445 (0.3575) data time 0.0007 (0.0025) model time 0.3438 (0.3522) loss 4.0165 (3.3111) grad_norm 1.2337 (1.4339) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][280/625] eta 0:02:03 lr 0.001364 wd 0.0500 time 0.3464 (0.3572) data time 0.0008 (0.0025) model time 0.3456 (0.3519) loss 3.9397 (3.3121) grad_norm 1.2617 (1.4283) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][290/625] eta 0:01:59 lr 0.001363 wd 0.0500 time 0.5347 (0.3582) data time 0.0007 (0.0024) model time 0.5341 (0.3533) loss 3.9047 (3.3126) grad_norm 1.5185 (1.4445) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][300/625] eta 0:01:56 lr 0.001363 wd 0.0500 time 0.3442 (0.3578) data time 0.0008 (0.0024) model time 0.3433 (0.3530) loss 2.3421 (3.2971) grad_norm 1.3446 (1.4435) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][310/625] eta 0:01:52 lr 0.001363 wd 0.0500 time 0.3489 (0.3574) data time 0.0007 (0.0023) model time 0.3482 (0.3527) loss 3.9351 (3.2957) grad_norm 1.5757 (1.4446) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][320/625] eta 0:01:48 lr 0.001363 wd 0.0500 time 0.3465 (0.3571) data time 0.0006 (0.0023) model time 0.3459 (0.3524) loss 2.0984 (3.2902) grad_norm 1.1861 (1.4423) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:37:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][330/625] eta 0:01:45 lr 0.001363 wd 0.0500 time 0.3449 (0.3571) data time 0.0007 (0.0023) model time 0.3442 (0.3526) loss 3.6305 (3.2927) grad_norm 0.8167 (1.4413) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][340/625] eta 0:01:41 lr 0.001363 wd 0.0500 time 0.3447 (0.3568) data time 0.0007 (0.0022) model time 0.3441 (0.3524) loss 3.6068 (3.2927) grad_norm 1.3476 (1.4398) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][350/625] eta 0:01:38 lr 0.001362 wd 0.0500 time 0.3511 (0.3566) data time 0.0007 (0.0022) model time 0.3504 (0.3522) loss 3.5102 (3.2884) grad_norm 1.3552 (1.4432) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][360/625] eta 0:01:34 lr 0.001362 wd 0.0500 time 0.3467 (0.3563) data time 0.0010 (0.0022) model time 0.3458 (0.3520) loss 3.4735 (3.2913) grad_norm 1.1852 (1.4408) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][370/625] eta 0:01:30 lr 0.001362 wd 0.0500 time 0.3464 (0.3560) data time 0.0006 (0.0021) model time 0.3458 (0.3517) loss 2.9645 (3.2826) grad_norm 1.0103 (1.4380) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][380/625] eta 0:01:27 lr 0.001362 wd 0.0500 time 0.3451 (0.3557) data time 0.0007 (0.0021) model time 0.3445 (0.3515) loss 3.2847 (3.2810) grad_norm 1.4434 (1.4360) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][390/625] eta 0:01:23 lr 0.001362 wd 0.0500 time 0.3434 (0.3559) data time 0.0007 (0.0021) model time 0.3428 (0.3519) loss 3.3072 (3.2802) grad_norm 1.6800 (1.4378) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][400/625] eta 0:01:20 lr 0.001362 wd 0.0500 time 0.3430 (0.3556) data time 0.0006 (0.0020) model time 0.3424 (0.3516) loss 3.2363 (3.2738) grad_norm 0.9861 (1.4379) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][410/625] eta 0:01:16 lr 0.001361 wd 0.0500 time 0.3454 (0.3554) data time 0.0006 (0.0020) model time 0.3447 (0.3514) loss 3.3627 (3.2775) grad_norm 1.6697 (1.4615) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][420/625] eta 0:01:13 lr 0.001361 wd 0.0500 time 0.3419 (0.3565) data time 0.0008 (0.0020) model time 0.3410 (0.3528) loss 3.3556 (3.2808) grad_norm 1.1407 (1.4634) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][430/625] eta 0:01:09 lr 0.001361 wd 0.0500 time 0.3409 (0.3567) data time 0.0009 (0.0020) model time 0.3400 (0.3531) loss 2.1599 (3.2738) grad_norm 1.0731 (1.4688) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][440/625] eta 0:01:05 lr 0.001361 wd 0.0500 time 0.3463 (0.3565) data time 0.0007 (0.0019) model time 0.3456 (0.3529) loss 2.5376 (3.2744) grad_norm 1.2813 (1.4717) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][450/625] eta 0:01:02 lr 0.001361 wd 0.0500 time 0.3437 (0.3566) data time 0.0007 (0.0019) model time 0.3430 (0.3531) loss 3.3053 (3.2757) grad_norm 0.9886 (1.4694) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][460/625] eta 0:00:58 lr 0.001361 wd 0.0500 time 0.3454 (0.3564) data time 0.0008 (0.0019) model time 0.3446 (0.3529) loss 3.5816 (3.2764) grad_norm 2.4975 (1.4735) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][470/625] eta 0:00:55 lr 0.001360 wd 0.0500 time 0.3410 (0.3562) data time 0.0008 (0.0019) model time 0.3402 (0.3527) loss 3.1263 (3.2827) grad_norm 1.4212 (1.4756) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][480/625] eta 0:00:51 lr 0.001360 wd 0.0500 time 0.3445 (0.3560) data time 0.0007 (0.0019) model time 0.3438 (0.3526) loss 3.8170 (3.2812) grad_norm 1.8243 (1.4798) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][490/625] eta 0:00:48 lr 0.001360 wd 0.0500 time 0.3431 (0.3558) data time 0.0008 (0.0019) model time 0.3423 (0.3524) loss 3.2523 (3.2791) grad_norm 1.3120 (1.4802) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:38:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][500/625] eta 0:00:44 lr 0.001360 wd 0.0500 time 0.3448 (0.3560) data time 0.0009 (0.0018) model time 0.3440 (0.3527) loss 3.0960 (3.2765) grad_norm 1.4633 (1.4773) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][510/625] eta 0:00:40 lr 0.001360 wd 0.0500 time 0.3429 (0.3562) data time 0.0009 (0.0018) model time 0.3420 (0.3529) loss 3.4475 (3.2749) grad_norm 1.2270 (1.4744) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][520/625] eta 0:00:37 lr 0.001360 wd 0.0500 time 0.3536 (0.3560) data time 0.0006 (0.0018) model time 0.3529 (0.3528) loss 2.2676 (3.2775) grad_norm 1.8001 (1.4712) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][530/625] eta 0:00:33 lr 0.001359 wd 0.0500 time 0.3476 (0.3559) data time 0.0008 (0.0018) model time 0.3468 (0.3527) loss 3.3929 (3.2830) grad_norm 1.2908 (1.4711) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][540/625] eta 0:00:30 lr 0.001359 wd 0.0500 time 0.3461 (0.3568) data time 0.0006 (0.0018) model time 0.3455 (0.3538) loss 2.6493 (3.2828) grad_norm 1.4745 (1.4697) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][550/625] eta 0:00:26 lr 0.001359 wd 0.0500 time 0.3483 (0.3571) data time 0.0008 (0.0018) model time 0.3474 (0.3541) loss 3.6297 (3.2817) grad_norm 1.5358 (1.4705) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][560/625] eta 0:00:23 lr 0.001359 wd 0.0500 time 0.3474 (0.3569) data time 0.0009 (0.0017) model time 0.3465 (0.3540) loss 3.8522 (3.2841) grad_norm 1.8727 (1.4697) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][570/625] eta 0:00:19 lr 0.001359 wd 0.0500 time 0.3476 (0.3568) data time 0.0008 (0.0017) model time 0.3468 (0.3539) loss 3.4366 (3.2829) grad_norm 1.2169 (1.4695) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][580/625] eta 0:00:16 lr 0.001359 wd 0.0500 time 0.3409 (0.3566) data time 0.0009 (0.0017) model time 0.3399 (0.3537) loss 2.3634 (3.2775) grad_norm 1.6131 (1.4703) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][590/625] eta 0:00:12 lr 0.001358 wd 0.0500 time 0.3504 (0.3564) data time 0.0006 (0.0017) model time 0.3498 (0.3536) loss 3.7297 (3.2785) grad_norm 1.3608 (1.4705) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][600/625] eta 0:00:08 lr 0.001358 wd 0.0500 time 0.3445 (0.3563) data time 0.0009 (0.0017) model time 0.3436 (0.3535) loss 3.4650 (3.2759) grad_norm 1.1805 (1.4694) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][610/625] eta 0:00:05 lr 0.001358 wd 0.0500 time 0.3374 (0.3562) data time 0.0004 (0.0017) model time 0.3370 (0.3533) loss 4.2952 (3.2771) grad_norm 1.5719 (1.4685) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [127/300][620/625] eta 0:00:01 lr 0.001358 wd 0.0500 time 0.3553 (0.3559) data time 0.0006 (0.0017) model time 0.3547 (0.3531) loss 3.3028 (3.2794) grad_norm 1.2906 (1.4671) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 127 training takes 0:03:42 [2024-08-26 11:39:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:39:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:39:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.450 (0.450) Loss 0.6270 (0.6270) Acc@1 86.230 (86.230) Acc@5 97.949 (97.949) Mem 11224MB [2024-08-26 11:39:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 1.0664 (0.7639) Acc@1 74.072 (82.573) Acc@5 93.311 (96.586) Mem 11224MB [2024-08-26 11:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.2168 (0.9128) Acc@1 70.117 (78.995) Acc@5 91.309 (94.773) Mem 11224MB [2024-08-26 11:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.809 Acc@5 94.754 [2024-08-26 11:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.8% [2024-08-26 11:39:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.785 (0.785) Loss 0.5273 (0.5273) Acc@1 88.330 (88.330) Acc@5 98.438 (98.438) Mem 11224MB [2024-08-26 11:39:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.166) Loss 0.8828 (0.6625) Acc@1 79.248 (85.032) Acc@5 95.410 (97.323) Mem 11224MB [2024-08-26 11:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.130) Loss 1.0205 (0.7965) Acc@1 74.854 (81.676) Acc@5 93.750 (95.931) Mem 11224MB [2024-08-26 11:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.398 Acc@5 95.925 [2024-08-26 11:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.4% [2024-08-26 11:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.40% [2024-08-26 11:39:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:39:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:39:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][0/625] eta 0:06:54 lr 0.001358 wd 0.0500 time 0.6635 (0.6635) data time 0.3390 (0.3390) model time 0.0000 (0.0000) loss 2.6977 (2.6977) grad_norm 1.0247 (1.0247) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][10/625] eta 0:03:50 lr 0.001358 wd 0.0500 time 0.3432 (0.3745) data time 0.0006 (0.0316) model time 0.0000 (0.0000) loss 4.0761 (3.4224) grad_norm 1.4751 (1.2292) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:39:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][20/625] eta 0:03:38 lr 0.001358 wd 0.0500 time 0.3508 (0.3614) data time 0.0008 (0.0171) model time 0.0000 (0.0000) loss 3.4151 (3.4600) grad_norm 1.1342 (1.2955) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][30/625] eta 0:03:32 lr 0.001357 wd 0.0500 time 0.3534 (0.3572) data time 0.0006 (0.0118) model time 0.0000 (0.0000) loss 3.7485 (3.3528) grad_norm 1.6650 (1.3334) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][40/625] eta 0:03:27 lr 0.001357 wd 0.0500 time 0.3450 (0.3545) data time 0.0007 (0.0092) model time 0.0000 (0.0000) loss 3.7754 (3.4198) grad_norm 0.9908 (1.3941) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][50/625] eta 0:03:27 lr 0.001357 wd 0.0500 time 0.3507 (0.3604) data time 0.0006 (0.0075) model time 0.0000 (0.0000) loss 2.8559 (3.4319) grad_norm 1.1804 (1.4000) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][60/625] eta 0:03:22 lr 0.001357 wd 0.0500 time 0.3495 (0.3582) data time 0.0009 (0.0065) model time 0.3485 (0.3459) loss 3.2331 (3.4115) grad_norm 1.1746 (1.4014) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][70/625] eta 0:03:24 lr 0.001357 wd 0.0500 time 0.5281 (0.3679) data time 0.0006 (0.0057) model time 0.5275 (0.3859) loss 4.1626 (3.4154) grad_norm 1.5569 (1.4244) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][80/625] eta 0:03:20 lr 0.001357 wd 0.0500 time 0.3429 (0.3678) data time 0.0008 (0.0051) model time 0.3420 (0.3793) loss 3.5403 (3.3951) grad_norm 2.0939 (1.4453) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][90/625] eta 0:03:15 lr 0.001356 wd 0.0500 time 0.3592 (0.3657) data time 0.0006 (0.0048) model time 0.3585 (0.3712) loss 4.2230 (3.4050) grad_norm 1.0919 (1.4398) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][100/625] eta 0:03:10 lr 0.001356 wd 0.0500 time 0.3423 (0.3637) data time 0.0009 (0.0044) model time 0.3414 (0.3659) loss 3.5151 (3.3970) grad_norm 1.1010 (1.4263) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][110/625] eta 0:03:06 lr 0.001356 wd 0.0500 time 0.3461 (0.3619) data time 0.0008 (0.0041) model time 0.3453 (0.3621) loss 3.1365 (3.4004) grad_norm 1.5583 (1.4238) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][120/625] eta 0:03:02 lr 0.001356 wd 0.0500 time 0.3630 (0.3607) data time 0.0009 (0.0038) model time 0.3621 (0.3598) loss 3.8720 (3.4030) grad_norm 1.2717 (1.4269) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][130/625] eta 0:02:57 lr 0.001356 wd 0.0500 time 0.3456 (0.3595) data time 0.0006 (0.0036) model time 0.3450 (0.3578) loss 2.4132 (3.3542) grad_norm 1.3585 (1.4519) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][140/625] eta 0:02:53 lr 0.001356 wd 0.0500 time 0.3457 (0.3586) data time 0.0007 (0.0034) model time 0.3450 (0.3566) loss 3.0203 (3.3272) grad_norm 1.3273 (1.4412) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][150/625] eta 0:02:49 lr 0.001355 wd 0.0500 time 0.3448 (0.3576) data time 0.0009 (0.0032) model time 0.3439 (0.3552) loss 3.7372 (3.3282) grad_norm 1.1143 (1.4497) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][160/625] eta 0:02:45 lr 0.001355 wd 0.0500 time 0.3398 (0.3570) data time 0.0007 (0.0031) model time 0.3391 (0.3543) loss 3.2084 (3.3396) grad_norm 1.0290 (1.4371) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][170/625] eta 0:02:42 lr 0.001355 wd 0.0500 time 0.3490 (0.3570) data time 0.0006 (0.0030) model time 0.3484 (0.3546) loss 2.3871 (3.3333) grad_norm 1.4016 (1.4352) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][180/625] eta 0:02:38 lr 0.001355 wd 0.0500 time 0.3417 (0.3564) data time 0.0009 (0.0028) model time 0.3409 (0.3538) loss 3.6289 (3.3292) grad_norm 1.2494 (1.4346) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:40:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][190/625] eta 0:02:34 lr 0.001355 wd 0.0500 time 0.3425 (0.3557) data time 0.0007 (0.0027) model time 0.3418 (0.3530) loss 3.8178 (3.3190) grad_norm 1.1296 (1.4232) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][200/625] eta 0:02:30 lr 0.001355 wd 0.0500 time 0.3430 (0.3552) data time 0.0007 (0.0027) model time 0.3423 (0.3524) loss 3.7640 (3.3216) grad_norm 1.9792 (1.4190) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][210/625] eta 0:02:27 lr 0.001354 wd 0.0500 time 0.3739 (0.3549) data time 0.0006 (0.0026) model time 0.3733 (0.3522) loss 3.7632 (3.3140) grad_norm 1.2619 (1.4222) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][220/625] eta 0:02:23 lr 0.001354 wd 0.0500 time 0.3456 (0.3545) data time 0.0008 (0.0025) model time 0.3449 (0.3517) loss 3.4649 (3.3040) grad_norm 1.1044 (1.4231) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][230/625] eta 0:02:19 lr 0.001354 wd 0.0500 time 0.3405 (0.3541) data time 0.0011 (0.0024) model time 0.3394 (0.3513) loss 2.8135 (3.3007) grad_norm 1.7680 (1.4270) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][240/625] eta 0:02:16 lr 0.001354 wd 0.0500 time 0.3424 (0.3538) data time 0.0007 (0.0024) model time 0.3417 (0.3511) loss 2.6413 (3.3010) grad_norm 1.2528 (1.4239) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][250/625] eta 0:02:13 lr 0.001354 wd 0.0500 time 0.3512 (0.3550) data time 0.0007 (0.0023) model time 0.3505 (0.3527) loss 4.2908 (3.3045) grad_norm 1.8055 (1.4189) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][260/625] eta 0:02:10 lr 0.001354 wd 0.0500 time 0.5292 (0.3565) data time 0.0007 (0.0023) model time 0.5285 (0.3546) loss 3.2645 (3.3007) grad_norm 2.0587 (1.4258) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][270/625] eta 0:02:06 lr 0.001353 wd 0.0500 time 0.3464 (0.3561) data time 0.0009 (0.0022) model time 0.3455 (0.3541) loss 3.1330 (3.2910) grad_norm 1.1560 (1.4342) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][280/625] eta 0:02:03 lr 0.001353 wd 0.0500 time 0.5720 (0.3581) data time 0.0008 (0.0022) model time 0.5712 (0.3566) loss 2.8625 (3.2959) grad_norm 1.5045 (1.4281) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:41:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][290/625] eta 0:02:00 lr 0.001353 wd 0.0500 time 0.3435 (0.3582) data time 0.0009 (0.0021) model time 0.3426 (0.3568) loss 3.9103 (3.3024) grad_norm 1.1259 (1.4211) loss_scale 2048.0000 (1052.1512) mem 11224MB [2024-08-26 11:41:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][300/625] eta 0:01:56 lr 0.001353 wd 0.0500 time 0.3384 (0.3578) data time 0.0009 (0.0021) model time 0.3376 (0.3563) loss 3.5949 (3.3128) grad_norm 1.2268 (1.4172) loss_scale 2048.0000 (1085.2359) mem 11224MB [2024-08-26 11:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][310/625] eta 0:01:52 lr 0.001353 wd 0.0500 time 0.3491 (0.3574) data time 0.0006 (0.0020) model time 0.3485 (0.3559) loss 3.2553 (3.3163) grad_norm 1.1362 (1.4178) loss_scale 2048.0000 (1116.1929) mem 11224MB [2024-08-26 11:41:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][320/625] eta 0:01:48 lr 0.001353 wd 0.0500 time 0.3503 (0.3571) data time 0.0009 (0.0020) model time 0.3494 (0.3555) loss 3.7392 (3.3198) grad_norm 2.3685 (1.4257) loss_scale 2048.0000 (1145.2212) mem 11224MB [2024-08-26 11:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][330/625] eta 0:01:45 lr 0.001352 wd 0.0500 time 0.3434 (0.3567) data time 0.0006 (0.0020) model time 0.3428 (0.3551) loss 3.7947 (3.3245) grad_norm 1.2611 (1.4209) loss_scale 2048.0000 (1172.4955) mem 11224MB [2024-08-26 11:41:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][340/625] eta 0:01:41 lr 0.001352 wd 0.0500 time 0.3424 (0.3563) data time 0.0010 (0.0019) model time 0.3414 (0.3547) loss 3.4670 (3.3249) grad_norm 1.1290 (1.4310) loss_scale 2048.0000 (1198.1701) mem 11224MB [2024-08-26 11:41:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][350/625] eta 0:01:37 lr 0.001352 wd 0.0500 time 0.3466 (0.3560) data time 0.0006 (0.0019) model time 0.3460 (0.3543) loss 3.4255 (3.3180) grad_norm 1.1184 (1.4303) loss_scale 2048.0000 (1222.3818) mem 11224MB [2024-08-26 11:41:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][360/625] eta 0:01:34 lr 0.001352 wd 0.0500 time 0.3412 (0.3557) data time 0.0009 (0.0019) model time 0.3403 (0.3540) loss 3.7112 (3.3197) grad_norm 1.0801 (1.4240) loss_scale 2048.0000 (1245.2521) mem 11224MB [2024-08-26 11:42:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][370/625] eta 0:01:30 lr 0.001352 wd 0.0500 time 0.3529 (0.3560) data time 0.0008 (0.0019) model time 0.3521 (0.3544) loss 3.1830 (3.3189) grad_norm 1.5060 (1.4248) loss_scale 2048.0000 (1266.8895) mem 11224MB [2024-08-26 11:42:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][380/625] eta 0:01:27 lr 0.001352 wd 0.0500 time 0.3436 (0.3564) data time 0.0009 (0.0018) model time 0.3427 (0.3548) loss 3.6185 (3.3182) grad_norm 1.8236 (1.4294) loss_scale 2048.0000 (1287.3911) mem 11224MB [2024-08-26 11:42:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][390/625] eta 0:01:23 lr 0.001351 wd 0.0500 time 0.3470 (0.3561) data time 0.0007 (0.0018) model time 0.3463 (0.3546) loss 2.3944 (3.3140) grad_norm 1.4801 (1.4394) loss_scale 2048.0000 (1306.8440) mem 11224MB [2024-08-26 11:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][400/625] eta 0:01:20 lr 0.001351 wd 0.0500 time 0.3578 (0.3559) data time 0.0006 (0.0018) model time 0.3571 (0.3543) loss 3.1894 (3.3151) grad_norm 1.0865 (1.4353) loss_scale 2048.0000 (1325.3267) mem 11224MB [2024-08-26 11:42:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][410/625] eta 0:01:16 lr 0.001351 wd 0.0500 time 0.3413 (0.3557) data time 0.0009 (0.0018) model time 0.3404 (0.3541) loss 3.4666 (3.3135) grad_norm 1.3779 (1.4335) loss_scale 2048.0000 (1342.9100) mem 11224MB [2024-08-26 11:42:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][420/625] eta 0:01:12 lr 0.001351 wd 0.0500 time 0.3421 (0.3555) data time 0.0008 (0.0018) model time 0.3413 (0.3539) loss 2.4520 (3.3163) grad_norm 1.7184 (1.4355) loss_scale 2048.0000 (1359.6580) mem 11224MB [2024-08-26 11:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][430/625] eta 0:01:09 lr 0.001351 wd 0.0500 time 0.3612 (0.3553) data time 0.0008 (0.0017) model time 0.3604 (0.3537) loss 3.4887 (3.3191) grad_norm 1.6510 (1.4431) loss_scale 2048.0000 (1375.6288) mem 11224MB [2024-08-26 11:42:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][440/625] eta 0:01:05 lr 0.001351 wd 0.0500 time 0.3466 (0.3552) data time 0.0009 (0.0017) model time 0.3457 (0.3536) loss 4.3576 (3.3191) grad_norm 1.1141 (1.4414) loss_scale 2048.0000 (1390.8753) mem 11224MB [2024-08-26 11:42:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][450/625] eta 0:01:02 lr 0.001350 wd 0.0500 time 0.3469 (0.3558) data time 0.0007 (0.0017) model time 0.3462 (0.3543) loss 3.7166 (3.3180) grad_norm 1.5275 (1.4417) loss_scale 2048.0000 (1405.4457) mem 11224MB [2024-08-26 11:42:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][460/625] eta 0:00:58 lr 0.001350 wd 0.0500 time 0.5530 (0.3566) data time 0.0007 (0.0017) model time 0.5523 (0.3551) loss 3.7021 (3.3143) grad_norm 1.2939 (1.4424) loss_scale 2048.0000 (1419.3839) mem 11224MB [2024-08-26 11:42:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][470/625] eta 0:00:55 lr 0.001350 wd 0.0500 time 0.3331 (0.3575) data time 0.0009 (0.0017) model time 0.3322 (0.3562) loss 2.9900 (3.3156) grad_norm inf (inf) loss_scale 1024.0000 (1430.5563) mem 11224MB [2024-08-26 11:42:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][480/625] eta 0:00:51 lr 0.001350 wd 0.0500 time 0.3479 (0.3573) data time 0.0008 (0.0017) model time 0.3471 (0.3559) loss 2.9855 (3.3083) grad_norm 1.3663 (inf) loss_scale 1024.0000 (1422.1040) mem 11224MB [2024-08-26 11:42:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][490/625] eta 0:00:48 lr 0.001350 wd 0.0500 time 0.3484 (0.3570) data time 0.0006 (0.0016) model time 0.3478 (0.3557) loss 2.7410 (3.3057) grad_norm 1.4866 (inf) loss_scale 1024.0000 (1413.9959) mem 11224MB [2024-08-26 11:42:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][500/625] eta 0:00:44 lr 0.001350 wd 0.0500 time 0.3436 (0.3572) data time 0.0006 (0.0016) model time 0.3430 (0.3559) loss 3.1612 (3.3060) grad_norm 1.1631 (inf) loss_scale 1024.0000 (1406.2116) mem 11224MB [2024-08-26 11:42:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][510/625] eta 0:00:41 lr 0.001349 wd 0.0500 time 0.3438 (0.3570) data time 0.0006 (0.0016) model time 0.3432 (0.3557) loss 3.5387 (3.3084) grad_norm 1.5039 (inf) loss_scale 1024.0000 (1398.7319) mem 11224MB [2024-08-26 11:42:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][520/625] eta 0:00:37 lr 0.001349 wd 0.0500 time 0.3476 (0.3568) data time 0.0008 (0.0016) model time 0.3467 (0.3555) loss 3.0918 (3.3071) grad_norm 1.7419 (inf) loss_scale 1024.0000 (1391.5393) mem 11224MB [2024-08-26 11:42:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][530/625] eta 0:00:33 lr 0.001349 wd 0.0500 time 0.3418 (0.3566) data time 0.0008 (0.0016) model time 0.3410 (0.3552) loss 3.8133 (3.3081) grad_norm 1.4382 (inf) loss_scale 1024.0000 (1384.6177) mem 11224MB [2024-08-26 11:43:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][540/625] eta 0:00:30 lr 0.001349 wd 0.0500 time 0.3481 (0.3564) data time 0.0006 (0.0016) model time 0.3474 (0.3550) loss 3.8246 (3.3091) grad_norm 1.3395 (inf) loss_scale 1024.0000 (1377.9519) mem 11224MB [2024-08-26 11:43:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][550/625] eta 0:00:26 lr 0.001349 wd 0.0500 time 0.3457 (0.3562) data time 0.0009 (0.0016) model time 0.3448 (0.3549) loss 3.3150 (3.3086) grad_norm 1.1683 (inf) loss_scale 1024.0000 (1371.5281) mem 11224MB [2024-08-26 11:43:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][560/625] eta 0:00:23 lr 0.001349 wd 0.0500 time 0.5202 (0.3564) data time 0.0009 (0.0016) model time 0.5193 (0.3550) loss 3.4366 (3.3097) grad_norm 1.9479 (inf) loss_scale 1024.0000 (1365.3333) mem 11224MB [2024-08-26 11:43:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][570/625] eta 0:00:19 lr 0.001348 wd 0.0500 time 0.3485 (0.3566) data time 0.0008 (0.0015) model time 0.3476 (0.3552) loss 3.4303 (3.3109) grad_norm 1.1860 (inf) loss_scale 1024.0000 (1359.3555) mem 11224MB [2024-08-26 11:43:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][580/625] eta 0:00:16 lr 0.001348 wd 0.0500 time 0.5453 (0.3578) data time 0.0008 (0.0015) model time 0.5445 (0.3566) loss 3.3190 (3.3069) grad_norm 1.6313 (inf) loss_scale 1024.0000 (1353.5835) mem 11224MB [2024-08-26 11:43:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][590/625] eta 0:00:12 lr 0.001348 wd 0.0500 time 0.3390 (0.3583) data time 0.0007 (0.0015) model time 0.3384 (0.3571) loss 3.9216 (3.3049) grad_norm 1.3549 (inf) loss_scale 1024.0000 (1348.0068) mem 11224MB [2024-08-26 11:43:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][600/625] eta 0:00:08 lr 0.001348 wd 0.0500 time 0.3419 (0.3581) data time 0.0006 (0.0015) model time 0.3412 (0.3569) loss 4.2752 (3.3065) grad_norm 1.0595 (inf) loss_scale 1024.0000 (1342.6156) mem 11224MB [2024-08-26 11:43:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][610/625] eta 0:00:05 lr 0.001348 wd 0.0500 time 0.3357 (0.3579) data time 0.0006 (0.0015) model time 0.3351 (0.3567) loss 2.9399 (3.3038) grad_norm 1.6883 (inf) loss_scale 1024.0000 (1337.4010) mem 11224MB [2024-08-26 11:43:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [128/300][620/625] eta 0:00:01 lr 0.001348 wd 0.0500 time 0.3379 (0.3576) data time 0.0006 (0.0015) model time 0.3373 (0.3564) loss 3.1940 (3.3027) grad_norm 1.4313 (inf) loss_scale 1024.0000 (1332.3543) mem 11224MB [2024-08-26 11:43:32 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 128 training takes 0:03:43 [2024-08-26 11:43:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:43:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:43:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.464 (0.464) Loss 0.6274 (0.6274) Acc@1 86.670 (86.670) Acc@5 98.096 (98.096) Mem 11224MB [2024-08-26 11:43:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.133) Loss 1.0195 (0.7732) Acc@1 76.025 (82.773) Acc@5 94.238 (96.697) Mem 11224MB [2024-08-26 11:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 1.1816 (0.9192) Acc@1 71.973 (79.153) Acc@5 91.943 (94.920) Mem 11224MB [2024-08-26 11:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.931 Acc@5 94.890 [2024-08-26 11:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 78.9% [2024-08-26 11:43:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.760 (0.760) Loss 0.5254 (0.5254) Acc@1 88.379 (88.379) Acc@5 98.438 (98.438) Mem 11224MB [2024-08-26 11:43:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.162) Loss 0.8828 (0.6618) Acc@1 79.248 (85.054) Acc@5 95.312 (97.341) Mem 11224MB [2024-08-26 11:43:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.129) Loss 1.0176 (0.7952) Acc@1 75.098 (81.727) Acc@5 93.848 (95.945) Mem 11224MB [2024-08-26 11:43:39 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.462 Acc@5 95.931 [2024-08-26 11:43:39 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.5% [2024-08-26 11:43:39 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.46% [2024-08-26 11:43:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:43:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:43:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][0/625] eta 0:07:12 lr 0.001347 wd 0.0500 time 0.6915 (0.6915) data time 0.3792 (0.3792) model time 0.0000 (0.0000) loss 3.7768 (3.7768) grad_norm 1.3818 (1.3818) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:43:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][10/625] eta 0:04:03 lr 0.001347 wd 0.0500 time 0.3746 (0.3961) data time 0.0007 (0.0353) model time 0.0000 (0.0000) loss 3.4400 (3.4036) grad_norm 1.4316 (1.4130) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:43:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][20/625] eta 0:03:47 lr 0.001347 wd 0.0500 time 0.3420 (0.3765) data time 0.0007 (0.0223) model time 0.0000 (0.0000) loss 4.3362 (3.4347) grad_norm 2.4635 (1.8033) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:43:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][30/625] eta 0:03:37 lr 0.001347 wd 0.0500 time 0.3487 (0.3663) data time 0.0008 (0.0156) model time 0.0000 (0.0000) loss 2.8691 (3.4388) grad_norm 1.1869 (1.7691) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:43:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][40/625] eta 0:03:31 lr 0.001347 wd 0.0500 time 0.3521 (0.3618) data time 0.0008 (0.0120) model time 0.0000 (0.0000) loss 3.4207 (3.4907) grad_norm 1.3365 (1.6613) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][50/625] eta 0:03:26 lr 0.001347 wd 0.0500 time 0.3375 (0.3584) data time 0.0009 (0.0098) model time 0.0000 (0.0000) loss 2.8737 (3.3919) grad_norm 1.0634 (1.6591) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][60/625] eta 0:03:21 lr 0.001346 wd 0.0500 time 0.3442 (0.3563) data time 0.0009 (0.0084) model time 0.3434 (0.3449) loss 3.5840 (3.4242) grad_norm 1.3311 (1.6124) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][70/625] eta 0:03:17 lr 0.001346 wd 0.0500 time 0.3441 (0.3551) data time 0.0007 (0.0073) model time 0.3435 (0.3459) loss 3.8451 (3.4146) grad_norm 1.2018 (1.5488) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][80/625] eta 0:03:12 lr 0.001346 wd 0.0500 time 0.3428 (0.3540) data time 0.0006 (0.0065) model time 0.3421 (0.3455) loss 3.8317 (3.4125) grad_norm 1.1569 (1.5256) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][90/625] eta 0:03:10 lr 0.001346 wd 0.0500 time 0.3498 (0.3553) data time 0.0007 (0.0059) model time 0.3491 (0.3505) loss 3.0226 (3.3906) grad_norm 1.0198 (1.4878) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][100/625] eta 0:03:09 lr 0.001346 wd 0.0500 time 0.3389 (0.3604) data time 0.0007 (0.0057) model time 0.3382 (0.3611) loss 2.4362 (3.3895) grad_norm 1.6716 (1.4903) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][110/625] eta 0:03:05 lr 0.001346 wd 0.0500 time 0.3389 (0.3593) data time 0.0007 (0.0053) model time 0.3382 (0.3587) loss 4.1509 (3.3867) grad_norm 1.6891 (1.5215) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][120/625] eta 0:03:01 lr 0.001345 wd 0.0500 time 0.5779 (0.3603) data time 0.0008 (0.0050) model time 0.5771 (0.3602) loss 3.5335 (3.3964) grad_norm 1.1699 (1.5175) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][130/625] eta 0:02:57 lr 0.001345 wd 0.0500 time 0.3510 (0.3590) data time 0.0009 (0.0047) model time 0.3501 (0.3581) loss 3.5124 (3.3930) grad_norm 0.9644 (1.4861) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][140/625] eta 0:02:53 lr 0.001345 wd 0.0500 time 0.3437 (0.3582) data time 0.0007 (0.0044) model time 0.3430 (0.3568) loss 3.9089 (3.3835) grad_norm 1.2868 (1.4672) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][150/625] eta 0:02:49 lr 0.001345 wd 0.0500 time 0.3479 (0.3576) data time 0.0009 (0.0042) model time 0.3470 (0.3559) loss 2.9193 (3.3672) grad_norm 1.8998 (1.4669) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][160/625] eta 0:02:47 lr 0.001345 wd 0.0500 time 0.3466 (0.3593) data time 0.0009 (0.0040) model time 0.3457 (0.3584) loss 3.2220 (3.3475) grad_norm 1.2439 (1.4540) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][170/625] eta 0:02:43 lr 0.001345 wd 0.0500 time 0.3436 (0.3584) data time 0.0007 (0.0038) model time 0.3429 (0.3572) loss 3.7081 (3.3532) grad_norm 1.7260 (1.4550) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][180/625] eta 0:02:39 lr 0.001344 wd 0.0500 time 0.3626 (0.3578) data time 0.0008 (0.0037) model time 0.3619 (0.3563) loss 3.5796 (3.3621) grad_norm 1.2162 (1.4450) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][190/625] eta 0:02:35 lr 0.001344 wd 0.0500 time 0.3466 (0.3572) data time 0.0007 (0.0035) model time 0.3460 (0.3555) loss 3.1442 (3.3518) grad_norm 1.4836 (1.4463) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][200/625] eta 0:02:31 lr 0.001344 wd 0.0500 time 0.3475 (0.3567) data time 0.0006 (0.0034) model time 0.3469 (0.3549) loss 3.6297 (3.3584) grad_norm 1.9269 (1.4522) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][210/625] eta 0:02:27 lr 0.001344 wd 0.0500 time 0.3425 (0.3561) data time 0.0008 (0.0033) model time 0.3417 (0.3541) loss 3.6420 (3.3603) grad_norm 1.1805 (1.4507) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:44:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][220/625] eta 0:02:23 lr 0.001344 wd 0.0500 time 0.3411 (0.3555) data time 0.0006 (0.0032) model time 0.3405 (0.3535) loss 2.9651 (3.3466) grad_norm 1.9458 (1.4569) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][230/625] eta 0:02:20 lr 0.001344 wd 0.0500 time 0.3471 (0.3551) data time 0.0007 (0.0031) model time 0.3464 (0.3530) loss 2.8778 (3.3357) grad_norm 1.1494 (1.4528) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][240/625] eta 0:02:16 lr 0.001343 wd 0.0500 time 0.3448 (0.3547) data time 0.0007 (0.0030) model time 0.3442 (0.3526) loss 2.9062 (3.3314) grad_norm 1.0990 (1.4417) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][250/625] eta 0:02:12 lr 0.001343 wd 0.0500 time 0.3463 (0.3544) data time 0.0006 (0.0029) model time 0.3457 (0.3522) loss 3.4620 (3.3499) grad_norm 1.3430 (1.4319) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][260/625] eta 0:02:09 lr 0.001343 wd 0.0500 time 0.3458 (0.3539) data time 0.0007 (0.0028) model time 0.3451 (0.3517) loss 3.7719 (3.3590) grad_norm 1.9250 (1.4258) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][270/625] eta 0:02:05 lr 0.001343 wd 0.0500 time 0.3461 (0.3536) data time 0.0008 (0.0028) model time 0.3453 (0.3513) loss 2.9408 (3.3595) grad_norm 1.4795 (1.4326) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][280/625] eta 0:02:01 lr 0.001343 wd 0.0500 time 0.3519 (0.3533) data time 0.0008 (0.0027) model time 0.3511 (0.3511) loss 3.2949 (3.3527) grad_norm 1.3366 (1.4364) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][290/625] eta 0:01:58 lr 0.001343 wd 0.0500 time 0.3400 (0.3530) data time 0.0008 (0.0026) model time 0.3392 (0.3508) loss 3.7042 (3.3531) grad_norm 1.3268 (1.4412) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][300/625] eta 0:01:54 lr 0.001342 wd 0.0500 time 0.3399 (0.3534) data time 0.0007 (0.0026) model time 0.3392 (0.3513) loss 3.8185 (3.3543) grad_norm 1.1715 (1.4390) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][310/625] eta 0:01:51 lr 0.001342 wd 0.0500 time 0.3418 (0.3546) data time 0.0007 (0.0025) model time 0.3411 (0.3527) loss 3.0635 (3.3545) grad_norm 1.6493 (1.4700) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][320/625] eta 0:01:48 lr 0.001342 wd 0.0500 time 0.3436 (0.3543) data time 0.0009 (0.0025) model time 0.3427 (0.3525) loss 3.1462 (3.3455) grad_norm 1.7221 (1.4688) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][330/625] eta 0:01:44 lr 0.001342 wd 0.0500 time 0.3523 (0.3541) data time 0.0008 (0.0024) model time 0.3515 (0.3523) loss 3.5350 (3.3455) grad_norm 1.4947 (1.4652) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][340/625] eta 0:01:41 lr 0.001342 wd 0.0500 time 0.3425 (0.3555) data time 0.0009 (0.0024) model time 0.3417 (0.3540) loss 3.6933 (3.3524) grad_norm 1.2572 (1.4605) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][350/625] eta 0:01:37 lr 0.001342 wd 0.0500 time 0.3464 (0.3558) data time 0.0009 (0.0023) model time 0.3454 (0.3543) loss 3.1335 (3.3529) grad_norm 1.3696 (1.4633) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][360/625] eta 0:01:34 lr 0.001341 wd 0.0500 time 0.3456 (0.3555) data time 0.0009 (0.0023) model time 0.3447 (0.3540) loss 4.0480 (3.3616) grad_norm 1.4075 (1.4655) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][370/625] eta 0:01:30 lr 0.001341 wd 0.0500 time 0.3418 (0.3553) data time 0.0006 (0.0023) model time 0.3412 (0.3537) loss 2.3620 (3.3576) grad_norm 1.9883 (1.4636) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][380/625] eta 0:01:26 lr 0.001341 wd 0.0500 time 0.3493 (0.3550) data time 0.0008 (0.0022) model time 0.3485 (0.3534) loss 3.5678 (3.3529) grad_norm 1.4578 (1.4759) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:45:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][390/625] eta 0:01:23 lr 0.001341 wd 0.0500 time 0.3501 (0.3549) data time 0.0006 (0.0022) model time 0.3494 (0.3533) loss 3.9452 (3.3495) grad_norm 1.4814 (1.4762) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][400/625] eta 0:01:19 lr 0.001341 wd 0.0500 time 0.3447 (0.3547) data time 0.0008 (0.0022) model time 0.3439 (0.3531) loss 1.8711 (3.3403) grad_norm 1.6081 (1.4708) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][410/625] eta 0:01:16 lr 0.001341 wd 0.0500 time 0.3475 (0.3545) data time 0.0007 (0.0021) model time 0.3468 (0.3528) loss 3.5653 (3.3381) grad_norm 1.1783 (1.4699) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][420/625] eta 0:01:12 lr 0.001340 wd 0.0500 time 0.3458 (0.3542) data time 0.0006 (0.0021) model time 0.3451 (0.3526) loss 3.9647 (3.3369) grad_norm 2.7396 (1.4736) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][430/625] eta 0:01:09 lr 0.001340 wd 0.0500 time 0.3409 (0.3540) data time 0.0009 (0.0021) model time 0.3400 (0.3524) loss 3.5503 (3.3275) grad_norm 1.0949 (1.4815) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][440/625] eta 0:01:05 lr 0.001340 wd 0.0500 time 0.3436 (0.3538) data time 0.0009 (0.0021) model time 0.3426 (0.3522) loss 3.2511 (3.3241) grad_norm 1.1233 (1.4843) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][450/625] eta 0:01:01 lr 0.001340 wd 0.0500 time 0.3625 (0.3538) data time 0.0008 (0.0021) model time 0.3618 (0.3520) loss 3.3820 (3.3254) grad_norm 1.2504 (1.4828) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][460/625] eta 0:00:58 lr 0.001340 wd 0.0500 time 0.3492 (0.3544) data time 0.0008 (0.0021) model time 0.3484 (0.3528) loss 2.7347 (3.3215) grad_norm 1.2163 (1.4773) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][470/625] eta 0:00:54 lr 0.001340 wd 0.0500 time 0.3432 (0.3547) data time 0.0008 (0.0021) model time 0.3424 (0.3531) loss 3.7987 (3.3252) grad_norm 1.5625 (1.4771) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][480/625] eta 0:00:51 lr 0.001339 wd 0.0500 time 0.3455 (0.3553) data time 0.0009 (0.0021) model time 0.3446 (0.3538) loss 1.6910 (3.3261) grad_norm 2.6640 (1.4788) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][490/625] eta 0:00:48 lr 0.001339 wd 0.0500 time 0.5643 (0.3565) data time 0.0006 (0.0020) model time 0.5637 (0.3552) loss 3.6830 (3.3267) grad_norm 1.3759 (1.4797) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][500/625] eta 0:00:44 lr 0.001339 wd 0.0500 time 0.3448 (0.3564) data time 0.0007 (0.0020) model time 0.3441 (0.3550) loss 2.9212 (3.3216) grad_norm 1.9047 (1.4820) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][510/625] eta 0:00:40 lr 0.001339 wd 0.0500 time 0.3439 (0.3562) data time 0.0006 (0.0020) model time 0.3432 (0.3548) loss 3.9846 (3.3262) grad_norm 2.3057 (1.4836) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][520/625] eta 0:00:37 lr 0.001339 wd 0.0500 time 0.3420 (0.3563) data time 0.0009 (0.0020) model time 0.3411 (0.3549) loss 3.5569 (3.3235) grad_norm 1.7686 (1.4874) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][530/625] eta 0:00:33 lr 0.001339 wd 0.0500 time 0.3446 (0.3561) data time 0.0009 (0.0020) model time 0.3438 (0.3547) loss 3.0462 (3.3241) grad_norm 1.0193 (1.4841) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][540/625] eta 0:00:30 lr 0.001338 wd 0.0500 time 0.3449 (0.3559) data time 0.0008 (0.0019) model time 0.3441 (0.3546) loss 4.2556 (3.3204) grad_norm 1.2959 (1.4817) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:46:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][550/625] eta 0:00:26 lr 0.001338 wd 0.0500 time 0.3466 (0.3558) data time 0.0006 (0.0019) model time 0.3460 (0.3544) loss 3.8396 (3.3221) grad_norm 1.7527 (1.4857) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][560/625] eta 0:00:23 lr 0.001338 wd 0.0500 time 0.3371 (0.3556) data time 0.0007 (0.0019) model time 0.3364 (0.3541) loss 2.8004 (3.3175) grad_norm 1.0056 (1.4804) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][570/625] eta 0:00:19 lr 0.001338 wd 0.0500 time 0.3471 (0.3554) data time 0.0006 (0.0019) model time 0.3465 (0.3540) loss 3.8671 (3.3137) grad_norm 1.6858 (1.4778) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][580/625] eta 0:00:15 lr 0.001338 wd 0.0500 time 0.3458 (0.3552) data time 0.0008 (0.0019) model time 0.3449 (0.3538) loss 3.3795 (3.3132) grad_norm 1.8323 (1.4865) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][590/625] eta 0:00:12 lr 0.001338 wd 0.0500 time 0.3439 (0.3551) data time 0.0007 (0.0019) model time 0.3433 (0.3536) loss 2.9619 (3.3113) grad_norm 1.6314 (1.4853) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][600/625] eta 0:00:08 lr 0.001337 wd 0.0500 time 0.3435 (0.3556) data time 0.0006 (0.0019) model time 0.3428 (0.3542) loss 2.9404 (3.3090) grad_norm 1.4594 (1.4828) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][610/625] eta 0:00:05 lr 0.001337 wd 0.0500 time 0.3602 (0.3564) data time 0.0006 (0.0019) model time 0.3596 (0.3551) loss 3.0088 (3.3086) grad_norm 1.5012 (1.4845) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [129/300][620/625] eta 0:00:01 lr 0.001337 wd 0.0500 time 0.3378 (0.3561) data time 0.0004 (0.0018) model time 0.3374 (0.3548) loss 3.0972 (3.3067) grad_norm 1.2276 (1.4856) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 129 training takes 0:03:42 [2024-08-26 11:47:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:47:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:47:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.463 (0.463) Loss 0.6318 (0.6318) Acc@1 87.207 (87.207) Acc@5 97.754 (97.754) Mem 11224MB [2024-08-26 11:47:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.132) Loss 1.0713 (0.7780) Acc@1 74.219 (82.804) Acc@5 93.896 (96.586) Mem 11224MB [2024-08-26 11:47:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 1.1406 (0.9274) Acc@1 72.949 (79.295) Acc@5 93.018 (94.859) Mem 11224MB [2024-08-26 11:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.097 Acc@5 94.808 [2024-08-26 11:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.1% [2024-08-26 11:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.10% [2024-08-26 11:47:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 11:47:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 11:47:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.457 (0.457) Loss 0.5259 (0.5259) Acc@1 88.281 (88.281) Acc@5 98.291 (98.291) Mem 11224MB [2024-08-26 11:47:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.8799 (0.6607) Acc@1 79.150 (85.045) Acc@5 95.361 (97.306) Mem 11224MB [2024-08-26 11:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.111) Loss 1.0176 (0.7936) Acc@1 75.146 (81.780) Acc@5 93.945 (95.931) Mem 11224MB [2024-08-26 11:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.502 Acc@5 95.923 [2024-08-26 11:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.5% [2024-08-26 11:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.50% [2024-08-26 11:47:30 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 11:47:31 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 11:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][0/625] eta 0:07:22 lr 0.001337 wd 0.0500 time 0.7088 (0.7088) data time 0.3941 (0.3941) model time 0.0000 (0.0000) loss 2.9358 (2.9358) grad_norm 1.1593 (1.1593) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][10/625] eta 0:03:52 lr 0.001337 wd 0.0500 time 0.3429 (0.3775) data time 0.0007 (0.0367) model time 0.0000 (0.0000) loss 3.8672 (3.6720) grad_norm 1.2121 (1.3467) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][20/625] eta 0:03:38 lr 0.001337 wd 0.0500 time 0.3437 (0.3616) data time 0.0007 (0.0197) model time 0.0000 (0.0000) loss 2.2411 (3.3907) grad_norm 1.2993 (1.4786) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][30/625] eta 0:03:42 lr 0.001336 wd 0.0500 time 0.3459 (0.3741) data time 0.0008 (0.0137) model time 0.0000 (0.0000) loss 3.5896 (3.3483) grad_norm 1.2821 (1.4215) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][40/625] eta 0:03:35 lr 0.001336 wd 0.0500 time 0.3910 (0.3681) data time 0.0006 (0.0106) model time 0.0000 (0.0000) loss 3.4477 (3.2865) grad_norm 2.2040 (1.4513) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][50/625] eta 0:03:29 lr 0.001336 wd 0.0500 time 0.3627 (0.3644) data time 0.0008 (0.0088) model time 0.0000 (0.0000) loss 2.7462 (3.2922) grad_norm 1.1817 (1.4394) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][60/625] eta 0:03:24 lr 0.001336 wd 0.0500 time 0.3465 (0.3614) data time 0.0006 (0.0075) model time 0.3458 (0.3456) loss 4.1578 (3.3288) grad_norm 1.4699 (1.4366) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:47:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][70/625] eta 0:03:19 lr 0.001336 wd 0.0500 time 0.3553 (0.3597) data time 0.0006 (0.0066) model time 0.3547 (0.3469) loss 4.0549 (3.3339) grad_norm 1.4853 (1.4701) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][80/625] eta 0:03:15 lr 0.001336 wd 0.0500 time 0.3392 (0.3578) data time 0.0008 (0.0059) model time 0.3384 (0.3459) loss 3.5807 (3.3474) grad_norm 2.2898 (1.5033) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][90/625] eta 0:03:10 lr 0.001335 wd 0.0500 time 0.3475 (0.3567) data time 0.0008 (0.0053) model time 0.3468 (0.3460) loss 3.3939 (3.3531) grad_norm 2.4394 (1.5715) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][100/625] eta 0:03:07 lr 0.001335 wd 0.0500 time 0.5287 (0.3578) data time 0.0006 (0.0049) model time 0.5281 (0.3501) loss 2.1684 (3.3219) grad_norm 1.2088 (1.5579) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][110/625] eta 0:03:04 lr 0.001335 wd 0.0500 time 0.3455 (0.3580) data time 0.0006 (0.0045) model time 0.3449 (0.3517) loss 3.6624 (3.3483) grad_norm 1.4849 (1.5532) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][120/625] eta 0:03:00 lr 0.001335 wd 0.0500 time 0.3443 (0.3571) data time 0.0007 (0.0042) model time 0.3435 (0.3510) loss 3.2041 (3.3336) grad_norm 1.5769 (1.5317) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][130/625] eta 0:02:56 lr 0.001335 wd 0.0500 time 0.3527 (0.3563) data time 0.0006 (0.0040) model time 0.3521 (0.3503) loss 3.8012 (3.3324) grad_norm 1.4722 (1.5260) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][140/625] eta 0:02:52 lr 0.001335 wd 0.0500 time 0.3398 (0.3558) data time 0.0008 (0.0038) model time 0.3390 (0.3500) loss 4.0529 (3.3218) grad_norm 1.4172 (1.5255) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][150/625] eta 0:02:48 lr 0.001334 wd 0.0500 time 0.3425 (0.3553) data time 0.0007 (0.0036) model time 0.3418 (0.3497) loss 2.6967 (3.3246) grad_norm 1.1472 (1.5296) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][160/625] eta 0:02:45 lr 0.001334 wd 0.0500 time 0.3461 (0.3558) data time 0.0006 (0.0034) model time 0.3455 (0.3509) loss 3.4483 (3.3197) grad_norm 1.5809 (1.5260) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][170/625] eta 0:02:42 lr 0.001334 wd 0.0500 time 0.3419 (0.3575) data time 0.0007 (0.0033) model time 0.3413 (0.3536) loss 3.1153 (3.3149) grad_norm 1.2905 (1.5141) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][180/625] eta 0:02:39 lr 0.001334 wd 0.0500 time 0.3506 (0.3578) data time 0.0006 (0.0031) model time 0.3500 (0.3543) loss 2.5536 (3.3017) grad_norm 1.0312 (1.5233) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][190/625] eta 0:02:35 lr 0.001334 wd 0.0500 time 0.3689 (0.3573) data time 0.0009 (0.0030) model time 0.3680 (0.3539) loss 2.4882 (3.3006) grad_norm 1.0050 (1.5290) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-26 11:48:40 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-26 11:48:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:48:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 11:50:23 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-26 11:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-26 11:50:38 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-26 11:52:26 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-26 11:52:29 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-26 11:52:46 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-26 11:53:03 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-26 11:53:03 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-26 11:53:06 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-26 11:53:08 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-26 11:53:08 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 130) [2024-08-26 11:53:08 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-26 11:53:28 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-26 11:53:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 11:53:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 12:52:29 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-26 12:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-26 12:52:43 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-26 12:52:56 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-26 12:52:56 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-26 12:52:59 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-26 12:53:01 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-26 12:53:01 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 130) [2024-08-26 12:53:01 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-26 12:53:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][200/625] eta 0:15:42 lr 0.001334 wd 0.0500 time 0.3855 (2.2176) data time 0.0009 (0.1131) model time 0.3846 (2.1045) loss 3.5267 (3.7185) grad_norm 1.2157 (1.5588) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 12:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][210/625] eta 0:08:15 lr 0.001333 wd 0.0500 time 0.3750 (1.1943) data time 0.0009 (0.0511) model time 0.3741 (1.1431) loss 4.0945 (3.5127) grad_norm 1.2881 (1.4519) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 12:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][220/625] eta 0:06:08 lr 0.001333 wd 0.0500 time 0.3823 (0.9097) data time 0.0011 (0.0334) model time 0.3812 (0.8763) loss 3.6061 (3.4967) grad_norm 1.0349 (1.4201) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 12:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][230/625] eta 0:05:03 lr 0.001333 wd 0.0500 time 0.3749 (0.7693) data time 0.0012 (0.0251) model time 0.3737 (0.7443) loss 3.5938 (3.4623) grad_norm 2.4620 (1.5379) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 12:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][240/625] eta 0:04:27 lr 0.001333 wd 0.0500 time 0.3798 (0.6936) data time 0.0009 (0.0201) model time 0.3789 (0.6735) loss 3.7685 (3.4468) grad_norm 1.3907 (1.6013) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 12:53:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][250/625] eta 0:03:59 lr 0.001333 wd 0.0500 time 0.3707 (0.6388) data time 0.0009 (0.0170) model time 0.3698 (0.6218) loss 2.6540 (3.4261) grad_norm 1.1787 (1.5526) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 12:53:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][260/625] eta 0:03:38 lr 0.001333 wd 0.0500 time 0.3669 (0.5997) data time 0.0009 (0.0148) model time 0.3660 (0.5850) loss 2.5504 (3.3991) grad_norm 1.1193 (1.4961) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 12:53:49 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-26 12:53:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 12:53:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 13:01:13 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-26 13:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-26 13:01:29 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-26 13:01:40 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-26 13:01:40 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-26 13:01:43 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-26 13:01:45 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-26 13:01:45 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 130) [2024-08-26 13:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-26 13:02:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][270/625] eta 1:34:23 lr 0.001332 wd 0.0500 time 15.9532 (15.9532) data time 0.8118 (0.8118) model time 15.1414 (15.1414) loss 3.6080 (3.6080) grad_norm 2.3419 (2.3419) loss_scale 1024.0000 (1024.0000) mem 23874MB [2024-08-26 13:02:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][280/625] eta 0:10:26 lr 0.001332 wd 0.0500 time 0.3754 (1.8174) data time 0.0011 (0.0749) model time 0.3743 (1.7425) loss 2.8652 (3.5882) grad_norm 1.3793 (1.4614) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][290/625] eta 0:06:18 lr 0.001332 wd 0.0500 time 0.3649 (1.1303) data time 0.0011 (0.0399) model time 0.3638 (1.0904) loss 2.9016 (3.5089) grad_norm 1.1981 (1.4451) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][300/625] eta 0:04:51 lr 0.001332 wd 0.0500 time 0.3818 (0.8959) data time 0.0008 (0.0274) model time 0.3810 (0.8686) loss 2.4138 (3.5014) grad_norm 1.2413 (1.3868) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][310/625] eta 0:04:04 lr 0.001332 wd 0.0500 time 0.6798 (0.7770) data time 0.0011 (0.0209) model time 0.6787 (0.7560) loss 3.3283 (3.4615) grad_norm 1.0390 (1.3902) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][320/625] eta 0:03:32 lr 0.001332 wd 0.0500 time 0.3830 (0.6983) data time 0.0008 (0.0171) model time 0.3822 (0.6812) loss 3.6303 (3.4744) grad_norm 1.3932 (1.4529) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][330/625] eta 0:03:10 lr 0.001331 wd 0.0500 time 0.3763 (0.6453) data time 0.0010 (0.0145) model time 0.3752 (0.6308) loss 3.1573 (3.4293) grad_norm 1.2940 (1.4387) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][340/625] eta 0:02:53 lr 0.001331 wd 0.0500 time 0.3793 (0.6076) data time 0.0012 (0.0126) model time 0.3782 (0.5950) loss 3.2725 (3.3905) grad_norm 1.0726 (1.3887) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][350/625] eta 0:02:39 lr 0.001331 wd 0.0500 time 0.3851 (0.5790) data time 0.0011 (0.0112) model time 0.3840 (0.5678) loss 2.7840 (3.3777) grad_norm 1.0409 (1.3820) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][360/625] eta 0:02:27 lr 0.001331 wd 0.0500 time 0.3809 (0.5569) data time 0.0008 (0.0101) model time 0.3801 (0.5468) loss 3.6084 (3.3551) grad_norm 1.9999 (1.4000) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][370/625] eta 0:02:17 lr 0.001331 wd 0.0500 time 0.4163 (0.5394) data time 0.0008 (0.0092) model time 0.4155 (0.5302) loss 3.4358 (3.3559) grad_norm 1.3478 (1.3980) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][380/625] eta 0:02:08 lr 0.001331 wd 0.0500 time 0.3672 (0.5248) data time 0.0011 (0.0085) model time 0.3661 (0.5163) loss 2.8507 (3.3578) grad_norm 1.2987 (1.4152) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][390/625] eta 0:02:00 lr 0.001330 wd 0.0500 time 0.3814 (0.5127) data time 0.0008 (0.0078) model time 0.3806 (0.5049) loss 2.1590 (3.3649) grad_norm 1.1338 (1.4209) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][400/625] eta 0:01:53 lr 0.001330 wd 0.0500 time 0.3776 (0.5024) data time 0.0010 (0.0073) model time 0.3766 (0.4951) loss 3.6663 (3.3599) grad_norm 1.5297 (1.4222) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][410/625] eta 0:01:46 lr 0.001330 wd 0.0500 time 0.3803 (0.4943) data time 0.0009 (0.0069) model time 0.3794 (0.4874) loss 3.2818 (3.3492) grad_norm 1.5176 (1.4159) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][420/625] eta 0:01:39 lr 0.001330 wd 0.0500 time 0.3700 (0.4866) data time 0.0010 (0.0066) model time 0.3690 (0.4800) loss 2.5724 (3.3393) grad_norm 1.4564 (1.4103) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][430/625] eta 0:01:33 lr 0.001330 wd 0.0500 time 0.3653 (0.4796) data time 0.0010 (0.0062) model time 0.3643 (0.4734) loss 3.7529 (3.3492) grad_norm 1.2404 (1.4028) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][440/625] eta 0:01:27 lr 0.001330 wd 0.0500 time 0.3737 (0.4737) data time 0.0011 (0.0059) model time 0.3726 (0.4678) loss 3.2668 (3.3443) grad_norm 1.2177 (1.4019) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][450/625] eta 0:01:22 lr 0.001329 wd 0.0500 time 0.3747 (0.4687) data time 0.0012 (0.0057) model time 0.3735 (0.4630) loss 3.3434 (3.3214) grad_norm 1.2136 (1.3964) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][460/625] eta 0:01:16 lr 0.001329 wd 0.0500 time 0.3899 (0.4640) data time 0.0011 (0.0054) model time 0.3888 (0.4586) loss 2.9767 (3.3120) grad_norm 1.4297 (1.4088) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][470/625] eta 0:01:11 lr 0.001329 wd 0.0500 time 0.3656 (0.4596) data time 0.0011 (0.0052) model time 0.3645 (0.4544) loss 3.2862 (3.3031) grad_norm 1.3360 (1.4101) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][480/625] eta 0:01:06 lr 0.001329 wd 0.0500 time 0.3791 (0.4559) data time 0.0011 (0.0050) model time 0.3781 (0.4509) loss 3.6096 (3.2980) grad_norm 1.2242 (1.4043) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][490/625] eta 0:01:01 lr 0.001329 wd 0.0500 time 0.3733 (0.4524) data time 0.0008 (0.0048) model time 0.3725 (0.4475) loss 3.3477 (3.2905) grad_norm 1.1547 (1.4084) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][500/625] eta 0:00:56 lr 0.001329 wd 0.0500 time 0.3782 (0.4510) data time 0.0008 (0.0047) model time 0.3774 (0.4463) loss 2.2782 (3.2915) grad_norm 1.5240 (1.4067) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][510/625] eta 0:00:51 lr 0.001328 wd 0.0500 time 0.3647 (0.4487) data time 0.0008 (0.0045) model time 0.3639 (0.4442) loss 3.4474 (3.2959) grad_norm 1.7294 (1.4159) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][520/625] eta 0:00:46 lr 0.001328 wd 0.0500 time 0.3791 (0.4458) data time 0.0009 (0.0044) model time 0.3782 (0.4414) loss 3.1003 (3.2848) grad_norm 1.5478 (1.4188) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][530/625] eta 0:00:42 lr 0.001328 wd 0.0500 time 0.3805 (0.4432) data time 0.0009 (0.0043) model time 0.3796 (0.4389) loss 3.1152 (3.2750) grad_norm 1.7783 (1.4186) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][540/625] eta 0:00:37 lr 0.001328 wd 0.0500 time 0.3744 (0.4407) data time 0.0008 (0.0042) model time 0.3735 (0.4365) loss 3.8254 (3.2705) grad_norm 1.3100 (1.4301) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][550/625] eta 0:00:32 lr 0.001328 wd 0.0500 time 0.3741 (0.4384) data time 0.0011 (0.0041) model time 0.3730 (0.4344) loss 2.9838 (3.2774) grad_norm 1.0357 (1.4285) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:03:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][560/625] eta 0:00:28 lr 0.001328 wd 0.0500 time 0.3779 (0.4363) data time 0.0008 (0.0040) model time 0.3771 (0.4323) loss 2.1696 (3.2694) grad_norm 1.0820 (1.4281) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][570/625] eta 0:00:23 lr 0.001327 wd 0.0500 time 0.3749 (0.4343) data time 0.0010 (0.0039) model time 0.3740 (0.4305) loss 3.4696 (3.2596) grad_norm 1.1125 (1.4304) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][580/625] eta 0:00:19 lr 0.001327 wd 0.0500 time 0.3853 (0.4325) data time 0.0010 (0.0038) model time 0.3842 (0.4287) loss 3.5021 (3.2575) grad_norm 1.5748 (1.4344) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:04:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][590/625] eta 0:00:15 lr 0.001327 wd 0.0500 time 0.3818 (0.4307) data time 0.0008 (0.0037) model time 0.3810 (0.4270) loss 3.9426 (3.2702) grad_norm 1.1410 (1.4303) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][600/625] eta 0:00:10 lr 0.001327 wd 0.0500 time 0.3744 (0.4291) data time 0.0008 (0.0036) model time 0.3735 (0.4255) loss 2.1556 (3.2658) grad_norm 1.3307 (1.4303) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][610/625] eta 0:00:06 lr 0.001327 wd 0.0500 time 0.3606 (0.4276) data time 0.0008 (0.0036) model time 0.3598 (0.4240) loss 3.2312 (3.2704) grad_norm 1.5010 (1.4264) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [130/300][620/625] eta 0:00:02 lr 0.001327 wd 0.0500 time 0.3568 (0.4269) data time 0.0005 (0.0035) model time 0.3563 (0.4235) loss 3.7075 (3.2702) grad_norm 1.9518 (1.4236) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-26 13:04:21 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 130 training takes 0:02:31 [2024-08-26 13:04:21 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 13:04:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 13:04:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 0.6323 (0.6323) Acc@1 86.719 (86.719) Acc@5 97.900 (97.900) Mem 11219MB [2024-08-26 13:04:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 1.0791 (0.7714) Acc@1 74.854 (82.715) Acc@5 93.652 (96.689) Mem 11219MB [2024-08-26 13:04:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.1455 (0.9207) Acc@1 72.949 (79.234) Acc@5 92.383 (94.864) Mem 11219MB [2024-08-26 13:04:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.065 Acc@5 94.880 [2024-08-26 13:04:29 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.1% [2024-08-26 13:04:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.807 (0.807) Loss 0.5249 (0.5249) Acc@1 88.428 (88.428) Acc@5 98.389 (98.389) Mem 11219MB [2024-08-26 13:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.176) Loss 0.8818 (0.6603) Acc@1 79.150 (85.072) Acc@5 95.557 (97.310) Mem 11219MB [2024-08-26 13:04:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.0146 (0.7927) Acc@1 74.951 (81.801) Acc@5 94.043 (95.945) Mem 11219MB [2024-08-26 13:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.512 Acc@5 95.951 [2024-08-26 13:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.5% [2024-08-26 13:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.51% [2024-08-26 13:04:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 13:04:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 13:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][0/625] eta 0:09:08 lr 0.001326 wd 0.0500 time 0.8781 (0.8781) data time 0.4766 (0.4766) model time 0.0000 (0.0000) loss 2.6879 (2.6879) grad_norm 1.3254 (1.3254) loss_scale 1024.0000 (1024.0000) mem 11228MB [2024-08-26 13:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][10/625] eta 0:04:32 lr 0.001326 wd 0.0500 time 0.3729 (0.4424) data time 0.0010 (0.0443) model time 0.0000 (0.0000) loss 3.6613 (3.2056) grad_norm 1.2002 (1.4391) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][20/625] eta 0:04:09 lr 0.001326 wd 0.0500 time 0.3878 (0.4119) data time 0.0009 (0.0237) model time 0.0000 (0.0000) loss 2.5840 (3.0877) grad_norm 1.3683 (1.3988) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][30/625] eta 0:03:59 lr 0.001326 wd 0.0500 time 0.3785 (0.4022) data time 0.0012 (0.0175) model time 0.0000 (0.0000) loss 3.6016 (3.0467) grad_norm 1.4103 (1.4707) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:04:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][40/625] eta 0:03:52 lr 0.001326 wd 0.0500 time 0.3868 (0.3977) data time 0.0009 (0.0135) model time 0.0000 (0.0000) loss 3.6798 (3.1218) grad_norm 1.3310 (1.4397) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][50/625] eta 0:03:46 lr 0.001326 wd 0.0500 time 0.3733 (0.3936) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 3.4777 (3.1815) grad_norm 1.8500 (1.4858) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][60/625] eta 0:03:41 lr 0.001325 wd 0.0500 time 0.3791 (0.3916) data time 0.0008 (0.0094) model time 0.3783 (0.3801) loss 3.3310 (3.1948) grad_norm 1.9439 (1.4853) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][70/625] eta 0:03:36 lr 0.001325 wd 0.0500 time 0.3796 (0.3896) data time 0.0011 (0.0083) model time 0.3785 (0.3782) loss 3.6010 (3.2121) grad_norm 1.2874 (1.4744) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][80/625] eta 0:03:31 lr 0.001325 wd 0.0500 time 0.3710 (0.3879) data time 0.0011 (0.0074) model time 0.3698 (0.3773) loss 2.7873 (3.2442) grad_norm 1.5262 (1.4889) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][90/625] eta 0:03:27 lr 0.001325 wd 0.0500 time 0.3765 (0.3870) data time 0.0010 (0.0067) model time 0.3756 (0.3776) loss 3.3790 (3.2500) grad_norm 1.4706 (1.4889) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][100/625] eta 0:03:22 lr 0.001325 wd 0.0500 time 0.3793 (0.3861) data time 0.0008 (0.0062) model time 0.3785 (0.3773) loss 2.8955 (3.2371) grad_norm 1.7417 (1.4728) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][110/625] eta 0:03:18 lr 0.001325 wd 0.0500 time 0.3774 (0.3854) data time 0.0009 (0.0057) model time 0.3765 (0.3772) loss 2.6504 (3.2100) grad_norm 1.7972 (1.4944) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][120/625] eta 0:03:14 lr 0.001324 wd 0.0500 time 0.3910 (0.3850) data time 0.0008 (0.0053) model time 0.3902 (0.3776) loss 2.8173 (3.2028) grad_norm 1.3285 (1.4943) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][130/625] eta 0:03:10 lr 0.001324 wd 0.0500 time 0.3712 (0.3859) data time 0.0010 (0.0050) model time 0.3702 (0.3798) loss 3.7203 (3.2229) grad_norm 1.9249 (1.4871) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][140/625] eta 0:03:06 lr 0.001324 wd 0.0500 time 0.3746 (0.3854) data time 0.0008 (0.0047) model time 0.3738 (0.3796) loss 3.0629 (3.2209) grad_norm 1.2911 (1.4907) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][150/625] eta 0:03:02 lr 0.001324 wd 0.0500 time 0.3775 (0.3850) data time 0.0010 (0.0045) model time 0.3765 (0.3794) loss 3.5085 (3.2246) grad_norm 1.0609 (1.4813) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][160/625] eta 0:02:58 lr 0.001324 wd 0.0500 time 0.3724 (0.3846) data time 0.0010 (0.0043) model time 0.3714 (0.3791) loss 3.2084 (3.2515) grad_norm 1.8416 (1.4815) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][170/625] eta 0:02:54 lr 0.001324 wd 0.0500 time 0.3708 (0.3842) data time 0.0008 (0.0042) model time 0.3699 (0.3790) loss 2.7137 (3.2363) grad_norm 1.3024 (1.4724) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][180/625] eta 0:02:51 lr 0.001323 wd 0.0500 time 0.3761 (0.3862) data time 0.0011 (0.0040) model time 0.3750 (0.3821) loss 3.1899 (3.2331) grad_norm 1.6643 (1.4717) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][190/625] eta 0:02:47 lr 0.001323 wd 0.0500 time 0.3764 (0.3857) data time 0.0008 (0.0038) model time 0.3757 (0.3817) loss 2.6177 (3.2186) grad_norm 1.0225 (1.4722) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][200/625] eta 0:02:43 lr 0.001323 wd 0.0500 time 0.3781 (0.3852) data time 0.0010 (0.0037) model time 0.3771 (0.3812) loss 3.7938 (3.2411) grad_norm 1.0636 (1.4631) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][210/625] eta 0:02:39 lr 0.001323 wd 0.0500 time 0.3756 (0.3851) data time 0.0008 (0.0036) model time 0.3748 (0.3812) loss 2.6150 (3.2469) grad_norm 1.2358 (1.4515) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:05:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][220/625] eta 0:02:35 lr 0.001323 wd 0.0500 time 0.3776 (0.3849) data time 0.0010 (0.0035) model time 0.3766 (0.3811) loss 2.5191 (3.2493) grad_norm 2.0281 (1.4558) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][230/625] eta 0:02:31 lr 0.001323 wd 0.0500 time 0.3775 (0.3846) data time 0.0012 (0.0034) model time 0.3764 (0.3809) loss 3.6210 (3.2595) grad_norm 2.0874 (1.4664) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][240/625] eta 0:02:28 lr 0.001322 wd 0.0500 time 0.3832 (0.3845) data time 0.0008 (0.0033) model time 0.3823 (0.3809) loss 2.4051 (3.2625) grad_norm 1.2400 (1.4646) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][250/625] eta 0:02:24 lr 0.001322 wd 0.0500 time 0.3816 (0.3844) data time 0.0009 (0.0032) model time 0.3806 (0.3808) loss 2.6971 (3.2572) grad_norm 1.2884 (1.4587) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][260/625] eta 0:02:20 lr 0.001322 wd 0.0500 time 0.3761 (0.3842) data time 0.0011 (0.0031) model time 0.3750 (0.3808) loss 3.2232 (3.2598) grad_norm 1.2137 (1.4570) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][270/625] eta 0:02:16 lr 0.001322 wd 0.0500 time 0.3771 (0.3840) data time 0.0010 (0.0030) model time 0.3761 (0.3805) loss 3.2461 (3.2720) grad_norm 1.1918 (1.4499) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][280/625] eta 0:02:12 lr 0.001322 wd 0.0500 time 0.3773 (0.3838) data time 0.0008 (0.0030) model time 0.3764 (0.3804) loss 3.4131 (3.2736) grad_norm 1.3896 (1.4512) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][290/625] eta 0:02:08 lr 0.001322 wd 0.0500 time 0.3755 (0.3836) data time 0.0010 (0.0029) model time 0.3744 (0.3803) loss 3.4988 (3.2641) grad_norm 0.9048 (1.4521) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][300/625] eta 0:02:05 lr 0.001321 wd 0.0500 time 0.3798 (0.3849) data time 0.0011 (0.0029) model time 0.3787 (0.3819) loss 2.8599 (3.2656) grad_norm 8.6311 (1.4694) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][310/625] eta 0:02:01 lr 0.001321 wd 0.0500 time 0.3749 (0.3847) data time 0.0008 (0.0028) model time 0.3741 (0.3818) loss 3.3294 (3.2609) grad_norm 1.4108 (1.4872) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][320/625] eta 0:01:57 lr 0.001321 wd 0.0500 time 0.3860 (0.3846) data time 0.0008 (0.0028) model time 0.3852 (0.3817) loss 3.1297 (3.2717) grad_norm 1.2623 (1.4823) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][330/625] eta 0:01:53 lr 0.001321 wd 0.0500 time 0.3736 (0.3844) data time 0.0009 (0.0027) model time 0.3727 (0.3815) loss 3.2751 (3.2737) grad_norm 1.2160 (1.4819) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][340/625] eta 0:01:49 lr 0.001321 wd 0.0500 time 0.3902 (0.3850) data time 0.0008 (0.0027) model time 0.3894 (0.3822) loss 2.3709 (3.2660) grad_norm 1.2841 (1.4842) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][350/625] eta 0:01:45 lr 0.001321 wd 0.0500 time 0.3768 (0.3847) data time 0.0008 (0.0026) model time 0.3760 (0.3820) loss 4.0276 (3.2637) grad_norm 1.1304 (1.4810) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][360/625] eta 0:01:41 lr 0.001320 wd 0.0500 time 0.3809 (0.3846) data time 0.0012 (0.0026) model time 0.3797 (0.3819) loss 3.8349 (3.2554) grad_norm 1.5749 (1.4865) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][370/625] eta 0:01:38 lr 0.001320 wd 0.0500 time 0.3734 (0.3844) data time 0.0008 (0.0026) model time 0.3726 (0.3817) loss 3.3373 (3.2580) grad_norm 1.5391 (1.4878) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][380/625] eta 0:01:34 lr 0.001320 wd 0.0500 time 0.3740 (0.3842) data time 0.0009 (0.0025) model time 0.3731 (0.3815) loss 3.5897 (3.2656) grad_norm 1.6574 (1.4873) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][390/625] eta 0:01:30 lr 0.001320 wd 0.0500 time 0.3778 (0.3841) data time 0.0008 (0.0025) model time 0.3770 (0.3814) loss 3.6510 (3.2658) grad_norm 1.2478 (1.4936) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][400/625] eta 0:01:26 lr 0.001320 wd 0.0500 time 0.3824 (0.3839) data time 0.0011 (0.0025) model time 0.3812 (0.3813) loss 3.9296 (3.2616) grad_norm 1.2039 (1.4901) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][410/625] eta 0:01:22 lr 0.001320 wd 0.0500 time 0.3671 (0.3839) data time 0.0011 (0.0024) model time 0.3660 (0.3813) loss 3.5858 (3.2703) grad_norm 1.2495 (1.4872) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][420/625] eta 0:01:18 lr 0.001319 wd 0.0500 time 0.3778 (0.3838) data time 0.0011 (0.0024) model time 0.3767 (0.3812) loss 3.4603 (3.2736) grad_norm 1.0950 (1.4835) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][430/625] eta 0:01:14 lr 0.001319 wd 0.0500 time 0.3902 (0.3837) data time 0.0010 (0.0024) model time 0.3893 (0.3811) loss 3.4425 (3.2739) grad_norm 1.0719 (1.4812) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][440/625] eta 0:01:10 lr 0.001319 wd 0.0500 time 0.3734 (0.3836) data time 0.0009 (0.0024) model time 0.3725 (0.3810) loss 1.7750 (3.2716) grad_norm 1.7072 (1.4796) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][450/625] eta 0:01:07 lr 0.001319 wd 0.0500 time 0.3797 (0.3835) data time 0.0008 (0.0023) model time 0.3788 (0.3810) loss 2.2592 (3.2685) grad_norm 1.1870 (1.4777) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][460/625] eta 0:01:03 lr 0.001319 wd 0.0500 time 0.3944 (0.3834) data time 0.0008 (0.0023) model time 0.3936 (0.3810) loss 4.2196 (3.2634) grad_norm 1.0665 (1.4762) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][470/625] eta 0:00:59 lr 0.001319 wd 0.0500 time 0.3781 (0.3833) data time 0.0010 (0.0023) model time 0.3771 (0.3809) loss 1.9020 (3.2581) grad_norm 1.0361 (1.4771) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][480/625] eta 0:00:55 lr 0.001318 wd 0.0500 time 0.3837 (0.3833) data time 0.0008 (0.0023) model time 0.3829 (0.3809) loss 4.0805 (3.2576) grad_norm 1.4592 (1.4761) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][490/625] eta 0:00:51 lr 0.001318 wd 0.0500 time 0.3774 (0.3832) data time 0.0008 (0.0022) model time 0.3766 (0.3808) loss 3.8714 (3.2548) grad_norm 1.0795 (1.4772) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][500/625] eta 0:00:47 lr 0.001318 wd 0.0500 time 0.5863 (0.3835) data time 0.0010 (0.0022) model time 0.5853 (0.3811) loss 3.7761 (3.2580) grad_norm 1.4486 (1.4774) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][510/625] eta 0:00:44 lr 0.001318 wd 0.0500 time 0.3820 (0.3837) data time 0.0010 (0.0022) model time 0.3810 (0.3814) loss 3.4436 (3.2553) grad_norm 1.3162 (1.4820) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][520/625] eta 0:00:40 lr 0.001318 wd 0.0500 time 0.3340 (0.3840) data time 0.0011 (0.0022) model time 0.3329 (0.3818) loss 3.2944 (3.2575) grad_norm 1.5198 (1.4793) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:07:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][530/625] eta 0:00:36 lr 0.001318 wd 0.0500 time 0.3703 (0.3839) data time 0.0008 (0.0022) model time 0.3695 (0.3816) loss 3.7364 (3.2552) grad_norm 1.8808 (1.4795) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:08:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][540/625] eta 0:00:32 lr 0.001317 wd 0.0500 time 0.3758 (0.3837) data time 0.0010 (0.0022) model time 0.3748 (0.3815) loss 3.5435 (3.2540) grad_norm 1.6240 (1.4772) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:08:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][550/625] eta 0:00:28 lr 0.001317 wd 0.0500 time 0.3815 (0.3836) data time 0.0010 (0.0021) model time 0.3805 (0.3813) loss 3.9558 (3.2575) grad_norm 1.4423 (1.4794) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:08:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][560/625] eta 0:00:24 lr 0.001317 wd 0.0500 time 0.3739 (0.3835) data time 0.0009 (0.0021) model time 0.3730 (0.3812) loss 4.2810 (3.2580) grad_norm 2.0812 (1.4789) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:08:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][570/625] eta 0:00:21 lr 0.001317 wd 0.0500 time 0.3790 (0.3834) data time 0.0011 (0.0021) model time 0.3778 (0.3812) loss 3.4047 (3.2641) grad_norm 1.0313 (1.4798) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:08:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][580/625] eta 0:00:17 lr 0.001317 wd 0.0500 time 0.3828 (0.3833) data time 0.0009 (0.0021) model time 0.3818 (0.3811) loss 2.4289 (3.2643) grad_norm 1.4604 (1.4772) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][590/625] eta 0:00:13 lr 0.001317 wd 0.0500 time 0.3737 (0.3832) data time 0.0011 (0.0021) model time 0.3727 (0.3810) loss 3.6056 (3.2648) grad_norm 2.1725 (1.4778) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-08-26 13:08:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][600/625] eta 0:00:09 lr 0.001316 wd 0.0500 time 0.3887 (0.3832) data time 0.0008 (0.0021) model time 0.3879 (0.3810) loss 3.5460 (3.2594) grad_norm 2.1942 (1.4785) loss_scale 2048.0000 (1034.2230) mem 11225MB [2024-08-26 13:08:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][610/625] eta 0:00:05 lr 0.001316 wd 0.0500 time 0.3710 (0.3831) data time 0.0008 (0.0020) model time 0.3702 (0.3809) loss 3.8669 (3.2606) grad_norm 1.3248 (1.4821) loss_scale 2048.0000 (1050.8151) mem 11225MB [2024-08-26 13:08:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [131/300][620/625] eta 0:00:01 lr 0.001316 wd 0.0500 time 0.3605 (0.3828) data time 0.0008 (0.0020) model time 0.3597 (0.3806) loss 3.3517 (3.2646) grad_norm 1.0817 (1.4784) loss_scale 2048.0000 (1066.8728) mem 11225MB [2024-08-26 13:08:32 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 131 training takes 0:03:59 [2024-08-26 13:08:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 13:08:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-26 13:08:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.489 (0.489) Loss 0.6182 (0.6182) Acc@1 86.914 (86.914) Acc@5 97.705 (97.705) Mem 11225MB [2024-08-26 13:08:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 1.0049 (0.7518) Acc@1 75.928 (83.008) Acc@5 94.727 (96.826) Mem 11225MB [2024-08-26 13:08:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.0928 (0.9017) Acc@1 74.463 (79.550) Acc@5 92.480 (95.033) Mem 11225MB [2024-08-26 13:08:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.291 Acc@5 95.016 [2024-08-26 13:08:36 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.3% [2024-08-26 13:08:36 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.29% [2024-08-26 13:08:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-26 13:08:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-26 13:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.5244 (0.5244) Acc@1 88.379 (88.379) Acc@5 98.389 (98.389) Mem 11225MB [2024-08-26 13:08:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 0.8809 (0.6593) Acc@1 79.102 (85.019) Acc@5 95.557 (97.310) Mem 11225MB [2024-08-26 13:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.0127 (0.7913) Acc@1 75.293 (81.810) Acc@5 94.043 (95.954) Mem 11225MB [2024-08-26 13:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.518 Acc@5 95.953 [2024-08-26 13:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.5% [2024-08-26 13:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.52% [2024-08-26 13:08:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-26 13:08:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-26 13:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][0/625] eta 0:08:24 lr 0.001316 wd 0.0500 time 0.8075 (0.8075) data time 0.4724 (0.4724) model time 0.0000 (0.0000) loss 3.5596 (3.5596) grad_norm 1.6285 (1.6285) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:08:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][10/625] eta 0:04:15 lr 0.001316 wd 0.0500 time 0.3818 (0.4162) data time 0.0011 (0.0440) model time 0.0000 (0.0000) loss 3.1008 (3.3200) grad_norm 1.1385 (1.6787) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:08:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][20/625] eta 0:04:00 lr 0.001316 wd 0.0500 time 0.3723 (0.3983) data time 0.0011 (0.0236) model time 0.0000 (0.0000) loss 3.0393 (3.2490) grad_norm 1.3325 (1.5772) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:08:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][30/625] eta 0:03:52 lr 0.001315 wd 0.0500 time 0.3783 (0.3911) data time 0.0010 (0.0165) model time 0.0000 (0.0000) loss 3.8182 (3.2797) grad_norm 1.2551 (1.5076) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:08:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][40/625] eta 0:03:46 lr 0.001315 wd 0.0500 time 0.3785 (0.3877) data time 0.0008 (0.0127) model time 0.0000 (0.0000) loss 3.7253 (3.2770) grad_norm 1.2835 (1.4457) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][50/625] eta 0:03:42 lr 0.001315 wd 0.0500 time 0.3892 (0.3861) data time 0.0011 (0.0104) model time 0.0000 (0.0000) loss 3.4310 (3.2339) grad_norm 1.6596 (1.4413) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][60/625] eta 0:03:37 lr 0.001315 wd 0.0500 time 0.3812 (0.3853) data time 0.0008 (0.0089) model time 0.3804 (0.3797) loss 3.8070 (3.2703) grad_norm 1.3314 (1.4322) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][70/625] eta 0:03:35 lr 0.001315 wd 0.0500 time 0.3714 (0.3875) data time 0.0012 (0.0080) model time 0.3702 (0.3893) loss 2.1171 (3.2579) grad_norm 1.6268 (1.4238) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][80/625] eta 0:03:32 lr 0.001315 wd 0.0500 time 0.3769 (0.3891) data time 0.0007 (0.0072) model time 0.3761 (0.3926) loss 2.7522 (3.2741) grad_norm 1.5845 (1.4253) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][90/625] eta 0:03:27 lr 0.001314 wd 0.0500 time 0.3759 (0.3879) data time 0.0011 (0.0065) model time 0.3748 (0.3888) loss 3.5905 (3.2674) grad_norm 1.2442 (1.4209) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][100/625] eta 0:03:23 lr 0.001314 wd 0.0500 time 0.3744 (0.3870) data time 0.0010 (0.0060) model time 0.3734 (0.3865) loss 2.5858 (3.2656) grad_norm 2.2338 (1.4468) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][110/625] eta 0:03:18 lr 0.001314 wd 0.0500 time 0.3730 (0.3861) data time 0.0010 (0.0055) model time 0.3719 (0.3847) loss 3.3508 (3.2546) grad_norm 2.0175 (1.4442) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][120/625] eta 0:03:14 lr 0.001314 wd 0.0500 time 0.3817 (0.3858) data time 0.0010 (0.0052) model time 0.3807 (0.3842) loss 3.3020 (3.2474) grad_norm 1.1548 (1.4497) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][130/625] eta 0:03:10 lr 0.001314 wd 0.0500 time 0.3846 (0.3856) data time 0.0008 (0.0049) model time 0.3837 (0.3838) loss 3.5025 (3.2511) grad_norm 1.7410 (1.4385) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][140/625] eta 0:03:06 lr 0.001314 wd 0.0500 time 0.3763 (0.3851) data time 0.0011 (0.0047) model time 0.3752 (0.3832) loss 2.2671 (3.2654) grad_norm 1.3304 (1.4441) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][150/625] eta 0:03:02 lr 0.001313 wd 0.0500 time 0.3826 (0.3847) data time 0.0008 (0.0044) model time 0.3819 (0.3827) loss 3.8444 (3.2701) grad_norm 1.3758 (1.4330) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][160/625] eta 0:02:58 lr 0.001313 wd 0.0500 time 0.3712 (0.3849) data time 0.0011 (0.0042) model time 0.3702 (0.3830) loss 2.3561 (3.2609) grad_norm 1.5371 (1.4491) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][170/625] eta 0:02:54 lr 0.001313 wd 0.0500 time 0.3769 (0.3846) data time 0.0010 (0.0040) model time 0.3759 (0.3826) loss 3.0979 (3.2600) grad_norm 1.8147 (1.4495) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][180/625] eta 0:02:51 lr 0.001313 wd 0.0500 time 0.3741 (0.3843) data time 0.0008 (0.0039) model time 0.3733 (0.3823) loss 4.1121 (3.2578) grad_norm 1.2982 (1.4422) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][190/625] eta 0:02:47 lr 0.001313 wd 0.0500 time 0.3829 (0.3840) data time 0.0010 (0.0037) model time 0.3819 (0.3819) loss 3.0162 (3.2536) grad_norm 1.2044 (1.4307) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:09:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][200/625] eta 0:02:43 lr 0.001313 wd 0.0500 time 0.3597 (0.3837) data time 0.0010 (0.0036) model time 0.3586 (0.3816) loss 3.0800 (3.2646) grad_norm 2.0810 (1.4220) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][210/625] eta 0:02:39 lr 0.001312 wd 0.0500 time 0.3719 (0.3835) data time 0.0009 (0.0035) model time 0.3710 (0.3814) loss 2.4485 (3.2752) grad_norm 1.4645 (1.4162) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][220/625] eta 0:02:35 lr 0.001312 wd 0.0500 time 0.3879 (0.3835) data time 0.0008 (0.0034) model time 0.3871 (0.3814) loss 3.4669 (3.2831) grad_norm 1.3765 (1.4209) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][230/625] eta 0:02:31 lr 0.001312 wd 0.0500 time 0.3751 (0.3833) data time 0.0008 (0.0033) model time 0.3744 (0.3812) loss 2.7377 (3.2866) grad_norm 1.4916 (1.4220) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][240/625] eta 0:02:27 lr 0.001312 wd 0.0500 time 0.3713 (0.3831) data time 0.0008 (0.0032) model time 0.3705 (0.3810) loss 2.4077 (3.2845) grad_norm 1.7338 (1.4368) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][250/625] eta 0:02:23 lr 0.001312 wd 0.0500 time 0.5752 (0.3838) data time 0.0011 (0.0031) model time 0.5741 (0.3820) loss 3.5848 (3.2839) grad_norm 2.2151 (1.4470) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][260/625] eta 0:02:20 lr 0.001312 wd 0.0500 time 0.3758 (0.3853) data time 0.0008 (0.0031) model time 0.3750 (0.3838) loss 3.7484 (3.2817) grad_norm 1.2988 (1.4614) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][270/625] eta 0:02:16 lr 0.001311 wd 0.0500 time 0.3941 (0.3851) data time 0.0008 (0.0030) model time 0.3933 (0.3836) loss 3.2090 (3.2735) grad_norm 1.3330 (1.4517) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][280/625] eta 0:02:12 lr 0.001311 wd 0.0500 time 0.3678 (0.3847) data time 0.0010 (0.0029) model time 0.3668 (0.3831) loss 3.1256 (3.2839) grad_norm 1.5068 (1.4500) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][290/625] eta 0:02:08 lr 0.001311 wd 0.0500 time 0.3763 (0.3846) data time 0.0009 (0.0029) model time 0.3754 (0.3829) loss 2.7512 (3.2887) grad_norm 1.1911 (1.4490) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][300/625] eta 0:02:04 lr 0.001311 wd 0.0500 time 0.3721 (0.3844) data time 0.0008 (0.0029) model time 0.3712 (0.3827) loss 3.9833 (3.3006) grad_norm 2.0268 (1.4585) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][310/625] eta 0:02:01 lr 0.001311 wd 0.0500 time 0.3746 (0.3843) data time 0.0008 (0.0028) model time 0.3738 (0.3826) loss 3.9021 (3.3077) grad_norm 1.4079 (1.4591) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][320/625] eta 0:01:57 lr 0.001311 wd 0.0500 time 0.3729 (0.3841) data time 0.0008 (0.0027) model time 0.3721 (0.3825) loss 3.9869 (3.3120) grad_norm 1.1708 (1.4561) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][330/625] eta 0:01:53 lr 0.001310 wd 0.0500 time 0.3727 (0.3842) data time 0.0008 (0.0027) model time 0.3719 (0.3825) loss 2.1596 (3.3051) grad_norm 1.9464 (1.4531) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][340/625] eta 0:01:49 lr 0.001310 wd 0.0500 time 0.3733 (0.3841) data time 0.0010 (0.0027) model time 0.3723 (0.3824) loss 2.1045 (3.2938) grad_norm 1.0688 (1.4485) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][350/625] eta 0:01:45 lr 0.001310 wd 0.0500 time 0.3748 (0.3844) data time 0.0008 (0.0026) model time 0.3740 (0.3828) loss 3.7416 (3.2959) grad_norm 1.1447 (1.4465) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][360/625] eta 0:01:41 lr 0.001310 wd 0.0500 time 0.3980 (0.3844) data time 0.0009 (0.0026) model time 0.3971 (0.3828) loss 3.9109 (3.2955) grad_norm 2.0570 (1.4441) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][370/625] eta 0:01:38 lr 0.001310 wd 0.0500 time 0.5834 (0.3854) data time 0.0011 (0.0026) model time 0.5822 (0.3840) loss 3.7014 (3.2975) grad_norm 2.3273 (1.4562) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][380/625] eta 0:01:34 lr 0.001309 wd 0.0500 time 0.3823 (0.3858) data time 0.0010 (0.0025) model time 0.3813 (0.3845) loss 3.4442 (3.3024) grad_norm 1.7611 (1.4582) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][390/625] eta 0:01:30 lr 0.001309 wd 0.0500 time 0.3843 (0.3857) data time 0.0011 (0.0025) model time 0.3832 (0.3843) loss 3.8621 (3.3059) grad_norm 1.9038 (1.4581) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][400/625] eta 0:01:26 lr 0.001309 wd 0.0500 time 0.3733 (0.3855) data time 0.0010 (0.0025) model time 0.3723 (0.3841) loss 3.6430 (3.3068) grad_norm 1.0672 (1.4552) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][410/625] eta 0:01:22 lr 0.001309 wd 0.0500 time 0.3852 (0.3853) data time 0.0008 (0.0024) model time 0.3844 (0.3839) loss 3.4991 (3.3097) grad_norm 1.1227 (1.4503) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][420/625] eta 0:01:18 lr 0.001309 wd 0.0500 time 0.3671 (0.3851) data time 0.0011 (0.0024) model time 0.3660 (0.3837) loss 3.9618 (3.3108) grad_norm 1.3519 (1.4506) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][430/625] eta 0:01:15 lr 0.001309 wd 0.0500 time 0.3758 (0.3850) data time 0.0007 (0.0024) model time 0.3750 (0.3835) loss 3.5335 (3.3141) grad_norm 2.6706 (1.4550) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][440/625] eta 0:01:11 lr 0.001308 wd 0.0500 time 0.3787 (0.3848) data time 0.0010 (0.0023) model time 0.3777 (0.3834) loss 3.4255 (3.3137) grad_norm 2.0060 (1.4665) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][450/625] eta 0:01:07 lr 0.001308 wd 0.0500 time 0.3785 (0.3848) data time 0.0008 (0.0023) model time 0.3776 (0.3833) loss 3.5459 (3.3119) grad_norm 1.3775 (1.4642) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][460/625] eta 0:01:03 lr 0.001308 wd 0.0500 time 0.3776 (0.3846) data time 0.0009 (0.0023) model time 0.3767 (0.3832) loss 3.0061 (3.3087) grad_norm 1.3870 (1.4623) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][470/625] eta 0:00:59 lr 0.001308 wd 0.0500 time 0.3743 (0.3849) data time 0.0008 (0.0022) model time 0.3735 (0.3835) loss 3.6996 (3.3111) grad_norm 1.7657 (1.4636) loss_scale 2048.0000 (2048.0000) mem 11225MB [2024-08-26 13:11:44 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-26 13:11:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-26 13:11:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 06:59:43 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-27 06:59:44 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-27 06:59:58 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-27 07:00:12 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-27 07:00:12 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-27 07:00:15 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-27 07:00:17 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-27 07:00:17 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 132) [2024-08-27 07:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-27 07:00:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][480/625] eta 0:05:34 lr 0.001308 wd 0.0500 time 0.3617 (2.3043) data time 0.0008 (0.0975) model time 0.3609 (2.2068) loss 3.7425 (3.7354) grad_norm 1.3375 (1.2550) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][490/625] eta 0:02:45 lr 0.001308 wd 0.0500 time 0.3654 (1.2279) data time 0.0008 (0.0439) model time 0.3646 (1.1840) loss 4.2248 (3.5330) grad_norm 1.2422 (1.3908) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:00:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][500/625] eta 0:01:55 lr 0.001307 wd 0.0500 time 0.3683 (0.9278) data time 0.0010 (0.0286) model time 0.3672 (0.8992) loss 3.3993 (3.5136) grad_norm 1.0522 (1.3312) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:00:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][510/625] eta 0:01:29 lr 0.001307 wd 0.0500 time 0.3606 (0.7805) data time 0.0011 (0.0214) model time 0.3595 (0.7592) loss 3.3469 (3.4883) grad_norm 1.0016 (1.3560) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:00:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][520/625] eta 0:01:13 lr 0.001307 wd 0.0500 time 0.3684 (0.7008) data time 0.0009 (0.0172) model time 0.3675 (0.6836) loss 3.4332 (3.4737) grad_norm 1.0744 (1.3354) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:00:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][530/625] eta 0:01:01 lr 0.001307 wd 0.0500 time 0.3952 (0.6439) data time 0.0009 (0.0144) model time 0.3943 (0.6295) loss 2.8783 (3.4642) grad_norm 1.2930 (1.3470) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][540/625] eta 0:00:51 lr 0.001307 wd 0.0500 time 0.3708 (0.6043) data time 0.0009 (0.0124) model time 0.3700 (0.5919) loss 2.3821 (3.4119) grad_norm 1.0118 (1.4528) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][550/625] eta 0:00:43 lr 0.001307 wd 0.0500 time 0.3657 (0.5741) data time 0.0009 (0.0110) model time 0.3649 (0.5631) loss 3.2228 (3.4009) grad_norm 1.0336 (1.4554) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][560/625] eta 0:00:35 lr 0.001306 wd 0.0500 time 0.3689 (0.5509) data time 0.0011 (0.0099) model time 0.3678 (0.5410) loss 3.7958 (3.3773) grad_norm 1.2733 (1.4307) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][570/625] eta 0:00:29 lr 0.001306 wd 0.0500 time 0.3675 (0.5322) data time 0.0008 (0.0090) model time 0.3667 (0.5232) loss 3.7395 (3.3884) grad_norm 1.4441 (1.4114) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][580/625] eta 0:00:23 lr 0.001306 wd 0.0500 time 0.3653 (0.5166) data time 0.0008 (0.0082) model time 0.3645 (0.5084) loss 2.4317 (3.3950) grad_norm 1.3233 (1.4025) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][590/625] eta 0:00:17 lr 0.001306 wd 0.0500 time 0.3685 (0.5040) data time 0.0011 (0.0076) model time 0.3674 (0.4963) loss 3.0563 (3.3965) grad_norm 1.0640 (1.3942) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][600/625] eta 0:00:12 lr 0.001306 wd 0.0500 time 0.3593 (0.4932) data time 0.0008 (0.0071) model time 0.3585 (0.4861) loss 3.3281 (3.3776) grad_norm 1.3319 (1.3890) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][610/625] eta 0:00:07 lr 0.001306 wd 0.0500 time 0.3649 (0.4840) data time 0.0008 (0.0067) model time 0.3641 (0.4773) loss 3.0825 (3.3619) grad_norm 1.2211 (1.3844) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [132/300][620/625] eta 0:00:02 lr 0.001305 wd 0.0500 time 0.3585 (0.4759) data time 0.0006 (0.0063) model time 0.3579 (0.4696) loss 3.6914 (3.3618) grad_norm 1.2933 (1.3765) loss_scale 2048.0000 (2048.0000) mem 11217MB [2024-08-27 07:01:33 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 132 training takes 0:01:11 [2024-08-27 07:01:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 07:01:36 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 07:01:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.891 (0.891) Loss 0.6431 (0.6431) Acc@1 86.377 (86.377) Acc@5 97.900 (97.900) Mem 11217MB [2024-08-27 07:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.179) Loss 1.0244 (0.7735) Acc@1 76.123 (83.026) Acc@5 93.896 (96.644) Mem 11217MB [2024-08-27 07:01:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.097 (0.140) Loss 1.1484 (0.9252) Acc@1 71.924 (79.455) Acc@5 92.334 (94.820) Mem 11217MB [2024-08-27 07:01:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.125 Acc@5 94.836 [2024-08-27 07:01:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.1% [2024-08-27 07:01:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 2.155 (2.155) Loss 0.5249 (0.5249) Acc@1 88.330 (88.330) Acc@5 98.438 (98.438) Mem 11217MB [2024-08-27 07:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.298) Loss 0.8799 (0.6586) Acc@1 79.248 (85.116) Acc@5 95.557 (97.341) Mem 11217MB [2024-08-27 07:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.096 (0.201) Loss 1.0098 (0.7901) Acc@1 75.684 (81.894) Acc@5 93.945 (95.961) Mem 11217MB [2024-08-27 07:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.616 Acc@5 95.959 [2024-08-27 07:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.6% [2024-08-27 07:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.62% [2024-08-27 07:01:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 07:01:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 07:01:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][0/625] eta 1:34:08 lr 0.001305 wd 0.0500 time 9.0370 (9.0370) data time 7.3391 (7.3391) model time 0.0000 (0.0000) loss 3.3769 (3.3769) grad_norm 1.3713 (1.3713) loss_scale 2048.0000 (2048.0000) mem 11228MB [2024-08-27 07:02:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][10/625] eta 0:11:51 lr 0.001305 wd 0.0500 time 0.3674 (1.1573) data time 0.0008 (0.6682) model time 0.0000 (0.0000) loss 2.6977 (3.2935) grad_norm 1.6626 (1.4113) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][20/625] eta 0:07:57 lr 0.001305 wd 0.0500 time 0.3658 (0.7889) data time 0.0010 (0.3506) model time 0.0000 (0.0000) loss 4.0139 (3.2911) grad_norm 1.3387 (1.4755) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][30/625] eta 0:06:29 lr 0.001305 wd 0.0500 time 0.3794 (0.6551) data time 0.0011 (0.2379) model time 0.0000 (0.0000) loss 3.6739 (3.2081) grad_norm 1.2553 (1.5295) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][40/625] eta 0:05:42 lr 0.001305 wd 0.0500 time 0.3717 (0.5851) data time 0.0010 (0.1801) model time 0.0000 (0.0000) loss 3.0362 (3.1941) grad_norm 1.4293 (1.5278) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][50/625] eta 0:05:12 lr 0.001304 wd 0.0500 time 0.3686 (0.5429) data time 0.0009 (0.1450) model time 0.0000 (0.0000) loss 2.3633 (3.1605) grad_norm 1.6625 (1.5064) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][60/625] eta 0:04:50 lr 0.001304 wd 0.0500 time 0.3938 (0.5145) data time 0.0011 (0.1214) model time 0.3927 (0.3686) loss 2.4859 (3.1778) grad_norm 1.3558 (1.4907) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][70/625] eta 0:04:34 lr 0.001304 wd 0.0500 time 0.3653 (0.4950) data time 0.0008 (0.1050) model time 0.3645 (0.3701) loss 2.6839 (3.1924) grad_norm 1.0808 (1.4950) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][80/625] eta 0:04:21 lr 0.001304 wd 0.0500 time 0.3683 (0.4794) data time 0.0010 (0.0921) model time 0.3672 (0.3692) loss 2.5968 (3.1888) grad_norm 1.6172 (1.4882) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][90/625] eta 0:04:11 lr 0.001304 wd 0.0500 time 0.3645 (0.4699) data time 0.0011 (0.0821) model time 0.3635 (0.3750) loss 3.5672 (3.2115) grad_norm 1.3762 (1.4847) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][100/625] eta 0:04:01 lr 0.001304 wd 0.0500 time 0.3699 (0.4598) data time 0.0011 (0.0741) model time 0.3688 (0.3734) loss 3.9538 (3.1962) grad_norm 1.9468 (1.5006) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][110/625] eta 0:03:52 lr 0.001303 wd 0.0500 time 0.3733 (0.4516) data time 0.0010 (0.0675) model time 0.3723 (0.3725) loss 2.9739 (3.1819) grad_norm 1.7440 (1.5132) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][120/625] eta 0:03:44 lr 0.001303 wd 0.0500 time 0.3794 (0.4450) data time 0.0008 (0.0620) model time 0.3785 (0.3722) loss 3.7940 (3.1770) grad_norm 1.2313 (1.4999) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][130/625] eta 0:03:37 lr 0.001303 wd 0.0500 time 0.3717 (0.4394) data time 0.0010 (0.0574) model time 0.3706 (0.3719) loss 3.3950 (3.1860) grad_norm 1.1006 (1.4930) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][140/625] eta 0:03:30 lr 0.001303 wd 0.0500 time 0.3721 (0.4347) data time 0.0012 (0.0534) model time 0.3709 (0.3718) loss 3.2119 (3.1837) grad_norm 1.2501 (1.4798) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][150/625] eta 0:03:24 lr 0.001303 wd 0.0500 time 0.3664 (0.4300) data time 0.0008 (0.0500) model time 0.3656 (0.3710) loss 3.2390 (3.1669) grad_norm 1.3457 (1.4752) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 07:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][160/625] eta 0:03:18 lr 0.001303 wd 0.0500 time 0.3549 (0.4263) data time 0.0008 (0.0469) model time 0.3541 (0.3708) loss 3.8027 (3.1787) grad_norm inf (inf) loss_scale 1024.0000 (2041.6398) mem 11224MB [2024-08-27 07:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][170/625] eta 0:03:12 lr 0.001302 wd 0.0500 time 0.3681 (0.4232) data time 0.0010 (0.0443) model time 0.3671 (0.3709) loss 3.5670 (3.1972) grad_norm 1.5920 (inf) loss_scale 1024.0000 (1982.1287) mem 11224MB [2024-08-27 07:03:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][180/625] eta 0:03:07 lr 0.001302 wd 0.0500 time 0.3649 (0.4203) data time 0.0011 (0.0419) model time 0.3638 (0.3709) loss 3.5507 (3.2090) grad_norm 1.0993 (inf) loss_scale 1024.0000 (1929.1934) mem 11224MB [2024-08-27 07:03:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][190/625] eta 0:03:01 lr 0.001302 wd 0.0500 time 0.3626 (0.4177) data time 0.0012 (0.0397) model time 0.3614 (0.3707) loss 2.9205 (3.2155) grad_norm 1.5446 (inf) loss_scale 1024.0000 (1881.8010) mem 11224MB [2024-08-27 07:03:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][200/625] eta 0:02:56 lr 0.001302 wd 0.0500 time 0.3739 (0.4152) data time 0.0008 (0.0378) model time 0.3731 (0.3705) loss 3.4935 (3.2223) grad_norm 1.9249 (inf) loss_scale 1024.0000 (1839.1244) mem 11224MB [2024-08-27 07:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][210/625] eta 0:02:51 lr 0.001302 wd 0.0500 time 0.3656 (0.4138) data time 0.0008 (0.0361) model time 0.3648 (0.3713) loss 3.5528 (3.2286) grad_norm 1.0879 (inf) loss_scale 1024.0000 (1800.4929) mem 11224MB [2024-08-27 07:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][220/625] eta 0:02:46 lr 0.001302 wd 0.0500 time 0.3801 (0.4117) data time 0.0009 (0.0345) model time 0.3792 (0.3710) loss 3.4549 (3.2247) grad_norm 1.1317 (inf) loss_scale 1024.0000 (1765.3575) mem 11224MB [2024-08-27 07:03:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][230/625] eta 0:02:41 lr 0.001301 wd 0.0500 time 0.3667 (0.4098) data time 0.0012 (0.0330) model time 0.3655 (0.3708) loss 2.1864 (3.2182) grad_norm 1.4597 (inf) loss_scale 1024.0000 (1733.2641) mem 11224MB [2024-08-27 07:03:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][240/625] eta 0:02:37 lr 0.001301 wd 0.0500 time 0.3704 (0.4080) data time 0.0011 (0.0317) model time 0.3693 (0.3705) loss 3.8274 (3.2204) grad_norm 1.4537 (inf) loss_scale 1024.0000 (1703.8340) mem 11224MB [2024-08-27 07:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][250/625] eta 0:02:32 lr 0.001301 wd 0.0500 time 0.3630 (0.4063) data time 0.0011 (0.0305) model time 0.3619 (0.3702) loss 3.1602 (3.2320) grad_norm 0.9869 (inf) loss_scale 1024.0000 (1676.7490) mem 11224MB [2024-08-27 07:03:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][260/625] eta 0:02:27 lr 0.001301 wd 0.0500 time 0.3750 (0.4048) data time 0.0012 (0.0294) model time 0.3739 (0.3700) loss 3.4461 (3.2414) grad_norm 1.0210 (inf) loss_scale 1024.0000 (1651.7395) mem 11224MB [2024-08-27 07:03:34 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-27 07:03:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 07:03:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 07:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-27 07:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-27 07:50:04 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-27 07:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-27 07:50:17 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-27 07:50:19 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-27 07:50:21 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-27 07:50:21 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 133) [2024-08-27 07:50:21 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-27 07:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][270/625] eta 0:14:12 lr 0.001301 wd 0.0500 time 0.3722 (2.4025) data time 0.0012 (0.1238) model time 0.3710 (2.2787) loss 3.2929 (3.5723) grad_norm 1.4371 (1.5620) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-27 07:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][280/625] eta 0:06:58 lr 0.001301 wd 0.0500 time 0.3882 (1.2127) data time 0.0011 (0.0516) model time 0.3871 (1.1611) loss 3.2947 (3.4765) grad_norm 1.2260 (1.4044) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-27 07:50:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][290/625] eta 0:05:05 lr 0.001300 wd 0.0500 time 0.3847 (0.9113) data time 0.0008 (0.0329) model time 0.3839 (0.8784) loss 3.9476 (3.5127) grad_norm 1.0353 (1.4124) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-27 07:50:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][300/625] eta 0:04:09 lr 0.001300 wd 0.0500 time 0.3798 (0.7669) data time 0.0011 (0.0243) model time 0.3786 (0.7426) loss 3.0101 (3.4501) grad_norm 1.7585 (1.4978) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-27 07:50:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][310/625] eta 0:03:37 lr 0.001300 wd 0.0500 time 0.3760 (0.6897) data time 0.0009 (0.0194) model time 0.3751 (0.6703) loss 3.8542 (3.4061) grad_norm 1.1281 (1.5220) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-27 07:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][320/625] eta 0:03:13 lr 0.001300 wd 0.0500 time 0.3736 (0.6348) data time 0.0010 (0.0162) model time 0.3725 (0.6186) loss 3.2597 (3.3973) grad_norm 1.3258 (1.5120) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-27 07:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][330/625] eta 0:02:55 lr 0.001300 wd 0.0500 time 0.3724 (0.5962) data time 0.0011 (0.0140) model time 0.3713 (0.5822) loss 3.4866 (3.3720) grad_norm 0.9927 (1.4792) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-27 07:51:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][340/625] eta 0:02:41 lr 0.001300 wd 0.0500 time 0.3590 (0.5674) data time 0.0014 (0.0123) model time 0.3577 (0.5551) loss 3.4441 (3.3448) grad_norm nan (nan) loss_scale 512.0000 (1017.3506) mem 11219MB [2024-08-27 07:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][350/625] eta 0:02:29 lr 0.001299 wd 0.0500 time 0.3738 (0.5454) data time 0.0011 (0.0110) model time 0.3727 (0.5343) loss 3.4843 (3.3065) grad_norm 1.4581 (nan) loss_scale 512.0000 (959.2644) mem 11219MB [2024-08-27 07:51:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][360/625] eta 0:02:19 lr 0.001299 wd 0.0500 time 0.3764 (0.5279) data time 0.0011 (0.0100) model time 0.3753 (0.5179) loss 3.2211 (3.3196) grad_norm 1.8661 (nan) loss_scale 512.0000 (913.1546) mem 11219MB [2024-08-27 07:51:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][370/625] eta 0:02:10 lr 0.001299 wd 0.0500 time 0.3683 (0.5137) data time 0.0009 (0.0092) model time 0.3673 (0.5045) loss 3.2647 (3.3443) grad_norm 1.2498 (nan) loss_scale 512.0000 (875.6636) mem 11219MB [2024-08-27 07:51:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][380/625] eta 0:02:02 lr 0.001299 wd 0.0500 time 0.3778 (0.5019) data time 0.0011 (0.0085) model time 0.3767 (0.4934) loss 3.2183 (3.3295) grad_norm 1.1171 (nan) loss_scale 512.0000 (844.5812) mem 11219MB [2024-08-27 07:51:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][390/625] eta 0:01:55 lr 0.001299 wd 0.0500 time 0.3719 (0.4920) data time 0.0010 (0.0079) model time 0.3709 (0.4841) loss 3.5504 (3.3215) grad_norm 1.5350 (nan) loss_scale 512.0000 (818.3937) mem 11219MB [2024-08-27 07:51:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][400/625] eta 0:01:48 lr 0.001299 wd 0.0500 time 0.3752 (0.4835) data time 0.0008 (0.0074) model time 0.3744 (0.4761) loss 2.9940 (3.3188) grad_norm 1.8979 (nan) loss_scale 512.0000 (796.0292) mem 11219MB [2024-08-27 07:51:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][410/625] eta 0:01:42 lr 0.001298 wd 0.0500 time 0.3798 (0.4762) data time 0.0010 (0.0070) model time 0.3788 (0.4692) loss 3.2848 (3.3151) grad_norm 1.4429 (nan) loss_scale 512.0000 (776.7075) mem 11219MB [2024-08-27 07:51:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][420/625] eta 0:01:36 lr 0.001298 wd 0.0500 time 0.3717 (0.4699) data time 0.0011 (0.0066) model time 0.3706 (0.4632) loss 2.8906 (3.3176) grad_norm 1.3650 (nan) loss_scale 512.0000 (759.8471) mem 11219MB [2024-08-27 07:51:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][430/625] eta 0:01:30 lr 0.001298 wd 0.0500 time 0.3687 (0.4642) data time 0.0010 (0.0063) model time 0.3676 (0.4579) loss 3.1511 (3.3234) grad_norm 0.7737 (nan) loss_scale 512.0000 (745.0060) mem 11219MB [2024-08-27 07:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][440/625] eta 0:01:24 lr 0.001298 wd 0.0500 time 0.3767 (0.4592) data time 0.0010 (0.0060) model time 0.3757 (0.4532) loss 3.2447 (3.3197) grad_norm 1.1919 (nan) loss_scale 512.0000 (731.8418) mem 11219MB [2024-08-27 07:51:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][450/625] eta 0:01:19 lr 0.001298 wd 0.0500 time 0.3880 (0.4548) data time 0.0009 (0.0057) model time 0.3871 (0.4490) loss 2.9938 (3.3106) grad_norm 1.5441 (nan) loss_scale 512.0000 (720.0856) mem 11219MB [2024-08-27 07:51:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][460/625] eta 0:01:14 lr 0.001298 wd 0.0500 time 0.3707 (0.4508) data time 0.0008 (0.0055) model time 0.3699 (0.4453) loss 3.4799 (3.3041) grad_norm 1.1082 (nan) loss_scale 512.0000 (709.5228) mem 11219MB [2024-08-27 07:51:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][470/625] eta 0:01:09 lr 0.001297 wd 0.0500 time 0.3793 (0.4472) data time 0.0008 (0.0053) model time 0.3785 (0.4419) loss 3.4145 (3.2862) grad_norm 1.4351 (nan) loss_scale 512.0000 (699.9807) mem 11219MB [2024-08-27 07:52:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][480/625] eta 0:01:04 lr 0.001297 wd 0.0500 time 0.3701 (0.4438) data time 0.0008 (0.0051) model time 0.3693 (0.4387) loss 3.5504 (3.2856) grad_norm 3.2724 (nan) loss_scale 512.0000 (691.3180) mem 11219MB [2024-08-27 07:52:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][490/625] eta 0:00:59 lr 0.001297 wd 0.0500 time 0.3785 (0.4427) data time 0.0010 (0.0049) model time 0.3775 (0.4378) loss 3.3254 (3.2880) grad_norm 1.0273 (nan) loss_scale 512.0000 (683.4185) mem 11219MB [2024-08-27 07:52:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][500/625] eta 0:00:55 lr 0.001297 wd 0.0500 time 0.3693 (0.4406) data time 0.0009 (0.0048) model time 0.3685 (0.4358) loss 3.1566 (3.2826) grad_norm 1.1299 (nan) loss_scale 512.0000 (676.1857) mem 11219MB [2024-08-27 07:52:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][510/625] eta 0:00:50 lr 0.001297 wd 0.0500 time 0.3918 (0.4381) data time 0.0011 (0.0046) model time 0.3907 (0.4335) loss 3.6722 (3.2844) grad_norm 1.5115 (nan) loss_scale 512.0000 (669.5385) mem 11219MB [2024-08-27 07:52:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][520/625] eta 0:00:45 lr 0.001297 wd 0.0500 time 0.3735 (0.4358) data time 0.0009 (0.0045) model time 0.3726 (0.4313) loss 2.5827 (3.2692) grad_norm 1.3526 (nan) loss_scale 512.0000 (663.4086) mem 11219MB [2024-08-27 07:52:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][530/625] eta 0:00:41 lr 0.001296 wd 0.0500 time 0.3769 (0.4334) data time 0.0009 (0.0043) model time 0.3760 (0.4291) loss 2.2621 (3.2593) grad_norm 1.5686 (nan) loss_scale 512.0000 (657.7378) mem 11219MB [2024-08-27 07:52:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][540/625] eta 0:00:36 lr 0.001296 wd 0.0500 time 0.3759 (0.4314) data time 0.0013 (0.0042) model time 0.3746 (0.4272) loss 3.2222 (3.2655) grad_norm 2.0886 (nan) loss_scale 512.0000 (652.4765) mem 11219MB [2024-08-27 07:52:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][550/625] eta 0:00:32 lr 0.001296 wd 0.0500 time 0.3744 (0.4294) data time 0.0009 (0.0041) model time 0.3735 (0.4252) loss 3.9220 (3.2630) grad_norm 1.6296 (nan) loss_scale 512.0000 (647.5819) mem 11219MB [2024-08-27 07:52:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][560/625] eta 0:00:27 lr 0.001296 wd 0.0500 time 0.3807 (0.4276) data time 0.0011 (0.0040) model time 0.3796 (0.4236) loss 1.8915 (3.2483) grad_norm 1.7618 (nan) loss_scale 512.0000 (643.0168) mem 11219MB [2024-08-27 07:52:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][570/625] eta 0:00:23 lr 0.001296 wd 0.0500 time 0.3706 (0.4259) data time 0.0008 (0.0039) model time 0.3698 (0.4220) loss 4.1139 (3.2477) grad_norm 1.3050 (nan) loss_scale 512.0000 (638.7492) mem 11219MB [2024-08-27 07:52:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][580/625] eta 0:00:19 lr 0.001295 wd 0.0500 time 0.3863 (0.4245) data time 0.0011 (0.0038) model time 0.3853 (0.4206) loss 3.8695 (3.2579) grad_norm 1.4526 (nan) loss_scale 512.0000 (634.7508) mem 11219MB [2024-08-27 07:52:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][590/625] eta 0:00:14 lr 0.001295 wd 0.0500 time 0.3745 (0.4230) data time 0.0009 (0.0038) model time 0.3736 (0.4192) loss 2.2365 (3.2646) grad_norm 2.2522 (nan) loss_scale 512.0000 (630.9969) mem 11219MB [2024-08-27 07:52:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][600/625] eta 0:00:10 lr 0.001295 wd 0.0500 time 0.3767 (0.4216) data time 0.0011 (0.0037) model time 0.3756 (0.4180) loss 2.6914 (3.2670) grad_norm 1.4988 (nan) loss_scale 512.0000 (627.4659) mem 11219MB [2024-08-27 07:52:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][610/625] eta 0:00:06 lr 0.001295 wd 0.0500 time 0.3672 (0.4215) data time 0.0006 (0.0036) model time 0.3667 (0.4179) loss 2.7247 (3.2693) grad_norm 1.5310 (nan) loss_scale 512.0000 (624.1383) mem 11219MB [2024-08-27 07:52:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [133/300][620/625] eta 0:00:02 lr 0.001295 wd 0.0500 time 0.3656 (0.4199) data time 0.0008 (0.0035) model time 0.3649 (0.4164) loss 2.6376 (3.2696) grad_norm 1.1840 (nan) loss_scale 512.0000 (620.9972) mem 11219MB [2024-08-27 07:52:57 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 133 training takes 0:02:31 [2024-08-27 07:52:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 07:53:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 07:53:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.477 (0.477) Loss 0.6221 (0.6221) Acc@1 86.279 (86.279) Acc@5 97.705 (97.705) Mem 11219MB [2024-08-27 07:53:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 1.0088 (0.7476) Acc@1 76.953 (82.986) Acc@5 93.262 (96.533) Mem 11219MB [2024-08-27 07:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.1855 (0.9054) Acc@1 71.680 (79.295) Acc@5 92.529 (94.757) Mem 11219MB [2024-08-27 07:53:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 78.989 Acc@5 94.716 [2024-08-27 07:53:05 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.0% [2024-08-27 07:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.806 (0.806) Loss 0.5254 (0.5254) Acc@1 88.379 (88.379) Acc@5 98.486 (98.486) Mem 11219MB [2024-08-27 07:53:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.176) Loss 0.8779 (0.6573) Acc@1 79.150 (85.094) Acc@5 95.459 (97.381) Mem 11219MB [2024-08-27 07:53:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 1.0068 (0.7888) Acc@1 75.635 (81.862) Acc@5 94.043 (96.001) Mem 11219MB [2024-08-27 07:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.586 Acc@5 95.985 [2024-08-27 07:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.6% [2024-08-27 07:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.59% [2024-08-27 07:53:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 07:53:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 07:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][0/625] eta 0:07:52 lr 0.001295 wd 0.0500 time 0.7558 (0.7558) data time 0.3775 (0.3775) model time 0.0000 (0.0000) loss 4.1156 (4.1156) grad_norm 1.7722 (1.7722) loss_scale 512.0000 (512.0000) mem 11226MB [2024-08-27 07:53:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][10/625] eta 0:04:20 lr 0.001295 wd 0.0500 time 0.3437 (0.4230) data time 0.0010 (0.0354) model time 0.0000 (0.0000) loss 3.3376 (3.2029) grad_norm 1.3604 (1.5255) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][20/625] eta 0:04:02 lr 0.001294 wd 0.0500 time 0.3769 (0.4013) data time 0.0008 (0.0190) model time 0.0000 (0.0000) loss 3.9605 (3.2207) grad_norm 1.3273 (1.4400) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][30/625] eta 0:03:54 lr 0.001294 wd 0.0500 time 0.3780 (0.3934) data time 0.0008 (0.0132) model time 0.0000 (0.0000) loss 3.2268 (3.1533) grad_norm 1.4377 (1.4375) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][40/625] eta 0:03:48 lr 0.001294 wd 0.0500 time 0.3851 (0.3902) data time 0.0008 (0.0103) model time 0.0000 (0.0000) loss 3.9879 (3.2302) grad_norm 1.5756 (1.4817) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][50/625] eta 0:03:42 lr 0.001294 wd 0.0500 time 0.3764 (0.3877) data time 0.0010 (0.0085) model time 0.0000 (0.0000) loss 3.5785 (3.2536) grad_norm 1.1873 (1.4621) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][60/625] eta 0:03:38 lr 0.001294 wd 0.0500 time 0.3818 (0.3862) data time 0.0008 (0.0073) model time 0.3810 (0.3774) loss 3.6461 (3.2616) grad_norm 1.4829 (1.4520) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][70/625] eta 0:03:33 lr 0.001294 wd 0.0500 time 0.3759 (0.3848) data time 0.0009 (0.0064) model time 0.3751 (0.3762) loss 3.3624 (3.3073) grad_norm 1.9848 (1.4644) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][80/625] eta 0:03:29 lr 0.001293 wd 0.0500 time 0.3748 (0.3837) data time 0.0008 (0.0058) model time 0.3740 (0.3757) loss 3.4422 (3.3219) grad_norm 1.3652 (1.4504) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][90/625] eta 0:03:24 lr 0.001293 wd 0.0500 time 0.3825 (0.3829) data time 0.0011 (0.0053) model time 0.3814 (0.3757) loss 2.5007 (3.3016) grad_norm 1.6326 (1.4402) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][100/625] eta 0:03:20 lr 0.001293 wd 0.0500 time 0.3874 (0.3825) data time 0.0009 (0.0048) model time 0.3866 (0.3760) loss 2.2871 (3.2730) grad_norm 1.1582 (1.4316) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:53:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][110/625] eta 0:03:16 lr 0.001293 wd 0.0500 time 0.3729 (0.3820) data time 0.0013 (0.0045) model time 0.3716 (0.3760) loss 3.4163 (3.2477) grad_norm 1.3574 (1.4055) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][120/625] eta 0:03:12 lr 0.001293 wd 0.0500 time 0.3762 (0.3816) data time 0.0008 (0.0042) model time 0.3754 (0.3760) loss 4.1545 (3.2443) grad_norm 1.2321 (1.4226) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][130/625] eta 0:03:09 lr 0.001293 wd 0.0500 time 0.3394 (0.3829) data time 0.0012 (0.0040) model time 0.3383 (0.3787) loss 3.6791 (3.2634) grad_norm 1.7122 (1.4449) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][140/625] eta 0:03:05 lr 0.001292 wd 0.0500 time 0.3688 (0.3822) data time 0.0011 (0.0038) model time 0.3677 (0.3780) loss 3.7090 (3.2618) grad_norm 1.3465 (1.4298) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][150/625] eta 0:03:01 lr 0.001292 wd 0.0500 time 0.3761 (0.3820) data time 0.0009 (0.0036) model time 0.3752 (0.3780) loss 3.7700 (3.2734) grad_norm 1.1884 (1.4355) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][160/625] eta 0:02:57 lr 0.001292 wd 0.0500 time 0.3959 (0.3819) data time 0.0008 (0.0035) model time 0.3951 (0.3781) loss 2.3232 (3.2629) grad_norm 1.2138 (1.4449) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][170/625] eta 0:02:55 lr 0.001292 wd 0.0500 time 0.3664 (0.3850) data time 0.0011 (0.0033) model time 0.3653 (0.3827) loss 3.1923 (3.2511) grad_norm 1.1104 (1.4348) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][180/625] eta 0:02:51 lr 0.001292 wd 0.0500 time 0.3755 (0.3845) data time 0.0008 (0.0032) model time 0.3747 (0.3822) loss 2.7240 (3.2447) grad_norm 1.6804 (1.4343) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][190/625] eta 0:02:47 lr 0.001291 wd 0.0500 time 0.3798 (0.3843) data time 0.0008 (0.0031) model time 0.3790 (0.3819) loss 4.4860 (3.2550) grad_norm 1.2908 (1.4314) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][200/625] eta 0:02:43 lr 0.001291 wd 0.0500 time 0.3765 (0.3838) data time 0.0009 (0.0030) model time 0.3756 (0.3813) loss 3.9448 (3.2672) grad_norm 1.5488 (1.4247) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][210/625] eta 0:02:39 lr 0.001291 wd 0.0500 time 0.3791 (0.3836) data time 0.0011 (0.0029) model time 0.3781 (0.3811) loss 2.8342 (3.2682) grad_norm 1.0467 (1.4203) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][220/625] eta 0:02:35 lr 0.001291 wd 0.0500 time 0.3787 (0.3833) data time 0.0008 (0.0028) model time 0.3779 (0.3809) loss 3.6633 (3.2749) grad_norm 1.4209 (1.4242) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][230/625] eta 0:02:31 lr 0.001291 wd 0.0500 time 0.3804 (0.3831) data time 0.0011 (0.0027) model time 0.3793 (0.3806) loss 3.7619 (3.2821) grad_norm 1.5821 (1.4271) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][240/625] eta 0:02:27 lr 0.001291 wd 0.0500 time 0.3878 (0.3830) data time 0.0009 (0.0027) model time 0.3870 (0.3806) loss 3.3247 (3.2801) grad_norm 1.1292 (1.4315) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][250/625] eta 0:02:23 lr 0.001290 wd 0.0500 time 0.3895 (0.3828) data time 0.0011 (0.0026) model time 0.3885 (0.3804) loss 2.3458 (3.2802) grad_norm 1.4039 (1.4483) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][260/625] eta 0:02:19 lr 0.001290 wd 0.0500 time 0.3678 (0.3826) data time 0.0012 (0.0026) model time 0.3665 (0.3802) loss 3.3508 (3.2877) grad_norm 1.2294 (1.4545) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][270/625] eta 0:02:15 lr 0.001290 wd 0.0500 time 0.3810 (0.3825) data time 0.0012 (0.0025) model time 0.3798 (0.3801) loss 3.5162 (3.2900) grad_norm 1.3770 (1.4585) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][280/625] eta 0:02:11 lr 0.001290 wd 0.0500 time 0.3785 (0.3823) data time 0.0011 (0.0025) model time 0.3775 (0.3800) loss 2.5109 (3.2810) grad_norm 2.0949 (1.4549) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][290/625] eta 0:02:08 lr 0.001290 wd 0.0500 time 0.3752 (0.3836) data time 0.0012 (0.0024) model time 0.3740 (0.3816) loss 2.4592 (3.2798) grad_norm 1.3689 (1.4563) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][300/625] eta 0:02:04 lr 0.001290 wd 0.0500 time 0.3733 (0.3834) data time 0.0014 (0.0024) model time 0.3719 (0.3814) loss 3.6368 (3.2711) grad_norm 1.6031 (1.4591) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][310/625] eta 0:02:00 lr 0.001289 wd 0.0500 time 0.4052 (0.3834) data time 0.0012 (0.0023) model time 0.4040 (0.3815) loss 3.0479 (3.2748) grad_norm 1.1136 (1.4586) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][320/625] eta 0:01:56 lr 0.001289 wd 0.0500 time 0.3807 (0.3832) data time 0.0011 (0.0023) model time 0.3796 (0.3813) loss 2.9516 (3.2778) grad_norm 1.9509 (1.4672) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][330/625] eta 0:01:53 lr 0.001289 wd 0.0500 time 0.3798 (0.3831) data time 0.0010 (0.0023) model time 0.3788 (0.3811) loss 3.7083 (3.2745) grad_norm 1.1496 (1.4632) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][340/625] eta 0:01:49 lr 0.001289 wd 0.0500 time 0.3758 (0.3835) data time 0.0009 (0.0022) model time 0.3749 (0.3816) loss 2.7225 (3.2699) grad_norm 1.5209 (1.4585) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][350/625] eta 0:01:45 lr 0.001289 wd 0.0500 time 0.3841 (0.3834) data time 0.0009 (0.0022) model time 0.3832 (0.3815) loss 2.2537 (3.2661) grad_norm 2.0136 (1.4652) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][360/625] eta 0:01:41 lr 0.001289 wd 0.0500 time 0.3760 (0.3833) data time 0.0008 (0.0022) model time 0.3752 (0.3814) loss 3.3484 (3.2631) grad_norm 1.1731 (1.4646) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][370/625] eta 0:01:37 lr 0.001288 wd 0.0500 time 0.3783 (0.3832) data time 0.0010 (0.0021) model time 0.3773 (0.3813) loss 3.6963 (3.2659) grad_norm 0.9807 (1.4596) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][380/625] eta 0:01:33 lr 0.001288 wd 0.0500 time 0.3696 (0.3830) data time 0.0011 (0.0021) model time 0.3684 (0.3812) loss 2.4734 (3.2713) grad_norm 1.5504 (1.4864) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][390/625] eta 0:01:29 lr 0.001288 wd 0.0500 time 0.3790 (0.3829) data time 0.0009 (0.0021) model time 0.3781 (0.3810) loss 4.0556 (3.2736) grad_norm 1.7158 (1.4843) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][400/625] eta 0:01:26 lr 0.001288 wd 0.0500 time 0.3876 (0.3827) data time 0.0009 (0.0021) model time 0.3867 (0.3809) loss 4.0878 (3.2761) grad_norm 1.9695 (1.4828) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][410/625] eta 0:01:22 lr 0.001288 wd 0.0500 time 0.3759 (0.3826) data time 0.0009 (0.0020) model time 0.3751 (0.3808) loss 3.3962 (3.2765) grad_norm 1.2711 (1.4812) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][420/625] eta 0:01:18 lr 0.001288 wd 0.0500 time 0.3718 (0.3825) data time 0.0008 (0.0020) model time 0.3710 (0.3807) loss 3.4201 (3.2786) grad_norm 1.3123 (1.4786) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][430/625] eta 0:01:14 lr 0.001287 wd 0.0500 time 0.3755 (0.3824) data time 0.0011 (0.0020) model time 0.3744 (0.3806) loss 3.8501 (3.2787) grad_norm 1.3845 (1.4743) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][440/625] eta 0:01:10 lr 0.001287 wd 0.0500 time 0.3825 (0.3824) data time 0.0008 (0.0020) model time 0.3817 (0.3806) loss 2.8770 (3.2791) grad_norm 1.6062 (1.4741) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][450/625] eta 0:01:06 lr 0.001287 wd 0.0500 time 0.3811 (0.3824) data time 0.0011 (0.0020) model time 0.3800 (0.3805) loss 3.7201 (3.2721) grad_norm 1.3458 (1.4724) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][460/625] eta 0:01:03 lr 0.001287 wd 0.0500 time 0.3738 (0.3823) data time 0.0008 (0.0019) model time 0.3730 (0.3805) loss 2.6426 (3.2709) grad_norm 1.2542 (1.4687) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][470/625] eta 0:00:59 lr 0.001287 wd 0.0500 time 0.3706 (0.3822) data time 0.0008 (0.0019) model time 0.3698 (0.3804) loss 2.8956 (3.2631) grad_norm 1.2810 (1.4720) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][480/625] eta 0:00:55 lr 0.001287 wd 0.0500 time 0.3741 (0.3821) data time 0.0010 (0.0019) model time 0.3730 (0.3803) loss 2.7637 (3.2628) grad_norm 1.5169 (1.4729) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][490/625] eta 0:00:51 lr 0.001286 wd 0.0500 time 0.3855 (0.3820) data time 0.0009 (0.0019) model time 0.3846 (0.3802) loss 3.0582 (3.2620) grad_norm 1.1435 (1.4689) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][500/625] eta 0:00:47 lr 0.001286 wd 0.0500 time 0.3781 (0.3828) data time 0.0010 (0.0019) model time 0.3771 (0.3811) loss 2.9778 (3.2633) grad_norm 1.1342 (1.4671) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][510/625] eta 0:00:44 lr 0.001286 wd 0.0500 time 0.3743 (0.3827) data time 0.0009 (0.0019) model time 0.3734 (0.3810) loss 2.4426 (3.2630) grad_norm 1.7297 (1.4706) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][520/625] eta 0:00:40 lr 0.001286 wd 0.0500 time 0.3697 (0.3826) data time 0.0009 (0.0018) model time 0.3688 (0.3810) loss 2.4322 (3.2608) grad_norm 1.4342 (1.4661) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][530/625] eta 0:00:36 lr 0.001286 wd 0.0500 time 0.3712 (0.3828) data time 0.0009 (0.0018) model time 0.3703 (0.3812) loss 3.7573 (3.2617) grad_norm 1.4594 (1.4636) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][540/625] eta 0:00:32 lr 0.001286 wd 0.0500 time 0.3764 (0.3827) data time 0.0009 (0.0018) model time 0.3755 (0.3811) loss 3.8420 (3.2618) grad_norm 1.9510 (1.4637) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][550/625] eta 0:00:28 lr 0.001285 wd 0.0500 time 0.3919 (0.3826) data time 0.0008 (0.0018) model time 0.3910 (0.3810) loss 2.5570 (3.2604) grad_norm 1.6017 (1.4661) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][560/625] eta 0:00:24 lr 0.001285 wd 0.0500 time 0.3622 (0.3826) data time 0.0011 (0.0018) model time 0.3611 (0.3810) loss 3.4462 (3.2634) grad_norm 1.2350 (1.4653) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][570/625] eta 0:00:21 lr 0.001285 wd 0.0500 time 0.3796 (0.3825) data time 0.0008 (0.0018) model time 0.3788 (0.3809) loss 3.7456 (3.2691) grad_norm 2.2460 (1.4653) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][580/625] eta 0:00:17 lr 0.001285 wd 0.0500 time 0.3859 (0.3824) data time 0.0010 (0.0018) model time 0.3848 (0.3808) loss 2.0261 (3.2620) grad_norm 2.7971 (1.4690) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][590/625] eta 0:00:13 lr 0.001285 wd 0.0500 time 0.3799 (0.3824) data time 0.0011 (0.0018) model time 0.3788 (0.3807) loss 3.5272 (3.2554) grad_norm 1.2633 (1.4685) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][600/625] eta 0:00:09 lr 0.001284 wd 0.0500 time 0.3733 (0.3823) data time 0.0010 (0.0017) model time 0.3722 (0.3806) loss 3.0019 (3.2555) grad_norm 1.2451 (1.4672) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][610/625] eta 0:00:05 lr 0.001284 wd 0.0500 time 0.3752 (0.3822) data time 0.0005 (0.0017) model time 0.3746 (0.3806) loss 4.5043 (3.2583) grad_norm 2.2930 (1.4671) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [134/300][620/625] eta 0:00:01 lr 0.001284 wd 0.0500 time 0.3633 (0.3820) data time 0.0006 (0.0017) model time 0.3627 (0.3803) loss 3.7281 (3.2585) grad_norm 1.6949 (1.4687) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:12 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 134 training takes 0:03:58 [2024-08-27 07:57:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 07:57:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 07:57:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.490 (0.490) Loss 0.6113 (0.6113) Acc@1 85.986 (85.986) Acc@5 97.803 (97.803) Mem 11225MB [2024-08-27 07:57:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 0.9897 (0.7376) Acc@1 76.904 (83.026) Acc@5 93.799 (96.569) Mem 11225MB [2024-08-27 07:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.1572 (0.8928) Acc@1 73.145 (79.622) Acc@5 92.871 (94.834) Mem 11225MB [2024-08-27 07:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.393 Acc@5 94.852 [2024-08-27 07:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.4% [2024-08-27 07:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.39% [2024-08-27 07:57:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 07:57:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 07:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.5254 (0.5254) Acc@1 88.281 (88.281) Acc@5 98.389 (98.389) Mem 11225MB [2024-08-27 07:57:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.139) Loss 0.8765 (0.6565) Acc@1 79.102 (85.107) Acc@5 95.508 (97.403) Mem 11225MB [2024-08-27 07:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.117) Loss 1.0068 (0.7878) Acc@1 75.732 (81.899) Acc@5 94.189 (96.040) Mem 11225MB [2024-08-27 07:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.632 Acc@5 96.025 [2024-08-27 07:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.6% [2024-08-27 07:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.63% [2024-08-27 07:57:20 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 07:57:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 07:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][0/625] eta 0:08:04 lr 0.001284 wd 0.0500 time 0.7758 (0.7758) data time 0.4454 (0.4454) model time 0.0000 (0.0000) loss 4.0287 (4.0287) grad_norm 2.1186 (2.1186) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][10/625] eta 0:04:14 lr 0.001284 wd 0.0500 time 0.3758 (0.4140) data time 0.0011 (0.0415) model time 0.0000 (0.0000) loss 1.7105 (3.1702) grad_norm 1.4759 (1.6663) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][20/625] eta 0:03:59 lr 0.001284 wd 0.0500 time 0.3806 (0.3967) data time 0.0010 (0.0223) model time 0.0000 (0.0000) loss 3.7984 (3.2646) grad_norm 1.6885 (1.6399) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][30/625] eta 0:03:52 lr 0.001284 wd 0.0500 time 0.3771 (0.3900) data time 0.0010 (0.0154) model time 0.0000 (0.0000) loss 2.2537 (3.3091) grad_norm 1.3008 (1.5386) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][40/625] eta 0:03:46 lr 0.001283 wd 0.0500 time 0.3844 (0.3872) data time 0.0010 (0.0120) model time 0.0000 (0.0000) loss 3.1913 (3.2402) grad_norm 1.4583 (1.4968) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][50/625] eta 0:03:42 lr 0.001283 wd 0.0500 time 0.3769 (0.3861) data time 0.0011 (0.0098) model time 0.0000 (0.0000) loss 3.4299 (3.2549) grad_norm 1.4285 (1.4651) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][60/625] eta 0:03:40 lr 0.001283 wd 0.0500 time 0.5878 (0.3905) data time 0.0008 (0.0084) model time 0.5870 (0.4116) loss 3.1219 (3.2667) grad_norm 2.6262 (1.4578) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][70/625] eta 0:03:37 lr 0.001283 wd 0.0500 time 0.3800 (0.3921) data time 0.0011 (0.0074) model time 0.3790 (0.4061) loss 3.5367 (3.2783) grad_norm 1.1504 (1.4341) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][80/625] eta 0:03:32 lr 0.001283 wd 0.0500 time 0.3757 (0.3902) data time 0.0010 (0.0066) model time 0.3746 (0.3960) loss 3.0549 (3.2699) grad_norm 1.2285 (1.4182) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:57:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][90/625] eta 0:03:28 lr 0.001283 wd 0.0500 time 0.3898 (0.3888) data time 0.0010 (0.0060) model time 0.3888 (0.3911) loss 3.8100 (3.2435) grad_norm 1.5590 (1.4140) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][100/625] eta 0:03:23 lr 0.001282 wd 0.0500 time 0.3822 (0.3876) data time 0.0009 (0.0055) model time 0.3814 (0.3880) loss 3.3427 (3.2391) grad_norm 1.2396 (1.4319) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][110/625] eta 0:03:19 lr 0.001282 wd 0.0500 time 0.3757 (0.3866) data time 0.0008 (0.0051) model time 0.3749 (0.3859) loss 4.1869 (3.2514) grad_norm 0.9573 (1.4094) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][120/625] eta 0:03:15 lr 0.001282 wd 0.0500 time 0.3725 (0.3863) data time 0.0008 (0.0048) model time 0.3717 (0.3852) loss 3.4150 (3.2447) grad_norm 1.9291 (1.4086) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][130/625] eta 0:03:10 lr 0.001282 wd 0.0500 time 0.3899 (0.3858) data time 0.0011 (0.0045) model time 0.3888 (0.3844) loss 3.3789 (3.2575) grad_norm 1.2012 (1.4021) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][140/625] eta 0:03:06 lr 0.001282 wd 0.0500 time 0.3817 (0.3854) data time 0.0009 (0.0043) model time 0.3808 (0.3838) loss 3.6868 (3.2663) grad_norm 1.4074 (1.3916) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][150/625] eta 0:03:02 lr 0.001282 wd 0.0500 time 0.3813 (0.3851) data time 0.0011 (0.0041) model time 0.3803 (0.3835) loss 2.8195 (3.2596) grad_norm 1.7737 (1.4215) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][160/625] eta 0:02:59 lr 0.001281 wd 0.0500 time 0.3800 (0.3861) data time 0.0009 (0.0039) model time 0.3792 (0.3849) loss 4.2416 (3.2625) grad_norm 1.0946 (1.4300) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][170/625] eta 0:02:55 lr 0.001281 wd 0.0500 time 0.3839 (0.3857) data time 0.0008 (0.0037) model time 0.3831 (0.3844) loss 3.1622 (3.2586) grad_norm 0.8874 (1.4411) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][180/625] eta 0:02:51 lr 0.001281 wd 0.0500 time 0.3783 (0.3853) data time 0.0011 (0.0036) model time 0.3772 (0.3839) loss 3.2502 (3.2411) grad_norm 1.1500 (1.4470) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][190/625] eta 0:02:47 lr 0.001281 wd 0.0500 time 0.3782 (0.3850) data time 0.0008 (0.0035) model time 0.3774 (0.3835) loss 4.3357 (3.2621) grad_norm 1.5791 (1.4454) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][200/625] eta 0:02:43 lr 0.001281 wd 0.0500 time 0.3899 (0.3848) data time 0.0011 (0.0033) model time 0.3888 (0.3832) loss 3.7473 (3.2811) grad_norm 1.3377 (1.4352) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][210/625] eta 0:02:39 lr 0.001280 wd 0.0500 time 0.3790 (0.3846) data time 0.0009 (0.0032) model time 0.3781 (0.3830) loss 3.5354 (3.2817) grad_norm 1.2290 (1.4393) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][220/625] eta 0:02:35 lr 0.001280 wd 0.0500 time 0.3787 (0.3844) data time 0.0008 (0.0031) model time 0.3779 (0.3827) loss 3.6936 (3.2938) grad_norm 1.7784 (1.4360) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][230/625] eta 0:02:31 lr 0.001280 wd 0.0500 time 0.3787 (0.3842) data time 0.0008 (0.0030) model time 0.3779 (0.3825) loss 3.7110 (3.2921) grad_norm 1.9031 (1.4436) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][240/625] eta 0:02:27 lr 0.001280 wd 0.0500 time 0.3761 (0.3840) data time 0.0011 (0.0030) model time 0.3750 (0.3823) loss 3.5375 (3.2788) grad_norm 1.3637 (1.4431) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:58:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][250/625] eta 0:02:25 lr 0.001280 wd 0.0500 time 0.3741 (0.3871) data time 0.0011 (0.0029) model time 0.3731 (0.3862) loss 3.4482 (3.2721) grad_norm 2.0866 (1.4493) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][260/625] eta 0:02:21 lr 0.001280 wd 0.0500 time 0.3742 (0.3867) data time 0.0010 (0.0028) model time 0.3732 (0.3858) loss 3.3442 (3.2764) grad_norm 1.4141 (1.4410) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][270/625] eta 0:02:17 lr 0.001279 wd 0.0500 time 0.3745 (0.3864) data time 0.0011 (0.0028) model time 0.3734 (0.3853) loss 2.8038 (3.2661) grad_norm 1.1942 (1.4399) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][280/625] eta 0:02:13 lr 0.001279 wd 0.0500 time 0.3772 (0.3861) data time 0.0009 (0.0027) model time 0.3763 (0.3850) loss 3.7936 (3.2779) grad_norm 1.2036 (1.4389) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][290/625] eta 0:02:09 lr 0.001279 wd 0.0500 time 0.3756 (0.3859) data time 0.0011 (0.0027) model time 0.3745 (0.3847) loss 3.6980 (3.2807) grad_norm 1.3669 (1.4467) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][300/625] eta 0:02:05 lr 0.001279 wd 0.0500 time 0.3746 (0.3858) data time 0.0011 (0.0026) model time 0.3735 (0.3846) loss 3.1137 (3.2845) grad_norm 1.2931 (1.4400) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][310/625] eta 0:02:01 lr 0.001279 wd 0.0500 time 0.3768 (0.3857) data time 0.0008 (0.0026) model time 0.3760 (0.3844) loss 2.9364 (3.2837) grad_norm 1.1892 (1.4517) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][320/625] eta 0:01:57 lr 0.001279 wd 0.0500 time 0.3805 (0.3855) data time 0.0008 (0.0025) model time 0.3797 (0.3842) loss 2.9440 (3.2777) grad_norm 1.7569 (1.4578) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][330/625] eta 0:01:53 lr 0.001278 wd 0.0500 time 0.3800 (0.3853) data time 0.0008 (0.0025) model time 0.3792 (0.3840) loss 3.2459 (3.2691) grad_norm 0.9665 (1.4531) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][340/625] eta 0:01:49 lr 0.001278 wd 0.0500 time 0.5849 (0.3857) data time 0.0010 (0.0024) model time 0.5839 (0.3845) loss 2.8275 (3.2656) grad_norm 1.1855 (1.4484) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][350/625] eta 0:01:45 lr 0.001278 wd 0.0500 time 0.3833 (0.3854) data time 0.0010 (0.0024) model time 0.3822 (0.3841) loss 3.4916 (3.2676) grad_norm 1.3561 (1.4452) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][360/625] eta 0:01:42 lr 0.001278 wd 0.0500 time 0.3813 (0.3852) data time 0.0011 (0.0024) model time 0.3802 (0.3839) loss 3.2086 (3.2726) grad_norm 1.7124 (1.4473) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][370/625] eta 0:01:38 lr 0.001278 wd 0.0500 time 0.3662 (0.3867) data time 0.0008 (0.0023) model time 0.3654 (0.3857) loss 3.7250 (3.2800) grad_norm 1.9704 (1.4476) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][380/625] eta 0:01:34 lr 0.001278 wd 0.0500 time 0.3791 (0.3865) data time 0.0010 (0.0023) model time 0.3781 (0.3854) loss 3.6816 (3.2847) grad_norm 2.9819 (1.4579) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][390/625] eta 0:01:30 lr 0.001277 wd 0.0500 time 0.3832 (0.3863) data time 0.0010 (0.0023) model time 0.3822 (0.3852) loss 2.5961 (3.2839) grad_norm 1.5955 (1.4637) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 07:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][400/625] eta 0:01:26 lr 0.001277 wd 0.0500 time 0.3818 (0.3862) data time 0.0013 (0.0022) model time 0.3804 (0.3851) loss 3.6034 (3.2884) grad_norm 1.9230 (1.4714) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][410/625] eta 0:01:22 lr 0.001277 wd 0.0500 time 0.3792 (0.3860) data time 0.0011 (0.0022) model time 0.3781 (0.3849) loss 3.6459 (3.2861) grad_norm 1.2306 (1.4649) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][420/625] eta 0:01:19 lr 0.001277 wd 0.0500 time 0.3774 (0.3859) data time 0.0009 (0.0022) model time 0.3765 (0.3847) loss 3.5597 (3.2911) grad_norm 1.4884 (1.4635) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][430/625] eta 0:01:15 lr 0.001277 wd 0.0500 time 0.3816 (0.3857) data time 0.0010 (0.0022) model time 0.3806 (0.3845) loss 2.4304 (3.2893) grad_norm 1.4128 (1.4615) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][440/625] eta 0:01:11 lr 0.001277 wd 0.0500 time 0.3784 (0.3856) data time 0.0010 (0.0022) model time 0.3774 (0.3844) loss 2.7820 (3.2884) grad_norm 1.0576 (1.4546) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][450/625] eta 0:01:07 lr 0.001276 wd 0.0500 time 0.3820 (0.3854) data time 0.0008 (0.0021) model time 0.3812 (0.3842) loss 2.8489 (3.2880) grad_norm 1.7323 (1.4547) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][460/625] eta 0:01:03 lr 0.001276 wd 0.0500 time 0.3334 (0.3856) data time 0.0009 (0.0021) model time 0.3325 (0.3844) loss 3.6072 (3.2899) grad_norm 1.5634 (1.4655) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][470/625] eta 0:00:59 lr 0.001276 wd 0.0500 time 0.3804 (0.3855) data time 0.0011 (0.0021) model time 0.3793 (0.3843) loss 3.3982 (3.2880) grad_norm 1.7518 (1.4706) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][480/625] eta 0:00:55 lr 0.001276 wd 0.0500 time 0.3786 (0.3854) data time 0.0010 (0.0021) model time 0.3776 (0.3841) loss 2.7853 (3.2887) grad_norm 1.0397 (1.4735) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][490/625] eta 0:00:52 lr 0.001276 wd 0.0500 time 0.3740 (0.3852) data time 0.0011 (0.0020) model time 0.3729 (0.3840) loss 2.4358 (3.2851) grad_norm 1.6544 (1.4748) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][500/625] eta 0:00:48 lr 0.001276 wd 0.0500 time 0.3732 (0.3851) data time 0.0009 (0.0020) model time 0.3723 (0.3838) loss 2.1555 (3.2770) grad_norm 1.8889 (1.4767) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][510/625] eta 0:00:44 lr 0.001275 wd 0.0500 time 0.3777 (0.3849) data time 0.0009 (0.0020) model time 0.3768 (0.3837) loss 2.6074 (3.2709) grad_norm 1.8104 (1.4798) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][520/625] eta 0:00:40 lr 0.001275 wd 0.0500 time 0.3831 (0.3848) data time 0.0008 (0.0020) model time 0.3823 (0.3835) loss 4.0273 (3.2758) grad_norm 2.4014 (1.4840) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][530/625] eta 0:00:36 lr 0.001275 wd 0.0500 time 0.3801 (0.3847) data time 0.0012 (0.0020) model time 0.3789 (0.3834) loss 3.3462 (3.2774) grad_norm 1.3003 (1.4810) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][540/625] eta 0:00:32 lr 0.001275 wd 0.0500 time 0.3805 (0.3846) data time 0.0012 (0.0020) model time 0.3793 (0.3833) loss 2.9301 (3.2836) grad_norm 2.1542 (1.4988) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][550/625] eta 0:00:28 lr 0.001275 wd 0.0500 time 0.3802 (0.3845) data time 0.0010 (0.0019) model time 0.3792 (0.3832) loss 3.4356 (3.2813) grad_norm 1.3454 (1.5021) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:00:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][560/625] eta 0:00:24 lr 0.001274 wd 0.0500 time 0.3842 (0.3844) data time 0.0011 (0.0019) model time 0.3831 (0.3831) loss 3.3420 (3.2790) grad_norm 1.5386 (1.4994) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][570/625] eta 0:00:21 lr 0.001274 wd 0.0500 time 0.3784 (0.3843) data time 0.0010 (0.0019) model time 0.3774 (0.3830) loss 3.5346 (3.2789) grad_norm 1.4213 (1.4984) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][580/625] eta 0:00:17 lr 0.001274 wd 0.0500 time 0.3766 (0.3854) data time 0.0009 (0.0019) model time 0.3757 (0.3841) loss 3.8363 (3.2754) grad_norm 1.6055 (1.5004) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][590/625] eta 0:00:13 lr 0.001274 wd 0.0500 time 0.3774 (0.3852) data time 0.0009 (0.0019) model time 0.3765 (0.3840) loss 2.5940 (3.2752) grad_norm 1.1025 (1.4966) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][600/625] eta 0:00:09 lr 0.001274 wd 0.0500 time 0.3774 (0.3851) data time 0.0008 (0.0019) model time 0.3766 (0.3839) loss 3.0846 (3.2778) grad_norm 2.0727 (1.4970) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][610/625] eta 0:00:05 lr 0.001274 wd 0.0500 time 0.3662 (0.3849) data time 0.0006 (0.0019) model time 0.3656 (0.3836) loss 3.3112 (3.2808) grad_norm 1.7678 (1.4983) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [135/300][620/625] eta 0:00:01 lr 0.001273 wd 0.0500 time 0.3642 (0.3846) data time 0.0006 (0.0019) model time 0.3636 (0.3834) loss 3.9889 (3.2864) grad_norm 1.4824 (1.5002) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:22 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 135 training takes 0:04:00 [2024-08-27 08:01:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 08:01:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 08:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.6279 (0.6279) Acc@1 87.256 (87.256) Acc@5 97.607 (97.607) Mem 11225MB [2024-08-27 08:01:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 1.0654 (0.7707) Acc@1 74.805 (83.119) Acc@5 94.043 (96.733) Mem 11225MB [2024-08-27 08:01:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.1484 (0.9127) Acc@1 72.949 (79.511) Acc@5 92.920 (95.033) Mem 11225MB [2024-08-27 08:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.267 Acc@5 95.028 [2024-08-27 08:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.3% [2024-08-27 08:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.821 (0.821) Loss 0.5254 (0.5254) Acc@1 88.379 (88.379) Acc@5 98.389 (98.389) Mem 11225MB [2024-08-27 08:01:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.099 (0.177) Loss 0.8735 (0.6549) Acc@1 79.297 (85.130) Acc@5 95.557 (97.377) Mem 11225MB [2024-08-27 08:01:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.139) Loss 1.0029 (0.7862) Acc@1 75.586 (81.913) Acc@5 94.189 (96.022) Mem 11225MB [2024-08-27 08:01:29 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.630 Acc@5 96.019 [2024-08-27 08:01:29 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.6% [2024-08-27 08:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][0/625] eta 0:13:06 lr 0.001273 wd 0.0500 time 1.2580 (1.2580) data time 0.5448 (0.5448) model time 0.0000 (0.0000) loss 3.5972 (3.5972) grad_norm 1.3078 (1.3078) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][10/625] eta 0:04:41 lr 0.001273 wd 0.0500 time 0.3833 (0.4584) data time 0.0010 (0.0506) model time 0.0000 (0.0000) loss 3.1261 (3.2510) grad_norm 1.1506 (1.5437) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][20/625] eta 0:04:19 lr 0.001273 wd 0.0500 time 0.3843 (0.4287) data time 0.0011 (0.0270) model time 0.0000 (0.0000) loss 3.6153 (3.3306) grad_norm 1.3063 (1.5551) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][30/625] eta 0:04:05 lr 0.001273 wd 0.0500 time 0.3819 (0.4131) data time 0.0010 (0.0187) model time 0.0000 (0.0000) loss 3.1001 (3.3227) grad_norm 1.5125 (1.5707) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][40/625] eta 0:03:57 lr 0.001273 wd 0.0500 time 0.3793 (0.4052) data time 0.0008 (0.0144) model time 0.0000 (0.0000) loss 2.2605 (3.1784) grad_norm 1.4144 (1.5555) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][50/625] eta 0:03:50 lr 0.001273 wd 0.0500 time 0.3710 (0.4003) data time 0.0011 (0.0118) model time 0.0000 (0.0000) loss 3.2036 (3.1416) grad_norm 1.7972 (1.7164) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][60/625] eta 0:03:44 lr 0.001272 wd 0.0500 time 0.3792 (0.3966) data time 0.0008 (0.0101) model time 0.3783 (0.3766) loss 1.8467 (3.1548) grad_norm 1.3804 (1.6518) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][70/625] eta 0:03:38 lr 0.001272 wd 0.0500 time 0.3809 (0.3945) data time 0.0011 (0.0088) model time 0.3799 (0.3784) loss 2.9579 (3.1925) grad_norm 1.1203 (1.6291) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][80/625] eta 0:03:34 lr 0.001272 wd 0.0500 time 0.3889 (0.3928) data time 0.0009 (0.0079) model time 0.3880 (0.3789) loss 3.9019 (3.2224) grad_norm 1.4190 (1.6172) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][90/625] eta 0:03:29 lr 0.001272 wd 0.0500 time 0.3751 (0.3913) data time 0.0011 (0.0071) model time 0.3740 (0.3786) loss 3.3754 (3.2651) grad_norm 1.4398 (1.6102) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][100/625] eta 0:03:24 lr 0.001272 wd 0.0500 time 0.3882 (0.3901) data time 0.0010 (0.0065) model time 0.3871 (0.3785) loss 3.4866 (3.2600) grad_norm 1.8123 (1.6115) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][110/625] eta 0:03:20 lr 0.001271 wd 0.0500 time 0.3866 (0.3894) data time 0.0010 (0.0060) model time 0.3856 (0.3790) loss 3.4776 (3.2436) grad_norm 2.0454 (1.6062) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][120/625] eta 0:03:16 lr 0.001271 wd 0.0500 time 0.3794 (0.3884) data time 0.0008 (0.0056) model time 0.3787 (0.3787) loss 3.5603 (3.2601) grad_norm 1.1743 (1.5915) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][130/625] eta 0:03:11 lr 0.001271 wd 0.0500 time 0.3783 (0.3876) data time 0.0011 (0.0053) model time 0.3772 (0.3784) loss 3.5399 (3.2554) grad_norm 4.1425 (1.5987) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][140/625] eta 0:03:07 lr 0.001271 wd 0.0500 time 0.3872 (0.3870) data time 0.0011 (0.0050) model time 0.3861 (0.3784) loss 2.9222 (3.2464) grad_norm 1.3514 (1.5942) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][150/625] eta 0:03:04 lr 0.001271 wd 0.0500 time 0.5864 (0.3879) data time 0.0010 (0.0047) model time 0.5854 (0.3805) loss 3.6875 (3.2513) grad_norm 1.4516 (1.5792) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][160/625] eta 0:03:01 lr 0.001271 wd 0.0500 time 0.3759 (0.3899) data time 0.0010 (0.0045) model time 0.3749 (0.3839) loss 2.6480 (3.2499) grad_norm 1.2272 (1.5812) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][170/625] eta 0:02:57 lr 0.001270 wd 0.0500 time 0.3768 (0.3891) data time 0.0011 (0.0043) model time 0.3757 (0.3831) loss 3.2761 (3.2582) grad_norm 2.1809 (1.5758) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][180/625] eta 0:02:52 lr 0.001270 wd 0.0500 time 0.3797 (0.3885) data time 0.0011 (0.0041) model time 0.3786 (0.3827) loss 3.2651 (3.2555) grad_norm 1.8652 (1.5781) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][190/625] eta 0:02:48 lr 0.001270 wd 0.0500 time 0.3756 (0.3880) data time 0.0011 (0.0040) model time 0.3745 (0.3824) loss 3.4241 (3.2720) grad_norm 1.8226 (1.5867) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][200/625] eta 0:02:45 lr 0.001270 wd 0.0500 time 0.3775 (0.3883) data time 0.0008 (0.0038) model time 0.3766 (0.3831) loss 3.6482 (3.2691) grad_norm 1.8379 (1.5782) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][210/625] eta 0:02:40 lr 0.001270 wd 0.0500 time 0.3844 (0.3879) data time 0.0011 (0.0037) model time 0.3833 (0.3829) loss 2.7115 (3.2674) grad_norm 1.3029 (1.5698) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][220/625] eta 0:02:36 lr 0.001270 wd 0.0500 time 0.3789 (0.3875) data time 0.0011 (0.0036) model time 0.3779 (0.3825) loss 3.3798 (3.2755) grad_norm 1.5352 (1.5682) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][230/625] eta 0:02:32 lr 0.001269 wd 0.0500 time 0.3773 (0.3871) data time 0.0008 (0.0035) model time 0.3765 (0.3822) loss 2.4027 (3.2587) grad_norm 1.5856 (1.5792) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:03:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][240/625] eta 0:02:28 lr 0.001269 wd 0.0500 time 0.3954 (0.3869) data time 0.0008 (0.0034) model time 0.3946 (0.3822) loss 2.8059 (3.2479) grad_norm 1.0794 (1.5648) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:03:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][250/625] eta 0:02:24 lr 0.001269 wd 0.0500 time 0.3762 (0.3865) data time 0.0011 (0.0033) model time 0.3750 (0.3819) loss 3.5233 (3.2483) grad_norm 1.1299 (1.5515) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:03:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][260/625] eta 0:02:20 lr 0.001269 wd 0.0500 time 0.3834 (0.3862) data time 0.0008 (0.0032) model time 0.3826 (0.3816) loss 3.0265 (3.2373) grad_norm 1.3642 (1.5591) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:03:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][270/625] eta 0:02:16 lr 0.001269 wd 0.0500 time 0.3805 (0.3859) data time 0.0010 (0.0031) model time 0.3794 (0.3815) loss 3.7208 (3.2515) grad_norm 1.2291 (1.5540) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:03:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][280/625] eta 0:02:13 lr 0.001269 wd 0.0500 time 0.3793 (0.3856) data time 0.0008 (0.0030) model time 0.3785 (0.3813) loss 2.8321 (3.2486) grad_norm 1.7075 (1.5741) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:03:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][290/625] eta 0:02:09 lr 0.001268 wd 0.0500 time 0.3776 (0.3854) data time 0.0011 (0.0030) model time 0.3765 (0.3811) loss 3.4374 (3.2510) grad_norm 1.5224 (1.5709) loss_scale 512.0000 (512.0000) mem 11225MB [2024-08-27 08:03:23 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-27 08:03:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 08:03:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 08:34:28 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-27 08:34:29 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-27 08:34:43 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-27 08:34:54 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-27 08:34:54 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-27 08:34:57 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-27 08:34:59 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-27 08:34:59 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 136) [2024-08-27 08:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-27 08:35:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][300/625] eta 0:14:43 lr 0.001268 wd 0.0500 time 0.3685 (2.7173) data time 0.0009 (0.1234) model time 0.3676 (2.5939) loss 4.1872 (3.8164) grad_norm 1.3716 (1.6374) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][310/625] eta 0:06:34 lr 0.001268 wd 0.0500 time 0.3733 (1.2531) data time 0.0010 (0.0470) model time 0.3724 (1.2061) loss 3.3169 (3.5568) grad_norm 1.4629 (1.6172) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][320/625] eta 0:04:42 lr 0.001268 wd 0.0500 time 0.3912 (0.9248) data time 0.0009 (0.0294) model time 0.3903 (0.8954) loss 3.4355 (3.5480) grad_norm 1.3524 (1.5325) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][330/625] eta 0:03:47 lr 0.001268 wd 0.0500 time 0.3698 (0.7719) data time 0.0013 (0.0215) model time 0.3685 (0.7504) loss 3.2643 (3.4872) grad_norm 1.1803 (1.5431) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][340/625] eta 0:03:17 lr 0.001268 wd 0.0500 time 0.3782 (0.6916) data time 0.0011 (0.0171) model time 0.3772 (0.6745) loss 3.1268 (3.4427) grad_norm 1.1765 (1.5209) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][350/625] eta 0:02:54 lr 0.001267 wd 0.0500 time 0.3758 (0.6348) data time 0.0010 (0.0143) model time 0.3748 (0.6206) loss 4.1874 (3.4467) grad_norm 1.1994 (1.5258) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][360/625] eta 0:02:37 lr 0.001267 wd 0.0500 time 0.3714 (0.5955) data time 0.0008 (0.0123) model time 0.3707 (0.5832) loss 2.8845 (3.4102) grad_norm 1.4432 (1.5102) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][370/625] eta 0:02:24 lr 0.001267 wd 0.0500 time 0.3781 (0.5667) data time 0.0011 (0.0108) model time 0.3770 (0.5559) loss 3.7099 (3.3896) grad_norm 2.5477 (1.5043) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][380/625] eta 0:02:13 lr 0.001267 wd 0.0500 time 0.3745 (0.5445) data time 0.0009 (0.0097) model time 0.3736 (0.5348) loss 2.5007 (3.3464) grad_norm 0.9209 (1.5179) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][390/625] eta 0:02:03 lr 0.001267 wd 0.0500 time 0.3760 (0.5268) data time 0.0008 (0.0088) model time 0.3752 (0.5180) loss 3.4881 (3.3579) grad_norm 1.0783 (1.5176) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:35:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][400/625] eta 0:01:55 lr 0.001267 wd 0.0500 time 0.3752 (0.5127) data time 0.0011 (0.0081) model time 0.3741 (0.5046) loss 3.6985 (3.3867) grad_norm 1.0940 (1.5194) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:36:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][410/625] eta 0:01:47 lr 0.001266 wd 0.0500 time 0.3797 (0.5009) data time 0.0009 (0.0075) model time 0.3787 (0.4934) loss 4.0943 (3.3811) grad_norm 1.4544 (1.5027) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:36:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][420/625] eta 0:01:40 lr 0.001266 wd 0.0500 time 0.3707 (0.4909) data time 0.0008 (0.0070) model time 0.3699 (0.4839) loss 2.0209 (3.3616) grad_norm 1.2611 (1.4922) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][430/625] eta 0:01:34 lr 0.001266 wd 0.0500 time 0.3780 (0.4823) data time 0.0013 (0.0066) model time 0.3767 (0.4758) loss 2.9227 (3.3565) grad_norm 1.6730 (1.4802) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:36:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][440/625] eta 0:01:27 lr 0.001266 wd 0.0500 time 0.3769 (0.4750) data time 0.0009 (0.0062) model time 0.3759 (0.4688) loss 2.6494 (3.3373) grad_norm 1.3690 (1.4815) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:36:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][450/625] eta 0:01:22 lr 0.001266 wd 0.0500 time 0.3726 (0.4686) data time 0.0011 (0.0059) model time 0.3715 (0.4628) loss 3.3555 (3.3332) grad_norm 1.1259 (1.4667) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:36:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][460/625] eta 0:01:16 lr 0.001265 wd 0.0500 time 0.3850 (0.4629) data time 0.0012 (0.0056) model time 0.3839 (0.4573) loss 3.3151 (3.3339) grad_norm 1.1939 (1.4571) loss_scale 512.0000 (512.0000) mem 11218MB [2024-08-27 08:36:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][470/625] eta 0:01:10 lr 0.001265 wd 0.0500 time 0.3727 (0.4580) data time 0.0009 (0.0053) model time 0.3718 (0.4527) loss 3.5019 (3.3232) grad_norm 1.8513 (1.4611) loss_scale 1024.0000 (529.4545) mem 11218MB [2024-08-27 08:36:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][480/625] eta 0:01:05 lr 0.001265 wd 0.0500 time 0.3668 (0.4535) data time 0.0011 (0.0051) model time 0.3657 (0.4484) loss 2.6209 (3.3124) grad_norm 1.2036 (1.4505) loss_scale 1024.0000 (556.0430) mem 11218MB [2024-08-27 08:36:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][490/625] eta 0:01:00 lr 0.001265 wd 0.0500 time 0.3674 (0.4494) data time 0.0009 (0.0049) model time 0.3665 (0.4445) loss 3.1252 (3.3071) grad_norm 1.3316 (1.4546) loss_scale 1024.0000 (579.9184) mem 11218MB [2024-08-27 08:36:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][500/625] eta 0:00:55 lr 0.001265 wd 0.0500 time 0.3782 (0.4457) data time 0.0008 (0.0047) model time 0.3774 (0.4410) loss 2.3878 (3.2985) grad_norm 1.4005 (1.4444) loss_scale 1024.0000 (601.4757) mem 11218MB [2024-08-27 08:36:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][510/625] eta 0:00:50 lr 0.001265 wd 0.0500 time 0.3731 (0.4424) data time 0.0008 (0.0046) model time 0.3723 (0.4378) loss 2.2969 (3.2936) grad_norm 1.4880 (1.4446) loss_scale 1024.0000 (621.0370) mem 11218MB [2024-08-27 08:36:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][520/625] eta 0:00:46 lr 0.001264 wd 0.0500 time 0.3813 (0.4413) data time 0.0008 (0.0044) model time 0.3805 (0.4369) loss 3.0177 (3.2930) grad_norm 1.7357 (1.4437) loss_scale 1024.0000 (638.8673) mem 11218MB [2024-08-27 08:36:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][530/625] eta 0:00:41 lr 0.001264 wd 0.0500 time 0.3837 (0.4393) data time 0.0008 (0.0043) model time 0.3829 (0.4350) loss 3.7547 (3.2892) grad_norm 1.3624 (1.4464) loss_scale 1024.0000 (655.1864) mem 11218MB [2024-08-27 08:36:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][540/625] eta 0:00:37 lr 0.001264 wd 0.0500 time 0.3694 (0.4366) data time 0.0008 (0.0041) model time 0.3686 (0.4324) loss 2.2471 (3.2858) grad_norm 1.4554 (1.4415) loss_scale 1024.0000 (670.1789) mem 11218MB [2024-08-27 08:36:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][550/625] eta 0:00:32 lr 0.001264 wd 0.0500 time 0.3763 (0.4342) data time 0.0010 (0.0040) model time 0.3754 (0.4301) loss 2.7837 (3.2815) grad_norm 1.7322 (1.4393) loss_scale 1024.0000 (684.0000) mem 11218MB [2024-08-27 08:36:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][560/625] eta 0:00:28 lr 0.001264 wd 0.0500 time 0.3772 (0.4319) data time 0.0009 (0.0039) model time 0.3763 (0.4280) loss 3.0853 (3.2759) grad_norm 2.5053 (1.4500) loss_scale 1024.0000 (696.7820) mem 11218MB [2024-08-27 08:37:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][570/625] eta 0:00:23 lr 0.001264 wd 0.0500 time 0.3675 (0.4298) data time 0.0011 (0.0038) model time 0.3664 (0.4260) loss 3.5681 (3.2802) grad_norm 1.3099 (1.4568) loss_scale 1024.0000 (708.6377) mem 11218MB [2024-08-27 08:37:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][580/625] eta 0:00:19 lr 0.001263 wd 0.0500 time 0.3672 (0.4277) data time 0.0012 (0.0037) model time 0.3660 (0.4240) loss 3.0933 (3.2755) grad_norm 1.4075 (1.4606) loss_scale 1024.0000 (719.6643) mem 11218MB [2024-08-27 08:37:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][590/625] eta 0:00:14 lr 0.001263 wd 0.0500 time 0.3747 (0.4259) data time 0.0011 (0.0036) model time 0.3736 (0.4223) loss 2.4857 (3.2722) grad_norm 2.1292 (1.4697) loss_scale 1024.0000 (729.9459) mem 11218MB [2024-08-27 08:37:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][600/625] eta 0:00:10 lr 0.001263 wd 0.0500 time 0.3666 (0.4242) data time 0.0008 (0.0036) model time 0.3658 (0.4206) loss 2.3921 (3.2632) grad_norm 1.4062 (1.4695) loss_scale 1024.0000 (739.5556) mem 11218MB [2024-08-27 08:37:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][610/625] eta 0:00:06 lr 0.001263 wd 0.0500 time 0.3630 (0.4225) data time 0.0006 (0.0035) model time 0.3624 (0.4190) loss 4.1778 (3.2748) grad_norm 1.8796 (1.4659) loss_scale 1024.0000 (748.5570) mem 11218MB [2024-08-27 08:37:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [136/300][620/625] eta 0:00:02 lr 0.001263 wd 0.0500 time 0.3634 (0.4206) data time 0.0008 (0.0034) model time 0.3626 (0.4172) loss 3.6133 (3.2835) grad_norm 1.2046 (1.4604) loss_scale 1024.0000 (757.0061) mem 11218MB [2024-08-27 08:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 136 training takes 0:02:18 [2024-08-27 08:37:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 08:37:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 08:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.493 (0.493) Loss 0.6162 (0.6162) Acc@1 86.816 (86.816) Acc@5 97.705 (97.705) Mem 11218MB [2024-08-27 08:37:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.099 (0.142) Loss 0.9707 (0.7476) Acc@1 75.781 (83.141) Acc@5 94.727 (96.711) Mem 11218MB [2024-08-27 08:37:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.0898 (0.8930) Acc@1 73.584 (79.611) Acc@5 93.164 (95.003) Mem 11218MB [2024-08-27 08:37:33 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.371 Acc@5 95.020 [2024-08-27 08:37:33 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.4% [2024-08-27 08:37:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.795 (0.795) Loss 0.5244 (0.5244) Acc@1 88.574 (88.574) Acc@5 98.389 (98.389) Mem 11218MB [2024-08-27 08:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.174) Loss 0.8726 (0.6540) Acc@1 79.150 (85.174) Acc@5 95.557 (97.385) Mem 11218MB [2024-08-27 08:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.136) Loss 0.9995 (0.7848) Acc@1 75.635 (81.978) Acc@5 94.336 (96.024) Mem 11218MB [2024-08-27 08:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.700 Acc@5 96.023 [2024-08-27 08:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.7% [2024-08-27 08:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.70% [2024-08-27 08:37:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 08:37:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 08:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][0/625] eta 0:08:36 lr 0.001263 wd 0.0500 time 0.8256 (0.8256) data time 0.4074 (0.4074) model time 0.0000 (0.0000) loss 2.3648 (2.3648) grad_norm 1.0540 (1.0540) loss_scale 1024.0000 (1024.0000) mem 11226MB [2024-08-27 08:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][10/625] eta 0:04:17 lr 0.001262 wd 0.0500 time 0.3775 (0.4183) data time 0.0010 (0.0383) model time 0.0000 (0.0000) loss 3.3557 (3.2228) grad_norm 1.1662 (1.2573) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:37:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][20/625] eta 0:04:18 lr 0.001262 wd 0.0500 time 0.5583 (0.4279) data time 0.0008 (0.0206) model time 0.0000 (0.0000) loss 3.8389 (3.2780) grad_norm 1.1563 (1.2412) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][30/625] eta 0:04:04 lr 0.001262 wd 0.0500 time 0.3740 (0.4112) data time 0.0007 (0.0143) model time 0.0000 (0.0000) loss 3.0815 (3.2701) grad_norm 1.2974 (1.2916) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:37:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][40/625] eta 0:03:57 lr 0.001262 wd 0.0500 time 0.3749 (0.4054) data time 0.0010 (0.0111) model time 0.0000 (0.0000) loss 2.3455 (3.2260) grad_norm 1.6016 (1.3205) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][50/625] eta 0:03:49 lr 0.001262 wd 0.0500 time 0.3769 (0.3997) data time 0.0008 (0.0091) model time 0.0000 (0.0000) loss 2.3675 (3.1970) grad_norm 1.1807 (1.3347) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][60/625] eta 0:03:43 lr 0.001262 wd 0.0500 time 0.3783 (0.3959) data time 0.0008 (0.0078) model time 0.3776 (0.3756) loss 3.7526 (3.1909) grad_norm 1.1795 (1.3553) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][70/625] eta 0:03:38 lr 0.001261 wd 0.0500 time 0.3753 (0.3931) data time 0.0011 (0.0069) model time 0.3742 (0.3752) loss 3.4031 (3.2353) grad_norm 1.2367 (1.3804) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][80/625] eta 0:03:33 lr 0.001261 wd 0.0500 time 0.3742 (0.3910) data time 0.0010 (0.0062) model time 0.3732 (0.3751) loss 3.9436 (3.2586) grad_norm 1.5034 (1.3899) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][90/625] eta 0:03:28 lr 0.001261 wd 0.0500 time 0.3688 (0.3893) data time 0.0008 (0.0056) model time 0.3680 (0.3749) loss 3.8609 (3.2520) grad_norm 1.3465 (1.3856) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][100/625] eta 0:03:23 lr 0.001261 wd 0.0500 time 0.3765 (0.3879) data time 0.0008 (0.0052) model time 0.3757 (0.3748) loss 3.5353 (3.2690) grad_norm 1.8237 (1.3911) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][110/625] eta 0:03:19 lr 0.001261 wd 0.0500 time 0.3761 (0.3867) data time 0.0010 (0.0048) model time 0.3750 (0.3745) loss 4.0401 (3.2724) grad_norm 1.3222 (1.4035) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][120/625] eta 0:03:14 lr 0.001261 wd 0.0500 time 0.3769 (0.3859) data time 0.0009 (0.0045) model time 0.3761 (0.3748) loss 3.4348 (3.2691) grad_norm 1.0481 (1.3979) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][130/625] eta 0:03:10 lr 0.001260 wd 0.0500 time 0.3838 (0.3852) data time 0.0011 (0.0042) model time 0.3827 (0.3749) loss 3.7100 (3.2506) grad_norm 1.5299 (1.3990) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][140/625] eta 0:03:08 lr 0.001260 wd 0.0500 time 0.5881 (0.3890) data time 0.0012 (0.0040) model time 0.5869 (0.3819) loss 3.3726 (3.2272) grad_norm 1.1386 (1.3998) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][150/625] eta 0:03:04 lr 0.001260 wd 0.0500 time 0.3698 (0.3880) data time 0.0008 (0.0038) model time 0.3689 (0.3810) loss 3.6532 (3.2252) grad_norm 1.2606 (1.4033) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][160/625] eta 0:03:00 lr 0.001260 wd 0.0500 time 0.3762 (0.3883) data time 0.0010 (0.0037) model time 0.3752 (0.3819) loss 3.5928 (3.2421) grad_norm 1.5369 (1.4808) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][170/625] eta 0:02:56 lr 0.001260 wd 0.0500 time 0.3795 (0.3876) data time 0.0010 (0.0035) model time 0.3784 (0.3813) loss 3.3151 (3.2443) grad_norm 1.0690 (1.4759) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][180/625] eta 0:02:52 lr 0.001260 wd 0.0500 time 0.3680 (0.3869) data time 0.0009 (0.0034) model time 0.3671 (0.3807) loss 3.6913 (3.2546) grad_norm 1.4105 (1.4736) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][190/625] eta 0:02:48 lr 0.001259 wd 0.0500 time 0.3693 (0.3863) data time 0.0009 (0.0033) model time 0.3684 (0.3803) loss 2.8167 (3.2593) grad_norm 1.9407 (1.5138) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][200/625] eta 0:02:43 lr 0.001259 wd 0.0500 time 0.3775 (0.3858) data time 0.0010 (0.0032) model time 0.3765 (0.3800) loss 3.1095 (3.2443) grad_norm 1.4987 (1.5304) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][210/625] eta 0:02:39 lr 0.001259 wd 0.0500 time 0.3721 (0.3853) data time 0.0011 (0.0031) model time 0.3710 (0.3796) loss 3.3365 (3.2453) grad_norm 1.6405 (1.5241) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][220/625] eta 0:02:35 lr 0.001259 wd 0.0500 time 0.3697 (0.3849) data time 0.0008 (0.0030) model time 0.3689 (0.3793) loss 3.3590 (3.2480) grad_norm 1.0035 (1.5159) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][230/625] eta 0:02:31 lr 0.001259 wd 0.0500 time 0.3747 (0.3846) data time 0.0013 (0.0029) model time 0.3734 (0.3792) loss 3.5511 (3.2612) grad_norm 1.2204 (1.5064) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][240/625] eta 0:02:27 lr 0.001259 wd 0.0500 time 0.3752 (0.3843) data time 0.0010 (0.0028) model time 0.3742 (0.3790) loss 3.8883 (3.2709) grad_norm 0.9340 (1.5033) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][250/625] eta 0:02:24 lr 0.001258 wd 0.0500 time 0.3776 (0.3840) data time 0.0009 (0.0028) model time 0.3768 (0.3789) loss 3.8054 (3.2768) grad_norm 1.4481 (1.4990) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][260/625] eta 0:02:20 lr 0.001258 wd 0.0500 time 0.3775 (0.3837) data time 0.0011 (0.0027) model time 0.3764 (0.3787) loss 3.7939 (3.2815) grad_norm 1.1856 (1.4907) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][270/625] eta 0:02:16 lr 0.001258 wd 0.0500 time 0.3742 (0.3835) data time 0.0010 (0.0026) model time 0.3731 (0.3786) loss 2.2960 (3.2742) grad_norm 2.1281 (1.4895) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][280/625] eta 0:02:12 lr 0.001258 wd 0.0500 time 0.3776 (0.3833) data time 0.0010 (0.0026) model time 0.3766 (0.3786) loss 3.4832 (3.2760) grad_norm 1.8420 (1.4928) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][290/625] eta 0:02:08 lr 0.001258 wd 0.0500 time 0.3746 (0.3831) data time 0.0008 (0.0025) model time 0.3738 (0.3785) loss 3.8846 (3.2827) grad_norm 1.2635 (1.5030) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][300/625] eta 0:02:04 lr 0.001257 wd 0.0500 time 0.3746 (0.3829) data time 0.0010 (0.0025) model time 0.3735 (0.3784) loss 3.4944 (3.2848) grad_norm 1.4594 (1.4993) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][310/625] eta 0:02:00 lr 0.001257 wd 0.0500 time 0.3763 (0.3828) data time 0.0009 (0.0024) model time 0.3755 (0.3783) loss 2.0573 (3.2782) grad_norm 1.6231 (1.4978) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][320/625] eta 0:01:56 lr 0.001257 wd 0.0500 time 0.3820 (0.3826) data time 0.0008 (0.0024) model time 0.3812 (0.3782) loss 4.5054 (3.2843) grad_norm 1.7641 (1.4957) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][330/625] eta 0:01:52 lr 0.001257 wd 0.0500 time 0.3769 (0.3824) data time 0.0009 (0.0024) model time 0.3760 (0.3781) loss 1.9955 (3.2730) grad_norm 2.6243 (1.5034) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][340/625] eta 0:01:48 lr 0.001257 wd 0.0500 time 0.3783 (0.3823) data time 0.0008 (0.0023) model time 0.3775 (0.3781) loss 3.7722 (3.2793) grad_norm 1.9515 (1.5018) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:39:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][350/625] eta 0:01:45 lr 0.001257 wd 0.0500 time 0.3808 (0.3841) data time 0.0008 (0.0023) model time 0.3800 (0.3803) loss 3.8363 (3.2801) grad_norm 1.3445 (1.4961) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][360/625] eta 0:01:41 lr 0.001256 wd 0.0500 time 0.3788 (0.3839) data time 0.0011 (0.0023) model time 0.3777 (0.3802) loss 2.3633 (3.2723) grad_norm 1.2788 (1.4914) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][370/625] eta 0:01:38 lr 0.001256 wd 0.0500 time 0.3819 (0.3844) data time 0.0011 (0.0022) model time 0.3808 (0.3808) loss 3.5501 (3.2674) grad_norm 1.7950 (1.4894) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][380/625] eta 0:01:34 lr 0.001256 wd 0.0500 time 0.3770 (0.3842) data time 0.0008 (0.0022) model time 0.3763 (0.3807) loss 2.1308 (3.2651) grad_norm 1.2251 (1.4952) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][390/625] eta 0:01:30 lr 0.001256 wd 0.0500 time 0.3907 (0.3842) data time 0.0008 (0.0022) model time 0.3900 (0.3807) loss 3.9266 (3.2631) grad_norm 1.6499 (1.4929) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][400/625] eta 0:01:26 lr 0.001256 wd 0.0500 time 0.3781 (0.3839) data time 0.0010 (0.0021) model time 0.3770 (0.3805) loss 3.6968 (3.2668) grad_norm 1.2822 (1.4909) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][410/625] eta 0:01:22 lr 0.001256 wd 0.0500 time 0.3847 (0.3838) data time 0.0011 (0.0021) model time 0.3836 (0.3804) loss 3.6909 (3.2760) grad_norm 1.4022 (1.4867) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][420/625] eta 0:01:18 lr 0.001255 wd 0.0500 time 0.3786 (0.3837) data time 0.0010 (0.0021) model time 0.3776 (0.3803) loss 2.9444 (3.2715) grad_norm 1.8268 (1.4898) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][430/625] eta 0:01:14 lr 0.001255 wd 0.0500 time 0.3845 (0.3835) data time 0.0010 (0.0021) model time 0.3835 (0.3802) loss 3.5327 (3.2736) grad_norm 1.9066 (1.4895) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][440/625] eta 0:01:10 lr 0.001255 wd 0.0500 time 0.3684 (0.3834) data time 0.0008 (0.0021) model time 0.3676 (0.3801) loss 4.3249 (3.2776) grad_norm 1.2284 (1.4890) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][450/625] eta 0:01:07 lr 0.001255 wd 0.0500 time 0.3773 (0.3833) data time 0.0008 (0.0020) model time 0.3764 (0.3801) loss 3.9521 (3.2806) grad_norm 1.3552 (1.4877) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][460/625] eta 0:01:03 lr 0.001255 wd 0.0500 time 0.3747 (0.3831) data time 0.0011 (0.0020) model time 0.3736 (0.3799) loss 2.8195 (3.2776) grad_norm 1.1576 (1.4810) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][470/625] eta 0:00:59 lr 0.001255 wd 0.0500 time 0.3716 (0.3830) data time 0.0009 (0.0020) model time 0.3707 (0.3798) loss 3.7317 (3.2784) grad_norm 1.5701 (1.4835) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][480/625] eta 0:00:55 lr 0.001254 wd 0.0500 time 0.3743 (0.3829) data time 0.0008 (0.0020) model time 0.3735 (0.3798) loss 2.4660 (3.2696) grad_norm 2.3275 (1.4843) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][490/625] eta 0:00:51 lr 0.001254 wd 0.0500 time 0.3800 (0.3827) data time 0.0008 (0.0020) model time 0.3792 (0.3796) loss 3.3236 (3.2721) grad_norm 1.4215 (1.4830) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][500/625] eta 0:00:47 lr 0.001254 wd 0.0500 time 0.3729 (0.3826) data time 0.0010 (0.0019) model time 0.3719 (0.3795) loss 2.3164 (3.2666) grad_norm 1.6931 (1.4792) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:40:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][510/625] eta 0:00:43 lr 0.001254 wd 0.0500 time 0.3749 (0.3824) data time 0.0009 (0.0019) model time 0.3741 (0.3794) loss 3.7545 (3.2647) grad_norm 1.1525 (1.4789) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][520/625] eta 0:00:40 lr 0.001254 wd 0.0500 time 0.3799 (0.3823) data time 0.0009 (0.0019) model time 0.3790 (0.3793) loss 4.0042 (3.2616) grad_norm 1.6164 (1.4844) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][530/625] eta 0:00:36 lr 0.001254 wd 0.0500 time 0.3713 (0.3825) data time 0.0010 (0.0019) model time 0.3704 (0.3795) loss 4.0448 (3.2657) grad_norm 1.3853 (1.4832) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][540/625] eta 0:00:32 lr 0.001253 wd 0.0500 time 0.3697 (0.3835) data time 0.0008 (0.0019) model time 0.3689 (0.3807) loss 3.2552 (3.2668) grad_norm 1.4444 (1.4847) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][550/625] eta 0:00:28 lr 0.001253 wd 0.0500 time 0.3745 (0.3837) data time 0.0008 (0.0019) model time 0.3736 (0.3809) loss 3.6751 (3.2633) grad_norm 1.3173 (1.4795) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][560/625] eta 0:00:24 lr 0.001253 wd 0.0500 time 0.3767 (0.3836) data time 0.0010 (0.0019) model time 0.3757 (0.3808) loss 2.3049 (3.2611) grad_norm 1.3777 (1.4750) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][570/625] eta 0:00:21 lr 0.001253 wd 0.0500 time 0.3743 (0.3834) data time 0.0011 (0.0018) model time 0.3732 (0.3807) loss 3.8192 (3.2629) grad_norm 1.0947 (1.4758) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][580/625] eta 0:00:17 lr 0.001253 wd 0.0500 time 0.3835 (0.3832) data time 0.0010 (0.0018) model time 0.3825 (0.3805) loss 2.7406 (3.2634) grad_norm 1.3116 (1.4718) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][590/625] eta 0:00:13 lr 0.001253 wd 0.0500 time 0.3710 (0.3831) data time 0.0008 (0.0018) model time 0.3702 (0.3804) loss 3.7577 (3.2649) grad_norm 3.3542 (1.4837) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][600/625] eta 0:00:09 lr 0.001252 wd 0.0500 time 0.3759 (0.3830) data time 0.0010 (0.0018) model time 0.3749 (0.3803) loss 3.5798 (3.2710) grad_norm 1.2580 (1.4905) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][610/625] eta 0:00:05 lr 0.001252 wd 0.0500 time 0.3735 (0.3828) data time 0.0005 (0.0018) model time 0.3730 (0.3801) loss 3.5317 (3.2691) grad_norm 1.5361 (1.4905) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [137/300][620/625] eta 0:00:01 lr 0.001252 wd 0.0500 time 0.3560 (0.3825) data time 0.0008 (0.0018) model time 0.3552 (0.3799) loss 2.4561 (3.2608) grad_norm 1.0979 (1.4892) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 137 training takes 0:03:59 [2024-08-27 08:41:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 08:41:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 08:41:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 0.5938 (0.5938) Acc@1 87.158 (87.158) Acc@5 98.047 (98.047) Mem 11224MB [2024-08-27 08:41:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 1.0264 (0.7340) Acc@1 75.391 (83.181) Acc@5 93.994 (96.808) Mem 11224MB [2024-08-27 08:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.1006 (0.8896) Acc@1 72.852 (79.467) Acc@5 92.822 (94.950) Mem 11224MB [2024-08-27 08:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.197 Acc@5 94.952 [2024-08-27 08:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.2% [2024-08-27 08:41:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.851 (0.851) Loss 0.5234 (0.5234) Acc@1 88.525 (88.525) Acc@5 98.486 (98.486) Mem 11224MB [2024-08-27 08:41:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.182) Loss 0.8691 (0.6527) Acc@1 79.053 (85.178) Acc@5 95.654 (97.412) Mem 11224MB [2024-08-27 08:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.140) Loss 0.9985 (0.7836) Acc@1 75.684 (81.989) Acc@5 94.336 (96.026) Mem 11224MB [2024-08-27 08:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.712 Acc@5 96.031 [2024-08-27 08:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.7% [2024-08-27 08:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.71% [2024-08-27 08:41:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 08:41:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 08:41:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][0/625] eta 0:07:29 lr 0.001252 wd 0.0500 time 0.7196 (0.7196) data time 0.3791 (0.3791) model time 0.0000 (0.0000) loss 3.1676 (3.1676) grad_norm 1.0701 (1.0701) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][10/625] eta 0:04:21 lr 0.001252 wd 0.0500 time 0.3824 (0.4245) data time 0.0010 (0.0355) model time 0.0000 (0.0000) loss 3.4216 (3.2449) grad_norm 1.2812 (1.4114) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:41:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][20/625] eta 0:04:03 lr 0.001252 wd 0.0500 time 0.3722 (0.4022) data time 0.0011 (0.0192) model time 0.0000 (0.0000) loss 3.5990 (3.3646) grad_norm 1.0203 (1.4674) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][30/625] eta 0:03:54 lr 0.001251 wd 0.0500 time 0.3701 (0.3942) data time 0.0011 (0.0134) model time 0.0000 (0.0000) loss 2.5216 (3.2723) grad_norm 1.4076 (1.4899) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][40/625] eta 0:03:56 lr 0.001251 wd 0.0500 time 0.5935 (0.4049) data time 0.0011 (0.0104) model time 0.0000 (0.0000) loss 3.4444 (3.2680) grad_norm 1.2219 (1.5090) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][50/625] eta 0:03:51 lr 0.001251 wd 0.0500 time 0.3772 (0.4024) data time 0.0010 (0.0086) model time 0.0000 (0.0000) loss 3.3233 (3.2457) grad_norm 1.0904 (1.5271) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][60/625] eta 0:03:44 lr 0.001251 wd 0.0500 time 0.3695 (0.3979) data time 0.0008 (0.0074) model time 0.3687 (0.3734) loss 2.6684 (3.2805) grad_norm 1.1774 (1.5145) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][70/625] eta 0:03:39 lr 0.001251 wd 0.0500 time 0.3707 (0.3950) data time 0.0011 (0.0065) model time 0.3696 (0.3748) loss 3.4011 (3.2412) grad_norm 1.8423 (1.5207) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][80/625] eta 0:03:34 lr 0.001251 wd 0.0500 time 0.3759 (0.3928) data time 0.0008 (0.0059) model time 0.3751 (0.3752) loss 3.8726 (3.2472) grad_norm 1.8048 (1.5966) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][90/625] eta 0:03:29 lr 0.001250 wd 0.0500 time 0.3680 (0.3909) data time 0.0010 (0.0053) model time 0.3670 (0.3750) loss 3.5725 (3.2557) grad_norm 1.3444 (1.5895) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][100/625] eta 0:03:24 lr 0.001250 wd 0.0500 time 0.3818 (0.3897) data time 0.0010 (0.0049) model time 0.3808 (0.3755) loss 3.3457 (3.2629) grad_norm 1.7903 (1.5674) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][110/625] eta 0:03:20 lr 0.001250 wd 0.0500 time 0.3731 (0.3887) data time 0.0008 (0.0046) model time 0.3723 (0.3759) loss 2.8002 (3.2775) grad_norm 1.4032 (1.5475) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][120/625] eta 0:03:15 lr 0.001250 wd 0.0500 time 0.3750 (0.3878) data time 0.0011 (0.0043) model time 0.3739 (0.3759) loss 2.6720 (3.2486) grad_norm 1.7344 (1.5299) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][130/625] eta 0:03:11 lr 0.001250 wd 0.0500 time 0.3897 (0.3870) data time 0.0008 (0.0041) model time 0.3889 (0.3760) loss 3.5867 (3.2498) grad_norm 1.8473 (1.6056) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][140/625] eta 0:03:07 lr 0.001249 wd 0.0500 time 0.3691 (0.3863) data time 0.0009 (0.0039) model time 0.3682 (0.3760) loss 4.2562 (3.2455) grad_norm 1.0857 (1.5947) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][150/625] eta 0:03:03 lr 0.001249 wd 0.0500 time 0.3755 (0.3857) data time 0.0010 (0.0037) model time 0.3745 (0.3760) loss 3.4849 (3.2506) grad_norm 1.0731 (1.5841) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][160/625] eta 0:03:00 lr 0.001249 wd 0.0500 time 0.3685 (0.3890) data time 0.0011 (0.0035) model time 0.3675 (0.3817) loss 4.1014 (3.2667) grad_norm 1.5327 (1.5727) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][170/625] eta 0:02:57 lr 0.001249 wd 0.0500 time 0.3753 (0.3894) data time 0.0011 (0.0034) model time 0.3742 (0.3828) loss 3.2870 (3.2668) grad_norm 0.9946 (1.5694) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:42:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][180/625] eta 0:02:52 lr 0.001249 wd 0.0500 time 0.3790 (0.3888) data time 0.0010 (0.0033) model time 0.3779 (0.3822) loss 3.3621 (3.2647) grad_norm 0.9168 (1.5582) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][190/625] eta 0:02:48 lr 0.001249 wd 0.0500 time 0.3705 (0.3880) data time 0.0008 (0.0031) model time 0.3697 (0.3816) loss 2.5914 (3.2608) grad_norm 1.3880 (1.5534) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][200/625] eta 0:02:44 lr 0.001248 wd 0.0500 time 0.3731 (0.3874) data time 0.0008 (0.0030) model time 0.3723 (0.3812) loss 4.0816 (3.2650) grad_norm 1.5302 (1.5669) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][210/625] eta 0:02:40 lr 0.001248 wd 0.0500 time 0.4127 (0.3873) data time 0.0008 (0.0030) model time 0.4119 (0.3813) loss 3.6703 (3.2539) grad_norm 1.5488 (1.5761) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][220/625] eta 0:02:37 lr 0.001248 wd 0.0500 time 0.3814 (0.3878) data time 0.0010 (0.0029) model time 0.3803 (0.3823) loss 3.4794 (3.2652) grad_norm 1.5792 (1.5712) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][230/625] eta 0:02:32 lr 0.001248 wd 0.0500 time 0.3848 (0.3873) data time 0.0011 (0.0028) model time 0.3837 (0.3819) loss 3.3582 (3.2799) grad_norm 1.4463 (1.5578) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][240/625] eta 0:02:28 lr 0.001248 wd 0.0500 time 0.3844 (0.3869) data time 0.0010 (0.0027) model time 0.3834 (0.3815) loss 3.4095 (3.2762) grad_norm 2.5006 (1.5616) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][250/625] eta 0:02:24 lr 0.001248 wd 0.0500 time 0.4010 (0.3866) data time 0.0009 (0.0027) model time 0.4001 (0.3814) loss 3.4263 (3.2857) grad_norm 1.7107 (1.5608) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][260/625] eta 0:02:21 lr 0.001247 wd 0.0500 time 0.3721 (0.3863) data time 0.0008 (0.0026) model time 0.3713 (0.3813) loss 2.6664 (3.2911) grad_norm 1.2120 (1.5587) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][270/625] eta 0:02:17 lr 0.001247 wd 0.0500 time 0.3740 (0.3861) data time 0.0008 (0.0026) model time 0.3732 (0.3812) loss 2.7084 (3.2860) grad_norm 1.2667 (1.5456) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][280/625] eta 0:02:13 lr 0.001247 wd 0.0500 time 0.3764 (0.3858) data time 0.0008 (0.0025) model time 0.3756 (0.3810) loss 1.9939 (3.2793) grad_norm 1.4010 (1.5439) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][290/625] eta 0:02:09 lr 0.001247 wd 0.0500 time 0.3780 (0.3857) data time 0.0011 (0.0025) model time 0.3769 (0.3810) loss 3.5545 (3.2840) grad_norm 1.5006 (1.5507) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][300/625] eta 0:02:05 lr 0.001247 wd 0.0500 time 0.3789 (0.3855) data time 0.0008 (0.0024) model time 0.3781 (0.3809) loss 4.3571 (3.2755) grad_norm 1.2814 (1.5527) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][310/625] eta 0:02:01 lr 0.001247 wd 0.0500 time 0.3762 (0.3852) data time 0.0008 (0.0024) model time 0.3753 (0.3807) loss 4.0592 (3.2826) grad_norm 1.3468 (1.5446) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][320/625] eta 0:01:57 lr 0.001246 wd 0.0500 time 0.3802 (0.3850) data time 0.0011 (0.0023) model time 0.3792 (0.3806) loss 3.5065 (3.2845) grad_norm 1.9519 (1.5442) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:43:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][330/625] eta 0:01:53 lr 0.001246 wd 0.0500 time 0.3850 (0.3848) data time 0.0008 (0.0023) model time 0.3842 (0.3805) loss 3.6545 (3.2933) grad_norm 1.3985 (1.5447) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][340/625] eta 0:01:49 lr 0.001246 wd 0.0500 time 0.3792 (0.3846) data time 0.0007 (0.0023) model time 0.3785 (0.3803) loss 3.7575 (3.2981) grad_norm 1.2780 (1.5399) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][350/625] eta 0:01:45 lr 0.001246 wd 0.0500 time 0.3731 (0.3845) data time 0.0008 (0.0022) model time 0.3723 (0.3803) loss 3.1879 (3.2948) grad_norm 1.3627 (1.5332) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][360/625] eta 0:01:41 lr 0.001246 wd 0.0500 time 0.3774 (0.3843) data time 0.0010 (0.0022) model time 0.3764 (0.3802) loss 3.2225 (3.2854) grad_norm 1.7619 (1.5362) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][370/625] eta 0:01:38 lr 0.001246 wd 0.0500 time 0.5805 (0.3859) data time 0.0010 (0.0022) model time 0.5795 (0.3821) loss 3.3672 (3.2828) grad_norm 1.9838 (1.5339) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][380/625] eta 0:01:34 lr 0.001245 wd 0.0500 time 0.3827 (0.3861) data time 0.0010 (0.0022) model time 0.3817 (0.3825) loss 2.6224 (3.2847) grad_norm 1.2333 (1.5327) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][390/625] eta 0:01:30 lr 0.001245 wd 0.0500 time 0.3757 (0.3859) data time 0.0010 (0.0021) model time 0.3747 (0.3823) loss 3.4580 (3.2878) grad_norm 1.3782 (1.5324) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][400/625] eta 0:01:26 lr 0.001245 wd 0.0500 time 0.3848 (0.3861) data time 0.0011 (0.0021) model time 0.3837 (0.3826) loss 3.6117 (3.2944) grad_norm 2.5546 (1.5399) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][410/625] eta 0:01:22 lr 0.001245 wd 0.0500 time 0.3718 (0.3859) data time 0.0012 (0.0021) model time 0.3707 (0.3824) loss 3.3453 (3.3007) grad_norm 2.0154 (1.5626) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][420/625] eta 0:01:19 lr 0.001245 wd 0.0500 time 0.3848 (0.3858) data time 0.0011 (0.0021) model time 0.3837 (0.3823) loss 3.0949 (3.3036) grad_norm 1.6154 (1.5682) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][430/625] eta 0:01:15 lr 0.001244 wd 0.0500 time 0.3814 (0.3856) data time 0.0008 (0.0020) model time 0.3806 (0.3822) loss 3.0398 (3.3030) grad_norm 1.4949 (1.5642) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][440/625] eta 0:01:11 lr 0.001244 wd 0.0500 time 0.3734 (0.3854) data time 0.0011 (0.0020) model time 0.3723 (0.3820) loss 2.7567 (3.3026) grad_norm 1.9118 (1.5599) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][450/625] eta 0:01:07 lr 0.001244 wd 0.0500 time 0.3745 (0.3852) data time 0.0011 (0.0020) model time 0.3734 (0.3819) loss 3.3948 (3.3077) grad_norm 1.7309 (1.5618) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][460/625] eta 0:01:03 lr 0.001244 wd 0.0500 time 0.3778 (0.3851) data time 0.0008 (0.0020) model time 0.3770 (0.3817) loss 2.5795 (3.3083) grad_norm 1.2343 (1.5602) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][470/625] eta 0:00:59 lr 0.001244 wd 0.0500 time 0.3947 (0.3850) data time 0.0008 (0.0020) model time 0.3939 (0.3817) loss 3.1752 (3.3072) grad_norm 1.1328 (1.5557) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][480/625] eta 0:00:55 lr 0.001244 wd 0.0500 time 0.3736 (0.3848) data time 0.0010 (0.0019) model time 0.3726 (0.3815) loss 3.1875 (3.3052) grad_norm 1.4857 (1.5509) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:44:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][490/625] eta 0:00:51 lr 0.001243 wd 0.0500 time 0.3748 (0.3846) data time 0.0010 (0.0019) model time 0.3738 (0.3814) loss 3.5070 (3.3048) grad_norm 1.2525 (1.5497) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][500/625] eta 0:00:48 lr 0.001243 wd 0.0500 time 0.3806 (0.3845) data time 0.0008 (0.0019) model time 0.3798 (0.3813) loss 4.1822 (3.3057) grad_norm 1.6883 (1.5617) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][510/625] eta 0:00:44 lr 0.001243 wd 0.0500 time 0.3795 (0.3845) data time 0.0010 (0.0019) model time 0.3785 (0.3813) loss 1.9625 (3.3066) grad_norm 1.2108 (1.5573) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][520/625] eta 0:00:40 lr 0.001243 wd 0.0500 time 0.3765 (0.3847) data time 0.0010 (0.0019) model time 0.3755 (0.3816) loss 2.9155 (3.3029) grad_norm 1.2934 (1.5509) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][530/625] eta 0:00:36 lr 0.001243 wd 0.0500 time 0.3773 (0.3846) data time 0.0012 (0.0019) model time 0.3762 (0.3815) loss 3.4111 (3.2957) grad_norm 1.1259 (1.5446) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][540/625] eta 0:00:32 lr 0.001243 wd 0.0500 time 0.3800 (0.3845) data time 0.0010 (0.0019) model time 0.3790 (0.3814) loss 3.3868 (3.2872) grad_norm 1.3255 (1.5428) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][550/625] eta 0:00:28 lr 0.001242 wd 0.0500 time 0.5688 (0.3855) data time 0.0011 (0.0018) model time 0.5677 (0.3826) loss 3.7212 (3.2904) grad_norm 1.2929 (1.5385) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][560/625] eta 0:00:25 lr 0.001242 wd 0.0500 time 0.3738 (0.3864) data time 0.0010 (0.0018) model time 0.3727 (0.3836) loss 3.2742 (3.2913) grad_norm 1.2712 (1.5336) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][570/625] eta 0:00:21 lr 0.001242 wd 0.0500 time 0.3792 (0.3862) data time 0.0008 (0.0018) model time 0.3784 (0.3835) loss 3.7022 (3.2944) grad_norm 1.7386 (1.5353) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][580/625] eta 0:00:17 lr 0.001242 wd 0.0500 time 0.3723 (0.3861) data time 0.0010 (0.0018) model time 0.3713 (0.3833) loss 3.4890 (3.2893) grad_norm 1.6486 (1.5404) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][590/625] eta 0:00:13 lr 0.001242 wd 0.0500 time 0.3803 (0.3859) data time 0.0009 (0.0018) model time 0.3794 (0.3832) loss 3.3092 (3.2853) grad_norm 1.1638 (1.5362) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][600/625] eta 0:00:09 lr 0.001242 wd 0.0500 time 0.3781 (0.3858) data time 0.0012 (0.0018) model time 0.3769 (0.3831) loss 4.0844 (3.2845) grad_norm 2.0117 (1.5347) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][610/625] eta 0:00:05 lr 0.001241 wd 0.0500 time 0.3605 (0.3857) data time 0.0005 (0.0018) model time 0.3600 (0.3831) loss 3.1451 (3.2773) grad_norm 1.1268 (1.5306) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [138/300][620/625] eta 0:00:01 lr 0.001241 wd 0.0500 time 0.3579 (0.3854) data time 0.0005 (0.0018) model time 0.3574 (0.3827) loss 2.8817 (3.2771) grad_norm 1.1449 (1.5258) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:45:49 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 138 training takes 0:04:00 [2024-08-27 08:45:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 08:45:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 08:45:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.503 (0.503) Loss 0.6411 (0.6411) Acc@1 86.279 (86.279) Acc@5 97.900 (97.900) Mem 11224MB [2024-08-27 08:45:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 1.0381 (0.7559) Acc@1 75.879 (83.225) Acc@5 94.434 (96.782) Mem 11224MB [2024-08-27 08:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.1035 (0.9084) Acc@1 73.438 (79.727) Acc@5 92.578 (94.957) Mem 11224MB [2024-08-27 08:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.459 Acc@5 94.964 [2024-08-27 08:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.5% [2024-08-27 08:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.46% [2024-08-27 08:45:53 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 08:45:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 08:45:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.5225 (0.5225) Acc@1 88.721 (88.721) Acc@5 98.535 (98.535) Mem 11224MB [2024-08-27 08:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.8662 (0.6517) Acc@1 79.004 (85.161) Acc@5 95.605 (97.412) Mem 11224MB [2024-08-27 08:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9966 (0.7822) Acc@1 75.781 (82.006) Acc@5 94.287 (96.024) Mem 11224MB [2024-08-27 08:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.730 Acc@5 96.025 [2024-08-27 08:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.7% [2024-08-27 08:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.73% [2024-08-27 08:46:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 08:46:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 08:46:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][0/625] eta 0:07:45 lr 0.001241 wd 0.0500 time 0.7443 (0.7443) data time 0.3971 (0.3971) model time 0.0000 (0.0000) loss 2.3826 (2.3826) grad_norm 2.3262 (2.3262) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][10/625] eta 0:04:12 lr 0.001241 wd 0.0500 time 0.3862 (0.4102) data time 0.0010 (0.0370) model time 0.0000 (0.0000) loss 3.4332 (3.4233) grad_norm 1.3004 (1.4584) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][20/625] eta 0:04:01 lr 0.001241 wd 0.0500 time 0.3670 (0.3987) data time 0.0008 (0.0199) model time 0.0000 (0.0000) loss 3.6363 (3.3895) grad_norm 1.6185 (1.4601) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][30/625] eta 0:03:55 lr 0.001241 wd 0.0500 time 0.3776 (0.3958) data time 0.0010 (0.0139) model time 0.0000 (0.0000) loss 2.9193 (3.3812) grad_norm 0.9577 (1.5521) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][40/625] eta 0:03:48 lr 0.001240 wd 0.0500 time 0.3840 (0.3907) data time 0.0010 (0.0108) model time 0.0000 (0.0000) loss 3.1078 (3.3731) grad_norm 1.1330 (1.5692) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][50/625] eta 0:03:42 lr 0.001240 wd 0.0500 time 0.3706 (0.3877) data time 0.0011 (0.0089) model time 0.0000 (0.0000) loss 3.8671 (3.3805) grad_norm 1.3311 (1.5350) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][60/625] eta 0:03:38 lr 0.001240 wd 0.0500 time 0.3750 (0.3859) data time 0.0008 (0.0076) model time 0.3742 (0.3754) loss 3.6781 (3.3687) grad_norm 2.2548 (1.5234) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][70/625] eta 0:03:39 lr 0.001240 wd 0.0500 time 0.3812 (0.3953) data time 0.0010 (0.0067) model time 0.3801 (0.4134) loss 2.5860 (3.2902) grad_norm 1.2009 (1.5159) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][80/625] eta 0:03:36 lr 0.001240 wd 0.0500 time 0.3721 (0.3977) data time 0.0011 (0.0060) model time 0.3711 (0.4134) loss 2.7678 (3.2296) grad_norm 1.9161 (1.4883) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][90/625] eta 0:03:31 lr 0.001240 wd 0.0500 time 0.3780 (0.3955) data time 0.0011 (0.0055) model time 0.3768 (0.4043) loss 3.4009 (3.2391) grad_norm 1.5022 (1.5155) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][100/625] eta 0:03:26 lr 0.001239 wd 0.0500 time 0.3784 (0.3936) data time 0.0010 (0.0050) model time 0.3774 (0.3985) loss 3.4961 (3.2358) grad_norm 1.5179 (1.5129) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][110/625] eta 0:03:21 lr 0.001239 wd 0.0500 time 0.3799 (0.3921) data time 0.0008 (0.0047) model time 0.3791 (0.3947) loss 2.7377 (3.2287) grad_norm 1.0098 (1.4973) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][120/625] eta 0:03:17 lr 0.001239 wd 0.0500 time 0.3811 (0.3912) data time 0.0009 (0.0044) model time 0.3802 (0.3926) loss 4.3527 (3.2640) grad_norm 1.2240 (1.4815) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][130/625] eta 0:03:13 lr 0.001239 wd 0.0500 time 0.3788 (0.3904) data time 0.0008 (0.0041) model time 0.3780 (0.3910) loss 3.2618 (3.2644) grad_norm 1.3044 (1.4743) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:46:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][140/625] eta 0:03:08 lr 0.001239 wd 0.0500 time 0.3748 (0.3896) data time 0.0010 (0.0039) model time 0.3738 (0.3895) loss 2.6583 (3.2535) grad_norm 1.4139 (1.4697) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][150/625] eta 0:03:04 lr 0.001239 wd 0.0500 time 0.3830 (0.3888) data time 0.0012 (0.0037) model time 0.3818 (0.3882) loss 3.5694 (3.2760) grad_norm 1.1546 (1.4763) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][160/625] eta 0:03:00 lr 0.001238 wd 0.0500 time 0.3767 (0.3880) data time 0.0011 (0.0036) model time 0.3756 (0.3870) loss 2.8953 (3.2647) grad_norm 1.1447 (1.5037) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][170/625] eta 0:02:56 lr 0.001238 wd 0.0500 time 0.3718 (0.3874) data time 0.0011 (0.0034) model time 0.3707 (0.3862) loss 2.8992 (3.2537) grad_norm 1.2549 (1.5062) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][180/625] eta 0:02:52 lr 0.001238 wd 0.0500 time 0.5730 (0.3880) data time 0.0008 (0.0033) model time 0.5722 (0.3870) loss 3.1273 (3.2458) grad_norm 1.4846 (1.5071) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][190/625] eta 0:02:49 lr 0.001238 wd 0.0500 time 0.3745 (0.3896) data time 0.0010 (0.0032) model time 0.3735 (0.3892) loss 3.3421 (3.2545) grad_norm 1.2712 (1.4999) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][200/625] eta 0:02:46 lr 0.001238 wd 0.0500 time 0.3352 (0.3917) data time 0.0008 (0.0031) model time 0.3345 (0.3919) loss 3.6993 (3.2520) grad_norm 1.4804 (1.5096) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][210/625] eta 0:02:42 lr 0.001237 wd 0.0500 time 0.3746 (0.3909) data time 0.0008 (0.0030) model time 0.3737 (0.3908) loss 2.8769 (3.2466) grad_norm 1.3442 (1.5116) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][220/625] eta 0:02:38 lr 0.001237 wd 0.0500 time 0.3754 (0.3903) data time 0.0012 (0.0029) model time 0.3742 (0.3900) loss 3.0875 (3.2583) grad_norm 1.1221 (1.5074) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][230/625] eta 0:02:34 lr 0.001237 wd 0.0500 time 0.3841 (0.3899) data time 0.0009 (0.0028) model time 0.3833 (0.3894) loss 2.4515 (3.2531) grad_norm 2.3226 (1.5024) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][240/625] eta 0:02:29 lr 0.001237 wd 0.0500 time 0.3769 (0.3894) data time 0.0011 (0.0028) model time 0.3758 (0.3888) loss 3.3346 (3.2550) grad_norm 1.7522 (1.5156) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][250/625] eta 0:02:25 lr 0.001237 wd 0.0500 time 0.3738 (0.3889) data time 0.0011 (0.0027) model time 0.3728 (0.3881) loss 3.4751 (3.2592) grad_norm 1.7152 (1.5184) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][260/625] eta 0:02:21 lr 0.001237 wd 0.0500 time 0.3788 (0.3885) data time 0.0008 (0.0026) model time 0.3780 (0.3875) loss 2.6011 (3.2546) grad_norm 1.3539 (1.5122) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][270/625] eta 0:02:17 lr 0.001236 wd 0.0500 time 0.3782 (0.3880) data time 0.0008 (0.0026) model time 0.3774 (0.3870) loss 2.1856 (3.2425) grad_norm 2.3229 (1.5140) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][280/625] eta 0:02:13 lr 0.001236 wd 0.0500 time 0.3768 (0.3877) data time 0.0008 (0.0025) model time 0.3760 (0.3866) loss 3.0892 (3.2422) grad_norm 1.4216 (1.5199) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][290/625] eta 0:02:09 lr 0.001236 wd 0.0500 time 0.3759 (0.3874) data time 0.0011 (0.0025) model time 0.3748 (0.3862) loss 3.4150 (3.2308) grad_norm 1.3467 (1.5146) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][300/625] eta 0:02:05 lr 0.001236 wd 0.0500 time 0.3767 (0.3871) data time 0.0011 (0.0024) model time 0.3756 (0.3858) loss 3.2418 (3.2439) grad_norm 2.0407 (1.5167) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][310/625] eta 0:02:01 lr 0.001236 wd 0.0500 time 0.3743 (0.3868) data time 0.0011 (0.0024) model time 0.3732 (0.3855) loss 2.2837 (3.2342) grad_norm 1.0388 (1.5215) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][320/625] eta 0:01:58 lr 0.001236 wd 0.0500 time 0.3340 (0.3870) data time 0.0009 (0.0024) model time 0.3331 (0.3858) loss 3.9031 (3.2315) grad_norm 1.0965 (1.5120) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][330/625] eta 0:01:54 lr 0.001235 wd 0.0500 time 0.3862 (0.3868) data time 0.0011 (0.0023) model time 0.3850 (0.3855) loss 3.3821 (3.2353) grad_norm 1.1365 (1.5047) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][340/625] eta 0:01:50 lr 0.001235 wd 0.0500 time 0.3898 (0.3866) data time 0.0008 (0.0023) model time 0.3890 (0.3853) loss 2.9083 (3.2233) grad_norm 1.1308 (1.5010) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][350/625] eta 0:01:46 lr 0.001235 wd 0.0500 time 0.3722 (0.3863) data time 0.0008 (0.0023) model time 0.3714 (0.3849) loss 2.8601 (3.2217) grad_norm 1.4959 (1.4972) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][360/625] eta 0:01:42 lr 0.001235 wd 0.0500 time 0.3896 (0.3861) data time 0.0008 (0.0022) model time 0.3887 (0.3847) loss 3.4541 (3.2171) grad_norm 1.4200 (1.4976) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][370/625] eta 0:01:38 lr 0.001235 wd 0.0500 time 0.3758 (0.3859) data time 0.0011 (0.0022) model time 0.3747 (0.3844) loss 3.2423 (3.2124) grad_norm 0.9778 (1.4962) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][380/625] eta 0:01:34 lr 0.001235 wd 0.0500 time 0.3786 (0.3856) data time 0.0011 (0.0022) model time 0.3775 (0.3842) loss 3.3253 (3.2104) grad_norm 1.4380 (1.4941) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][390/625] eta 0:01:30 lr 0.001234 wd 0.0500 time 0.3782 (0.3860) data time 0.0010 (0.0021) model time 0.3771 (0.3846) loss 3.2969 (3.2071) grad_norm 1.0532 (1.5006) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][400/625] eta 0:01:27 lr 0.001234 wd 0.0500 time 0.3749 (0.3869) data time 0.0011 (0.0021) model time 0.3738 (0.3856) loss 2.1483 (3.2043) grad_norm 1.0548 (1.4964) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][410/625] eta 0:01:23 lr 0.001234 wd 0.0500 time 0.3760 (0.3877) data time 0.0010 (0.0021) model time 0.3751 (0.3865) loss 3.8339 (3.2037) grad_norm 1.3787 (1.4917) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][420/625] eta 0:01:19 lr 0.001234 wd 0.0500 time 0.3822 (0.3874) data time 0.0010 (0.0021) model time 0.3812 (0.3862) loss 3.3850 (3.2026) grad_norm 1.0632 (1.4849) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][430/625] eta 0:01:15 lr 0.001234 wd 0.0500 time 0.3726 (0.3871) data time 0.0008 (0.0020) model time 0.3718 (0.3859) loss 3.8450 (3.2097) grad_norm 2.0889 (1.4897) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][440/625] eta 0:01:11 lr 0.001233 wd 0.0500 time 0.3726 (0.3868) data time 0.0010 (0.0020) model time 0.3716 (0.3856) loss 2.5356 (3.2076) grad_norm 1.5974 (1.4939) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:48:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][450/625] eta 0:01:07 lr 0.001233 wd 0.0500 time 0.3776 (0.3866) data time 0.0011 (0.0020) model time 0.3765 (0.3854) loss 3.3775 (3.2099) grad_norm 1.6520 (1.4970) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][460/625] eta 0:01:03 lr 0.001233 wd 0.0500 time 0.3734 (0.3864) data time 0.0010 (0.0020) model time 0.3724 (0.3851) loss 3.3136 (3.2148) grad_norm 1.6184 (1.4972) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][470/625] eta 0:00:59 lr 0.001233 wd 0.0500 time 0.3739 (0.3862) data time 0.0008 (0.0020) model time 0.3731 (0.3849) loss 3.1510 (3.2119) grad_norm 1.1703 (1.4946) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][480/625] eta 0:00:55 lr 0.001233 wd 0.0500 time 0.3777 (0.3860) data time 0.0010 (0.0019) model time 0.3767 (0.3847) loss 3.2812 (3.2130) grad_norm 1.3457 (1.4904) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][490/625] eta 0:00:52 lr 0.001233 wd 0.0500 time 0.3751 (0.3858) data time 0.0010 (0.0019) model time 0.3740 (0.3845) loss 3.7061 (3.2176) grad_norm 1.6395 (1.5012) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][500/625] eta 0:00:48 lr 0.001232 wd 0.0500 time 0.3672 (0.3856) data time 0.0009 (0.0019) model time 0.3663 (0.3842) loss 2.9095 (3.2188) grad_norm 1.1897 (1.4967) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][510/625] eta 0:00:44 lr 0.001232 wd 0.0500 time 0.3764 (0.3854) data time 0.0011 (0.0019) model time 0.3753 (0.3840) loss 3.2842 (3.2232) grad_norm 2.0383 (1.4917) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][520/625] eta 0:00:40 lr 0.001232 wd 0.0500 time 0.3774 (0.3852) data time 0.0008 (0.0019) model time 0.3766 (0.3838) loss 3.1466 (3.2210) grad_norm 2.5126 (1.5070) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][530/625] eta 0:00:36 lr 0.001232 wd 0.0500 time 0.3740 (0.3854) data time 0.0010 (0.0019) model time 0.3730 (0.3840) loss 3.8523 (3.2236) grad_norm 1.5126 (1.5123) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][540/625] eta 0:00:32 lr 0.001232 wd 0.0500 time 0.3838 (0.3852) data time 0.0011 (0.0019) model time 0.3827 (0.3838) loss 3.4610 (3.2240) grad_norm 1.1767 (1.5079) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][550/625] eta 0:00:28 lr 0.001232 wd 0.0500 time 0.3807 (0.3850) data time 0.0008 (0.0018) model time 0.3800 (0.3836) loss 4.1418 (3.2270) grad_norm 1.2426 (1.5058) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][560/625] eta 0:00:25 lr 0.001231 wd 0.0500 time 0.3645 (0.3848) data time 0.0011 (0.0018) model time 0.3634 (0.3834) loss 2.7055 (3.2317) grad_norm 1.4101 (1.5002) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][570/625] eta 0:00:21 lr 0.001231 wd 0.0500 time 0.3732 (0.3847) data time 0.0008 (0.0018) model time 0.3724 (0.3833) loss 3.6464 (3.2319) grad_norm 1.4441 (1.5017) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][580/625] eta 0:00:17 lr 0.001231 wd 0.0500 time 0.5721 (0.3859) data time 0.0010 (0.0018) model time 0.5711 (0.3846) loss 3.3591 (3.2348) grad_norm 1.7887 (1.5006) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 08:49:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][590/625] eta 0:00:13 lr 0.001231 wd 0.0500 time 0.3839 (0.3868) data time 0.0008 (0.0018) model time 0.3831 (0.3856) loss 3.7376 (3.2409) grad_norm 1.5369 (1.4986) loss_scale 2048.0000 (1025.7327) mem 11224MB [2024-08-27 08:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][600/625] eta 0:00:09 lr 0.001231 wd 0.0500 time 0.3744 (0.3867) data time 0.0011 (0.0018) model time 0.3733 (0.3855) loss 2.8486 (3.2383) grad_norm 1.4574 (1.5004) loss_scale 2048.0000 (1042.7421) mem 11224MB [2024-08-27 08:49:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][610/625] eta 0:00:05 lr 0.001231 wd 0.0500 time 0.3656 (0.3865) data time 0.0006 (0.0018) model time 0.3650 (0.3852) loss 3.6532 (3.2419) grad_norm 1.2926 (1.5048) loss_scale 2048.0000 (1059.1948) mem 11224MB [2024-08-27 08:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [139/300][620/625] eta 0:00:01 lr 0.001230 wd 0.0500 time 0.3756 (0.3861) data time 0.0008 (0.0018) model time 0.3748 (0.3848) loss 3.7644 (3.2399) grad_norm 1.3179 (1.5041) loss_scale 2048.0000 (1075.1176) mem 11224MB [2024-08-27 08:50:03 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 139 training takes 0:04:01 [2024-08-27 08:50:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 08:50:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 08:50:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.513 (0.513) Loss 0.5918 (0.5918) Acc@1 87.012 (87.012) Acc@5 98.047 (98.047) Mem 11224MB [2024-08-27 08:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.9478 (0.7309) Acc@1 77.393 (83.478) Acc@5 94.482 (96.857) Mem 11224MB [2024-08-27 08:50:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.1084 (0.8799) Acc@1 72.949 (79.776) Acc@5 92.578 (95.073) Mem 11224MB [2024-08-27 08:50:07 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.533 Acc@5 95.064 [2024-08-27 08:50:07 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.5% [2024-08-27 08:50:07 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.53% [2024-08-27 08:50:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 08:50:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 08:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.508 (0.508) Loss 0.5210 (0.5210) Acc@1 88.770 (88.770) Acc@5 98.438 (98.438) Mem 11224MB [2024-08-27 08:50:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.8643 (0.6508) Acc@1 79.297 (85.165) Acc@5 95.557 (97.372) Mem 11224MB [2024-08-27 08:50:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.098 (0.120) Loss 0.9932 (0.7811) Acc@1 75.635 (81.987) Acc@5 94.238 (96.010) Mem 11224MB [2024-08-27 08:50:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.714 Acc@5 96.023 [2024-08-27 08:50:11 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.7% [2024-08-27 08:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][0/625] eta 0:12:36 lr 0.001230 wd 0.0500 time 1.2112 (1.2112) data time 0.5741 (0.5741) model time 0.0000 (0.0000) loss 2.6411 (2.6411) grad_norm 1.4935 (1.4935) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][10/625] eta 0:04:37 lr 0.001230 wd 0.0500 time 0.3740 (0.4507) data time 0.0010 (0.0533) model time 0.0000 (0.0000) loss 3.1263 (3.1835) grad_norm 1.8798 (1.7652) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][20/625] eta 0:04:10 lr 0.001230 wd 0.0500 time 0.3725 (0.4139) data time 0.0007 (0.0284) model time 0.0000 (0.0000) loss 4.1517 (3.1206) grad_norm 1.4382 (1.6576) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][30/625] eta 0:03:58 lr 0.001230 wd 0.0500 time 0.3713 (0.4014) data time 0.0010 (0.0196) model time 0.0000 (0.0000) loss 3.8634 (3.1540) grad_norm 0.9129 (1.5382) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][40/625] eta 0:03:50 lr 0.001230 wd 0.0500 time 0.3765 (0.3948) data time 0.0008 (0.0151) model time 0.0000 (0.0000) loss 3.4299 (3.1628) grad_norm 1.3503 (1.4564) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][50/625] eta 0:03:46 lr 0.001229 wd 0.0500 time 0.3769 (0.3941) data time 0.0008 (0.0123) model time 0.0000 (0.0000) loss 3.8528 (3.1787) grad_norm 1.5605 (1.4354) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][60/625] eta 0:03:40 lr 0.001229 wd 0.0500 time 0.3772 (0.3907) data time 0.0010 (0.0105) model time 0.3762 (0.3719) loss 3.2433 (3.1782) grad_norm 1.5499 (1.4383) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][70/625] eta 0:03:35 lr 0.001229 wd 0.0500 time 0.3707 (0.3882) data time 0.0008 (0.0092) model time 0.3700 (0.3718) loss 3.3433 (3.2143) grad_norm 1.1575 (1.4304) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][80/625] eta 0:03:30 lr 0.001229 wd 0.0500 time 0.3719 (0.3863) data time 0.0009 (0.0082) model time 0.3710 (0.3717) loss 3.9871 (3.1949) grad_norm 1.8589 (1.4375) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][90/625] eta 0:03:26 lr 0.001229 wd 0.0500 time 0.3773 (0.3854) data time 0.0011 (0.0075) model time 0.3761 (0.3730) loss 3.2558 (3.1888) grad_norm 1.3351 (1.4555) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][100/625] eta 0:03:21 lr 0.001229 wd 0.0500 time 0.3763 (0.3843) data time 0.0012 (0.0068) model time 0.3752 (0.3730) loss 3.5972 (3.1783) grad_norm 0.9966 (1.4424) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][110/625] eta 0:03:19 lr 0.001228 wd 0.0500 time 0.5291 (0.3877) data time 0.0009 (0.0063) model time 0.5282 (0.3811) loss 3.4954 (3.1905) grad_norm 1.1593 (1.4330) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:50:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][120/625] eta 0:03:17 lr 0.001228 wd 0.0500 time 0.3769 (0.3914) data time 0.0010 (0.0059) model time 0.3759 (0.3882) loss 3.2146 (3.2090) grad_norm 1.1524 (1.4194) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][130/625] eta 0:03:14 lr 0.001228 wd 0.0500 time 0.3736 (0.3935) data time 0.0008 (0.0055) model time 0.3728 (0.3918) loss 4.1684 (3.2295) grad_norm 1.3992 (1.4118) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][140/625] eta 0:03:10 lr 0.001228 wd 0.0500 time 0.3713 (0.3924) data time 0.0012 (0.0052) model time 0.3701 (0.3903) loss 3.2482 (3.2283) grad_norm 1.2919 (1.4608) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][150/625] eta 0:03:05 lr 0.001228 wd 0.0500 time 0.3790 (0.3913) data time 0.0009 (0.0050) model time 0.3780 (0.3886) loss 3.1347 (3.2360) grad_norm 1.4653 (1.4641) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][160/625] eta 0:03:01 lr 0.001228 wd 0.0500 time 0.3765 (0.3903) data time 0.0008 (0.0047) model time 0.3757 (0.3873) loss 4.4372 (3.2424) grad_norm 1.5272 (1.4801) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][170/625] eta 0:02:57 lr 0.001227 wd 0.0500 time 0.3692 (0.3903) data time 0.0008 (0.0045) model time 0.3684 (0.3875) loss 3.5249 (3.2320) grad_norm 1.3387 (1.4717) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][180/625] eta 0:02:53 lr 0.001227 wd 0.0500 time 0.3762 (0.3894) data time 0.0008 (0.0043) model time 0.3754 (0.3864) loss 3.7018 (3.2450) grad_norm 1.4521 (1.4725) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][190/625] eta 0:02:49 lr 0.001227 wd 0.0500 time 0.3720 (0.3887) data time 0.0010 (0.0042) model time 0.3711 (0.3856) loss 3.8410 (3.2361) grad_norm 1.1621 (1.4718) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][200/625] eta 0:02:44 lr 0.001227 wd 0.0500 time 0.3696 (0.3881) data time 0.0008 (0.0040) model time 0.3688 (0.3849) loss 3.5130 (3.2446) grad_norm 1.3603 (1.4686) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][210/625] eta 0:02:40 lr 0.001227 wd 0.0500 time 0.3716 (0.3875) data time 0.0009 (0.0039) model time 0.3707 (0.3842) loss 1.9257 (3.2363) grad_norm 1.5720 (1.4681) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][220/625] eta 0:02:36 lr 0.001226 wd 0.0500 time 0.3727 (0.3869) data time 0.0009 (0.0037) model time 0.3718 (0.3835) loss 2.1816 (3.2297) grad_norm 1.2625 (1.4596) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][230/625] eta 0:02:33 lr 0.001226 wd 0.0500 time 0.5716 (0.3888) data time 0.0010 (0.0036) model time 0.5706 (0.3862) loss 3.3690 (3.2388) grad_norm 1.4803 (1.4535) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][240/625] eta 0:02:30 lr 0.001226 wd 0.0500 time 0.3694 (0.3900) data time 0.0007 (0.0035) model time 0.3687 (0.3878) loss 3.1822 (3.2432) grad_norm 1.4528 (1.4850) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][250/625] eta 0:02:26 lr 0.001226 wd 0.0500 time 0.3733 (0.3910) data time 0.0008 (0.0034) model time 0.3724 (0.3891) loss 4.0200 (3.2575) grad_norm 0.8900 (1.4806) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][260/625] eta 0:02:22 lr 0.001226 wd 0.0500 time 0.3700 (0.3904) data time 0.0008 (0.0033) model time 0.3693 (0.3883) loss 1.7893 (3.2375) grad_norm 1.5353 (1.4895) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:51:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][270/625] eta 0:02:18 lr 0.001226 wd 0.0500 time 0.3790 (0.3897) data time 0.0008 (0.0033) model time 0.3782 (0.3876) loss 2.7243 (3.2364) grad_norm 1.5463 (1.4880) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][280/625] eta 0:02:14 lr 0.001225 wd 0.0500 time 0.3767 (0.3892) data time 0.0008 (0.0032) model time 0.3759 (0.3869) loss 4.3302 (3.2435) grad_norm 0.9495 (1.4782) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][290/625] eta 0:02:10 lr 0.001225 wd 0.0500 time 0.3737 (0.3888) data time 0.0010 (0.0031) model time 0.3726 (0.3865) loss 2.5904 (3.2435) grad_norm 2.8420 (1.4774) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][300/625] eta 0:02:06 lr 0.001225 wd 0.0500 time 0.3902 (0.3885) data time 0.0008 (0.0031) model time 0.3894 (0.3862) loss 2.7266 (3.2411) grad_norm 1.0872 (1.4776) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][310/625] eta 0:02:02 lr 0.001225 wd 0.0500 time 0.3683 (0.3880) data time 0.0011 (0.0030) model time 0.3672 (0.3856) loss 2.9220 (3.2431) grad_norm 1.4185 (1.4700) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][320/625] eta 0:01:58 lr 0.001225 wd 0.0500 time 0.3685 (0.3876) data time 0.0008 (0.0029) model time 0.3678 (0.3852) loss 4.0747 (3.2506) grad_norm 1.9275 (1.4815) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][330/625] eta 0:01:54 lr 0.001225 wd 0.0500 time 0.3702 (0.3873) data time 0.0011 (0.0029) model time 0.3691 (0.3849) loss 3.0018 (3.2542) grad_norm 1.3203 (1.4800) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][340/625] eta 0:01:50 lr 0.001224 wd 0.0500 time 0.4026 (0.3871) data time 0.0010 (0.0028) model time 0.4016 (0.3847) loss 2.7638 (3.2502) grad_norm 1.1726 (1.4831) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][350/625] eta 0:01:46 lr 0.001224 wd 0.0500 time 0.3744 (0.3870) data time 0.0009 (0.0028) model time 0.3735 (0.3846) loss 3.3036 (3.2589) grad_norm 1.4766 (1.4821) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][360/625] eta 0:01:42 lr 0.001224 wd 0.0500 time 0.3759 (0.3868) data time 0.0010 (0.0028) model time 0.3749 (0.3844) loss 3.2107 (3.2659) grad_norm 1.7764 (1.4891) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][370/625] eta 0:01:38 lr 0.001224 wd 0.0500 time 0.3884 (0.3865) data time 0.0010 (0.0027) model time 0.3874 (0.3841) loss 3.9257 (3.2693) grad_norm 1.0423 (1.4829) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][380/625] eta 0:01:34 lr 0.001224 wd 0.0500 time 0.3710 (0.3868) data time 0.0010 (0.0027) model time 0.3700 (0.3845) loss 3.2513 (3.2658) grad_norm 1.2380 (1.4835) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][390/625] eta 0:01:30 lr 0.001224 wd 0.0500 time 0.3766 (0.3865) data time 0.0010 (0.0026) model time 0.3756 (0.3842) loss 3.2697 (3.2601) grad_norm 1.5386 (1.4814) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][400/625] eta 0:01:26 lr 0.001223 wd 0.0500 time 0.3775 (0.3863) data time 0.0012 (0.0026) model time 0.3763 (0.3840) loss 3.2226 (3.2633) grad_norm 1.3640 (1.4812) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][410/625] eta 0:01:23 lr 0.001223 wd 0.0500 time 0.3848 (0.3862) data time 0.0011 (0.0026) model time 0.3836 (0.3838) loss 2.7292 (3.2589) grad_norm 1.1418 (1.4871) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][420/625] eta 0:01:19 lr 0.001223 wd 0.0500 time 0.3689 (0.3860) data time 0.0011 (0.0025) model time 0.3678 (0.3837) loss 3.1457 (3.2573) grad_norm 1.3133 (1.5057) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:52:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][430/625] eta 0:01:15 lr 0.001223 wd 0.0500 time 0.3758 (0.3858) data time 0.0007 (0.0025) model time 0.3751 (0.3835) loss 3.6023 (3.2619) grad_norm 1.3129 (1.5048) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][440/625] eta 0:01:11 lr 0.001223 wd 0.0500 time 0.3707 (0.3869) data time 0.0008 (0.0025) model time 0.3700 (0.3848) loss 2.4161 (3.2639) grad_norm 1.7674 (1.5043) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][450/625] eta 0:01:07 lr 0.001222 wd 0.0500 time 0.3695 (0.3876) data time 0.0008 (0.0024) model time 0.3687 (0.3856) loss 3.6999 (3.2603) grad_norm 1.3112 (1.5094) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][460/625] eta 0:01:04 lr 0.001222 wd 0.0500 time 0.3872 (0.3884) data time 0.0011 (0.0024) model time 0.3861 (0.3865) loss 3.5958 (3.2640) grad_norm 1.5323 (1.5087) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][470/625] eta 0:01:00 lr 0.001222 wd 0.0500 time 0.3659 (0.3882) data time 0.0011 (0.0024) model time 0.3648 (0.3863) loss 3.4993 (3.2642) grad_norm 1.1936 (1.5073) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][480/625] eta 0:00:56 lr 0.001222 wd 0.0500 time 0.3781 (0.3880) data time 0.0007 (0.0024) model time 0.3774 (0.3861) loss 3.8740 (3.2655) grad_norm 1.3578 (1.5028) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][490/625] eta 0:00:52 lr 0.001222 wd 0.0500 time 0.3895 (0.3878) data time 0.0010 (0.0024) model time 0.3885 (0.3859) loss 2.8108 (3.2669) grad_norm 1.4072 (1.4987) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][500/625] eta 0:00:48 lr 0.001222 wd 0.0500 time 0.3981 (0.3877) data time 0.0008 (0.0024) model time 0.3973 (0.3858) loss 4.0242 (3.2731) grad_norm 1.2809 (1.4933) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][510/625] eta 0:00:44 lr 0.001221 wd 0.0500 time 0.3759 (0.3876) data time 0.0015 (0.0024) model time 0.3744 (0.3856) loss 3.6429 (3.2760) grad_norm 3.1560 (1.4962) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][520/625] eta 0:00:40 lr 0.001221 wd 0.0500 time 0.3976 (0.3875) data time 0.0009 (0.0023) model time 0.3967 (0.3855) loss 3.3723 (3.2745) grad_norm 1.2919 (1.5003) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][530/625] eta 0:00:36 lr 0.001221 wd 0.0500 time 0.3863 (0.3873) data time 0.0011 (0.0023) model time 0.3852 (0.3853) loss 1.9413 (3.2732) grad_norm 1.0868 (1.4987) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][540/625] eta 0:00:32 lr 0.001221 wd 0.0500 time 0.3696 (0.3872) data time 0.0010 (0.0023) model time 0.3685 (0.3852) loss 3.3815 (3.2727) grad_norm 1.1557 (1.4966) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][550/625] eta 0:00:29 lr 0.001221 wd 0.0500 time 0.4022 (0.3871) data time 0.0011 (0.0023) model time 0.4011 (0.3851) loss 3.5466 (3.2689) grad_norm 1.2588 (1.4959) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][560/625] eta 0:00:25 lr 0.001221 wd 0.0500 time 0.3737 (0.3874) data time 0.0008 (0.0023) model time 0.3729 (0.3855) loss 3.9636 (3.2715) grad_norm 2.4244 (1.5023) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][570/625] eta 0:00:21 lr 0.001220 wd 0.0500 time 0.3715 (0.3872) data time 0.0009 (0.0023) model time 0.3706 (0.3853) loss 3.2169 (3.2738) grad_norm 1.2772 (1.5049) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:53:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][580/625] eta 0:00:17 lr 0.001220 wd 0.0500 time 0.3908 (0.3871) data time 0.0011 (0.0023) model time 0.3897 (0.3851) loss 3.3532 (3.2724) grad_norm 1.2943 (1.5077) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][590/625] eta 0:00:13 lr 0.001220 wd 0.0500 time 0.3664 (0.3870) data time 0.0011 (0.0022) model time 0.3653 (0.3850) loss 2.5358 (3.2676) grad_norm 1.4948 (1.5053) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][600/625] eta 0:00:09 lr 0.001220 wd 0.0500 time 0.3816 (0.3869) data time 0.0008 (0.0022) model time 0.3808 (0.3850) loss 3.7574 (3.2693) grad_norm 1.2985 (1.5020) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][610/625] eta 0:00:05 lr 0.001220 wd 0.0500 time 0.3770 (0.3868) data time 0.0008 (0.0022) model time 0.3762 (0.3848) loss 3.4709 (3.2711) grad_norm 1.0122 (1.5003) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [140/300][620/625] eta 0:00:01 lr 0.001220 wd 0.0500 time 0.3676 (0.3867) data time 0.0007 (0.0022) model time 0.3668 (0.3848) loss 3.2715 (3.2705) grad_norm 1.5693 (1.4999) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:13 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 140 training takes 0:04:02 [2024-08-27 08:54:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 08:54:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 08:54:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.5991 (0.5991) Acc@1 86.963 (86.963) Acc@5 98.047 (98.047) Mem 11224MB [2024-08-27 08:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 1.0146 (0.7391) Acc@1 75.928 (83.274) Acc@5 94.287 (96.773) Mem 11224MB [2024-08-27 08:54:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.1162 (0.8801) Acc@1 72.461 (79.790) Acc@5 93.115 (95.159) Mem 11224MB [2024-08-27 08:54:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.587 Acc@5 95.160 [2024-08-27 08:54:17 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.6% [2024-08-27 08:54:17 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.59% [2024-08-27 08:54:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 08:54:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 08:54:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.491 (0.491) Loss 0.5200 (0.5200) Acc@1 88.672 (88.672) Acc@5 98.340 (98.340) Mem 11224MB [2024-08-27 08:54:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.142) Loss 0.8633 (0.6499) Acc@1 79.639 (85.236) Acc@5 95.654 (97.368) Mem 11224MB [2024-08-27 08:54:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9907 (0.7799) Acc@1 75.439 (82.015) Acc@5 94.287 (96.047) Mem 11224MB [2024-08-27 08:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.746 Acc@5 96.057 [2024-08-27 08:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.7% [2024-08-27 08:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.75% [2024-08-27 08:54:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 08:54:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 08:54:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][0/625] eta 0:07:52 lr 0.001219 wd 0.0500 time 0.7567 (0.7567) data time 0.4082 (0.4082) model time 0.0000 (0.0000) loss 3.0658 (3.0658) grad_norm 1.2063 (1.2063) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][10/625] eta 0:05:02 lr 0.001219 wd 0.0500 time 0.5928 (0.4911) data time 0.0010 (0.0381) model time 0.0000 (0.0000) loss 2.5421 (3.1883) grad_norm 2.0914 (1.4702) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][20/625] eta 0:04:28 lr 0.001219 wd 0.0500 time 0.3728 (0.4438) data time 0.0010 (0.0205) model time 0.0000 (0.0000) loss 2.6982 (3.0757) grad_norm 1.6706 (1.5359) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][30/625] eta 0:04:11 lr 0.001219 wd 0.0500 time 0.3892 (0.4229) data time 0.0009 (0.0144) model time 0.0000 (0.0000) loss 2.7975 (3.0045) grad_norm 1.4662 (1.5363) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][40/625] eta 0:04:00 lr 0.001219 wd 0.0500 time 0.3650 (0.4118) data time 0.0008 (0.0112) model time 0.0000 (0.0000) loss 2.7716 (3.0544) grad_norm 1.4263 (1.5195) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][50/625] eta 0:03:53 lr 0.001219 wd 0.0500 time 0.3794 (0.4053) data time 0.0011 (0.0093) model time 0.0000 (0.0000) loss 3.7329 (3.0678) grad_norm 1.0374 (1.4907) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][60/625] eta 0:03:46 lr 0.001218 wd 0.0500 time 0.3945 (0.4010) data time 0.0010 (0.0080) model time 0.3936 (0.3780) loss 3.6147 (3.0437) grad_norm 1.2987 (1.4591) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][70/625] eta 0:03:40 lr 0.001218 wd 0.0500 time 0.3749 (0.3981) data time 0.0011 (0.0071) model time 0.3738 (0.3783) loss 2.9011 (3.0864) grad_norm 1.6452 (1.4676) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][80/625] eta 0:03:35 lr 0.001218 wd 0.0500 time 0.3809 (0.3954) data time 0.0010 (0.0064) model time 0.3799 (0.3771) loss 3.0560 (3.0536) grad_norm 2.2649 (1.5060) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:54:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][90/625] eta 0:03:30 lr 0.001218 wd 0.0500 time 0.3870 (0.3937) data time 0.0011 (0.0058) model time 0.3858 (0.3775) loss 3.0412 (3.0847) grad_norm 1.5693 (1.4953) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][100/625] eta 0:03:25 lr 0.001218 wd 0.0500 time 0.3787 (0.3919) data time 0.0009 (0.0053) model time 0.3778 (0.3770) loss 3.3825 (3.1210) grad_norm 2.0479 (1.5333) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][110/625] eta 0:03:21 lr 0.001218 wd 0.0500 time 0.3739 (0.3918) data time 0.0008 (0.0049) model time 0.3731 (0.3790) loss 3.7430 (3.1545) grad_norm 1.1940 (1.5406) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][120/625] eta 0:03:18 lr 0.001217 wd 0.0500 time 0.3788 (0.3938) data time 0.0009 (0.0047) model time 0.3779 (0.3842) loss 3.0452 (3.1363) grad_norm 1.3027 (1.5253) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][130/625] eta 0:03:14 lr 0.001217 wd 0.0500 time 0.3717 (0.3924) data time 0.0011 (0.0044) model time 0.3706 (0.3829) loss 3.3374 (3.1445) grad_norm 1.2218 (1.5112) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][140/625] eta 0:03:09 lr 0.001217 wd 0.0500 time 0.3714 (0.3913) data time 0.0009 (0.0042) model time 0.3704 (0.3821) loss 3.4301 (3.1513) grad_norm 1.5143 (1.5016) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][150/625] eta 0:03:05 lr 0.001217 wd 0.0500 time 0.3848 (0.3907) data time 0.0008 (0.0040) model time 0.3840 (0.3820) loss 1.6427 (3.1526) grad_norm 1.1863 (1.4960) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][160/625] eta 0:03:01 lr 0.001217 wd 0.0500 time 0.3722 (0.3897) data time 0.0010 (0.0038) model time 0.3713 (0.3812) loss 3.7373 (3.1622) grad_norm 1.5008 (1.4940) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][170/625] eta 0:02:57 lr 0.001216 wd 0.0500 time 0.3697 (0.3891) data time 0.0009 (0.0037) model time 0.3688 (0.3809) loss 3.2123 (3.1673) grad_norm 1.1044 (1.4867) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][180/625] eta 0:02:52 lr 0.001216 wd 0.0500 time 0.3794 (0.3886) data time 0.0009 (0.0035) model time 0.3785 (0.3808) loss 4.5310 (3.1706) grad_norm 1.9691 (1.4796) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][190/625] eta 0:02:50 lr 0.001216 wd 0.0500 time 0.3674 (0.3910) data time 0.0008 (0.0034) model time 0.3666 (0.3845) loss 3.5410 (3.1775) grad_norm 1.8105 (1.4955) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][200/625] eta 0:02:46 lr 0.001216 wd 0.0500 time 0.5439 (0.3923) data time 0.0008 (0.0033) model time 0.5431 (0.3865) loss 3.5072 (3.1635) grad_norm 1.1192 (1.5070) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][210/625] eta 0:02:42 lr 0.001216 wd 0.0500 time 0.3799 (0.3917) data time 0.0008 (0.0033) model time 0.3791 (0.3861) loss 2.0806 (3.1742) grad_norm 1.2721 (1.5028) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][220/625] eta 0:02:38 lr 0.001216 wd 0.0500 time 0.3778 (0.3918) data time 0.0013 (0.0032) model time 0.3764 (0.3864) loss 3.5811 (3.1824) grad_norm 1.1910 (1.4962) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][230/625] eta 0:02:34 lr 0.001215 wd 0.0500 time 0.3811 (0.3912) data time 0.0008 (0.0031) model time 0.3803 (0.3858) loss 3.5227 (3.1813) grad_norm 1.2230 (1.5007) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:55:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][240/625] eta 0:02:30 lr 0.001215 wd 0.0500 time 0.3792 (0.3907) data time 0.0010 (0.0031) model time 0.3782 (0.3854) loss 3.0826 (3.1776) grad_norm 1.5169 (1.5060) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][250/625] eta 0:02:26 lr 0.001215 wd 0.0500 time 0.3860 (0.3904) data time 0.0008 (0.0030) model time 0.3852 (0.3852) loss 2.6858 (3.1727) grad_norm 1.5161 (1.5150) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][260/625] eta 0:02:22 lr 0.001215 wd 0.0500 time 0.3727 (0.3899) data time 0.0010 (0.0029) model time 0.3716 (0.3848) loss 3.1434 (3.1723) grad_norm 1.0795 (1.5192) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][270/625] eta 0:02:18 lr 0.001215 wd 0.0500 time 0.3955 (0.3896) data time 0.0008 (0.0029) model time 0.3947 (0.3846) loss 3.7829 (3.1869) grad_norm 1.3924 (1.5088) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][280/625] eta 0:02:14 lr 0.001215 wd 0.0500 time 0.3738 (0.3892) data time 0.0011 (0.0029) model time 0.3727 (0.3842) loss 3.2958 (3.1878) grad_norm 1.2224 (1.5064) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][290/625] eta 0:02:10 lr 0.001214 wd 0.0500 time 0.3781 (0.3890) data time 0.0008 (0.0028) model time 0.3773 (0.3841) loss 3.2799 (3.1900) grad_norm 1.9099 (1.5102) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][300/625] eta 0:02:06 lr 0.001214 wd 0.0500 time 0.3727 (0.3886) data time 0.0014 (0.0027) model time 0.3713 (0.3838) loss 3.4181 (3.1875) grad_norm 1.3721 (1.5055) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][310/625] eta 0:02:02 lr 0.001214 wd 0.0500 time 0.3851 (0.3896) data time 0.0010 (0.0027) model time 0.3841 (0.3852) loss 2.8485 (3.1820) grad_norm 1.3654 (1.5027) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][320/625] eta 0:01:59 lr 0.001214 wd 0.0500 time 0.5665 (0.3912) data time 0.0008 (0.0027) model time 0.5657 (0.3872) loss 2.1592 (3.1876) grad_norm 2.1923 (1.5103) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][330/625] eta 0:01:55 lr 0.001214 wd 0.0500 time 0.3649 (0.3921) data time 0.0008 (0.0026) model time 0.3642 (0.3883) loss 2.8751 (3.1887) grad_norm 2.6829 (1.5193) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][340/625] eta 0:01:51 lr 0.001214 wd 0.0500 time 0.3695 (0.3917) data time 0.0011 (0.0026) model time 0.3684 (0.3879) loss 3.5556 (3.1936) grad_norm 1.3916 (1.5145) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][350/625] eta 0:01:47 lr 0.001213 wd 0.0500 time 0.3773 (0.3913) data time 0.0011 (0.0025) model time 0.3762 (0.3875) loss 3.3620 (3.1948) grad_norm 1.1965 (1.5137) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][360/625] eta 0:01:43 lr 0.001213 wd 0.0500 time 0.3716 (0.3909) data time 0.0010 (0.0025) model time 0.3705 (0.3871) loss 3.4454 (3.1900) grad_norm 1.6751 (1.5153) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][370/625] eta 0:01:39 lr 0.001213 wd 0.0500 time 0.3725 (0.3905) data time 0.0008 (0.0025) model time 0.3717 (0.3868) loss 3.8179 (3.1942) grad_norm 1.2442 (1.5144) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][380/625] eta 0:01:35 lr 0.001213 wd 0.0500 time 0.3785 (0.3902) data time 0.0009 (0.0024) model time 0.3776 (0.3865) loss 4.2290 (3.1979) grad_norm 1.6074 (1.5119) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][390/625] eta 0:01:31 lr 0.001213 wd 0.0500 time 0.3822 (0.3898) data time 0.0008 (0.0024) model time 0.3814 (0.3861) loss 3.9251 (3.2000) grad_norm 1.6725 (1.5091) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][400/625] eta 0:01:27 lr 0.001212 wd 0.0500 time 0.3347 (0.3899) data time 0.0011 (0.0024) model time 0.3337 (0.3863) loss 3.6092 (3.2037) grad_norm 1.3258 (1.5063) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][410/625] eta 0:01:23 lr 0.001212 wd 0.0500 time 0.4085 (0.3897) data time 0.0008 (0.0023) model time 0.4077 (0.3862) loss 4.0354 (3.2157) grad_norm 2.0848 (1.5105) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][420/625] eta 0:01:19 lr 0.001212 wd 0.0500 time 0.3763 (0.3894) data time 0.0008 (0.0023) model time 0.3756 (0.3859) loss 4.1322 (3.2173) grad_norm 1.2235 (1.5063) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][430/625] eta 0:01:15 lr 0.001212 wd 0.0500 time 0.3735 (0.3891) data time 0.0010 (0.0023) model time 0.3725 (0.3856) loss 2.9366 (3.2185) grad_norm 1.5037 (1.5026) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][440/625] eta 0:01:11 lr 0.001212 wd 0.0500 time 0.3889 (0.3890) data time 0.0010 (0.0023) model time 0.3879 (0.3855) loss 2.7504 (3.2156) grad_norm 1.7583 (1.5012) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][450/625] eta 0:01:08 lr 0.001212 wd 0.0500 time 0.3823 (0.3888) data time 0.0010 (0.0022) model time 0.3813 (0.3853) loss 3.7394 (3.2127) grad_norm 1.5186 (1.5029) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][460/625] eta 0:01:04 lr 0.001211 wd 0.0500 time 0.3767 (0.3885) data time 0.0010 (0.0022) model time 0.3757 (0.3851) loss 3.3607 (3.2152) grad_norm 1.6379 (1.5025) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][470/625] eta 0:01:00 lr 0.001211 wd 0.0500 time 0.3835 (0.3883) data time 0.0008 (0.0022) model time 0.3827 (0.3849) loss 2.0837 (3.2163) grad_norm 0.9226 (1.5010) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][480/625] eta 0:00:56 lr 0.001211 wd 0.0500 time 0.3768 (0.3881) data time 0.0009 (0.0022) model time 0.3760 (0.3847) loss 3.8578 (3.2222) grad_norm 0.9432 (1.4977) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][490/625] eta 0:00:52 lr 0.001211 wd 0.0500 time 0.3720 (0.3880) data time 0.0011 (0.0022) model time 0.3709 (0.3846) loss 3.5958 (3.2259) grad_norm 1.7638 (1.4971) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][500/625] eta 0:00:48 lr 0.001211 wd 0.0500 time 0.3824 (0.3878) data time 0.0008 (0.0022) model time 0.3816 (0.3844) loss 2.7944 (3.2204) grad_norm 1.5918 (1.4964) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][510/625] eta 0:00:44 lr 0.001211 wd 0.0500 time 0.3765 (0.3886) data time 0.0009 (0.0022) model time 0.3756 (0.3853) loss 4.0712 (3.2160) grad_norm 2.2901 (1.4938) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][520/625] eta 0:00:40 lr 0.001210 wd 0.0500 time 0.3786 (0.3895) data time 0.0012 (0.0022) model time 0.3774 (0.3864) loss 3.6669 (3.2173) grad_norm 1.1314 (1.4907) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][530/625] eta 0:00:37 lr 0.001210 wd 0.0500 time 0.3876 (0.3901) data time 0.0010 (0.0021) model time 0.3866 (0.3871) loss 3.0453 (3.2184) grad_norm 1.4374 (1.4923) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][540/625] eta 0:00:33 lr 0.001210 wd 0.0500 time 0.3702 (0.3898) data time 0.0010 (0.0021) model time 0.3692 (0.3868) loss 3.2879 (3.2180) grad_norm 1.6600 (1.4912) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][550/625] eta 0:00:29 lr 0.001210 wd 0.0500 time 0.3746 (0.3896) data time 0.0012 (0.0021) model time 0.3734 (0.3866) loss 3.8991 (3.2223) grad_norm 1.0719 (1.4879) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][560/625] eta 0:00:25 lr 0.001210 wd 0.0500 time 0.3944 (0.3894) data time 0.0010 (0.0021) model time 0.3933 (0.3864) loss 3.4055 (3.2238) grad_norm 1.2311 (1.4859) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][570/625] eta 0:00:21 lr 0.001210 wd 0.0500 time 0.3755 (0.3892) data time 0.0008 (0.0021) model time 0.3747 (0.3863) loss 2.4638 (3.2222) grad_norm 1.9882 (1.4949) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][580/625] eta 0:00:17 lr 0.001209 wd 0.0500 time 0.3742 (0.3890) data time 0.0007 (0.0021) model time 0.3735 (0.3861) loss 3.7750 (3.2246) grad_norm 1.4332 (1.4942) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][590/625] eta 0:00:13 lr 0.001209 wd 0.0500 time 0.3773 (0.3889) data time 0.0011 (0.0021) model time 0.3761 (0.3859) loss 2.9269 (3.2231) grad_norm 1.5219 (1.4897) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][600/625] eta 0:00:09 lr 0.001209 wd 0.0500 time 0.3766 (0.3887) data time 0.0008 (0.0021) model time 0.3758 (0.3857) loss 2.7630 (3.2209) grad_norm 1.8997 (1.4918) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][610/625] eta 0:00:05 lr 0.001209 wd 0.0500 time 0.3652 (0.3885) data time 0.0008 (0.0021) model time 0.3644 (0.3855) loss 2.7885 (3.2189) grad_norm 1.3160 (1.4954) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [141/300][620/625] eta 0:00:01 lr 0.001209 wd 0.0500 time 0.5843 (0.3885) data time 0.0005 (0.0021) model time 0.5838 (0.3856) loss 2.7419 (3.2192) grad_norm 1.5566 (1.4948) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 141 training takes 0:04:02 [2024-08-27 08:58:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 08:58:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 08:58:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 0.6001 (0.6001) Acc@1 85.547 (85.547) Acc@5 98.096 (98.096) Mem 11224MB [2024-08-27 08:58:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.147) Loss 0.9824 (0.7285) Acc@1 76.855 (83.523) Acc@5 94.189 (96.871) Mem 11224MB [2024-08-27 08:58:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.123) Loss 1.1016 (0.8780) Acc@1 74.609 (79.950) Acc@5 92.188 (95.115) Mem 11224MB [2024-08-27 08:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.655 Acc@5 95.104 [2024-08-27 08:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.7% [2024-08-27 08:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.65% [2024-08-27 08:58:30 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 08:58:31 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 08:58:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.510 (0.510) Loss 0.5195 (0.5195) Acc@1 88.672 (88.672) Acc@5 98.340 (98.340) Mem 11224MB [2024-08-27 08:58:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 0.8633 (0.6487) Acc@1 79.639 (85.325) Acc@5 95.703 (97.399) Mem 11224MB [2024-08-27 08:58:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 0.9883 (0.7787) Acc@1 75.635 (82.120) Acc@5 94.482 (96.066) Mem 11224MB [2024-08-27 08:58:34 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.858 Acc@5 96.075 [2024-08-27 08:58:34 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.9% [2024-08-27 08:58:34 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.86% [2024-08-27 08:58:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 08:58:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 08:58:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][0/625] eta 0:08:16 lr 0.001209 wd 0.0500 time 0.7946 (0.7946) data time 0.4572 (0.4572) model time 0.0000 (0.0000) loss 3.8005 (3.8005) grad_norm 1.0030 (1.0030) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][10/625] eta 0:04:36 lr 0.001208 wd 0.0500 time 0.3920 (0.4495) data time 0.0010 (0.0430) model time 0.0000 (0.0000) loss 2.5904 (3.1153) grad_norm 1.3248 (1.4502) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][20/625] eta 0:04:11 lr 0.001208 wd 0.0500 time 0.3723 (0.4163) data time 0.0012 (0.0240) model time 0.0000 (0.0000) loss 3.1924 (3.1610) grad_norm 1.2484 (1.5731) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][30/625] eta 0:04:04 lr 0.001208 wd 0.0500 time 0.3809 (0.4114) data time 0.0011 (0.0168) model time 0.0000 (0.0000) loss 3.2454 (3.1528) grad_norm 1.2221 (1.5463) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][40/625] eta 0:03:55 lr 0.001208 wd 0.0500 time 0.3855 (0.4028) data time 0.0010 (0.0131) model time 0.0000 (0.0000) loss 3.5544 (3.1837) grad_norm 1.2075 (1.5300) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][50/625] eta 0:03:48 lr 0.001208 wd 0.0500 time 0.3770 (0.3976) data time 0.0008 (0.0108) model time 0.0000 (0.0000) loss 4.2468 (3.2374) grad_norm 1.6264 (1.5297) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:58:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][60/625] eta 0:03:42 lr 0.001208 wd 0.0500 time 0.3639 (0.3942) data time 0.0009 (0.0092) model time 0.3630 (0.3753) loss 3.4761 (3.2340) grad_norm 1.4527 (1.5152) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][70/625] eta 0:03:37 lr 0.001207 wd 0.0500 time 0.3820 (0.3916) data time 0.0008 (0.0081) model time 0.3812 (0.3752) loss 2.3248 (3.2024) grad_norm 1.9107 (1.5216) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][80/625] eta 0:03:32 lr 0.001207 wd 0.0500 time 0.3713 (0.3897) data time 0.0008 (0.0072) model time 0.3705 (0.3750) loss 2.9476 (3.2099) grad_norm 1.5885 (1.5335) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][90/625] eta 0:03:30 lr 0.001207 wd 0.0500 time 0.3749 (0.3931) data time 0.0010 (0.0065) model time 0.3739 (0.3862) loss 2.8089 (3.1805) grad_norm 1.4354 (1.5564) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][100/625] eta 0:03:27 lr 0.001207 wd 0.0500 time 0.4050 (0.3943) data time 0.0010 (0.0060) model time 0.4039 (0.3898) loss 3.6088 (3.1835) grad_norm 1.6224 (1.6478) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][110/625] eta 0:03:23 lr 0.001207 wd 0.0500 time 0.3605 (0.3945) data time 0.0010 (0.0056) model time 0.3595 (0.3906) loss 2.2231 (3.2103) grad_norm 1.6364 (1.6377) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][120/625] eta 0:03:18 lr 0.001206 wd 0.0500 time 0.3738 (0.3933) data time 0.0011 (0.0052) model time 0.3728 (0.3890) loss 3.4462 (3.2054) grad_norm 1.4057 (1.6263) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][130/625] eta 0:03:15 lr 0.001206 wd 0.0500 time 0.4032 (0.3942) data time 0.0010 (0.0049) model time 0.4021 (0.3907) loss 3.1299 (3.2318) grad_norm 4.4217 (1.6393) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][140/625] eta 0:03:10 lr 0.001206 wd 0.0500 time 0.3705 (0.3930) data time 0.0008 (0.0047) model time 0.3697 (0.3891) loss 2.4103 (3.2333) grad_norm 1.2695 (1.6351) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][150/625] eta 0:03:06 lr 0.001206 wd 0.0500 time 0.3665 (0.3917) data time 0.0008 (0.0045) model time 0.3657 (0.3875) loss 3.5040 (3.2289) grad_norm 3.0861 (1.6794) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][160/625] eta 0:03:01 lr 0.001206 wd 0.0500 time 0.3832 (0.3909) data time 0.0010 (0.0043) model time 0.3822 (0.3865) loss 3.5032 (3.2424) grad_norm 1.4479 (1.6889) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][170/625] eta 0:02:57 lr 0.001206 wd 0.0500 time 0.3798 (0.3901) data time 0.0008 (0.0041) model time 0.3790 (0.3857) loss 3.7187 (3.2460) grad_norm 1.2322 (1.6691) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][180/625] eta 0:02:53 lr 0.001205 wd 0.0500 time 0.3665 (0.3892) data time 0.0013 (0.0040) model time 0.3653 (0.3847) loss 3.4637 (3.2604) grad_norm 1.1428 (1.6542) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][190/625] eta 0:02:49 lr 0.001205 wd 0.0500 time 0.3862 (0.3888) data time 0.0011 (0.0038) model time 0.3851 (0.3843) loss 3.3486 (3.2629) grad_norm 1.0429 (1.6419) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][200/625] eta 0:02:45 lr 0.001205 wd 0.0500 time 0.3681 (0.3884) data time 0.0010 (0.0037) model time 0.3670 (0.3840) loss 3.6936 (3.2637) grad_norm 1.3588 (1.6316) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 08:59:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][210/625] eta 0:02:41 lr 0.001205 wd 0.0500 time 0.3771 (0.3896) data time 0.0008 (0.0036) model time 0.3763 (0.3857) loss 3.4413 (3.2651) grad_norm 1.7599 (1.6182) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][220/625] eta 0:02:37 lr 0.001205 wd 0.0500 time 0.4073 (0.3891) data time 0.0008 (0.0035) model time 0.4065 (0.3853) loss 3.0294 (3.2652) grad_norm 1.3884 (1.6067) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][230/625] eta 0:02:33 lr 0.001205 wd 0.0500 time 0.3759 (0.3885) data time 0.0011 (0.0034) model time 0.3748 (0.3846) loss 2.6238 (3.2628) grad_norm 1.3709 (1.6053) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][240/625] eta 0:02:29 lr 0.001204 wd 0.0500 time 0.3754 (0.3880) data time 0.0008 (0.0033) model time 0.3745 (0.3841) loss 2.2804 (3.2596) grad_norm 1.2797 (1.6071) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][250/625] eta 0:02:25 lr 0.001204 wd 0.0500 time 0.4033 (0.3876) data time 0.0009 (0.0032) model time 0.4024 (0.3837) loss 2.7140 (3.2531) grad_norm 2.1574 (1.6016) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][260/625] eta 0:02:21 lr 0.001204 wd 0.0500 time 0.3734 (0.3874) data time 0.0012 (0.0032) model time 0.3723 (0.3836) loss 3.1469 (3.2568) grad_norm 1.6792 (1.6067) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][270/625] eta 0:02:17 lr 0.001204 wd 0.0500 time 0.3708 (0.3879) data time 0.0011 (0.0031) model time 0.3697 (0.3843) loss 3.4815 (3.2628) grad_norm 1.6711 (1.6025) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][280/625] eta 0:02:14 lr 0.001204 wd 0.0500 time 0.3683 (0.3893) data time 0.0011 (0.0030) model time 0.3672 (0.3862) loss 3.0015 (3.2637) grad_norm 1.0282 (1.5967) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][290/625] eta 0:02:10 lr 0.001204 wd 0.0500 time 0.3777 (0.3896) data time 0.0010 (0.0030) model time 0.3767 (0.3866) loss 2.4787 (3.2689) grad_norm 1.4520 (1.5928) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][300/625] eta 0:02:06 lr 0.001203 wd 0.0500 time 0.4009 (0.3895) data time 0.0010 (0.0029) model time 0.3999 (0.3864) loss 3.7061 (3.2697) grad_norm 1.0827 (1.5837) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][310/625] eta 0:02:02 lr 0.001203 wd 0.0500 time 0.3694 (0.3891) data time 0.0007 (0.0029) model time 0.3686 (0.3861) loss 3.5382 (3.2702) grad_norm 1.7000 (1.5961) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][320/625] eta 0:01:58 lr 0.001203 wd 0.0500 time 0.3764 (0.3890) data time 0.0011 (0.0029) model time 0.3753 (0.3860) loss 3.2408 (3.2685) grad_norm 2.3129 (1.6103) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][330/625] eta 0:01:54 lr 0.001203 wd 0.0500 time 0.4078 (0.3893) data time 0.0011 (0.0028) model time 0.4067 (0.3864) loss 3.2746 (3.2669) grad_norm 1.3313 (1.6048) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][340/625] eta 0:01:51 lr 0.001203 wd 0.0500 time 0.3605 (0.3896) data time 0.0009 (0.0028) model time 0.3596 (0.3868) loss 3.9523 (3.2685) grad_norm 1.3449 (1.5986) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][350/625] eta 0:01:47 lr 0.001202 wd 0.0500 time 0.3741 (0.3893) data time 0.0010 (0.0028) model time 0.3731 (0.3865) loss 3.5547 (3.2669) grad_norm 1.6243 (1.5878) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][360/625] eta 0:01:43 lr 0.001202 wd 0.0500 time 0.4003 (0.3892) data time 0.0009 (0.0027) model time 0.3994 (0.3864) loss 3.3555 (3.2625) grad_norm 1.2944 (1.5822) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:00:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][370/625] eta 0:01:39 lr 0.001202 wd 0.0500 time 0.3652 (0.3888) data time 0.0010 (0.0027) model time 0.3642 (0.3860) loss 3.5311 (3.2681) grad_norm 1.5308 (1.5807) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][380/625] eta 0:01:35 lr 0.001202 wd 0.0500 time 0.3678 (0.3886) data time 0.0011 (0.0027) model time 0.3668 (0.3858) loss 2.0948 (3.2647) grad_norm 1.3417 (1.5766) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][390/625] eta 0:01:31 lr 0.001202 wd 0.0500 time 0.6149 (0.3895) data time 0.0008 (0.0026) model time 0.6141 (0.3869) loss 2.4784 (3.2614) grad_norm 1.1473 (1.5705) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][400/625] eta 0:01:27 lr 0.001202 wd 0.0500 time 0.3676 (0.3897) data time 0.0012 (0.0026) model time 0.3664 (0.3872) loss 3.1079 (3.2616) grad_norm 1.2289 (1.5678) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][410/625] eta 0:01:23 lr 0.001201 wd 0.0500 time 0.3753 (0.3900) data time 0.0009 (0.0026) model time 0.3744 (0.3875) loss 2.3937 (3.2526) grad_norm 1.2015 (1.5626) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][420/625] eta 0:01:19 lr 0.001201 wd 0.0500 time 0.3958 (0.3897) data time 0.0010 (0.0025) model time 0.3949 (0.3872) loss 3.4398 (3.2569) grad_norm 1.9245 (1.5640) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][430/625] eta 0:01:15 lr 0.001201 wd 0.0500 time 0.3772 (0.3894) data time 0.0011 (0.0025) model time 0.3762 (0.3869) loss 3.4672 (3.2559) grad_norm 1.3978 (1.5754) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][440/625] eta 0:01:11 lr 0.001201 wd 0.0500 time 0.3751 (0.3891) data time 0.0011 (0.0025) model time 0.3740 (0.3866) loss 2.8509 (3.2518) grad_norm 1.3824 (1.5817) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][450/625] eta 0:01:08 lr 0.001201 wd 0.0500 time 0.3809 (0.3888) data time 0.0010 (0.0025) model time 0.3798 (0.3863) loss 3.0625 (3.2470) grad_norm 1.9725 (1.5773) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][460/625] eta 0:01:04 lr 0.001201 wd 0.0500 time 0.3702 (0.3885) data time 0.0008 (0.0024) model time 0.3694 (0.3860) loss 3.6438 (3.2491) grad_norm 1.5836 (1.5770) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][470/625] eta 0:01:00 lr 0.001200 wd 0.0500 time 0.3711 (0.3883) data time 0.0009 (0.0024) model time 0.3702 (0.3858) loss 2.1085 (3.2465) grad_norm 1.5197 (1.5728) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][480/625] eta 0:00:56 lr 0.001200 wd 0.0500 time 0.3754 (0.3881) data time 0.0011 (0.0024) model time 0.3743 (0.3856) loss 2.7613 (3.2451) grad_norm 1.0700 (1.5679) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][490/625] eta 0:00:52 lr 0.001200 wd 0.0500 time 0.3696 (0.3878) data time 0.0008 (0.0023) model time 0.3688 (0.3853) loss 3.8164 (3.2444) grad_norm 1.4575 (1.5638) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][500/625] eta 0:00:48 lr 0.001200 wd 0.0500 time 0.3737 (0.3876) data time 0.0011 (0.0023) model time 0.3726 (0.3851) loss 3.8195 (3.2418) grad_norm 1.7893 (1.5639) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][510/625] eta 0:00:44 lr 0.001200 wd 0.0500 time 0.3766 (0.3877) data time 0.0010 (0.0023) model time 0.3755 (0.3852) loss 3.3543 (3.2449) grad_norm 1.5360 (1.5616) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][520/625] eta 0:00:40 lr 0.001200 wd 0.0500 time 0.3772 (0.3878) data time 0.0010 (0.0023) model time 0.3762 (0.3854) loss 3.2823 (3.2468) grad_norm 1.4661 (1.5594) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][530/625] eta 0:00:36 lr 0.001199 wd 0.0500 time 0.3756 (0.3877) data time 0.0008 (0.0022) model time 0.3748 (0.3853) loss 3.9269 (3.2486) grad_norm 1.2520 (1.5565) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][540/625] eta 0:00:32 lr 0.001199 wd 0.0500 time 0.3757 (0.3878) data time 0.0007 (0.0022) model time 0.3750 (0.3854) loss 2.9894 (3.2480) grad_norm 1.1931 (1.5543) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][550/625] eta 0:00:29 lr 0.001199 wd 0.0500 time 0.3788 (0.3876) data time 0.0011 (0.0022) model time 0.3777 (0.3852) loss 2.2890 (3.2487) grad_norm 1.4468 (1.5524) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][560/625] eta 0:00:25 lr 0.001199 wd 0.0500 time 0.3779 (0.3874) data time 0.0010 (0.0022) model time 0.3769 (0.3850) loss 3.3410 (3.2474) grad_norm 1.8863 (1.5551) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][570/625] eta 0:00:21 lr 0.001199 wd 0.0500 time 0.4007 (0.3873) data time 0.0008 (0.0022) model time 0.3999 (0.3849) loss 3.1817 (3.2517) grad_norm 1.3439 (1.5565) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][580/625] eta 0:00:17 lr 0.001198 wd 0.0500 time 0.3692 (0.3870) data time 0.0008 (0.0022) model time 0.3685 (0.3847) loss 4.0263 (3.2496) grad_norm 1.4950 (1.5515) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][590/625] eta 0:00:13 lr 0.001198 wd 0.0500 time 0.3672 (0.3868) data time 0.0011 (0.0022) model time 0.3661 (0.3845) loss 3.3127 (3.2521) grad_norm 1.3850 (1.5493) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][600/625] eta 0:00:09 lr 0.001198 wd 0.0500 time 0.4005 (0.3874) data time 0.0008 (0.0021) model time 0.3997 (0.3851) loss 3.8124 (3.2555) grad_norm 1.5223 (1.5467) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][610/625] eta 0:00:05 lr 0.001198 wd 0.0500 time 0.3714 (0.3877) data time 0.0008 (0.0021) model time 0.3706 (0.3854) loss 2.2203 (3.2502) grad_norm 1.2840 (1.5445) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [142/300][620/625] eta 0:00:01 lr 0.001198 wd 0.0500 time 0.3599 (0.3876) data time 0.0005 (0.0021) model time 0.3593 (0.3854) loss 2.3447 (3.2462) grad_norm 1.1768 (1.5404) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 142 training takes 0:04:02 [2024-08-27 09:02:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:02:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:02:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.6099 (0.6099) Acc@1 86.377 (86.377) Acc@5 97.998 (97.998) Mem 11224MB [2024-08-27 09:02:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 1.0068 (0.7517) Acc@1 76.221 (83.341) Acc@5 94.434 (96.857) Mem 11224MB [2024-08-27 09:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.1250 (0.8860) Acc@1 73.584 (80.069) Acc@5 93.018 (95.285) Mem 11224MB [2024-08-27 09:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.607 Acc@5 95.254 [2024-08-27 09:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.6% [2024-08-27 09:02:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.783 (0.783) Loss 0.5190 (0.5190) Acc@1 88.770 (88.770) Acc@5 98.389 (98.389) Mem 11224MB [2024-08-27 09:02:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.174) Loss 0.8643 (0.6482) Acc@1 79.785 (85.369) Acc@5 95.654 (97.408) Mem 11224MB [2024-08-27 09:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.9878 (0.7776) Acc@1 75.781 (82.143) Acc@5 94.385 (96.066) Mem 11224MB [2024-08-27 09:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.858 Acc@5 96.087 [2024-08-27 09:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.9% [2024-08-27 09:02:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][0/625] eta 0:13:40 lr 0.001198 wd 0.0500 time 1.3123 (1.3123) data time 0.7396 (0.7396) model time 0.0000 (0.0000) loss 3.5226 (3.5226) grad_norm 1.6761 (1.6761) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][10/625] eta 0:04:45 lr 0.001198 wd 0.0500 time 0.3956 (0.4649) data time 0.0007 (0.0682) model time 0.0000 (0.0000) loss 3.7058 (3.0172) grad_norm 1.2376 (1.3455) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][20/625] eta 0:04:21 lr 0.001197 wd 0.0500 time 0.3754 (0.4315) data time 0.0008 (0.0364) model time 0.0000 (0.0000) loss 3.4036 (3.1148) grad_norm 1.1821 (1.3687) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:02:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][30/625] eta 0:04:06 lr 0.001197 wd 0.0500 time 0.3869 (0.4146) data time 0.0009 (0.0252) model time 0.0000 (0.0000) loss 3.5604 (3.1038) grad_norm 1.4412 (1.4128) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:03:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][40/625] eta 0:03:57 lr 0.001197 wd 0.0500 time 0.3793 (0.4054) data time 0.0008 (0.0194) model time 0.0000 (0.0000) loss 3.1048 (3.1091) grad_norm 1.2075 (1.4680) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:03:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][50/625] eta 0:03:50 lr 0.001197 wd 0.0500 time 0.3775 (0.4001) data time 0.0008 (0.0159) model time 0.0000 (0.0000) loss 2.7832 (3.1523) grad_norm 1.0817 (1.4328) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:03:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][60/625] eta 0:03:46 lr 0.001197 wd 0.0500 time 0.3918 (0.4001) data time 0.0010 (0.0135) model time 0.3908 (0.3983) loss 2.7351 (3.1529) grad_norm 2.1795 (1.5002) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:03:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][70/625] eta 0:03:41 lr 0.001196 wd 0.0500 time 0.3751 (0.3997) data time 0.0008 (0.0118) model time 0.3742 (0.3971) loss 4.0191 (3.1654) grad_norm 1.8790 (1.5131) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:03:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][80/625] eta 0:03:36 lr 0.001196 wd 0.0500 time 0.3780 (0.3971) data time 0.0011 (0.0105) model time 0.3769 (0.3904) loss 1.9262 (3.1444) grad_norm 1.4511 (1.4979) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:03:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][90/625] eta 0:03:31 lr 0.001196 wd 0.0500 time 0.3749 (0.3945) data time 0.0007 (0.0095) model time 0.3741 (0.3860) loss 3.7146 (3.1747) grad_norm 1.9273 (1.5113) loss_scale 4096.0000 (2070.5055) mem 11224MB [2024-08-27 09:03:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][100/625] eta 0:03:26 lr 0.001196 wd 0.0500 time 0.3759 (0.3927) data time 0.0011 (0.0087) model time 0.3748 (0.3837) loss 3.3756 (3.1951) grad_norm 1.3602 (1.5009) loss_scale 4096.0000 (2271.0495) mem 11224MB [2024-08-27 09:03:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][110/625] eta 0:03:23 lr 0.001196 wd 0.0500 time 0.3700 (0.3948) data time 0.0008 (0.0080) model time 0.3692 (0.3889) loss 3.7660 (3.1895) grad_norm 1.5821 (1.4943) loss_scale 4096.0000 (2435.4595) mem 11224MB [2024-08-27 09:03:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][120/625] eta 0:03:18 lr 0.001196 wd 0.0500 time 0.3764 (0.3932) data time 0.0007 (0.0074) model time 0.3757 (0.3868) loss 3.5048 (3.1765) grad_norm 1.4142 (1.4883) loss_scale 4096.0000 (2572.6942) mem 11224MB [2024-08-27 09:03:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][130/625] eta 0:03:13 lr 0.001195 wd 0.0500 time 0.3740 (0.3918) data time 0.0010 (0.0070) model time 0.3730 (0.3853) loss 3.6376 (3.1867) grad_norm 1.3279 (1.4893) loss_scale 4096.0000 (2688.9771) mem 11224MB [2024-08-27 09:03:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][140/625] eta 0:03:10 lr 0.001195 wd 0.0500 time 0.3810 (0.3922) data time 0.0009 (0.0066) model time 0.3801 (0.3864) loss 3.9258 (3.2026) grad_norm 1.4357 (1.4878) loss_scale 4096.0000 (2788.7660) mem 11224MB [2024-08-27 09:03:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][150/625] eta 0:03:05 lr 0.001195 wd 0.0500 time 0.3713 (0.3913) data time 0.0009 (0.0062) model time 0.3704 (0.3855) loss 3.0965 (3.1861) grad_norm 1.1604 (1.4751) loss_scale 4096.0000 (2875.3377) mem 11224MB [2024-08-27 09:03:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][160/625] eta 0:03:01 lr 0.001195 wd 0.0500 time 0.3850 (0.3905) data time 0.0010 (0.0059) model time 0.3840 (0.3847) loss 3.1701 (3.1887) grad_norm 1.0394 (1.4696) loss_scale 4096.0000 (2951.1553) mem 11224MB [2024-08-27 09:03:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][170/625] eta 0:02:57 lr 0.001195 wd 0.0500 time 0.3737 (0.3898) data time 0.0008 (0.0056) model time 0.3729 (0.3842) loss 3.5224 (3.1945) grad_norm 1.6002 (1.4616) loss_scale 4096.0000 (3018.1053) mem 11224MB [2024-08-27 09:03:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][180/625] eta 0:02:54 lr 0.001195 wd 0.0500 time 0.5706 (0.3923) data time 0.0010 (0.0054) model time 0.5695 (0.3879) loss 2.2621 (3.2043) grad_norm 1.1086 (1.4491) loss_scale 4096.0000 (3077.6575) mem 11224MB [2024-08-27 09:03:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][190/625] eta 0:02:50 lr 0.001194 wd 0.0500 time 0.3780 (0.3926) data time 0.0009 (0.0052) model time 0.3771 (0.3885) loss 2.2376 (3.1858) grad_norm 1.3918 (1.4504) loss_scale 4096.0000 (3130.9738) mem 11224MB [2024-08-27 09:04:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][200/625] eta 0:02:46 lr 0.001194 wd 0.0500 time 0.3720 (0.3917) data time 0.0011 (0.0050) model time 0.3710 (0.3876) loss 3.1082 (3.1911) grad_norm 1.4962 (1.4520) loss_scale 4096.0000 (3178.9851) mem 11224MB [2024-08-27 09:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][210/625] eta 0:02:42 lr 0.001194 wd 0.0500 time 0.3821 (0.3911) data time 0.0010 (0.0048) model time 0.3810 (0.3869) loss 3.5289 (3.1897) grad_norm 1.3633 (1.4607) loss_scale 4096.0000 (3222.4455) mem 11224MB [2024-08-27 09:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][220/625] eta 0:02:38 lr 0.001194 wd 0.0500 time 0.3714 (0.3905) data time 0.0008 (0.0046) model time 0.3706 (0.3863) loss 2.4796 (3.1805) grad_norm 1.2806 (1.4657) loss_scale 4096.0000 (3261.9729) mem 11224MB [2024-08-27 09:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][230/625] eta 0:02:33 lr 0.001194 wd 0.0500 time 0.3718 (0.3898) data time 0.0010 (0.0045) model time 0.3707 (0.3856) loss 3.6004 (3.1798) grad_norm 1.3710 (1.4696) loss_scale 4096.0000 (3298.0779) mem 11224MB [2024-08-27 09:04:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][240/625] eta 0:02:29 lr 0.001193 wd 0.0500 time 0.3853 (0.3893) data time 0.0008 (0.0043) model time 0.3845 (0.3851) loss 3.4324 (3.1836) grad_norm 1.1752 (1.4623) loss_scale 4096.0000 (3331.1867) mem 11224MB [2024-08-27 09:04:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][250/625] eta 0:02:25 lr 0.001193 wd 0.0500 time 0.3753 (0.3888) data time 0.0009 (0.0042) model time 0.3744 (0.3847) loss 3.0197 (3.1812) grad_norm 1.5508 (1.4600) loss_scale 4096.0000 (3361.6574) mem 11224MB [2024-08-27 09:04:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][260/625] eta 0:02:21 lr 0.001193 wd 0.0500 time 0.3708 (0.3884) data time 0.0011 (0.0041) model time 0.3697 (0.3843) loss 3.3806 (3.1782) grad_norm 2.4702 (1.4727) loss_scale 4096.0000 (3389.7931) mem 11224MB [2024-08-27 09:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][270/625] eta 0:02:18 lr 0.001193 wd 0.0500 time 0.3808 (0.3893) data time 0.0010 (0.0040) model time 0.3798 (0.3856) loss 3.5809 (3.1721) grad_norm 1.4533 (1.4855) loss_scale 4096.0000 (3415.8524) mem 11224MB [2024-08-27 09:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][280/625] eta 0:02:14 lr 0.001193 wd 0.0500 time 0.3764 (0.3889) data time 0.0009 (0.0039) model time 0.3756 (0.3853) loss 2.4984 (3.1782) grad_norm 1.5222 (1.4816) loss_scale 4096.0000 (3440.0569) mem 11224MB [2024-08-27 09:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][290/625] eta 0:02:10 lr 0.001193 wd 0.0500 time 0.3753 (0.3886) data time 0.0010 (0.0038) model time 0.3742 (0.3850) loss 3.7816 (3.1818) grad_norm 1.2302 (1.4845) loss_scale 4096.0000 (3462.5979) mem 11224MB [2024-08-27 09:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][300/625] eta 0:02:06 lr 0.001192 wd 0.0500 time 0.3862 (0.3890) data time 0.0008 (0.0037) model time 0.3854 (0.3855) loss 3.9786 (3.1865) grad_norm 1.2082 (1.4839) loss_scale 4096.0000 (3483.6412) mem 11224MB [2024-08-27 09:04:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][310/625] eta 0:02:02 lr 0.001192 wd 0.0500 time 0.5799 (0.3894) data time 0.0011 (0.0036) model time 0.5789 (0.3860) loss 3.6400 (3.1845) grad_norm 0.9776 (1.4797) loss_scale 4096.0000 (3503.3312) mem 11224MB [2024-08-27 09:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][320/625] eta 0:01:58 lr 0.001192 wd 0.0500 time 0.3800 (0.3898) data time 0.0010 (0.0036) model time 0.3789 (0.3866) loss 3.4273 (3.1939) grad_norm 1.6902 (1.4800) loss_scale 4096.0000 (3521.7944) mem 11224MB [2024-08-27 09:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][330/625] eta 0:01:54 lr 0.001192 wd 0.0500 time 0.3768 (0.3894) data time 0.0010 (0.0035) model time 0.3757 (0.3862) loss 3.8826 (3.2008) grad_norm 1.1023 (1.4771) loss_scale 4096.0000 (3539.1420) mem 11224MB [2024-08-27 09:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][340/625] eta 0:01:50 lr 0.001192 wd 0.0500 time 0.3725 (0.3890) data time 0.0008 (0.0034) model time 0.3717 (0.3858) loss 3.9888 (3.1968) grad_norm 1.5667 (1.4792) loss_scale 4096.0000 (3555.4721) mem 11224MB [2024-08-27 09:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][350/625] eta 0:01:47 lr 0.001192 wd 0.0500 time 0.3832 (0.3894) data time 0.0008 (0.0034) model time 0.3824 (0.3864) loss 3.8435 (3.1965) grad_norm 1.1981 (1.4833) loss_scale 4096.0000 (3570.8718) mem 11224MB [2024-08-27 09:05:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][360/625] eta 0:01:43 lr 0.001191 wd 0.0500 time 0.5452 (0.3897) data time 0.0010 (0.0033) model time 0.5442 (0.3867) loss 3.6274 (3.2022) grad_norm 1.2763 (1.4800) loss_scale 4096.0000 (3585.4183) mem 11224MB [2024-08-27 09:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][370/625] eta 0:01:39 lr 0.001191 wd 0.0500 time 0.3742 (0.3900) data time 0.0009 (0.0033) model time 0.3733 (0.3871) loss 1.8884 (3.1988) grad_norm 1.5914 (1.4829) loss_scale 4096.0000 (3599.1806) mem 11224MB [2024-08-27 09:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][380/625] eta 0:01:35 lr 0.001191 wd 0.0500 time 0.3688 (0.3897) data time 0.0010 (0.0032) model time 0.3678 (0.3869) loss 2.2872 (3.1927) grad_norm 1.8193 (1.4829) loss_scale 4096.0000 (3612.2205) mem 11224MB [2024-08-27 09:05:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][390/625] eta 0:01:31 lr 0.001191 wd 0.0500 time 0.3955 (0.3895) data time 0.0011 (0.0032) model time 0.3944 (0.3866) loss 3.7253 (3.2017) grad_norm 3.5291 (1.4932) loss_scale 4096.0000 (3624.5934) mem 11224MB [2024-08-27 09:05:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][400/625] eta 0:01:27 lr 0.001191 wd 0.0500 time 0.3758 (0.3896) data time 0.0008 (0.0031) model time 0.3750 (0.3868) loss 3.9255 (3.2074) grad_norm 1.2049 (1.4953) loss_scale 4096.0000 (3636.3491) mem 11224MB [2024-08-27 09:05:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][410/625] eta 0:01:23 lr 0.001191 wd 0.0500 time 0.3729 (0.3894) data time 0.0009 (0.0031) model time 0.3720 (0.3866) loss 2.5623 (3.2003) grad_norm 1.6732 (1.4952) loss_scale 4096.0000 (3647.5328) mem 11224MB [2024-08-27 09:05:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][420/625] eta 0:01:19 lr 0.001190 wd 0.0500 time 0.3801 (0.3891) data time 0.0009 (0.0031) model time 0.3792 (0.3863) loss 3.8475 (3.2018) grad_norm 1.2889 (1.4931) loss_scale 4096.0000 (3658.1853) mem 11224MB [2024-08-27 09:05:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][430/625] eta 0:01:15 lr 0.001190 wd 0.0500 time 0.3820 (0.3889) data time 0.0011 (0.0030) model time 0.3810 (0.3861) loss 3.5904 (3.2080) grad_norm 1.6412 (1.4939) loss_scale 4096.0000 (3668.3434) mem 11224MB [2024-08-27 09:05:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][440/625] eta 0:01:11 lr 0.001190 wd 0.0500 time 0.3778 (0.3887) data time 0.0008 (0.0030) model time 0.3770 (0.3859) loss 4.0493 (3.2059) grad_norm 1.9620 (1.4938) loss_scale 4096.0000 (3678.0408) mem 11224MB [2024-08-27 09:05:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][450/625] eta 0:01:08 lr 0.001190 wd 0.0500 time 0.3843 (0.3888) data time 0.0007 (0.0030) model time 0.3836 (0.3861) loss 2.5853 (3.2067) grad_norm 1.2114 (1.4963) loss_scale 4096.0000 (3687.3082) mem 11224MB [2024-08-27 09:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][460/625] eta 0:01:04 lr 0.001190 wd 0.0500 time 0.3727 (0.3886) data time 0.0010 (0.0029) model time 0.3717 (0.3858) loss 3.9128 (3.2071) grad_norm 1.2353 (1.4959) loss_scale 4096.0000 (3696.1735) mem 11224MB [2024-08-27 09:05:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][470/625] eta 0:01:00 lr 0.001189 wd 0.0500 time 0.3707 (0.3884) data time 0.0011 (0.0029) model time 0.3696 (0.3856) loss 3.7584 (3.2090) grad_norm 1.2079 (1.5007) loss_scale 4096.0000 (3704.6624) mem 11224MB [2024-08-27 09:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][480/625] eta 0:00:56 lr 0.001189 wd 0.0500 time 0.5628 (0.3885) data time 0.0010 (0.0029) model time 0.5618 (0.3858) loss 2.9553 (3.2123) grad_norm 1.1656 (1.4946) loss_scale 4096.0000 (3712.7983) mem 11224MB [2024-08-27 09:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][490/625] eta 0:00:52 lr 0.001189 wd 0.0500 time 0.3630 (0.3887) data time 0.0010 (0.0028) model time 0.3620 (0.3860) loss 2.2981 (3.2149) grad_norm 0.9446 (1.4905) loss_scale 4096.0000 (3720.6029) mem 11224MB [2024-08-27 09:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][500/625] eta 0:00:48 lr 0.001189 wd 0.0500 time 0.5706 (0.3896) data time 0.0011 (0.0028) model time 0.5695 (0.3871) loss 3.3027 (3.2162) grad_norm 1.3150 (1.4911) loss_scale 4096.0000 (3728.0958) mem 11224MB [2024-08-27 09:06:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][510/625] eta 0:00:44 lr 0.001189 wd 0.0500 time 0.3772 (0.3897) data time 0.0008 (0.0028) model time 0.3764 (0.3873) loss 3.5400 (3.2197) grad_norm 1.7059 (1.4937) loss_scale 4096.0000 (3735.2955) mem 11224MB [2024-08-27 09:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][520/625] eta 0:00:40 lr 0.001189 wd 0.0500 time 0.3686 (0.3895) data time 0.0010 (0.0027) model time 0.3675 (0.3870) loss 3.3477 (3.2199) grad_norm 1.7310 (1.4968) loss_scale 4096.0000 (3742.2188) mem 11224MB [2024-08-27 09:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][530/625] eta 0:00:37 lr 0.001188 wd 0.0500 time 0.3823 (0.3896) data time 0.0012 (0.0027) model time 0.3811 (0.3872) loss 2.9442 (3.2166) grad_norm 1.7725 (1.5007) loss_scale 4096.0000 (3748.8814) mem 11224MB [2024-08-27 09:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][540/625] eta 0:00:33 lr 0.001188 wd 0.0500 time 0.3720 (0.3894) data time 0.0011 (0.0027) model time 0.3709 (0.3870) loss 2.5458 (3.2184) grad_norm 1.1894 (1.4993) loss_scale 4096.0000 (3755.2976) mem 11224MB [2024-08-27 09:06:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][550/625] eta 0:00:29 lr 0.001188 wd 0.0500 time 0.3718 (0.3891) data time 0.0009 (0.0027) model time 0.3708 (0.3866) loss 2.5887 (3.2205) grad_norm 1.1813 (nan) loss_scale 2048.0000 (3728.0290) mem 11224MB [2024-08-27 09:06:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][560/625] eta 0:00:25 lr 0.001188 wd 0.0500 time 0.3664 (0.3889) data time 0.0008 (0.0026) model time 0.3656 (0.3864) loss 3.5280 (3.2240) grad_norm 1.4438 (nan) loss_scale 2048.0000 (3698.0820) mem 11224MB [2024-08-27 09:06:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][570/625] eta 0:00:21 lr 0.001188 wd 0.0500 time 0.3783 (0.3890) data time 0.0008 (0.0026) model time 0.3776 (0.3866) loss 3.6112 (3.2264) grad_norm 1.9538 (nan) loss_scale 2048.0000 (3669.1839) mem 11224MB [2024-08-27 09:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][580/625] eta 0:00:17 lr 0.001188 wd 0.0500 time 0.3774 (0.3890) data time 0.0011 (0.0026) model time 0.3763 (0.3866) loss 3.4713 (3.2314) grad_norm 1.1463 (nan) loss_scale 2048.0000 (3641.2806) mem 11224MB [2024-08-27 09:06:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][590/625] eta 0:00:13 lr 0.001187 wd 0.0500 time 0.3870 (0.3889) data time 0.0009 (0.0026) model time 0.3862 (0.3865) loss 3.4746 (3.2342) grad_norm 1.2893 (nan) loss_scale 2048.0000 (3614.3215) mem 11224MB [2024-08-27 09:06:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][600/625] eta 0:00:09 lr 0.001187 wd 0.0500 time 0.3805 (0.3887) data time 0.0008 (0.0026) model time 0.3797 (0.3864) loss 3.3073 (3.2343) grad_norm 1.6372 (nan) loss_scale 2048.0000 (3588.2596) mem 11224MB [2024-08-27 09:06:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][610/625] eta 0:00:05 lr 0.001187 wd 0.0500 time 0.3679 (0.3885) data time 0.0008 (0.0026) model time 0.3671 (0.3861) loss 3.8189 (3.2375) grad_norm 1.4388 (nan) loss_scale 2048.0000 (3563.0507) mem 11224MB [2024-08-27 09:06:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [143/300][620/625] eta 0:00:01 lr 0.001187 wd 0.0500 time 0.5692 (0.3892) data time 0.0006 (0.0025) model time 0.5686 (0.3869) loss 4.0966 (3.2434) grad_norm 1.1528 (nan) loss_scale 2048.0000 (3538.6538) mem 11224MB [2024-08-27 09:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 143 training takes 0:04:03 [2024-08-27 09:06:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:06:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:06:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 0.6572 (0.6572) Acc@1 86.182 (86.182) Acc@5 98.047 (98.047) Mem 11224MB [2024-08-27 09:06:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 1.0088 (0.7731) Acc@1 76.758 (83.438) Acc@5 94.287 (96.786) Mem 11224MB [2024-08-27 09:06:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 1.1270 (0.9100) Acc@1 73.340 (79.901) Acc@5 93.359 (95.122) Mem 11224MB [2024-08-27 09:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.688 Acc@5 95.142 [2024-08-27 09:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.7% [2024-08-27 09:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.69% [2024-08-27 09:06:52 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 09:06:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 09:06:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.498 (0.498) Loss 0.5186 (0.5186) Acc@1 88.672 (88.672) Acc@5 98.438 (98.438) Mem 11224MB [2024-08-27 09:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.142) Loss 0.8623 (0.6473) Acc@1 79.736 (85.409) Acc@5 95.703 (97.430) Mem 11224MB [2024-08-27 09:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9844 (0.7763) Acc@1 76.123 (82.182) Acc@5 94.482 (96.098) Mem 11224MB [2024-08-27 09:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.908 Acc@5 96.121 [2024-08-27 09:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.9% [2024-08-27 09:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.91% [2024-08-27 09:06:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 09:06:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 09:06:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][0/625] eta 0:08:03 lr 0.001187 wd 0.0500 time 0.7735 (0.7735) data time 0.4234 (0.4234) model time 0.0000 (0.0000) loss 3.1225 (3.1225) grad_norm 1.2785 (1.2785) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:07:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][10/625] eta 0:04:15 lr 0.001187 wd 0.0500 time 0.3829 (0.4152) data time 0.0010 (0.0396) model time 0.0000 (0.0000) loss 3.7628 (3.1870) grad_norm 1.4067 (1.5615) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:07:04 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-27 09:07:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:07:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-27 09:10:53 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-27 09:11:00 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-27 09:11:16 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-27 09:11:16 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-27 09:11:19 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-27 09:11:21 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-27 09:11:21 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 144) [2024-08-27 09:11:21 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-27 09:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-27 09:11:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:11:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:13:28 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-27 09:13:29 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-27 09:13:43 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-27 09:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-27 09:13:57 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-27 09:13:59 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-27 09:14:01 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-27 09:14:01 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 144) [2024-08-27 09:14:01 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-27 09:14:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][20/625] eta 2:33:35 lr 0.001186 wd 0.0500 time 15.2324 (15.2324) data time 0.6893 (0.6893) model time 0.0000 (0.0000) loss 3.9395 (3.9395) grad_norm 1.2862 (1.2862) loss_scale 2048.0000 (2048.0000) mem 23875MB [2024-08-27 09:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][30/625] eta 0:17:15 lr 0.001186 wd 0.0500 time 0.3718 (1.7401) data time 0.0010 (0.0636) model time 0.0000 (0.0000) loss 2.8166 (3.6120) grad_norm 1.6003 (1.4030) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][40/625] eta 0:10:37 lr 0.001186 wd 0.0500 time 0.3719 (1.0896) data time 0.0011 (0.0338) model time 0.0000 (0.0000) loss 3.3875 (3.5365) grad_norm 1.8160 (1.4575) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][50/625] eta 0:08:17 lr 0.001186 wd 0.0500 time 0.3707 (0.8652) data time 0.0009 (0.0233) model time 0.0000 (0.0000) loss 2.4945 (3.5079) grad_norm 2.4591 (1.4948) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][60/625] eta 0:07:04 lr 0.001186 wd 0.0500 time 0.6596 (0.7519) data time 0.0011 (0.0179) model time 0.6585 (0.3995) loss 3.2725 (3.4371) grad_norm 1.6473 (1.5482) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][70/625] eta 0:06:16 lr 0.001186 wd 0.0500 time 0.3863 (0.6779) data time 0.0008 (0.0146) model time 0.3855 (0.3864) loss 4.1244 (3.4333) grad_norm 1.2114 (1.5188) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][80/625] eta 0:05:42 lr 0.001185 wd 0.0500 time 0.3775 (0.6279) data time 0.0012 (0.0124) model time 0.3763 (0.3816) loss 3.0370 (3.3973) grad_norm 1.2420 (1.5227) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][90/625] eta 0:05:16 lr 0.001185 wd 0.0500 time 0.3713 (0.5920) data time 0.0012 (0.0108) model time 0.3701 (0.3792) loss 3.1732 (3.3580) grad_norm 1.9675 (1.5259) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][100/625] eta 0:04:56 lr 0.001185 wd 0.0500 time 0.3770 (0.5650) data time 0.0010 (0.0096) model time 0.3761 (0.3778) loss 2.9809 (3.3303) grad_norm 2.2673 (1.5153) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][110/625] eta 0:04:40 lr 0.001185 wd 0.0500 time 0.3745 (0.5439) data time 0.0009 (0.0087) model time 0.3737 (0.3768) loss 3.7287 (3.3059) grad_norm 1.4094 (1.5120) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:14:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][120/625] eta 0:04:26 lr 0.001185 wd 0.0500 time 0.3748 (0.5274) data time 0.0008 (0.0079) model time 0.3740 (0.3767) loss 3.5563 (3.3175) grad_norm 1.4516 (1.5181) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][130/625] eta 0:04:14 lr 0.001184 wd 0.0500 time 0.3678 (0.5133) data time 0.0013 (0.0073) model time 0.3665 (0.3758) loss 2.9149 (3.3225) grad_norm 1.3128 (1.5318) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][140/625] eta 0:04:03 lr 0.001184 wd 0.0500 time 0.3652 (0.5013) data time 0.0009 (0.0068) model time 0.3643 (0.3748) loss 2.1462 (3.3200) grad_norm 1.2094 (1.5160) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][150/625] eta 0:03:53 lr 0.001184 wd 0.0500 time 0.3728 (0.4914) data time 0.0012 (0.0064) model time 0.3716 (0.3744) loss 3.6593 (3.3128) grad_norm 1.1661 (1.4988) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][160/625] eta 0:03:44 lr 0.001184 wd 0.0500 time 0.3859 (0.4829) data time 0.0009 (0.0060) model time 0.3851 (0.3740) loss 3.1828 (3.3023) grad_norm 1.6883 (1.4991) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][170/625] eta 0:03:36 lr 0.001184 wd 0.0500 time 0.3699 (0.4754) data time 0.0009 (0.0057) model time 0.3690 (0.3736) loss 2.1535 (3.2942) grad_norm 1.5189 (1.5166) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][180/625] eta 0:03:28 lr 0.001184 wd 0.0500 time 0.3715 (0.4691) data time 0.0010 (0.0054) model time 0.3705 (0.3735) loss 3.5670 (3.3045) grad_norm 3.1314 (1.5320) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][190/625] eta 0:03:21 lr 0.001183 wd 0.0500 time 0.3776 (0.4634) data time 0.0010 (0.0051) model time 0.3766 (0.3733) loss 3.3668 (3.3041) grad_norm 1.5674 (1.5300) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][200/625] eta 0:03:14 lr 0.001183 wd 0.0500 time 0.3718 (0.4585) data time 0.0011 (0.0049) model time 0.3708 (0.3733) loss 3.3910 (3.2865) grad_norm 1.5242 (1.5223) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][210/625] eta 0:03:08 lr 0.001183 wd 0.0500 time 0.3719 (0.4540) data time 0.0011 (0.0047) model time 0.3708 (0.3732) loss 2.9777 (3.2853) grad_norm 1.1962 (1.5178) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][220/625] eta 0:03:02 lr 0.001183 wd 0.0500 time 0.3710 (0.4500) data time 0.0010 (0.0045) model time 0.3700 (0.3732) loss 3.1476 (3.2781) grad_norm 1.1983 (1.5172) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][230/625] eta 0:02:56 lr 0.001183 wd 0.0500 time 0.3708 (0.4463) data time 0.0011 (0.0044) model time 0.3698 (0.3731) loss 3.4982 (3.2723) grad_norm 1.2416 (1.5063) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][240/625] eta 0:02:50 lr 0.001183 wd 0.0500 time 0.3833 (0.4430) data time 0.0008 (0.0042) model time 0.3825 (0.3730) loss 3.1832 (3.2662) grad_norm 1.0659 (1.5009) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][250/625] eta 0:02:45 lr 0.001182 wd 0.0500 time 0.3796 (0.4419) data time 0.0007 (0.0041) model time 0.3788 (0.3752) loss 2.0244 (3.2598) grad_norm 1.3736 (1.5035) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][260/625] eta 0:02:40 lr 0.001182 wd 0.0500 time 0.3633 (0.4398) data time 0.0009 (0.0039) model time 0.3625 (0.3759) loss 3.3662 (3.2566) grad_norm 2.2471 (1.5109) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][270/625] eta 0:02:35 lr 0.001182 wd 0.0500 time 0.3720 (0.4372) data time 0.0009 (0.0038) model time 0.3711 (0.3758) loss 3.3397 (3.2487) grad_norm 1.6809 (1.5172) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:15:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][280/625] eta 0:02:29 lr 0.001182 wd 0.0500 time 0.3716 (0.4346) data time 0.0008 (0.0037) model time 0.3708 (0.3755) loss 3.2009 (3.2422) grad_norm 1.4597 (1.5276) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][290/625] eta 0:02:24 lr 0.001182 wd 0.0500 time 0.3675 (0.4324) data time 0.0007 (0.0036) model time 0.3668 (0.3754) loss 3.6543 (3.2404) grad_norm 1.4768 (1.5213) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][300/625] eta 0:02:19 lr 0.001182 wd 0.0500 time 0.3772 (0.4303) data time 0.0011 (0.0035) model time 0.3760 (0.3753) loss 3.3571 (3.2444) grad_norm 1.5851 (1.5147) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][310/625] eta 0:02:14 lr 0.001181 wd 0.0500 time 0.3790 (0.4285) data time 0.0008 (0.0035) model time 0.3782 (0.3754) loss 1.8575 (3.2325) grad_norm 1.4860 (1.5046) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][320/625] eta 0:02:10 lr 0.001181 wd 0.0500 time 0.3738 (0.4266) data time 0.0012 (0.0034) model time 0.3726 (0.3752) loss 3.0113 (3.2203) grad_norm 1.1332 (1.4985) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][330/625] eta 0:02:05 lr 0.001181 wd 0.0500 time 0.3722 (0.4249) data time 0.0011 (0.0033) model time 0.3712 (0.3750) loss 3.6990 (3.2197) grad_norm 1.1111 (1.4938) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][340/625] eta 0:02:00 lr 0.001181 wd 0.0500 time 0.3757 (0.4233) data time 0.0008 (0.0033) model time 0.3749 (0.3749) loss 4.2457 (3.2315) grad_norm 1.7001 (1.4923) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][350/625] eta 0:01:55 lr 0.001181 wd 0.0500 time 0.3710 (0.4218) data time 0.0008 (0.0032) model time 0.3703 (0.3749) loss 3.1233 (3.2370) grad_norm 1.1747 (1.4859) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][360/625] eta 0:01:51 lr 0.001180 wd 0.0500 time 0.3696 (0.4204) data time 0.0010 (0.0031) model time 0.3685 (0.3748) loss 3.5212 (3.2398) grad_norm 1.0139 (1.4799) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][370/625] eta 0:01:47 lr 0.001180 wd 0.0500 time 0.3793 (0.4203) data time 0.0008 (0.0031) model time 0.3784 (0.3761) loss 3.7063 (3.2417) grad_norm 2.4856 (1.4867) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][380/625] eta 0:01:42 lr 0.001180 wd 0.0500 time 0.3801 (0.4190) data time 0.0007 (0.0030) model time 0.3793 (0.3759) loss 3.1144 (3.2429) grad_norm 1.4585 (1.4855) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][390/625] eta 0:01:38 lr 0.001180 wd 0.0500 time 0.3792 (0.4177) data time 0.0010 (0.0030) model time 0.3782 (0.3758) loss 2.6439 (3.2414) grad_norm 1.7545 (1.4857) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][400/625] eta 0:01:33 lr 0.001180 wd 0.0500 time 0.3797 (0.4166) data time 0.0008 (0.0029) model time 0.3789 (0.3758) loss 2.0618 (3.2393) grad_norm 2.4478 (1.4923) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][410/625] eta 0:01:29 lr 0.001180 wd 0.0500 time 0.3704 (0.4155) data time 0.0009 (0.0029) model time 0.3695 (0.3757) loss 4.3019 (3.2351) grad_norm 1.4366 (1.5006) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][420/625] eta 0:01:24 lr 0.001179 wd 0.0500 time 0.3715 (0.4145) data time 0.0010 (0.0028) model time 0.3704 (0.3757) loss 3.3747 (3.2398) grad_norm 1.9488 (1.5020) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][430/625] eta 0:01:20 lr 0.001179 wd 0.0500 time 0.3744 (0.4135) data time 0.0010 (0.0028) model time 0.3733 (0.3756) loss 3.4934 (3.2473) grad_norm 1.6183 (1.5023) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][440/625] eta 0:01:16 lr 0.001179 wd 0.0500 time 0.3707 (0.4130) data time 0.0008 (0.0027) model time 0.3699 (0.3760) loss 3.5686 (3.2471) grad_norm 1.6619 (1.5038) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][450/625] eta 0:01:12 lr 0.001179 wd 0.0500 time 0.3698 (0.4121) data time 0.0007 (0.0027) model time 0.3691 (0.3758) loss 3.8361 (3.2540) grad_norm 1.1620 (1.5077) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][460/625] eta 0:01:07 lr 0.001179 wd 0.0500 time 0.3685 (0.4112) data time 0.0008 (0.0026) model time 0.3677 (0.3758) loss 3.9403 (3.2566) grad_norm 1.8892 (1.5130) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][470/625] eta 0:01:03 lr 0.001179 wd 0.0500 time 0.3741 (0.4103) data time 0.0009 (0.0026) model time 0.3733 (0.3757) loss 3.4059 (3.2568) grad_norm 1.9672 (1.5167) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][480/625] eta 0:00:59 lr 0.001178 wd 0.0500 time 0.3768 (0.4095) data time 0.0011 (0.0026) model time 0.3758 (0.3756) loss 3.1529 (3.2492) grad_norm 1.6460 (1.5113) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][490/625] eta 0:00:55 lr 0.001178 wd 0.0500 time 0.3734 (0.4088) data time 0.0010 (0.0025) model time 0.3724 (0.3756) loss 3.1534 (3.2410) grad_norm 1.3834 (1.5088) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][500/625] eta 0:00:51 lr 0.001178 wd 0.0500 time 0.3710 (0.4081) data time 0.0008 (0.0025) model time 0.3703 (0.3755) loss 3.7675 (3.2386) grad_norm 1.2058 (1.5096) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][510/625] eta 0:00:46 lr 0.001178 wd 0.0500 time 0.3740 (0.4074) data time 0.0011 (0.0025) model time 0.3729 (0.3754) loss 3.4300 (3.2455) grad_norm 1.2841 (1.5073) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][520/625] eta 0:00:42 lr 0.001178 wd 0.0500 time 0.3744 (0.4067) data time 0.0010 (0.0025) model time 0.3734 (0.3754) loss 3.3996 (3.2436) grad_norm 2.1571 (1.5084) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][530/625] eta 0:00:38 lr 0.001177 wd 0.0500 time 0.3727 (0.4061) data time 0.0007 (0.0024) model time 0.3720 (0.3754) loss 4.1567 (3.2485) grad_norm 1.7621 (1.5110) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][540/625] eta 0:00:34 lr 0.001177 wd 0.0500 time 0.3733 (0.4055) data time 0.0008 (0.0024) model time 0.3725 (0.3753) loss 2.8252 (3.2480) grad_norm 1.5684 (1.5151) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][550/625] eta 0:00:30 lr 0.001177 wd 0.0500 time 0.3688 (0.4048) data time 0.0012 (0.0024) model time 0.3676 (0.3752) loss 3.3400 (3.2413) grad_norm 1.2170 (1.5156) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][560/625] eta 0:00:26 lr 0.001177 wd 0.0500 time 0.3708 (0.4046) data time 0.0011 (0.0023) model time 0.3698 (0.3756) loss 3.4955 (3.2407) grad_norm 1.3694 (1.5131) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][570/625] eta 0:00:22 lr 0.001177 wd 0.0500 time 0.3741 (0.4041) data time 0.0007 (0.0023) model time 0.3733 (0.3755) loss 3.8031 (3.2406) grad_norm 1.8980 (1.5198) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][580/625] eta 0:00:18 lr 0.001177 wd 0.0500 time 0.3716 (0.4043) data time 0.0010 (0.0023) model time 0.3706 (0.3763) loss 3.2868 (3.2454) grad_norm 0.9570 (1.5194) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:17:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][590/625] eta 0:00:14 lr 0.001176 wd 0.0500 time 0.3771 (0.4038) data time 0.0010 (0.0023) model time 0.3761 (0.3762) loss 4.0479 (3.2474) grad_norm 2.8585 (1.5185) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:18:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][600/625] eta 0:00:10 lr 0.001176 wd 0.0500 time 0.3748 (0.4032) data time 0.0008 (0.0023) model time 0.3740 (0.3761) loss 4.0418 (3.2480) grad_norm 1.4694 (1.5174) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:18:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][610/625] eta 0:00:06 lr 0.001176 wd 0.0500 time 0.3565 (0.4027) data time 0.0007 (0.0022) model time 0.3558 (0.3760) loss 3.7152 (3.2506) grad_norm 2.5001 (1.5146) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:18:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [144/300][620/625] eta 0:00:02 lr 0.001176 wd 0.0500 time 0.3593 (0.4019) data time 0.0007 (0.0022) model time 0.3586 (0.3757) loss 2.0643 (3.2507) grad_norm 1.2811 (1.5127) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 144 training takes 0:04:03 [2024-08-27 09:18:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:18:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:18:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.489 (0.489) Loss 0.5645 (0.5645) Acc@1 87.451 (87.451) Acc@5 97.949 (97.949) Mem 11219MB [2024-08-27 09:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.142) Loss 0.9771 (0.7062) Acc@1 77.246 (83.749) Acc@5 94.092 (96.924) Mem 11219MB [2024-08-27 09:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.0889 (0.8567) Acc@1 73.975 (80.169) Acc@5 93.262 (95.245) Mem 11219MB [2024-08-27 09:18:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.894 Acc@5 95.228 [2024-08-27 09:18:16 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.9% [2024-08-27 09:18:16 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.89% [2024-08-27 09:18:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 09:18:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 09:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.489 (0.489) Loss 0.5181 (0.5181) Acc@1 88.770 (88.770) Acc@5 98.438 (98.438) Mem 11219MB [2024-08-27 09:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 0.8608 (0.6470) Acc@1 79.639 (85.383) Acc@5 95.508 (97.421) Mem 11219MB [2024-08-27 09:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9834 (0.7755) Acc@1 75.928 (82.203) Acc@5 94.482 (96.117) Mem 11219MB [2024-08-27 09:18:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.934 Acc@5 96.135 [2024-08-27 09:18:22 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.9% [2024-08-27 09:18:22 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.93% [2024-08-27 09:18:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 09:18:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 09:18:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][0/625] eta 0:08:25 lr 0.001176 wd 0.0500 time 0.8096 (0.8096) data time 0.4236 (0.4236) model time 0.0000 (0.0000) loss 3.0751 (3.0751) grad_norm 0.9390 (0.9390) loss_scale 2048.0000 (2048.0000) mem 11227MB [2024-08-27 09:18:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][10/625] eta 0:04:13 lr 0.001176 wd 0.0500 time 0.3800 (0.4125) data time 0.0010 (0.0395) model time 0.0000 (0.0000) loss 3.0058 (3.2609) grad_norm 1.1893 (1.2677) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:18:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][20/625] eta 0:03:58 lr 0.001175 wd 0.0500 time 0.3694 (0.3936) data time 0.0010 (0.0212) model time 0.0000 (0.0000) loss 2.4959 (3.3286) grad_norm 0.9072 (1.2756) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:18:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][30/625] eta 0:03:50 lr 0.001175 wd 0.0500 time 0.3738 (0.3872) data time 0.0009 (0.0147) model time 0.0000 (0.0000) loss 3.0960 (3.3128) grad_norm 2.8538 (1.5121) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:18:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][40/625] eta 0:03:44 lr 0.001175 wd 0.0500 time 0.3982 (0.3844) data time 0.0010 (0.0113) model time 0.0000 (0.0000) loss 3.5674 (3.2551) grad_norm 2.2744 (1.7143) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:18:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][50/625] eta 0:03:39 lr 0.001175 wd 0.0500 time 0.3817 (0.3826) data time 0.0015 (0.0093) model time 0.0000 (0.0000) loss 2.4579 (3.2283) grad_norm 2.2188 (1.7008) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:18:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][60/625] eta 0:03:36 lr 0.001175 wd 0.0500 time 0.3766 (0.3831) data time 0.0008 (0.0080) model time 0.3758 (0.3851) loss 3.7728 (3.1975) grad_norm 1.0245 (1.6810) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:18:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][70/625] eta 0:03:32 lr 0.001175 wd 0.0500 time 0.3757 (0.3820) data time 0.0007 (0.0070) model time 0.3750 (0.3795) loss 2.7125 (3.2289) grad_norm 1.3845 (1.6608) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:18:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][80/625] eta 0:03:27 lr 0.001174 wd 0.0500 time 0.3710 (0.3811) data time 0.0008 (0.0062) model time 0.3702 (0.3776) loss 2.8178 (3.2375) grad_norm 1.4331 (1.6376) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][90/625] eta 0:03:23 lr 0.001174 wd 0.0500 time 0.3683 (0.3804) data time 0.0007 (0.0057) model time 0.3676 (0.3767) loss 2.6720 (3.2230) grad_norm 1.9500 (1.6289) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][100/625] eta 0:03:19 lr 0.001174 wd 0.0500 time 0.3775 (0.3799) data time 0.0009 (0.0052) model time 0.3766 (0.3761) loss 3.6141 (3.2249) grad_norm 1.2712 (1.6046) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][110/625] eta 0:03:15 lr 0.001174 wd 0.0500 time 0.3729 (0.3794) data time 0.0011 (0.0048) model time 0.3718 (0.3757) loss 2.9685 (3.1856) grad_norm 2.2659 (1.5835) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][120/625] eta 0:03:11 lr 0.001174 wd 0.0500 time 0.3722 (0.3790) data time 0.0008 (0.0045) model time 0.3714 (0.3754) loss 3.2847 (3.1810) grad_norm 1.7314 (1.5776) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][130/625] eta 0:03:07 lr 0.001174 wd 0.0500 time 0.3688 (0.3784) data time 0.0007 (0.0042) model time 0.3680 (0.3748) loss 3.8461 (3.1953) grad_norm 1.9011 (1.5674) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][140/625] eta 0:03:03 lr 0.001173 wd 0.0500 time 0.3798 (0.3781) data time 0.0008 (0.0040) model time 0.3790 (0.3746) loss 3.7444 (3.2032) grad_norm 1.1845 (1.5635) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][150/625] eta 0:03:00 lr 0.001173 wd 0.0500 time 0.3791 (0.3791) data time 0.0008 (0.0038) model time 0.3782 (0.3763) loss 3.5600 (3.2001) grad_norm 1.3460 (1.5469) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][160/625] eta 0:02:56 lr 0.001173 wd 0.0500 time 0.3745 (0.3799) data time 0.0010 (0.0037) model time 0.3735 (0.3776) loss 3.3369 (3.2157) grad_norm 1.1735 (1.5326) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][170/625] eta 0:02:52 lr 0.001173 wd 0.0500 time 0.3670 (0.3795) data time 0.0010 (0.0035) model time 0.3660 (0.3773) loss 3.6320 (3.2205) grad_norm 1.3096 (1.5260) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][180/625] eta 0:02:48 lr 0.001173 wd 0.0500 time 0.3676 (0.3792) data time 0.0010 (0.0034) model time 0.3665 (0.3768) loss 3.4565 (3.2161) grad_norm 1.5975 (1.5230) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][190/625] eta 0:02:44 lr 0.001173 wd 0.0500 time 0.3764 (0.3789) data time 0.0008 (0.0032) model time 0.3756 (0.3765) loss 1.9681 (3.2169) grad_norm 1.4685 (1.5307) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][200/625] eta 0:02:40 lr 0.001172 wd 0.0500 time 0.3680 (0.3786) data time 0.0008 (0.0031) model time 0.3673 (0.3762) loss 2.2518 (3.2142) grad_norm 1.7933 (1.5294) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][210/625] eta 0:02:37 lr 0.001172 wd 0.0500 time 0.3789 (0.3785) data time 0.0007 (0.0030) model time 0.3782 (0.3762) loss 3.9183 (3.2075) grad_norm 1.0814 (1.5198) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][220/625] eta 0:02:33 lr 0.001172 wd 0.0500 time 0.3718 (0.3782) data time 0.0012 (0.0029) model time 0.3707 (0.3759) loss 2.2368 (3.1943) grad_norm 1.2641 (1.5151) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][230/625] eta 0:02:29 lr 0.001172 wd 0.0500 time 0.3671 (0.3779) data time 0.0008 (0.0028) model time 0.3663 (0.3756) loss 4.0579 (3.1900) grad_norm 1.5070 (1.5070) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][240/625] eta 0:02:25 lr 0.001172 wd 0.0500 time 0.5295 (0.3783) data time 0.0008 (0.0028) model time 0.5287 (0.3762) loss 3.4148 (3.1783) grad_norm 1.7544 (1.5084) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][250/625] eta 0:02:21 lr 0.001171 wd 0.0500 time 0.3751 (0.3779) data time 0.0010 (0.0027) model time 0.3740 (0.3757) loss 4.0539 (3.1857) grad_norm 1.3486 (1.5038) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][260/625] eta 0:02:17 lr 0.001171 wd 0.0500 time 0.3725 (0.3778) data time 0.0011 (0.0026) model time 0.3714 (0.3756) loss 3.3075 (3.1783) grad_norm 1.2539 (1.5024) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][270/625] eta 0:02:14 lr 0.001171 wd 0.0500 time 0.3703 (0.3776) data time 0.0009 (0.0026) model time 0.3694 (0.3755) loss 3.0588 (3.1814) grad_norm 1.8346 (1.5076) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][280/625] eta 0:02:10 lr 0.001171 wd 0.0500 time 0.3929 (0.3776) data time 0.0007 (0.0025) model time 0.3921 (0.3755) loss 3.8325 (3.1799) grad_norm 1.5615 (1.5058) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][290/625] eta 0:02:06 lr 0.001171 wd 0.0500 time 0.3754 (0.3775) data time 0.0013 (0.0025) model time 0.3741 (0.3754) loss 3.1492 (3.1813) grad_norm 1.5753 (1.5085) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][300/625] eta 0:02:02 lr 0.001171 wd 0.0500 time 0.3790 (0.3774) data time 0.0010 (0.0024) model time 0.3780 (0.3753) loss 3.1125 (3.1813) grad_norm 1.8930 (1.5177) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][310/625] eta 0:01:58 lr 0.001170 wd 0.0500 time 0.3710 (0.3773) data time 0.0010 (0.0024) model time 0.3701 (0.3752) loss 4.0310 (3.1797) grad_norm 2.2926 (1.5275) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][320/625] eta 0:01:55 lr 0.001170 wd 0.0500 time 0.3735 (0.3772) data time 0.0010 (0.0023) model time 0.3726 (0.3752) loss 2.9798 (3.1883) grad_norm 1.4726 (1.5592) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][330/625] eta 0:01:51 lr 0.001170 wd 0.0500 time 0.3741 (0.3772) data time 0.0008 (0.0023) model time 0.3733 (0.3752) loss 2.1391 (3.1884) grad_norm 0.9310 (1.5579) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][340/625] eta 0:01:47 lr 0.001170 wd 0.0500 time 0.3740 (0.3782) data time 0.0011 (0.0023) model time 0.3729 (0.3764) loss 3.7867 (3.1886) grad_norm 1.3690 (1.5544) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][350/625] eta 0:01:43 lr 0.001170 wd 0.0500 time 0.3676 (0.3780) data time 0.0008 (0.0022) model time 0.3668 (0.3763) loss 3.5226 (3.1811) grad_norm 1.3677 (1.5464) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][360/625] eta 0:01:40 lr 0.001170 wd 0.0500 time 0.5880 (0.3786) data time 0.0014 (0.0022) model time 0.5866 (0.3769) loss 3.7929 (3.1822) grad_norm 1.1732 (1.5372) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][370/625] eta 0:01:36 lr 0.001169 wd 0.0500 time 0.3819 (0.3784) data time 0.0011 (0.0022) model time 0.3808 (0.3767) loss 3.4263 (3.1901) grad_norm 1.6467 (1.5336) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][380/625] eta 0:01:32 lr 0.001169 wd 0.0500 time 0.3780 (0.3783) data time 0.0011 (0.0021) model time 0.3769 (0.3766) loss 2.2468 (3.1843) grad_norm 1.9802 (1.5394) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][390/625] eta 0:01:28 lr 0.001169 wd 0.0500 time 0.3730 (0.3781) data time 0.0010 (0.0021) model time 0.3720 (0.3765) loss 3.6544 (3.1884) grad_norm 1.3440 (1.5426) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:20:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][400/625] eta 0:01:25 lr 0.001169 wd 0.0500 time 0.3668 (0.3780) data time 0.0012 (0.0021) model time 0.3656 (0.3763) loss 3.7983 (3.1901) grad_norm 1.4095 (1.5388) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][410/625] eta 0:01:21 lr 0.001169 wd 0.0500 time 0.3774 (0.3779) data time 0.0008 (0.0020) model time 0.3766 (0.3762) loss 2.7921 (3.1965) grad_norm 1.8205 (1.5389) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][420/625] eta 0:01:17 lr 0.001168 wd 0.0500 time 0.3786 (0.3778) data time 0.0010 (0.0020) model time 0.3775 (0.3761) loss 3.0067 (3.1876) grad_norm 1.2389 (1.5422) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][430/625] eta 0:01:13 lr 0.001168 wd 0.0500 time 0.3809 (0.3777) data time 0.0008 (0.0020) model time 0.3802 (0.3760) loss 3.9647 (3.1892) grad_norm 1.6138 (1.5420) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][440/625] eta 0:01:09 lr 0.001168 wd 0.0500 time 0.3665 (0.3777) data time 0.0010 (0.0020) model time 0.3655 (0.3761) loss 3.5938 (3.1942) grad_norm 1.4808 (1.5386) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][450/625] eta 0:01:06 lr 0.001168 wd 0.0500 time 0.3730 (0.3776) data time 0.0009 (0.0020) model time 0.3721 (0.3760) loss 2.9298 (3.1989) grad_norm 0.9965 (1.5357) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][460/625] eta 0:01:02 lr 0.001168 wd 0.0500 time 0.3803 (0.3785) data time 0.0007 (0.0019) model time 0.3795 (0.3770) loss 2.4989 (3.2002) grad_norm 1.2014 (1.5363) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][470/625] eta 0:00:58 lr 0.001168 wd 0.0500 time 0.3699 (0.3784) data time 0.0009 (0.0019) model time 0.3689 (0.3769) loss 2.9794 (3.1946) grad_norm 1.1730 (1.5329) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][480/625] eta 0:00:54 lr 0.001167 wd 0.0500 time 0.3796 (0.3784) data time 0.0010 (0.0019) model time 0.3787 (0.3769) loss 3.3905 (3.1957) grad_norm 1.4778 (1.5315) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][490/625] eta 0:00:51 lr 0.001167 wd 0.0500 time 0.3798 (0.3783) data time 0.0009 (0.0019) model time 0.3789 (0.3768) loss 4.1067 (3.1942) grad_norm 1.5743 (1.5324) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][500/625] eta 0:00:47 lr 0.001167 wd 0.0500 time 0.3677 (0.3783) data time 0.0011 (0.0019) model time 0.3665 (0.3768) loss 3.4116 (3.1945) grad_norm 1.2464 (1.5302) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][510/625] eta 0:00:43 lr 0.001167 wd 0.0500 time 0.3769 (0.3782) data time 0.0010 (0.0018) model time 0.3759 (0.3767) loss 3.4866 (3.1996) grad_norm 1.7329 (1.5275) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][520/625] eta 0:00:39 lr 0.001167 wd 0.0500 time 0.3744 (0.3781) data time 0.0010 (0.0018) model time 0.3734 (0.3767) loss 3.4686 (3.1983) grad_norm 1.0300 (1.5230) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][530/625] eta 0:00:35 lr 0.001167 wd 0.0500 time 0.3781 (0.3781) data time 0.0010 (0.0018) model time 0.3772 (0.3766) loss 3.6179 (3.1982) grad_norm 1.1174 (1.5188) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][540/625] eta 0:00:32 lr 0.001166 wd 0.0500 time 0.3733 (0.3781) data time 0.0008 (0.0018) model time 0.3725 (0.3766) loss 2.7231 (3.1973) grad_norm 1.8231 (1.5171) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:21:54 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-27 09:21:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:21:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:23:48 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-27 09:23:49 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-27 09:24:02 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-27 09:24:14 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-27 09:24:14 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-27 09:24:17 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-27 09:24:19 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-27 09:24:19 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 145) [2024-08-27 09:24:19 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-27 09:24:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][550/625] eta 0:16:06 lr 0.001166 wd 0.0500 time 12.8868 (12.8868) data time 0.7558 (0.7558) model time 12.1311 (12.1311) loss 4.1311 (4.1311) grad_norm 1.5255 (1.5255) loss_scale 2048.0000 (2048.0000) mem 23875MB [2024-08-27 09:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][560/625] eta 0:01:37 lr 0.001166 wd 0.0500 time 0.3450 (1.4987) data time 0.0009 (0.0696) model time 0.3442 (1.4291) loss 2.8735 (3.6263) grad_norm 1.5661 (1.4595) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:24:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][570/625] eta 0:00:52 lr 0.001166 wd 0.0500 time 0.3439 (0.9501) data time 0.0009 (0.0369) model time 0.3430 (0.9132) loss 3.4139 (3.4927) grad_norm 1.5674 (1.4952) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:24:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][580/625] eta 0:00:34 lr 0.001166 wd 0.0500 time 0.3422 (0.7625) data time 0.0007 (0.0253) model time 0.3414 (0.7372) loss 2.4955 (3.4864) grad_norm 3.0547 (1.5651) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:24:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][590/625] eta 0:00:23 lr 0.001165 wd 0.0500 time 0.6506 (0.6681) data time 0.0008 (0.0194) model time 0.6497 (0.6487) loss 3.5703 (3.4318) grad_norm 1.3230 (1.6335) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:24:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][600/625] eta 0:00:15 lr 0.001165 wd 0.0500 time 0.3492 (0.6051) data time 0.0007 (0.0158) model time 0.3485 (0.5893) loss 3.5098 (3.4186) grad_norm 1.6387 (1.6170) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:24:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][610/625] eta 0:00:08 lr 0.001165 wd 0.0500 time 0.3319 (0.5621) data time 0.0006 (0.0134) model time 0.3313 (0.5487) loss 3.6054 (3.3807) grad_norm 1.9387 (1.6443) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:25:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [145/300][620/625] eta 0:00:02 lr 0.001165 wd 0.0500 time 0.3359 (0.5304) data time 0.0007 (0.0116) model time 0.3352 (0.5188) loss 3.1264 (3.3421) grad_norm 1.1941 (1.7331) loss_scale 2048.0000 (2048.0000) mem 11219MB [2024-08-27 09:25:02 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 145 training takes 0:00:39 [2024-08-27 09:25:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:25:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:25:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.424 (0.424) Loss 0.5928 (0.5928) Acc@1 86.768 (86.768) Acc@5 97.998 (97.998) Mem 11219MB [2024-08-27 09:25:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.128) Loss 1.0166 (0.7508) Acc@1 75.293 (83.079) Acc@5 94.287 (96.653) Mem 11219MB [2024-08-27 09:25:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 1.1250 (0.8953) Acc@1 73.291 (79.476) Acc@5 92.334 (94.947) Mem 11219MB [2024-08-27 09:25:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.251 Acc@5 94.980 [2024-08-27 09:25:11 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.3% [2024-08-27 09:25:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.876 (0.876) Loss 0.5186 (0.5186) Acc@1 88.721 (88.721) Acc@5 98.438 (98.438) Mem 11219MB [2024-08-27 09:25:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.169) Loss 0.8584 (0.6459) Acc@1 79.932 (85.449) Acc@5 95.361 (97.403) Mem 11219MB [2024-08-27 09:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.132) Loss 0.9824 (0.7741) Acc@1 75.830 (82.252) Acc@5 94.678 (96.122) Mem 11219MB [2024-08-27 09:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.976 Acc@5 96.145 [2024-08-27 09:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.0% [2024-08-27 09:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.98% [2024-08-27 09:25:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 09:25:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 09:25:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][0/625] eta 0:07:25 lr 0.001165 wd 0.0500 time 0.7134 (0.7134) data time 0.3471 (0.3471) model time 0.0000 (0.0000) loss 3.4413 (3.4413) grad_norm 1.9679 (1.9679) loss_scale 2048.0000 (2048.0000) mem 11227MB [2024-08-27 09:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][10/625] eta 0:03:52 lr 0.001165 wd 0.0500 time 0.3464 (0.3779) data time 0.0006 (0.0324) model time 0.0000 (0.0000) loss 2.5380 (3.1326) grad_norm 1.4484 (1.5244) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][20/625] eta 0:03:39 lr 0.001165 wd 0.0500 time 0.3420 (0.3620) data time 0.0007 (0.0174) model time 0.0000 (0.0000) loss 3.6435 (3.2766) grad_norm 1.2382 (1.4636) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][30/625] eta 0:03:32 lr 0.001164 wd 0.0500 time 0.3444 (0.3566) data time 0.0009 (0.0121) model time 0.0000 (0.0000) loss 3.4035 (3.3708) grad_norm 1.5430 (1.5291) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][40/625] eta 0:03:26 lr 0.001164 wd 0.0500 time 0.3411 (0.3536) data time 0.0007 (0.0094) model time 0.0000 (0.0000) loss 4.3330 (3.3531) grad_norm 2.9395 (1.5398) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][50/625] eta 0:03:22 lr 0.001164 wd 0.0500 time 0.3393 (0.3516) data time 0.0007 (0.0077) model time 0.0000 (0.0000) loss 2.5769 (3.3428) grad_norm 1.3943 (1.5242) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][60/625] eta 0:03:18 lr 0.001164 wd 0.0500 time 0.3441 (0.3507) data time 0.0008 (0.0066) model time 0.3433 (0.3446) loss 3.0164 (3.3406) grad_norm 1.6295 (1.4865) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][70/625] eta 0:03:14 lr 0.001164 wd 0.0500 time 0.3414 (0.3498) data time 0.0007 (0.0058) model time 0.3407 (0.3440) loss 2.4040 (3.3042) grad_norm 1.0281 (1.4464) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][80/625] eta 0:03:10 lr 0.001163 wd 0.0500 time 0.3454 (0.3493) data time 0.0010 (0.0052) model time 0.3444 (0.3443) loss 2.9979 (3.2927) grad_norm 1.5039 (1.4577) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][90/625] eta 0:03:06 lr 0.001163 wd 0.0500 time 0.3429 (0.3489) data time 0.0008 (0.0047) model time 0.3421 (0.3443) loss 2.9922 (3.2786) grad_norm 1.2412 (1.5082) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][100/625] eta 0:03:03 lr 0.001163 wd 0.0500 time 0.3084 (0.3495) data time 0.0007 (0.0044) model time 0.3077 (0.3464) loss 3.3274 (3.2668) grad_norm 1.2451 (1.5572) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:25:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][110/625] eta 0:02:59 lr 0.001163 wd 0.0500 time 0.3458 (0.3492) data time 0.0008 (0.0041) model time 0.3450 (0.3462) loss 3.1693 (3.2507) grad_norm 1.0570 (1.5489) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][120/625] eta 0:02:56 lr 0.001163 wd 0.0500 time 0.3488 (0.3490) data time 0.0007 (0.0038) model time 0.3480 (0.3461) loss 2.8833 (3.2343) grad_norm 1.4351 (1.5444) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][130/625] eta 0:02:52 lr 0.001163 wd 0.0500 time 0.3467 (0.3487) data time 0.0006 (0.0036) model time 0.3460 (0.3458) loss 2.4852 (3.2146) grad_norm 1.2653 (1.5435) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][140/625] eta 0:02:48 lr 0.001162 wd 0.0500 time 0.3454 (0.3484) data time 0.0007 (0.0034) model time 0.3447 (0.3456) loss 2.2795 (3.2127) grad_norm 1.7398 (1.5326) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][150/625] eta 0:02:45 lr 0.001162 wd 0.0500 time 0.3477 (0.3482) data time 0.0006 (0.0032) model time 0.3471 (0.3454) loss 2.7295 (3.2214) grad_norm 1.5395 (1.5195) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][160/625] eta 0:02:41 lr 0.001162 wd 0.0500 time 0.3456 (0.3479) data time 0.0007 (0.0031) model time 0.3449 (0.3453) loss 3.7639 (3.2164) grad_norm 1.1103 (1.5200) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][170/625] eta 0:02:38 lr 0.001162 wd 0.0500 time 0.3488 (0.3494) data time 0.0007 (0.0030) model time 0.3481 (0.3474) loss 1.8326 (3.2138) grad_norm 1.5951 (1.5255) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][180/625] eta 0:02:35 lr 0.001162 wd 0.0500 time 0.3442 (0.3491) data time 0.0009 (0.0028) model time 0.3434 (0.3472) loss 2.4090 (3.2019) grad_norm 1.3450 (1.5395) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][190/625] eta 0:02:31 lr 0.001162 wd 0.0500 time 0.3445 (0.3489) data time 0.0007 (0.0027) model time 0.3438 (0.3470) loss 3.0106 (3.1918) grad_norm 1.6550 (1.5376) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][200/625] eta 0:02:28 lr 0.001161 wd 0.0500 time 0.3424 (0.3488) data time 0.0008 (0.0027) model time 0.3416 (0.3468) loss 3.4627 (3.1964) grad_norm 1.4635 (1.5320) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][210/625] eta 0:02:24 lr 0.001161 wd 0.0500 time 0.3489 (0.3486) data time 0.0009 (0.0026) model time 0.3480 (0.3467) loss 3.2471 (3.1911) grad_norm 1.3378 (1.5278) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][220/625] eta 0:02:21 lr 0.001161 wd 0.0500 time 0.3459 (0.3484) data time 0.0006 (0.0025) model time 0.3453 (0.3465) loss 2.6546 (3.1865) grad_norm 1.6112 (1.5293) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][230/625] eta 0:02:17 lr 0.001161 wd 0.0500 time 0.3512 (0.3483) data time 0.0007 (0.0024) model time 0.3506 (0.3464) loss 2.3860 (3.1759) grad_norm 2.3444 (1.5428) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][240/625] eta 0:02:14 lr 0.001161 wd 0.0500 time 0.3477 (0.3482) data time 0.0007 (0.0024) model time 0.3470 (0.3463) loss 3.7226 (3.1868) grad_norm 1.5818 (1.5534) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][250/625] eta 0:02:10 lr 0.001160 wd 0.0500 time 0.3558 (0.3482) data time 0.0009 (0.0023) model time 0.3549 (0.3464) loss 3.3585 (3.1990) grad_norm 1.1576 (1.5472) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][260/625] eta 0:02:07 lr 0.001160 wd 0.0500 time 0.3466 (0.3482) data time 0.0009 (0.0023) model time 0.3457 (0.3464) loss 3.0905 (3.1932) grad_norm 1.5139 (1.5397) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][270/625] eta 0:02:03 lr 0.001160 wd 0.0500 time 0.3413 (0.3481) data time 0.0008 (0.0022) model time 0.3405 (0.3464) loss 3.9319 (3.1956) grad_norm 1.3785 (1.5402) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:26:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][280/625] eta 0:02:00 lr 0.001160 wd 0.0500 time 0.3463 (0.3481) data time 0.0006 (0.0022) model time 0.3457 (0.3464) loss 2.2543 (3.1955) grad_norm 2.1455 (1.5375) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][290/625] eta 0:01:56 lr 0.001160 wd 0.0500 time 0.3461 (0.3485) data time 0.0009 (0.0021) model time 0.3452 (0.3469) loss 3.7043 (3.1956) grad_norm 2.0352 (1.5382) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][300/625] eta 0:01:53 lr 0.001160 wd 0.0500 time 0.3480 (0.3485) data time 0.0006 (0.0021) model time 0.3474 (0.3469) loss 2.1899 (3.1921) grad_norm 1.5157 (1.5669) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][310/625] eta 0:01:49 lr 0.001159 wd 0.0500 time 0.3430 (0.3484) data time 0.0008 (0.0021) model time 0.3422 (0.3468) loss 3.2473 (3.1900) grad_norm 1.2243 (1.5600) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][320/625] eta 0:01:46 lr 0.001159 wd 0.0500 time 0.3456 (0.3483) data time 0.0007 (0.0020) model time 0.3449 (0.3467) loss 3.6828 (3.1943) grad_norm 1.1618 (1.5526) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][330/625] eta 0:01:42 lr 0.001159 wd 0.0500 time 0.3386 (0.3482) data time 0.0006 (0.0020) model time 0.3380 (0.3466) loss 3.8187 (3.2009) grad_norm 1.4817 (1.5430) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][340/625] eta 0:01:39 lr 0.001159 wd 0.0500 time 0.3465 (0.3481) data time 0.0007 (0.0020) model time 0.3459 (0.3465) loss 2.7219 (3.2030) grad_norm 1.2937 (1.5375) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][350/625] eta 0:01:35 lr 0.001159 wd 0.0500 time 0.3452 (0.3480) data time 0.0009 (0.0019) model time 0.3443 (0.3464) loss 3.6475 (3.2055) grad_norm 1.4203 (1.5359) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][360/625] eta 0:01:32 lr 0.001159 wd 0.0500 time 0.3457 (0.3491) data time 0.0009 (0.0019) model time 0.3449 (0.3477) loss 2.9662 (3.2141) grad_norm 1.7593 (1.5363) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][370/625] eta 0:01:28 lr 0.001158 wd 0.0500 time 0.3450 (0.3490) data time 0.0007 (0.0019) model time 0.3444 (0.3477) loss 3.4823 (3.2163) grad_norm 1.6714 (1.5399) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][380/625] eta 0:01:25 lr 0.001158 wd 0.0500 time 0.3473 (0.3489) data time 0.0007 (0.0018) model time 0.3466 (0.3476) loss 3.3156 (3.2133) grad_norm 2.5783 (1.5548) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][390/625] eta 0:01:21 lr 0.001158 wd 0.0500 time 0.3449 (0.3489) data time 0.0007 (0.0018) model time 0.3442 (0.3475) loss 2.7470 (3.2087) grad_norm 1.5328 (1.5638) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][400/625] eta 0:01:18 lr 0.001158 wd 0.0500 time 0.3491 (0.3488) data time 0.0007 (0.0018) model time 0.3484 (0.3474) loss 2.7637 (3.2032) grad_norm 1.5435 (1.5663) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][410/625] eta 0:01:15 lr 0.001158 wd 0.0500 time 0.3414 (0.3491) data time 0.0010 (0.0018) model time 0.3405 (0.3478) loss 3.2400 (3.2084) grad_norm 2.0074 (1.5731) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][420/625] eta 0:01:11 lr 0.001157 wd 0.0500 time 0.3483 (0.3491) data time 0.0007 (0.0018) model time 0.3477 (0.3478) loss 2.9595 (3.2096) grad_norm 1.5090 (1.5702) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][430/625] eta 0:01:08 lr 0.001157 wd 0.0500 time 0.3432 (0.3490) data time 0.0009 (0.0017) model time 0.3423 (0.3477) loss 3.2751 (3.2116) grad_norm 1.4692 (1.5710) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][440/625] eta 0:01:04 lr 0.001157 wd 0.0500 time 0.3401 (0.3490) data time 0.0008 (0.0017) model time 0.3392 (0.3477) loss 2.6386 (3.2163) grad_norm 2.2281 (1.5849) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:27:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][450/625] eta 0:01:01 lr 0.001157 wd 0.0500 time 0.3466 (0.3490) data time 0.0007 (0.0017) model time 0.3459 (0.3477) loss 2.3772 (3.2097) grad_norm 0.9953 (1.5794) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][460/625] eta 0:00:57 lr 0.001157 wd 0.0500 time 0.3461 (0.3489) data time 0.0008 (0.0017) model time 0.3453 (0.3476) loss 2.9766 (3.2091) grad_norm 1.4655 (1.5759) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][470/625] eta 0:00:54 lr 0.001157 wd 0.0500 time 0.3396 (0.3488) data time 0.0007 (0.0017) model time 0.3390 (0.3475) loss 3.0817 (3.2082) grad_norm 2.0349 (1.5906) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][480/625] eta 0:00:50 lr 0.001156 wd 0.0500 time 0.3438 (0.3497) data time 0.0008 (0.0017) model time 0.3430 (0.3485) loss 3.6999 (3.2170) grad_norm 1.2273 (1.5960) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][490/625] eta 0:00:47 lr 0.001156 wd 0.0500 time 0.3414 (0.3496) data time 0.0006 (0.0016) model time 0.3408 (0.3484) loss 2.4875 (3.2182) grad_norm 1.3176 (1.5932) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][500/625] eta 0:00:43 lr 0.001156 wd 0.0500 time 0.3444 (0.3495) data time 0.0008 (0.0016) model time 0.3436 (0.3483) loss 2.7003 (3.2191) grad_norm 1.0632 (1.5875) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][510/625] eta 0:00:40 lr 0.001156 wd 0.0500 time 0.3449 (0.3494) data time 0.0009 (0.0016) model time 0.3441 (0.3482) loss 3.6917 (3.2229) grad_norm 0.8225 (1.5827) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][520/625] eta 0:00:36 lr 0.001156 wd 0.0500 time 0.3441 (0.3493) data time 0.0007 (0.0016) model time 0.3435 (0.3481) loss 2.1559 (3.2230) grad_norm 1.7965 (1.5790) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][530/625] eta 0:00:33 lr 0.001156 wd 0.0500 time 0.3407 (0.3493) data time 0.0007 (0.0016) model time 0.3400 (0.3481) loss 2.7662 (3.2220) grad_norm 1.5299 (1.5758) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][540/625] eta 0:00:29 lr 0.001155 wd 0.0500 time 0.3451 (0.3492) data time 0.0009 (0.0016) model time 0.3441 (0.3480) loss 3.2916 (3.2226) grad_norm 2.0235 (1.5799) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][550/625] eta 0:00:26 lr 0.001155 wd 0.0500 time 0.3535 (0.3492) data time 0.0009 (0.0016) model time 0.3526 (0.3480) loss 2.8129 (3.2262) grad_norm 1.2138 (1.5762) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-08-27 09:28:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][560/625] eta 0:00:22 lr 0.001155 wd 0.0500 time 0.3415 (0.3491) data time 0.0006 (0.0015) model time 0.3409 (0.3479) loss 2.9931 (3.2281) grad_norm 1.1883 (inf) loss_scale 1024.0000 (2035.2228) mem 11224MB [2024-08-27 09:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][570/625] eta 0:00:19 lr 0.001155 wd 0.0500 time 0.3406 (0.3490) data time 0.0009 (0.0015) model time 0.3396 (0.3478) loss 3.3071 (3.2240) grad_norm 1.2078 (inf) loss_scale 1024.0000 (2017.5131) mem 11224MB [2024-08-27 09:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][580/625] eta 0:00:15 lr 0.001155 wd 0.0500 time 0.3461 (0.3489) data time 0.0010 (0.0015) model time 0.3451 (0.3477) loss 2.3036 (3.2236) grad_norm 1.3367 (inf) loss_scale 1024.0000 (2000.4131) mem 11224MB [2024-08-27 09:28:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][590/625] eta 0:00:12 lr 0.001155 wd 0.0500 time 0.3428 (0.3489) data time 0.0009 (0.0015) model time 0.3418 (0.3477) loss 3.4932 (3.2184) grad_norm 1.9041 (inf) loss_scale 1024.0000 (1983.8917) mem 11224MB [2024-08-27 09:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][600/625] eta 0:00:08 lr 0.001154 wd 0.0500 time 0.3434 (0.3488) data time 0.0006 (0.0015) model time 0.3428 (0.3476) loss 2.6329 (3.2207) grad_norm 1.4321 (inf) loss_scale 1024.0000 (1967.9201) mem 11224MB [2024-08-27 09:28:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][610/625] eta 0:00:05 lr 0.001154 wd 0.0500 time 0.3420 (0.3487) data time 0.0005 (0.0015) model time 0.3415 (0.3475) loss 2.9579 (3.2208) grad_norm 1.0464 (inf) loss_scale 1024.0000 (1952.4714) mem 11224MB [2024-08-27 09:28:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [146/300][620/625] eta 0:00:01 lr 0.001154 wd 0.0500 time 0.3399 (0.3488) data time 0.0003 (0.0015) model time 0.3395 (0.3476) loss 2.6809 (3.2162) grad_norm 1.6708 (inf) loss_scale 1024.0000 (1937.5201) mem 11224MB [2024-08-27 09:28:57 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 146 training takes 0:03:37 [2024-08-27 09:28:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:28:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:28:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.438 (0.438) Loss 0.5938 (0.5938) Acc@1 86.719 (86.719) Acc@5 98.193 (98.193) Mem 11224MB [2024-08-27 09:28:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.129) Loss 1.0166 (0.7219) Acc@1 75.049 (83.252) Acc@5 93.945 (96.804) Mem 11224MB [2024-08-27 09:29:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.111) Loss 1.1035 (0.8748) Acc@1 73.682 (79.806) Acc@5 93.359 (95.108) Mem 11224MB [2024-08-27 09:29:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.563 Acc@5 95.118 [2024-08-27 09:29:01 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.6% [2024-08-27 09:29:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.819 (0.819) Loss 0.5181 (0.5181) Acc@1 88.672 (88.672) Acc@5 98.486 (98.486) Mem 11224MB [2024-08-27 09:29:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.166) Loss 0.8579 (0.6451) Acc@1 79.834 (85.427) Acc@5 95.459 (97.421) Mem 11224MB [2024-08-27 09:29:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.130) Loss 0.9824 (0.7731) Acc@1 75.928 (82.220) Acc@5 94.629 (96.129) Mem 11224MB [2024-08-27 09:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.958 Acc@5 96.149 [2024-08-27 09:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.0% [2024-08-27 09:29:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][0/625] eta 0:12:31 lr 0.001154 wd 0.0500 time 1.2030 (1.2030) data time 0.6096 (0.6096) model time 0.0000 (0.0000) loss 3.7488 (3.7488) grad_norm 1.3188 (1.3188) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][10/625] eta 0:04:20 lr 0.001154 wd 0.0500 time 0.3442 (0.4236) data time 0.0007 (0.0563) model time 0.0000 (0.0000) loss 1.9695 (3.1727) grad_norm 0.8835 (1.4252) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][20/625] eta 0:03:53 lr 0.001154 wd 0.0500 time 0.3534 (0.3867) data time 0.0007 (0.0299) model time 0.0000 (0.0000) loss 3.6445 (3.1717) grad_norm 1.8284 (1.6016) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][30/625] eta 0:03:46 lr 0.001153 wd 0.0500 time 0.5548 (0.3801) data time 0.0008 (0.0206) model time 0.0000 (0.0000) loss 3.6063 (3.2228) grad_norm 1.4210 (1.5161) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][40/625] eta 0:03:42 lr 0.001153 wd 0.0500 time 0.3442 (0.3812) data time 0.0008 (0.0158) model time 0.0000 (0.0000) loss 3.4305 (3.2991) grad_norm 1.4841 (1.5640) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][50/625] eta 0:03:35 lr 0.001153 wd 0.0500 time 0.3414 (0.3743) data time 0.0008 (0.0129) model time 0.0000 (0.0000) loss 3.6234 (3.2735) grad_norm 1.2408 (1.5647) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][60/625] eta 0:03:28 lr 0.001153 wd 0.0500 time 0.3480 (0.3693) data time 0.0008 (0.0109) model time 0.3472 (0.3432) loss 3.2819 (3.2417) grad_norm 2.4159 (1.6126) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][70/625] eta 0:03:23 lr 0.001153 wd 0.0500 time 0.3409 (0.3658) data time 0.0006 (0.0095) model time 0.3402 (0.3433) loss 4.3223 (3.2763) grad_norm 1.2624 (1.6223) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][80/625] eta 0:03:17 lr 0.001152 wd 0.0500 time 0.3427 (0.3633) data time 0.0006 (0.0085) model time 0.3421 (0.3436) loss 3.9489 (3.2758) grad_norm 1.6054 (1.5906) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][90/625] eta 0:03:13 lr 0.001152 wd 0.0500 time 0.3459 (0.3613) data time 0.0008 (0.0077) model time 0.3451 (0.3439) loss 2.8255 (3.2703) grad_norm 0.9717 (1.5977) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][100/625] eta 0:03:09 lr 0.001152 wd 0.0500 time 0.3481 (0.3611) data time 0.0006 (0.0070) model time 0.3475 (0.3467) loss 3.6527 (3.2639) grad_norm 1.6277 (1.6021) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][110/625] eta 0:03:05 lr 0.001152 wd 0.0500 time 0.3453 (0.3598) data time 0.0007 (0.0064) model time 0.3446 (0.3465) loss 1.9530 (3.2140) grad_norm 1.2435 (1.5976) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][120/625] eta 0:03:01 lr 0.001152 wd 0.0500 time 0.3390 (0.3585) data time 0.0007 (0.0060) model time 0.3383 (0.3461) loss 3.5132 (3.2133) grad_norm 1.4232 (1.5790) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][130/625] eta 0:02:56 lr 0.001152 wd 0.0500 time 0.3407 (0.3576) data time 0.0008 (0.0056) model time 0.3399 (0.3460) loss 2.3390 (3.1949) grad_norm 1.3483 (1.5764) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][140/625] eta 0:02:52 lr 0.001151 wd 0.0500 time 0.3468 (0.3567) data time 0.0007 (0.0053) model time 0.3461 (0.3458) loss 3.3935 (3.1930) grad_norm 2.0823 (1.5645) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][150/625] eta 0:02:49 lr 0.001151 wd 0.0500 time 0.3453 (0.3560) data time 0.0006 (0.0050) model time 0.3446 (0.3458) loss 3.8001 (3.1882) grad_norm 1.3392 (1.5622) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][160/625] eta 0:02:45 lr 0.001151 wd 0.0500 time 0.3440 (0.3554) data time 0.0008 (0.0047) model time 0.3432 (0.3457) loss 3.6015 (3.1890) grad_norm 1.1846 (1.5456) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][170/625] eta 0:02:41 lr 0.001151 wd 0.0500 time 0.3438 (0.3549) data time 0.0006 (0.0045) model time 0.3432 (0.3458) loss 3.3561 (3.1926) grad_norm 1.1123 (1.5379) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][180/625] eta 0:02:37 lr 0.001151 wd 0.0500 time 0.3530 (0.3545) data time 0.0006 (0.0043) model time 0.3524 (0.3458) loss 3.5913 (3.1851) grad_norm 1.6363 (1.5290) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][190/625] eta 0:02:33 lr 0.001151 wd 0.0500 time 0.3430 (0.3540) data time 0.0008 (0.0041) model time 0.3422 (0.3457) loss 2.3880 (3.1806) grad_norm 1.5153 (1.5225) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][200/625] eta 0:02:30 lr 0.001150 wd 0.0500 time 0.3522 (0.3537) data time 0.0009 (0.0040) model time 0.3513 (0.3458) loss 3.7282 (3.1804) grad_norm 1.9061 (1.5631) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][210/625] eta 0:02:26 lr 0.001150 wd 0.0500 time 0.3447 (0.3534) data time 0.0008 (0.0038) model time 0.3439 (0.3458) loss 3.2943 (3.1821) grad_norm 1.6142 (1.5652) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][220/625] eta 0:02:23 lr 0.001150 wd 0.0500 time 0.3476 (0.3531) data time 0.0006 (0.0037) model time 0.3470 (0.3458) loss 4.1119 (3.1903) grad_norm 1.1022 (1.5730) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][230/625] eta 0:02:19 lr 0.001150 wd 0.0500 time 0.3463 (0.3530) data time 0.0008 (0.0036) model time 0.3455 (0.3460) loss 3.4211 (3.2038) grad_norm 1.5885 (1.5656) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][240/625] eta 0:02:16 lr 0.001150 wd 0.0500 time 0.3462 (0.3535) data time 0.0007 (0.0035) model time 0.3455 (0.3470) loss 3.4567 (3.2026) grad_norm 1.2106 (1.5583) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][250/625] eta 0:02:13 lr 0.001149 wd 0.0500 time 0.3395 (0.3549) data time 0.0008 (0.0034) model time 0.3387 (0.3491) loss 2.3404 (3.1860) grad_norm 1.8157 (1.5521) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][260/625] eta 0:02:09 lr 0.001149 wd 0.0500 time 0.3471 (0.3546) data time 0.0009 (0.0033) model time 0.3463 (0.3489) loss 3.6422 (3.1947) grad_norm 1.1779 (1.5798) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][270/625] eta 0:02:05 lr 0.001149 wd 0.0500 time 0.3419 (0.3543) data time 0.0009 (0.0032) model time 0.3410 (0.3487) loss 3.0978 (3.1963) grad_norm 1.2415 (1.5712) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][280/625] eta 0:02:02 lr 0.001149 wd 0.0500 time 0.3490 (0.3540) data time 0.0009 (0.0031) model time 0.3481 (0.3485) loss 3.3119 (3.1993) grad_norm 1.0367 (1.5641) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][290/625] eta 0:01:58 lr 0.001149 wd 0.0500 time 0.3440 (0.3537) data time 0.0009 (0.0030) model time 0.3431 (0.3483) loss 3.2105 (3.2029) grad_norm 1.2744 (1.5654) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][300/625] eta 0:01:54 lr 0.001149 wd 0.0500 time 0.3457 (0.3535) data time 0.0009 (0.0030) model time 0.3448 (0.3483) loss 2.5488 (3.1993) grad_norm 1.1880 (1.5661) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][310/625] eta 0:01:51 lr 0.001148 wd 0.0500 time 0.3486 (0.3538) data time 0.0009 (0.0029) model time 0.3477 (0.3489) loss 3.4800 (3.2058) grad_norm 1.1961 (1.5953) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][320/625] eta 0:01:47 lr 0.001148 wd 0.0500 time 0.3430 (0.3536) data time 0.0006 (0.0028) model time 0.3423 (0.3488) loss 3.3674 (3.2038) grad_norm 1.7532 (1.5931) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][330/625] eta 0:01:44 lr 0.001148 wd 0.0500 time 0.3485 (0.3535) data time 0.0011 (0.0028) model time 0.3474 (0.3487) loss 3.4578 (3.1964) grad_norm 1.4415 (1.5806) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][340/625] eta 0:01:40 lr 0.001148 wd 0.0500 time 0.3482 (0.3534) data time 0.0007 (0.0027) model time 0.3475 (0.3487) loss 3.9865 (3.2065) grad_norm 1.3368 (1.5801) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][350/625] eta 0:01:37 lr 0.001148 wd 0.0500 time 0.3493 (0.3533) data time 0.0009 (0.0027) model time 0.3484 (0.3488) loss 2.1593 (3.2034) grad_norm 1.3058 (1.5733) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][360/625] eta 0:01:33 lr 0.001148 wd 0.0500 time 0.3468 (0.3532) data time 0.0006 (0.0026) model time 0.3462 (0.3488) loss 2.7925 (3.2086) grad_norm 1.5593 (1.5699) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][370/625] eta 0:01:30 lr 0.001147 wd 0.0500 time 0.3505 (0.3531) data time 0.0008 (0.0026) model time 0.3497 (0.3488) loss 3.2495 (3.2072) grad_norm 2.1238 (1.5675) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][380/625] eta 0:01:26 lr 0.001147 wd 0.0500 time 0.3488 (0.3530) data time 0.0009 (0.0026) model time 0.3479 (0.3488) loss 2.2142 (3.2066) grad_norm 1.2397 (1.5718) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][390/625] eta 0:01:22 lr 0.001147 wd 0.0500 time 0.3589 (0.3529) data time 0.0009 (0.0025) model time 0.3580 (0.3488) loss 3.3005 (3.2060) grad_norm 1.2631 (1.5723) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][400/625] eta 0:01:19 lr 0.001147 wd 0.0500 time 0.3476 (0.3528) data time 0.0009 (0.0025) model time 0.3467 (0.3487) loss 3.5089 (3.2061) grad_norm 1.6937 (1.5691) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][410/625] eta 0:01:15 lr 0.001147 wd 0.0500 time 0.3507 (0.3528) data time 0.0007 (0.0024) model time 0.3500 (0.3488) loss 3.6245 (3.2077) grad_norm 1.4567 (1.5647) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][420/625] eta 0:01:12 lr 0.001147 wd 0.0500 time 0.3496 (0.3527) data time 0.0008 (0.0024) model time 0.3489 (0.3487) loss 2.2907 (3.2097) grad_norm 1.0067 (1.5591) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][430/625] eta 0:01:09 lr 0.001146 wd 0.0500 time 0.3583 (0.3544) data time 0.0007 (0.0024) model time 0.3576 (0.3508) loss 3.9181 (3.2142) grad_norm 1.9266 (1.5545) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][440/625] eta 0:01:05 lr 0.001146 wd 0.0500 time 0.3437 (0.3541) data time 0.0008 (0.0023) model time 0.3428 (0.3506) loss 2.6188 (3.2106) grad_norm 1.4015 (1.5498) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][450/625] eta 0:01:01 lr 0.001146 wd 0.0500 time 0.3454 (0.3540) data time 0.0008 (0.0023) model time 0.3446 (0.3504) loss 3.6365 (3.2144) grad_norm 1.1379 (1.5450) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][460/625] eta 0:00:58 lr 0.001146 wd 0.0500 time 0.3412 (0.3538) data time 0.0007 (0.0023) model time 0.3406 (0.3503) loss 3.8705 (3.2141) grad_norm 1.5614 (1.5438) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][470/625] eta 0:00:54 lr 0.001146 wd 0.0500 time 0.3469 (0.3536) data time 0.0008 (0.0022) model time 0.3461 (0.3502) loss 3.4714 (3.2175) grad_norm 1.3276 (1.5481) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][480/625] eta 0:00:51 lr 0.001145 wd 0.0500 time 0.3524 (0.3536) data time 0.0009 (0.0022) model time 0.3516 (0.3501) loss 2.8923 (3.2248) grad_norm 1.9396 (1.5459) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:31:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][490/625] eta 0:00:47 lr 0.001145 wd 0.0500 time 0.5651 (0.3539) data time 0.0008 (0.0022) model time 0.5643 (0.3506) loss 2.4489 (3.2275) grad_norm 1.7938 (1.5491) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][500/625] eta 0:00:44 lr 0.001145 wd 0.0500 time 0.3616 (0.3537) data time 0.0008 (0.0022) model time 0.3608 (0.3505) loss 3.6271 (3.2308) grad_norm 2.3131 (1.5526) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][510/625] eta 0:00:40 lr 0.001145 wd 0.0500 time 0.3453 (0.3536) data time 0.0007 (0.0021) model time 0.3446 (0.3504) loss 2.8411 (3.2308) grad_norm 1.5746 (1.5498) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][520/625] eta 0:00:37 lr 0.001145 wd 0.0500 time 0.3503 (0.3536) data time 0.0006 (0.0021) model time 0.3497 (0.3504) loss 2.5128 (3.2302) grad_norm 1.3366 (1.5467) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][530/625] eta 0:00:33 lr 0.001145 wd 0.0500 time 0.3492 (0.3535) data time 0.0008 (0.0021) model time 0.3484 (0.3503) loss 3.4404 (3.2286) grad_norm 1.2922 (1.5464) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][540/625] eta 0:00:30 lr 0.001144 wd 0.0500 time 0.3507 (0.3534) data time 0.0008 (0.0021) model time 0.3499 (0.3503) loss 3.4593 (3.2265) grad_norm 1.7834 (1.5470) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][550/625] eta 0:00:26 lr 0.001144 wd 0.0500 time 0.3460 (0.3544) data time 0.0007 (0.0021) model time 0.3454 (0.3514) loss 3.4718 (3.2232) grad_norm 1.3917 (1.5435) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][560/625] eta 0:00:23 lr 0.001144 wd 0.0500 time 0.3457 (0.3542) data time 0.0009 (0.0020) model time 0.3448 (0.3513) loss 3.1469 (3.2260) grad_norm 1.7636 (1.5432) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][570/625] eta 0:00:19 lr 0.001144 wd 0.0500 time 0.3438 (0.3540) data time 0.0007 (0.0020) model time 0.3431 (0.3511) loss 2.3894 (3.2281) grad_norm 1.3558 (1.5425) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][580/625] eta 0:00:15 lr 0.001144 wd 0.0500 time 0.3467 (0.3539) data time 0.0007 (0.0020) model time 0.3460 (0.3510) loss 3.6937 (3.2336) grad_norm 1.8578 (1.5450) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][590/625] eta 0:00:12 lr 0.001144 wd 0.0500 time 0.3476 (0.3538) data time 0.0008 (0.0020) model time 0.3468 (0.3509) loss 3.9620 (3.2353) grad_norm 1.1458 (1.5451) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][600/625] eta 0:00:08 lr 0.001143 wd 0.0500 time 0.3517 (0.3538) data time 0.0006 (0.0020) model time 0.3510 (0.3509) loss 3.5295 (3.2362) grad_norm 1.2552 (1.5447) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][610/625] eta 0:00:05 lr 0.001143 wd 0.0500 time 0.3112 (0.3540) data time 0.0005 (0.0019) model time 0.3108 (0.3512) loss 2.3045 (3.2328) grad_norm 1.5121 (1.5475) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [147/300][620/625] eta 0:00:01 lr 0.001143 wd 0.0500 time 0.3386 (0.3537) data time 0.0006 (0.0019) model time 0.3380 (0.3509) loss 2.2572 (3.2286) grad_norm 2.6817 (1.5531) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:45 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 147 training takes 0:03:41 [2024-08-27 09:32:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:32:46 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:32:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.451 (0.451) Loss 0.5938 (0.5938) Acc@1 86.670 (86.670) Acc@5 98.047 (98.047) Mem 11224MB [2024-08-27 09:32:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.130) Loss 1.0264 (0.7446) Acc@1 76.660 (83.332) Acc@5 93.848 (96.822) Mem 11224MB [2024-08-27 09:32:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.1104 (0.8881) Acc@1 72.998 (79.932) Acc@5 93.213 (95.147) Mem 11224MB [2024-08-27 09:32:48 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.659 Acc@5 95.166 [2024-08-27 09:32:48 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.7% [2024-08-27 09:32:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.782 (0.782) Loss 0.5171 (0.5171) Acc@1 88.818 (88.818) Acc@5 98.535 (98.535) Mem 11224MB [2024-08-27 09:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.164) Loss 0.8560 (0.6441) Acc@1 79.590 (85.431) Acc@5 95.557 (97.448) Mem 11224MB [2024-08-27 09:32:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.130) Loss 0.9795 (0.7719) Acc@1 75.684 (82.220) Acc@5 94.775 (96.136) Mem 11224MB [2024-08-27 09:32:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.960 Acc@5 96.159 [2024-08-27 09:32:51 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.0% [2024-08-27 09:32:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][0/625] eta 0:12:52 lr 0.001143 wd 0.0500 time 1.2358 (1.2358) data time 0.4135 (0.4135) model time 0.0000 (0.0000) loss 3.5888 (3.5888) grad_norm 1.3287 (1.3287) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][10/625] eta 0:04:24 lr 0.001143 wd 0.0500 time 0.3498 (0.4296) data time 0.0009 (0.0384) model time 0.0000 (0.0000) loss 2.7274 (3.2682) grad_norm 1.4200 (1.5512) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][20/625] eta 0:03:56 lr 0.001143 wd 0.0500 time 0.3514 (0.3908) data time 0.0008 (0.0206) model time 0.0000 (0.0000) loss 3.5694 (3.3164) grad_norm 1.8153 (1.6408) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][30/625] eta 0:03:45 lr 0.001142 wd 0.0500 time 0.3730 (0.3798) data time 0.0008 (0.0143) model time 0.0000 (0.0000) loss 3.4068 (3.3242) grad_norm 1.1470 (1.5917) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][40/625] eta 0:03:37 lr 0.001142 wd 0.0500 time 0.3519 (0.3723) data time 0.0009 (0.0113) model time 0.0000 (0.0000) loss 3.5862 (3.3402) grad_norm 1.4384 (1.5676) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][50/625] eta 0:03:31 lr 0.001142 wd 0.0500 time 0.3537 (0.3678) data time 0.0008 (0.0093) model time 0.0000 (0.0000) loss 3.0104 (3.3262) grad_norm 1.3085 (1.5465) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][60/625] eta 0:03:26 lr 0.001142 wd 0.0500 time 0.3478 (0.3646) data time 0.0007 (0.0079) model time 0.3471 (0.3476) loss 3.9357 (3.3325) grad_norm 1.8381 (1.5490) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][70/625] eta 0:03:21 lr 0.001142 wd 0.0500 time 0.3500 (0.3623) data time 0.0006 (0.0069) model time 0.3493 (0.3475) loss 2.6647 (3.3031) grad_norm 1.6168 (1.5441) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][80/625] eta 0:03:20 lr 0.001141 wd 0.0500 time 0.3392 (0.3679) data time 0.0008 (0.0062) model time 0.3384 (0.3672) loss 3.1254 (3.3244) grad_norm 1.5543 (1.5407) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][90/625] eta 0:03:15 lr 0.001141 wd 0.0500 time 0.3464 (0.3655) data time 0.0007 (0.0056) model time 0.3457 (0.3616) loss 2.2414 (3.3231) grad_norm 1.4915 (1.5222) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][100/625] eta 0:03:10 lr 0.001141 wd 0.0500 time 0.3398 (0.3633) data time 0.0006 (0.0051) model time 0.3392 (0.3578) loss 3.1899 (3.3198) grad_norm 1.4357 (1.5074) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][110/625] eta 0:03:06 lr 0.001141 wd 0.0500 time 0.3449 (0.3615) data time 0.0008 (0.0048) model time 0.3440 (0.3553) loss 3.6783 (3.3090) grad_norm 1.3870 (1.5151) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][120/625] eta 0:03:01 lr 0.001141 wd 0.0500 time 0.3493 (0.3603) data time 0.0008 (0.0045) model time 0.3485 (0.3538) loss 3.5704 (3.3130) grad_norm 1.8376 (1.5271) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][130/625] eta 0:02:57 lr 0.001141 wd 0.0500 time 0.3477 (0.3594) data time 0.0006 (0.0042) model time 0.3471 (0.3531) loss 3.7669 (3.3125) grad_norm 1.6402 (1.5241) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][140/625] eta 0:02:53 lr 0.001140 wd 0.0500 time 0.3440 (0.3586) data time 0.0010 (0.0040) model time 0.3430 (0.3524) loss 1.9775 (3.3014) grad_norm 1.7505 (1.5188) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][150/625] eta 0:02:50 lr 0.001140 wd 0.0500 time 0.3488 (0.3591) data time 0.0008 (0.0038) model time 0.3480 (0.3538) loss 3.0890 (3.2923) grad_norm 1.2476 (1.5233) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][160/625] eta 0:02:46 lr 0.001140 wd 0.0500 time 0.3491 (0.3585) data time 0.0008 (0.0036) model time 0.3483 (0.3532) loss 3.2224 (3.2571) grad_norm 2.1053 (1.5324) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][170/625] eta 0:02:42 lr 0.001140 wd 0.0500 time 0.3423 (0.3577) data time 0.0010 (0.0034) model time 0.3413 (0.3525) loss 3.4424 (3.2371) grad_norm 2.4227 (1.5412) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:33:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][180/625] eta 0:02:38 lr 0.001140 wd 0.0500 time 0.3527 (0.3571) data time 0.0010 (0.0033) model time 0.3518 (0.3520) loss 3.6202 (3.2408) grad_norm 1.5396 (1.5364) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][190/625] eta 0:02:35 lr 0.001140 wd 0.0500 time 0.3469 (0.3566) data time 0.0008 (0.0032) model time 0.3461 (0.3516) loss 3.1522 (3.2444) grad_norm 2.2804 (1.5376) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:34:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][200/625] eta 0:02:31 lr 0.001139 wd 0.0500 time 0.3516 (0.3562) data time 0.0007 (0.0031) model time 0.3509 (0.3513) loss 4.1518 (3.2509) grad_norm 1.6005 (1.5349) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-27 09:34:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:34:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:36:04 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-27 09:36:05 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-27 09:36:18 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-27 09:36:31 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-27 09:36:31 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-27 09:36:33 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-27 09:36:35 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-27 09:36:35 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 148) [2024-08-27 09:36:35 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-27 09:36:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][210/625] eta 0:13:27 lr 0.001139 wd 0.0500 time 0.3761 (1.9447) data time 0.0012 (0.0913) model time 0.3750 (1.8534) loss 3.5742 (3.5614) grad_norm 3.3395 (1.8641) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][220/625] eta 0:07:32 lr 0.001139 wd 0.0500 time 0.3717 (1.1177) data time 0.0013 (0.0440) model time 0.3704 (1.0737) loss 3.6966 (3.4129) grad_norm 3.9415 (1.8942) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][230/625] eta 0:05:43 lr 0.001139 wd 0.0500 time 0.3722 (0.8690) data time 0.0008 (0.0292) model time 0.3714 (0.8398) loss 3.8758 (3.4950) grad_norm 2.2113 (1.7732) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][240/625] eta 0:04:45 lr 0.001139 wd 0.0500 time 0.3705 (0.7427) data time 0.0014 (0.0220) model time 0.3691 (0.7207) loss 3.0469 (3.4095) grad_norm 1.7150 (1.7063) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][250/625] eta 0:04:12 lr 0.001138 wd 0.0500 time 0.3778 (0.6734) data time 0.0016 (0.0178) model time 0.3762 (0.6556) loss 3.1684 (3.3764) grad_norm 1.2007 (1.6619) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][260/625] eta 0:03:47 lr 0.001138 wd 0.0500 time 0.3763 (0.6235) data time 0.0008 (0.0150) model time 0.3755 (0.6086) loss 2.7593 (3.3618) grad_norm 1.0692 (1.5767) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][270/625] eta 0:03:28 lr 0.001138 wd 0.0500 time 0.3794 (0.5880) data time 0.0011 (0.0130) model time 0.3783 (0.5751) loss 3.0878 (3.3442) grad_norm 1.2263 (1.5651) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][280/625] eta 0:03:13 lr 0.001138 wd 0.0500 time 0.3783 (0.5614) data time 0.0011 (0.0115) model time 0.3771 (0.5499) loss 3.2344 (3.3230) grad_norm 9.0441 (1.6593) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][290/625] eta 0:03:01 lr 0.001138 wd 0.0500 time 0.3765 (0.5406) data time 0.0010 (0.0103) model time 0.3755 (0.5303) loss 3.2876 (3.3047) grad_norm 1.4679 (1.6655) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][300/625] eta 0:02:50 lr 0.001138 wd 0.0500 time 0.3795 (0.5237) data time 0.0012 (0.0094) model time 0.3783 (0.5143) loss 3.2054 (3.3155) grad_norm 1.3976 (1.6429) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][310/625] eta 0:02:40 lr 0.001137 wd 0.0500 time 0.3754 (0.5101) data time 0.0009 (0.0086) model time 0.3746 (0.5014) loss 3.5318 (3.3174) grad_norm 1.3016 (1.6336) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][320/625] eta 0:02:32 lr 0.001137 wd 0.0500 time 0.3807 (0.4986) data time 0.0010 (0.0080) model time 0.3797 (0.4906) loss 3.2942 (3.3119) grad_norm 1.3938 (1.6289) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][330/625] eta 0:02:24 lr 0.001137 wd 0.0500 time 0.3729 (0.4891) data time 0.0008 (0.0075) model time 0.3720 (0.4816) loss 3.1427 (3.3047) grad_norm 1.9280 (1.7684) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][340/625] eta 0:02:17 lr 0.001137 wd 0.0500 time 0.3789 (0.4809) data time 0.0009 (0.0070) model time 0.3780 (0.4739) loss 4.0322 (3.3061) grad_norm 1.3766 (1.7538) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][350/625] eta 0:02:10 lr 0.001137 wd 0.0500 time 0.3716 (0.4739) data time 0.0009 (0.0067) model time 0.3707 (0.4672) loss 3.1311 (3.3021) grad_norm 1.2259 (1.7302) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][360/625] eta 0:02:03 lr 0.001137 wd 0.0500 time 0.3620 (0.4675) data time 0.0009 (0.0063) model time 0.3611 (0.4612) loss 3.8942 (3.2973) grad_norm 1.4288 (1.7123) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:37:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][370/625] eta 0:01:57 lr 0.001136 wd 0.0500 time 0.3657 (0.4619) data time 0.0011 (0.0060) model time 0.3646 (0.4559) loss 3.0813 (3.2950) grad_norm 1.1268 (1.6891) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][380/625] eta 0:01:51 lr 0.001136 wd 0.0500 time 0.3729 (0.4570) data time 0.0008 (0.0057) model time 0.3721 (0.4513) loss 3.3251 (3.2777) grad_norm 1.1326 (1.6777) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][390/625] eta 0:01:46 lr 0.001136 wd 0.0500 time 0.3721 (0.4527) data time 0.0010 (0.0055) model time 0.3712 (0.4472) loss 3.7376 (3.2789) grad_norm 1.5077 (1.6686) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][400/625] eta 0:01:40 lr 0.001136 wd 0.0500 time 0.3778 (0.4488) data time 0.0011 (0.0053) model time 0.3767 (0.4435) loss 2.2409 (3.2593) grad_norm 1.6273 (1.6629) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][410/625] eta 0:01:35 lr 0.001136 wd 0.0500 time 0.3847 (0.4453) data time 0.0007 (0.0051) model time 0.3839 (0.4402) loss 3.5776 (3.2496) grad_norm 1.4115 (1.6615) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][420/625] eta 0:01:30 lr 0.001135 wd 0.0500 time 0.3729 (0.4421) data time 0.0009 (0.0049) model time 0.3721 (0.4372) loss 4.0491 (3.2467) grad_norm 1.4179 (1.6491) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][430/625] eta 0:01:26 lr 0.001135 wd 0.0500 time 0.3805 (0.4411) data time 0.0009 (0.0047) model time 0.3795 (0.4363) loss 2.5632 (3.2534) grad_norm 1.4945 (1.6390) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][440/625] eta 0:01:21 lr 0.001135 wd 0.0500 time 0.3746 (0.4390) data time 0.0008 (0.0046) model time 0.3738 (0.4344) loss 2.1176 (3.2465) grad_norm 1.4642 (1.6356) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][450/625] eta 0:01:16 lr 0.001135 wd 0.0500 time 0.3711 (0.4364) data time 0.0011 (0.0045) model time 0.3700 (0.4320) loss 3.2724 (3.2427) grad_norm 1.9154 (1.6357) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][460/625] eta 0:01:11 lr 0.001135 wd 0.0500 time 0.3709 (0.4340) data time 0.0011 (0.0043) model time 0.3698 (0.4297) loss 3.6917 (3.2338) grad_norm 1.0916 (1.6371) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][470/625] eta 0:01:06 lr 0.001135 wd 0.0500 time 0.3790 (0.4318) data time 0.0009 (0.0042) model time 0.3781 (0.4276) loss 2.1792 (3.2224) grad_norm 2.6291 (1.6358) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][480/625] eta 0:01:02 lr 0.001134 wd 0.0500 time 0.3721 (0.4297) data time 0.0011 (0.0041) model time 0.3711 (0.4256) loss 3.5848 (3.2310) grad_norm 1.2464 (1.6349) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][490/625] eta 0:00:57 lr 0.001134 wd 0.0500 time 0.3766 (0.4279) data time 0.0011 (0.0040) model time 0.3755 (0.4239) loss 2.9139 (3.2280) grad_norm 0.9097 (1.6295) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][500/625] eta 0:00:53 lr 0.001134 wd 0.0500 time 0.3758 (0.4261) data time 0.0010 (0.0039) model time 0.3748 (0.4222) loss 3.3314 (3.2122) grad_norm 1.2881 (1.6235) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][510/625] eta 0:00:48 lr 0.001134 wd 0.0500 time 0.3800 (0.4245) data time 0.0011 (0.0038) model time 0.3789 (0.4206) loss 3.5908 (3.2086) grad_norm 1.3792 (1.6180) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][520/625] eta 0:00:44 lr 0.001134 wd 0.0500 time 0.3759 (0.4229) data time 0.0009 (0.0037) model time 0.3750 (0.4192) loss 3.6668 (3.2197) grad_norm 1.4364 (1.6231) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][530/625] eta 0:00:40 lr 0.001134 wd 0.0500 time 0.3757 (0.4214) data time 0.0013 (0.0037) model time 0.3743 (0.4178) loss 2.3606 (3.2226) grad_norm 1.2925 (1.6158) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][540/625] eta 0:00:35 lr 0.001133 wd 0.0500 time 0.3799 (0.4201) data time 0.0011 (0.0036) model time 0.3788 (0.4165) loss 3.1092 (3.2235) grad_norm 1.4138 (1.6131) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][550/625] eta 0:00:31 lr 0.001133 wd 0.0500 time 0.3791 (0.4201) data time 0.0009 (0.0035) model time 0.3782 (0.4165) loss 3.7134 (3.2251) grad_norm 1.8022 (1.6120) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][560/625] eta 0:00:27 lr 0.001133 wd 0.0500 time 0.3760 (0.4188) data time 0.0009 (0.0035) model time 0.3750 (0.4153) loss 2.8961 (3.2225) grad_norm 1.1609 (1.6043) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][570/625] eta 0:00:22 lr 0.001133 wd 0.0500 time 0.3682 (0.4176) data time 0.0012 (0.0034) model time 0.3670 (0.4142) loss 3.7488 (3.2249) grad_norm 1.3142 (1.5998) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][580/625] eta 0:00:18 lr 0.001133 wd 0.0500 time 0.3725 (0.4164) data time 0.0008 (0.0033) model time 0.3717 (0.4131) loss 3.6833 (3.2260) grad_norm 1.4789 (1.6045) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][590/625] eta 0:00:14 lr 0.001132 wd 0.0500 time 0.3744 (0.4154) data time 0.0009 (0.0033) model time 0.3736 (0.4121) loss 3.3443 (3.2215) grad_norm 2.4825 (1.6058) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][600/625] eta 0:00:10 lr 0.001132 wd 0.0500 time 0.3733 (0.4143) data time 0.0008 (0.0032) model time 0.3724 (0.4111) loss 3.6249 (3.2258) grad_norm 1.3958 (1.6063) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][610/625] eta 0:00:06 lr 0.001132 wd 0.0500 time 0.3617 (0.4133) data time 0.0008 (0.0032) model time 0.3609 (0.4101) loss 3.1741 (3.2306) grad_norm 1.1907 (1.6023) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [148/300][620/625] eta 0:00:02 lr 0.001132 wd 0.0500 time 0.3666 (0.4125) data time 0.0005 (0.0031) model time 0.3661 (0.4094) loss 1.9179 (3.2260) grad_norm 1.5749 (1.6020) loss_scale 1024.0000 (1024.0000) mem 11218MB [2024-08-27 09:39:34 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 148 training takes 0:02:54 [2024-08-27 09:39:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:39:39 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.492 (0.492) Loss 0.5625 (0.5625) Acc@1 87.695 (87.695) Acc@5 98.193 (98.193) Mem 11218MB [2024-08-27 09:39:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.141) Loss 0.9580 (0.7108) Acc@1 77.734 (83.807) Acc@5 94.434 (96.950) Mem 11218MB [2024-08-27 09:39:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 1.0625 (0.8620) Acc@1 75.195 (80.215) Acc@5 93.506 (95.326) Mem 11218MB [2024-08-27 09:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.968 Acc@5 95.308 [2024-08-27 09:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.0% [2024-08-27 09:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.97% [2024-08-27 09:39:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 09:39:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 09:39:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.493 (0.493) Loss 0.5156 (0.5156) Acc@1 88.623 (88.623) Acc@5 98.535 (98.535) Mem 11218MB [2024-08-27 09:39:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.139) Loss 0.8540 (0.6432) Acc@1 79.492 (85.347) Acc@5 95.605 (97.452) Mem 11218MB [2024-08-27 09:39:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.117) Loss 0.9785 (0.7707) Acc@1 75.684 (82.175) Acc@5 94.727 (96.161) Mem 11218MB [2024-08-27 09:39:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.938 Acc@5 96.177 [2024-08-27 09:39:52 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 81.9% [2024-08-27 09:39:52 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.94% [2024-08-27 09:39:52 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 09:39:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 09:39:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][0/625] eta 0:08:38 lr 0.001132 wd 0.0500 time 0.8294 (0.8294) data time 0.4297 (0.4297) model time 0.0000 (0.0000) loss 4.0018 (4.0018) grad_norm 2.4315 (2.4315) loss_scale 1024.0000 (1024.0000) mem 11226MB [2024-08-27 09:40:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][10/625] eta 0:04:17 lr 0.001132 wd 0.0500 time 0.3713 (0.4181) data time 0.0012 (0.0401) model time 0.0000 (0.0000) loss 3.8457 (3.6251) grad_norm 2.1716 (1.7635) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][20/625] eta 0:04:01 lr 0.001132 wd 0.0500 time 0.3786 (0.4000) data time 0.0011 (0.0216) model time 0.0000 (0.0000) loss 3.4528 (3.4415) grad_norm 1.4159 (1.7678) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][30/625] eta 0:03:53 lr 0.001131 wd 0.0500 time 0.3780 (0.3923) data time 0.0009 (0.0150) model time 0.0000 (0.0000) loss 3.6502 (3.3073) grad_norm 1.2264 (1.5999) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][40/625] eta 0:03:46 lr 0.001131 wd 0.0500 time 0.3724 (0.3877) data time 0.0009 (0.0116) model time 0.0000 (0.0000) loss 3.4457 (3.2327) grad_norm 1.0595 (1.5362) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][50/625] eta 0:03:41 lr 0.001131 wd 0.0500 time 0.3740 (0.3861) data time 0.0014 (0.0095) model time 0.0000 (0.0000) loss 3.0781 (3.1542) grad_norm 1.8997 (1.5665) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][60/625] eta 0:03:37 lr 0.001131 wd 0.0500 time 0.3741 (0.3843) data time 0.0013 (0.0083) model time 0.3728 (0.3735) loss 3.0139 (3.1867) grad_norm 1.2353 (1.6765) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][70/625] eta 0:03:32 lr 0.001131 wd 0.0500 time 0.3842 (0.3831) data time 0.0012 (0.0073) model time 0.3831 (0.3741) loss 3.2400 (3.1975) grad_norm 1.5422 (1.6351) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][80/625] eta 0:03:31 lr 0.001130 wd 0.0500 time 0.3726 (0.3874) data time 0.0011 (0.0065) model time 0.3715 (0.3883) loss 2.9150 (3.1739) grad_norm 1.3884 (1.6290) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][90/625] eta 0:03:26 lr 0.001130 wd 0.0500 time 0.3762 (0.3859) data time 0.0011 (0.0059) model time 0.3752 (0.3843) loss 3.6939 (3.2255) grad_norm 1.2194 (1.6058) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][100/625] eta 0:03:22 lr 0.001130 wd 0.0500 time 0.3739 (0.3851) data time 0.0008 (0.0055) model time 0.3731 (0.3829) loss 2.3167 (3.1889) grad_norm 1.1748 (1.5960) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][110/625] eta 0:03:18 lr 0.001130 wd 0.0500 time 0.5510 (0.3863) data time 0.0009 (0.0051) model time 0.5501 (0.3852) loss 2.4082 (3.1938) grad_norm 1.4155 (1.5817) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][120/625] eta 0:03:14 lr 0.001130 wd 0.0500 time 0.3777 (0.3858) data time 0.0009 (0.0048) model time 0.3768 (0.3843) loss 3.7888 (3.2003) grad_norm 1.4796 (1.5605) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][130/625] eta 0:03:10 lr 0.001130 wd 0.0500 time 0.3793 (0.3852) data time 0.0009 (0.0045) model time 0.3784 (0.3833) loss 3.2261 (3.2189) grad_norm 1.4743 (1.5766) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][140/625] eta 0:03:06 lr 0.001129 wd 0.0500 time 0.3747 (0.3845) data time 0.0012 (0.0043) model time 0.3735 (0.3823) loss 3.3741 (3.2264) grad_norm 1.8942 (1.5802) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][150/625] eta 0:03:02 lr 0.001129 wd 0.0500 time 0.3786 (0.3840) data time 0.0008 (0.0041) model time 0.3778 (0.3816) loss 3.2721 (3.2258) grad_norm 1.2249 (1.5843) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:40:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][160/625] eta 0:02:58 lr 0.001129 wd 0.0500 time 0.3839 (0.3837) data time 0.0009 (0.0039) model time 0.3830 (0.3812) loss 2.6023 (3.2367) grad_norm 2.0783 (1.5807) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:41:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][170/625] eta 0:02:54 lr 0.001129 wd 0.0500 time 0.3792 (0.3834) data time 0.0010 (0.0038) model time 0.3783 (0.3809) loss 3.2646 (3.2374) grad_norm 1.7010 (inf) loss_scale 512.0000 (1018.0117) mem 11224MB [2024-08-27 09:41:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][180/625] eta 0:02:50 lr 0.001129 wd 0.0500 time 0.3782 (0.3832) data time 0.0011 (0.0037) model time 0.3772 (0.3808) loss 3.3299 (3.2312) grad_norm 1.8272 (inf) loss_scale 512.0000 (990.0552) mem 11224MB [2024-08-27 09:41:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][190/625] eta 0:02:46 lr 0.001129 wd 0.0500 time 0.3900 (0.3832) data time 0.0008 (0.0036) model time 0.3892 (0.3808) loss 3.1826 (3.2295) grad_norm 2.0414 (inf) loss_scale 512.0000 (965.0262) mem 11224MB [2024-08-27 09:41:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][200/625] eta 0:02:42 lr 0.001128 wd 0.0500 time 0.3755 (0.3831) data time 0.0008 (0.0034) model time 0.3747 (0.3807) loss 3.7050 (3.2434) grad_norm 1.5033 (inf) loss_scale 512.0000 (942.4876) mem 11224MB [2024-08-27 09:41:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][210/625] eta 0:02:38 lr 0.001128 wd 0.0500 time 0.3790 (0.3829) data time 0.0011 (0.0033) model time 0.3779 (0.3806) loss 3.3894 (3.2437) grad_norm 1.1651 (inf) loss_scale 512.0000 (922.0853) mem 11224MB [2024-08-27 09:41:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][220/625] eta 0:02:35 lr 0.001128 wd 0.0500 time 0.4326 (0.3831) data time 0.0008 (0.0033) model time 0.4318 (0.3808) loss 3.2072 (3.2302) grad_norm 1.9053 (inf) loss_scale 512.0000 (903.5294) mem 11224MB [2024-08-27 09:41:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][230/625] eta 0:02:31 lr 0.001128 wd 0.0500 time 0.3688 (0.3830) data time 0.0010 (0.0032) model time 0.3678 (0.3807) loss 3.2609 (3.2337) grad_norm 1.6404 (inf) loss_scale 512.0000 (886.5801) mem 11224MB [2024-08-27 09:41:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][240/625] eta 0:02:27 lr 0.001128 wd 0.0500 time 0.3679 (0.3828) data time 0.0011 (0.0031) model time 0.3668 (0.3806) loss 3.6039 (3.2233) grad_norm 1.0500 (inf) loss_scale 512.0000 (871.0373) mem 11224MB [2024-08-27 09:41:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][250/625] eta 0:02:23 lr 0.001127 wd 0.0500 time 0.3830 (0.3827) data time 0.0009 (0.0030) model time 0.3821 (0.3804) loss 3.7250 (3.2375) grad_norm 1.2426 (inf) loss_scale 512.0000 (856.7331) mem 11224MB [2024-08-27 09:41:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][260/625] eta 0:02:19 lr 0.001127 wd 0.0500 time 0.3625 (0.3825) data time 0.0008 (0.0030) model time 0.3617 (0.3802) loss 3.4991 (3.2354) grad_norm 1.3947 (inf) loss_scale 512.0000 (843.5249) mem 11224MB [2024-08-27 09:41:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][270/625] eta 0:02:15 lr 0.001127 wd 0.0500 time 0.3699 (0.3826) data time 0.0012 (0.0029) model time 0.3687 (0.3804) loss 3.9100 (3.2331) grad_norm 1.4365 (inf) loss_scale 512.0000 (831.2915) mem 11224MB [2024-08-27 09:41:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][280/625] eta 0:02:11 lr 0.001127 wd 0.0500 time 0.3827 (0.3824) data time 0.0011 (0.0028) model time 0.3816 (0.3802) loss 3.6726 (3.2263) grad_norm 2.3593 (inf) loss_scale 512.0000 (819.9288) mem 11224MB [2024-08-27 09:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][290/625] eta 0:02:08 lr 0.001127 wd 0.0500 time 0.3644 (0.3841) data time 0.0008 (0.0028) model time 0.3635 (0.3824) loss 2.3612 (3.2163) grad_norm 0.8968 (inf) loss_scale 512.0000 (809.3471) mem 11224MB [2024-08-27 09:41:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][300/625] eta 0:02:04 lr 0.001127 wd 0.0500 time 0.3782 (0.3845) data time 0.0008 (0.0027) model time 0.3774 (0.3828) loss 3.2038 (3.2167) grad_norm 2.0867 (inf) loss_scale 512.0000 (799.4684) mem 11224MB [2024-08-27 09:41:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][310/625] eta 0:02:01 lr 0.001126 wd 0.0500 time 0.3759 (0.3842) data time 0.0015 (0.0027) model time 0.3744 (0.3825) loss 3.6693 (3.2194) grad_norm 1.5172 (inf) loss_scale 512.0000 (790.2251) mem 11224MB [2024-08-27 09:41:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][320/625] eta 0:01:57 lr 0.001126 wd 0.0500 time 0.3821 (0.3840) data time 0.0009 (0.0026) model time 0.3812 (0.3823) loss 2.3710 (3.2228) grad_norm 2.0558 (inf) loss_scale 512.0000 (781.5576) mem 11224MB [2024-08-27 09:42:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][330/625] eta 0:01:53 lr 0.001126 wd 0.0500 time 0.3763 (0.3837) data time 0.0011 (0.0026) model time 0.3753 (0.3820) loss 2.3361 (3.2233) grad_norm 1.3756 (inf) loss_scale 512.0000 (773.4139) mem 11224MB [2024-08-27 09:42:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][340/625] eta 0:01:49 lr 0.001126 wd 0.0500 time 0.3775 (0.3835) data time 0.0010 (0.0026) model time 0.3765 (0.3817) loss 3.0381 (3.2264) grad_norm 1.3179 (inf) loss_scale 512.0000 (765.7478) mem 11224MB [2024-08-27 09:42:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][350/625] eta 0:01:45 lr 0.001126 wd 0.0500 time 0.3747 (0.3833) data time 0.0010 (0.0025) model time 0.3736 (0.3815) loss 3.5295 (3.2306) grad_norm 1.3454 (inf) loss_scale 512.0000 (758.5185) mem 11224MB [2024-08-27 09:42:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][360/625] eta 0:01:41 lr 0.001126 wd 0.0500 time 0.3779 (0.3831) data time 0.0008 (0.0025) model time 0.3771 (0.3813) loss 2.3859 (3.2263) grad_norm 1.9867 (inf) loss_scale 512.0000 (751.6898) mem 11224MB [2024-08-27 09:42:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][370/625] eta 0:01:37 lr 0.001125 wd 0.0500 time 0.3791 (0.3830) data time 0.0010 (0.0024) model time 0.3781 (0.3812) loss 3.2971 (3.2260) grad_norm 1.4621 (inf) loss_scale 512.0000 (745.2291) mem 11224MB [2024-08-27 09:42:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][380/625] eta 0:01:33 lr 0.001125 wd 0.0500 time 0.3718 (0.3829) data time 0.0010 (0.0024) model time 0.3707 (0.3810) loss 2.8823 (3.2179) grad_norm 1.6187 (inf) loss_scale 512.0000 (739.1076) mem 11224MB [2024-08-27 09:42:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][390/625] eta 0:01:29 lr 0.001125 wd 0.0500 time 0.3751 (0.3827) data time 0.0012 (0.0024) model time 0.3739 (0.3808) loss 3.4398 (3.2085) grad_norm 1.1687 (inf) loss_scale 512.0000 (733.2992) mem 11224MB [2024-08-27 09:42:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][400/625] eta 0:01:26 lr 0.001125 wd 0.0500 time 0.3835 (0.3825) data time 0.0011 (0.0023) model time 0.3825 (0.3807) loss 3.4273 (3.2130) grad_norm 1.8536 (inf) loss_scale 512.0000 (727.7805) mem 11224MB [2024-08-27 09:42:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][410/625] eta 0:01:22 lr 0.001125 wd 0.0500 time 0.3751 (0.3824) data time 0.0010 (0.0023) model time 0.3741 (0.3805) loss 3.2341 (3.2084) grad_norm 1.3481 (inf) loss_scale 512.0000 (722.5304) mem 11224MB [2024-08-27 09:42:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][420/625] eta 0:01:18 lr 0.001124 wd 0.0500 time 0.3842 (0.3826) data time 0.0008 (0.0023) model time 0.3834 (0.3808) loss 2.3979 (3.2035) grad_norm 1.3367 (inf) loss_scale 512.0000 (717.5297) mem 11224MB [2024-08-27 09:42:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][430/625] eta 0:01:14 lr 0.001124 wd 0.0500 time 0.3762 (0.3824) data time 0.0013 (0.0023) model time 0.3749 (0.3806) loss 3.5124 (3.2049) grad_norm 2.4781 (inf) loss_scale 512.0000 (712.7610) mem 11224MB [2024-08-27 09:42:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][440/625] eta 0:01:10 lr 0.001124 wd 0.0500 time 0.3811 (0.3824) data time 0.0011 (0.0022) model time 0.3800 (0.3806) loss 3.6460 (3.2021) grad_norm 1.8203 (inf) loss_scale 512.0000 (708.2086) mem 11224MB [2024-08-27 09:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][450/625] eta 0:01:06 lr 0.001124 wd 0.0500 time 0.3730 (0.3823) data time 0.0014 (0.0022) model time 0.3716 (0.3805) loss 3.4034 (3.2021) grad_norm 1.3680 (inf) loss_scale 512.0000 (703.8581) mem 11224MB [2024-08-27 09:42:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][460/625] eta 0:01:03 lr 0.001124 wd 0.0500 time 0.3760 (0.3821) data time 0.0008 (0.0022) model time 0.3752 (0.3803) loss 3.9450 (3.2007) grad_norm 1.5426 (inf) loss_scale 512.0000 (699.6963) mem 11224MB [2024-08-27 09:42:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][470/625] eta 0:00:59 lr 0.001124 wd 0.0500 time 0.3858 (0.3833) data time 0.0011 (0.0022) model time 0.3847 (0.3817) loss 3.2711 (3.1990) grad_norm 1.5683 (inf) loss_scale 512.0000 (695.7113) mem 11224MB [2024-08-27 09:42:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][480/625] eta 0:00:55 lr 0.001123 wd 0.0500 time 0.3730 (0.3832) data time 0.0008 (0.0022) model time 0.3722 (0.3816) loss 2.8157 (3.2017) grad_norm 1.1370 (inf) loss_scale 512.0000 (691.8919) mem 11224MB [2024-08-27 09:43:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][490/625] eta 0:00:51 lr 0.001123 wd 0.0500 time 0.3734 (0.3830) data time 0.0008 (0.0021) model time 0.3726 (0.3814) loss 2.8630 (3.1976) grad_norm 1.5089 (inf) loss_scale 512.0000 (688.2281) mem 11224MB [2024-08-27 09:43:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][500/625] eta 0:00:47 lr 0.001123 wd 0.0500 time 0.3766 (0.3828) data time 0.0010 (0.0021) model time 0.3757 (0.3812) loss 4.1567 (3.2028) grad_norm 1.2522 (inf) loss_scale 512.0000 (684.7106) mem 11224MB [2024-08-27 09:43:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][510/625] eta 0:00:44 lr 0.001123 wd 0.0500 time 0.3917 (0.3827) data time 0.0010 (0.0021) model time 0.3907 (0.3811) loss 2.4399 (3.2056) grad_norm 1.6872 (inf) loss_scale 512.0000 (681.3307) mem 11224MB [2024-08-27 09:43:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][520/625] eta 0:00:40 lr 0.001123 wd 0.0500 time 0.3722 (0.3826) data time 0.0011 (0.0021) model time 0.3711 (0.3809) loss 2.9542 (3.1994) grad_norm 1.1382 (inf) loss_scale 512.0000 (678.0806) mem 11224MB [2024-08-27 09:43:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][530/625] eta 0:00:36 lr 0.001123 wd 0.0500 time 0.3776 (0.3824) data time 0.0008 (0.0021) model time 0.3768 (0.3808) loss 2.4047 (3.1943) grad_norm 1.1122 (inf) loss_scale 512.0000 (674.9529) mem 11224MB [2024-08-27 09:43:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][540/625] eta 0:00:32 lr 0.001122 wd 0.0500 time 0.3711 (0.3823) data time 0.0009 (0.0020) model time 0.3702 (0.3806) loss 2.8160 (3.1962) grad_norm 1.4147 (inf) loss_scale 512.0000 (671.9409) mem 11224MB [2024-08-27 09:43:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][550/625] eta 0:00:28 lr 0.001122 wd 0.0500 time 0.3738 (0.3821) data time 0.0011 (0.0020) model time 0.3727 (0.3805) loss 2.8640 (3.1994) grad_norm 1.2259 (inf) loss_scale 512.0000 (669.0381) mem 11224MB [2024-08-27 09:43:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][560/625] eta 0:00:24 lr 0.001122 wd 0.0500 time 0.3710 (0.3820) data time 0.0010 (0.0020) model time 0.3700 (0.3803) loss 3.4640 (3.1997) grad_norm 1.8097 (inf) loss_scale 512.0000 (666.2389) mem 11224MB [2024-08-27 09:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][570/625] eta 0:00:21 lr 0.001122 wd 0.0500 time 0.3770 (0.3818) data time 0.0011 (0.0020) model time 0.3759 (0.3801) loss 2.5780 (3.1996) grad_norm 1.1430 (inf) loss_scale 512.0000 (663.5377) mem 11224MB [2024-08-27 09:43:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][580/625] eta 0:00:17 lr 0.001122 wd 0.0500 time 0.3703 (0.3817) data time 0.0008 (0.0020) model time 0.3695 (0.3800) loss 3.1812 (3.2015) grad_norm 1.1212 (inf) loss_scale 512.0000 (660.9294) mem 11224MB [2024-08-27 09:43:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][590/625] eta 0:00:13 lr 0.001121 wd 0.0500 time 0.3802 (0.3823) data time 0.0008 (0.0020) model time 0.3794 (0.3807) loss 3.3897 (3.2053) grad_norm 1.8356 (inf) loss_scale 512.0000 (658.4095) mem 11224MB [2024-08-27 09:43:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][600/625] eta 0:00:09 lr 0.001121 wd 0.0500 time 0.3777 (0.3823) data time 0.0010 (0.0020) model time 0.3767 (0.3806) loss 2.3084 (3.2014) grad_norm 1.5691 (inf) loss_scale 512.0000 (655.9734) mem 11224MB [2024-08-27 09:43:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][610/625] eta 0:00:05 lr 0.001121 wd 0.0500 time 0.3663 (0.3822) data time 0.0007 (0.0020) model time 0.3656 (0.3806) loss 3.1127 (3.1976) grad_norm 2.0553 (inf) loss_scale 512.0000 (653.6170) mem 11224MB [2024-08-27 09:43:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [149/300][620/625] eta 0:00:01 lr 0.001121 wd 0.0500 time 0.3675 (0.3819) data time 0.0008 (0.0019) model time 0.3667 (0.3803) loss 2.3434 (3.2018) grad_norm 1.4468 (inf) loss_scale 512.0000 (651.3366) mem 11224MB [2024-08-27 09:43:54 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 149 training takes 0:03:58 [2024-08-27 09:43:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:43:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:43:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.490 (0.490) Loss 0.5811 (0.5811) Acc@1 87.500 (87.500) Acc@5 98.193 (98.193) Mem 11224MB [2024-08-27 09:43:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.9580 (0.7169) Acc@1 77.881 (83.598) Acc@5 94.385 (96.933) Mem 11224MB [2024-08-27 09:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.0967 (0.8597) Acc@1 73.438 (80.143) Acc@5 93.262 (95.326) Mem 11224MB [2024-08-27 09:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.880 Acc@5 95.292 [2024-08-27 09:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.9% [2024-08-27 09:43:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.823 (0.823) Loss 0.5142 (0.5142) Acc@1 88.770 (88.770) Acc@5 98.584 (98.584) Mem 11224MB [2024-08-27 09:44:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.177) Loss 0.8511 (0.6423) Acc@1 79.639 (85.409) Acc@5 95.605 (97.448) Mem 11224MB [2024-08-27 09:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.9766 (0.7694) Acc@1 75.830 (82.227) Acc@5 94.678 (96.187) Mem 11224MB [2024-08-27 09:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.992 Acc@5 96.199 [2024-08-27 09:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.0% [2024-08-27 09:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 81.99% [2024-08-27 09:44:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 09:44:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 09:44:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][0/625] eta 0:08:35 lr 0.001121 wd 0.0500 time 0.8241 (0.8241) data time 0.4770 (0.4770) model time 0.0000 (0.0000) loss 4.0026 (4.0026) grad_norm 1.4266 (1.4266) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][10/625] eta 0:04:18 lr 0.001121 wd 0.0500 time 0.3762 (0.4210) data time 0.0012 (0.0444) model time 0.0000 (0.0000) loss 3.2607 (3.4106) grad_norm 1.9198 (1.3897) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][20/625] eta 0:04:01 lr 0.001120 wd 0.0500 time 0.3709 (0.3989) data time 0.0011 (0.0238) model time 0.0000 (0.0000) loss 1.8614 (3.2671) grad_norm 1.1185 (1.3676) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][30/625] eta 0:03:53 lr 0.001120 wd 0.0500 time 0.3818 (0.3919) data time 0.0009 (0.0165) model time 0.0000 (0.0000) loss 4.2049 (3.2527) grad_norm 1.3256 (1.4233) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][40/625] eta 0:03:47 lr 0.001120 wd 0.0500 time 0.3768 (0.3884) data time 0.0010 (0.0128) model time 0.0000 (0.0000) loss 3.6374 (3.1978) grad_norm 1.3438 (1.4152) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][50/625] eta 0:03:42 lr 0.001120 wd 0.0500 time 0.3803 (0.3865) data time 0.0008 (0.0105) model time 0.0000 (0.0000) loss 2.4916 (3.1873) grad_norm 1.4681 (1.4443) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][60/625] eta 0:03:37 lr 0.001120 wd 0.0500 time 0.3755 (0.3847) data time 0.0008 (0.0090) model time 0.3747 (0.3743) loss 1.8666 (3.1887) grad_norm 1.6825 (1.5209) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][70/625] eta 0:03:32 lr 0.001120 wd 0.0500 time 0.3866 (0.3837) data time 0.0012 (0.0079) model time 0.3854 (0.3755) loss 3.3241 (3.2472) grad_norm 2.1211 (1.5817) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][80/625] eta 0:03:28 lr 0.001119 wd 0.0500 time 0.3744 (0.3830) data time 0.0011 (0.0070) model time 0.3733 (0.3758) loss 3.3941 (3.2392) grad_norm 1.6606 (1.6244) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][90/625] eta 0:03:24 lr 0.001119 wd 0.0500 time 0.3764 (0.3824) data time 0.0008 (0.0064) model time 0.3756 (0.3761) loss 3.4291 (3.2440) grad_norm 1.2608 (1.5993) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][100/625] eta 0:03:22 lr 0.001119 wd 0.0500 time 0.3762 (0.3854) data time 0.0010 (0.0059) model time 0.3752 (0.3831) loss 2.5356 (3.2402) grad_norm 1.5002 (1.5878) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][110/625] eta 0:03:18 lr 0.001119 wd 0.0500 time 0.3698 (0.3845) data time 0.0011 (0.0054) model time 0.3687 (0.3816) loss 3.6781 (3.2312) grad_norm 1.1817 (1.5818) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][120/625] eta 0:03:13 lr 0.001119 wd 0.0500 time 0.3758 (0.3837) data time 0.0011 (0.0051) model time 0.3748 (0.3805) loss 3.4031 (3.2219) grad_norm 1.7343 (1.5840) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][130/625] eta 0:03:09 lr 0.001119 wd 0.0500 time 0.3904 (0.3834) data time 0.0011 (0.0048) model time 0.3893 (0.3804) loss 3.2485 (3.2372) grad_norm 1.3072 (1.5514) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:44:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][140/625] eta 0:03:05 lr 0.001118 wd 0.0500 time 0.3809 (0.3832) data time 0.0010 (0.0046) model time 0.3799 (0.3801) loss 3.3135 (3.2517) grad_norm 2.4187 (1.5850) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][150/625] eta 0:03:02 lr 0.001118 wd 0.0500 time 0.3806 (0.3843) data time 0.0010 (0.0044) model time 0.3796 (0.3819) loss 3.6370 (3.2549) grad_norm 1.5550 (1.5957) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][160/625] eta 0:02:58 lr 0.001118 wd 0.0500 time 0.3820 (0.3841) data time 0.0008 (0.0042) model time 0.3812 (0.3817) loss 3.0066 (3.2663) grad_norm 1.0666 (1.5800) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][170/625] eta 0:02:54 lr 0.001118 wd 0.0500 time 0.3692 (0.3837) data time 0.0010 (0.0040) model time 0.3681 (0.3812) loss 2.1545 (3.2617) grad_norm 1.6425 (1.5724) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][180/625] eta 0:02:50 lr 0.001118 wd 0.0500 time 0.3739 (0.3833) data time 0.0008 (0.0038) model time 0.3731 (0.3809) loss 3.0852 (3.2517) grad_norm 1.1280 (1.5588) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][190/625] eta 0:02:46 lr 0.001117 wd 0.0500 time 0.3806 (0.3831) data time 0.0012 (0.0037) model time 0.3795 (0.3806) loss 3.4158 (3.2529) grad_norm 1.7114 (1.5442) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][200/625] eta 0:02:42 lr 0.001117 wd 0.0500 time 0.3698 (0.3827) data time 0.0011 (0.0036) model time 0.3687 (0.3802) loss 2.1832 (3.2480) grad_norm 1.7714 (1.5496) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][210/625] eta 0:02:38 lr 0.001117 wd 0.0500 time 0.3766 (0.3824) data time 0.0010 (0.0035) model time 0.3756 (0.3798) loss 3.3416 (3.2458) grad_norm 4.0769 (1.5576) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][220/625] eta 0:02:34 lr 0.001117 wd 0.0500 time 0.3736 (0.3822) data time 0.0010 (0.0034) model time 0.3726 (0.3797) loss 3.5335 (3.2556) grad_norm 1.9070 (1.5626) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][230/625] eta 0:02:30 lr 0.001117 wd 0.0500 time 0.3784 (0.3820) data time 0.0008 (0.0033) model time 0.3776 (0.3795) loss 3.1004 (3.2614) grad_norm 1.2189 (1.5609) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][240/625] eta 0:02:26 lr 0.001117 wd 0.0500 time 0.3836 (0.3818) data time 0.0011 (0.0032) model time 0.3825 (0.3793) loss 3.0178 (3.2662) grad_norm 1.5355 (1.5466) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][250/625] eta 0:02:23 lr 0.001116 wd 0.0500 time 0.3834 (0.3816) data time 0.0010 (0.0031) model time 0.3824 (0.3792) loss 3.8754 (3.2714) grad_norm 1.2571 (1.5663) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][260/625] eta 0:02:19 lr 0.001116 wd 0.0500 time 0.3689 (0.3815) data time 0.0008 (0.0030) model time 0.3681 (0.3790) loss 2.7613 (3.2699) grad_norm 2.5547 (1.5916) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][270/625] eta 0:02:15 lr 0.001116 wd 0.0500 time 0.3788 (0.3814) data time 0.0010 (0.0029) model time 0.3777 (0.3790) loss 3.0355 (3.2597) grad_norm 1.2685 (1.5851) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][280/625] eta 0:02:11 lr 0.001116 wd 0.0500 time 0.3860 (0.3813) data time 0.0010 (0.0029) model time 0.3850 (0.3789) loss 3.3426 (3.2597) grad_norm 1.3475 (1.5740) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][290/625] eta 0:02:07 lr 0.001116 wd 0.0500 time 0.3779 (0.3812) data time 0.0009 (0.0028) model time 0.3770 (0.3788) loss 3.8492 (3.2604) grad_norm 1.0960 (1.5687) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:45:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][300/625] eta 0:02:03 lr 0.001116 wd 0.0500 time 0.3871 (0.3811) data time 0.0010 (0.0028) model time 0.3861 (0.3788) loss 3.9631 (3.2614) grad_norm 1.5024 (1.5712) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][310/625] eta 0:02:00 lr 0.001115 wd 0.0500 time 0.3747 (0.3817) data time 0.0011 (0.0027) model time 0.3736 (0.3796) loss 2.5630 (3.2649) grad_norm 1.3507 (1.5746) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][320/625] eta 0:01:56 lr 0.001115 wd 0.0500 time 0.3766 (0.3815) data time 0.0012 (0.0027) model time 0.3754 (0.3794) loss 2.9426 (3.2693) grad_norm 1.3130 (1.5737) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][330/625] eta 0:01:52 lr 0.001115 wd 0.0500 time 0.3755 (0.3818) data time 0.0011 (0.0026) model time 0.3744 (0.3798) loss 3.7135 (3.2726) grad_norm 2.5542 (1.5759) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][340/625] eta 0:01:48 lr 0.001115 wd 0.0500 time 0.3820 (0.3817) data time 0.0008 (0.0026) model time 0.3812 (0.3797) loss 3.7144 (3.2736) grad_norm 1.5059 (1.5716) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][350/625] eta 0:01:44 lr 0.001115 wd 0.0500 time 0.3756 (0.3816) data time 0.0008 (0.0025) model time 0.3748 (0.3795) loss 2.8147 (3.2689) grad_norm 1.6864 (1.5728) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][360/625] eta 0:01:41 lr 0.001114 wd 0.0500 time 0.3787 (0.3815) data time 0.0010 (0.0025) model time 0.3777 (0.3794) loss 3.2886 (3.2740) grad_norm 1.9099 (1.5760) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][370/625] eta 0:01:37 lr 0.001114 wd 0.0500 time 0.3748 (0.3814) data time 0.0010 (0.0025) model time 0.3738 (0.3794) loss 3.0325 (3.2716) grad_norm 1.0862 (1.5762) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][380/625] eta 0:01:33 lr 0.001114 wd 0.0500 time 0.3818 (0.3813) data time 0.0008 (0.0024) model time 0.3809 (0.3793) loss 3.7381 (3.2702) grad_norm 1.4982 (1.5797) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][390/625] eta 0:01:29 lr 0.001114 wd 0.0500 time 0.3766 (0.3813) data time 0.0015 (0.0024) model time 0.3751 (0.3794) loss 2.8930 (3.2711) grad_norm 2.1340 (1.5767) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][400/625] eta 0:01:25 lr 0.001114 wd 0.0500 time 0.3831 (0.3813) data time 0.0010 (0.0024) model time 0.3821 (0.3794) loss 2.6931 (3.2733) grad_norm 1.3737 (1.5739) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][410/625] eta 0:01:21 lr 0.001114 wd 0.0500 time 0.3784 (0.3813) data time 0.0012 (0.0023) model time 0.3773 (0.3794) loss 2.3687 (3.2751) grad_norm 1.6262 (1.5718) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][420/625] eta 0:01:18 lr 0.001113 wd 0.0500 time 0.3822 (0.3813) data time 0.0011 (0.0023) model time 0.3812 (0.3793) loss 3.5207 (3.2755) grad_norm 1.1647 (1.5746) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][430/625] eta 0:01:14 lr 0.001113 wd 0.0500 time 0.3767 (0.3812) data time 0.0008 (0.0023) model time 0.3759 (0.3793) loss 2.6605 (3.2684) grad_norm 2.2315 (1.5806) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][440/625] eta 0:01:10 lr 0.001113 wd 0.0500 time 0.3858 (0.3811) data time 0.0008 (0.0023) model time 0.3850 (0.3792) loss 3.3563 (3.2588) grad_norm 1.5337 (1.5772) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][450/625] eta 0:01:06 lr 0.001113 wd 0.0500 time 0.3769 (0.3814) data time 0.0008 (0.0022) model time 0.3760 (0.3796) loss 3.2193 (3.2511) grad_norm 1.7496 (1.5828) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][460/625] eta 0:01:02 lr 0.001113 wd 0.0500 time 0.3790 (0.3813) data time 0.0011 (0.0022) model time 0.3779 (0.3795) loss 2.9855 (3.2532) grad_norm 2.0751 (1.5846) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][470/625] eta 0:00:59 lr 0.001113 wd 0.0500 time 0.3732 (0.3811) data time 0.0010 (0.0022) model time 0.3723 (0.3793) loss 3.0831 (3.2540) grad_norm 1.0245 (1.5871) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][480/625] eta 0:00:55 lr 0.001112 wd 0.0500 time 0.3797 (0.3810) data time 0.0010 (0.0022) model time 0.3787 (0.3792) loss 3.0704 (3.2539) grad_norm 1.2821 (1.5839) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][490/625] eta 0:00:51 lr 0.001112 wd 0.0500 time 0.5854 (0.3817) data time 0.0010 (0.0021) model time 0.5845 (0.3800) loss 3.3824 (3.2535) grad_norm 1.1758 (1.5771) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][500/625] eta 0:00:47 lr 0.001112 wd 0.0500 time 0.3723 (0.3816) data time 0.0014 (0.0021) model time 0.3709 (0.3799) loss 2.2489 (3.2486) grad_norm 2.4256 (1.5750) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][510/625] eta 0:00:43 lr 0.001112 wd 0.0500 time 0.3742 (0.3815) data time 0.0011 (0.0021) model time 0.3731 (0.3798) loss 3.3445 (3.2476) grad_norm 1.5642 (1.5696) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][520/625] eta 0:00:40 lr 0.001112 wd 0.0500 time 0.3876 (0.3814) data time 0.0011 (0.0021) model time 0.3866 (0.3797) loss 3.3503 (3.2387) grad_norm 2.0027 (1.5725) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][530/625] eta 0:00:36 lr 0.001111 wd 0.0500 time 0.3781 (0.3814) data time 0.0012 (0.0021) model time 0.3770 (0.3796) loss 3.1725 (3.2377) grad_norm 1.9097 (1.5788) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][540/625] eta 0:00:32 lr 0.001111 wd 0.0500 time 0.3815 (0.3813) data time 0.0008 (0.0021) model time 0.3807 (0.3796) loss 3.4488 (3.2361) grad_norm 1.1267 (1.5791) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][550/625] eta 0:00:28 lr 0.001111 wd 0.0500 time 0.3883 (0.3813) data time 0.0012 (0.0020) model time 0.3872 (0.3796) loss 3.2830 (3.2384) grad_norm 1.1452 (1.5797) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][560/625] eta 0:00:24 lr 0.001111 wd 0.0500 time 0.3702 (0.3812) data time 0.0014 (0.0020) model time 0.3688 (0.3795) loss 3.4574 (3.2399) grad_norm 1.6796 (1.5763) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][570/625] eta 0:00:20 lr 0.001111 wd 0.0500 time 0.3805 (0.3812) data time 0.0011 (0.0020) model time 0.3794 (0.3795) loss 2.2719 (3.2404) grad_norm 1.3598 (1.5704) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][580/625] eta 0:00:17 lr 0.001111 wd 0.0500 time 0.3794 (0.3811) data time 0.0010 (0.0020) model time 0.3784 (0.3794) loss 3.7023 (3.2412) grad_norm 1.7123 (1.5704) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][590/625] eta 0:00:13 lr 0.001110 wd 0.0500 time 0.3802 (0.3810) data time 0.0008 (0.0020) model time 0.3794 (0.3793) loss 2.5874 (3.2443) grad_norm 1.7142 (1.5669) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][600/625] eta 0:00:09 lr 0.001110 wd 0.0500 time 0.3736 (0.3810) data time 0.0009 (0.0020) model time 0.3728 (0.3793) loss 3.8277 (3.2372) grad_norm 1.9940 (1.5668) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][610/625] eta 0:00:05 lr 0.001110 wd 0.0500 time 0.3637 (0.3817) data time 0.0006 (0.0020) model time 0.3631 (0.3800) loss 3.5177 (3.2320) grad_norm 1.3516 (1.5640) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [150/300][620/625] eta 0:00:01 lr 0.001110 wd 0.0500 time 0.3730 (0.3814) data time 0.0006 (0.0019) model time 0.3724 (0.3798) loss 4.0708 (3.2320) grad_norm 1.0854 (1.5630) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:01 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 150 training takes 0:03:58 [2024-08-27 09:48:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:48:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:48:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.494 (0.494) Loss 0.5796 (0.5796) Acc@1 88.037 (88.037) Acc@5 98.047 (98.047) Mem 11224MB [2024-08-27 09:48:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 0.9604 (0.7207) Acc@1 77.197 (83.789) Acc@5 94.775 (96.968) Mem 11224MB [2024-08-27 09:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.0586 (0.8639) Acc@1 75.586 (80.264) Acc@5 93.164 (95.375) Mem 11224MB [2024-08-27 09:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.978 Acc@5 95.345 [2024-08-27 09:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.0% [2024-08-27 09:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 79.98% [2024-08-27 09:48:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 09:48:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 09:48:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.500 (0.500) Loss 0.5142 (0.5142) Acc@1 88.721 (88.721) Acc@5 98.584 (98.584) Mem 11224MB [2024-08-27 09:48:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.8486 (0.6416) Acc@1 79.834 (85.467) Acc@5 95.703 (97.474) Mem 11224MB [2024-08-27 09:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9771 (0.7687) Acc@1 75.781 (82.301) Acc@5 94.678 (96.208) Mem 11224MB [2024-08-27 09:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.060 Acc@5 96.213 [2024-08-27 09:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-08-27 09:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.06% [2024-08-27 09:48:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 09:48:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 09:48:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][0/625] eta 0:07:51 lr 0.001110 wd 0.0500 time 0.7548 (0.7548) data time 0.4102 (0.4102) model time 0.0000 (0.0000) loss 2.2656 (2.2656) grad_norm 1.4646 (1.4646) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][10/625] eta 0:04:12 lr 0.001110 wd 0.0500 time 0.3787 (0.4105) data time 0.0011 (0.0384) model time 0.0000 (0.0000) loss 3.1538 (3.1104) grad_norm 0.9672 (1.5518) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][20/625] eta 0:03:58 lr 0.001109 wd 0.0500 time 0.3674 (0.3947) data time 0.0008 (0.0206) model time 0.0000 (0.0000) loss 3.7692 (3.2093) grad_norm 1.4866 (1.6243) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][30/625] eta 0:03:51 lr 0.001109 wd 0.0500 time 0.3686 (0.3897) data time 0.0009 (0.0144) model time 0.0000 (0.0000) loss 3.3501 (3.2988) grad_norm 1.8724 (1.7458) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][40/625] eta 0:03:46 lr 0.001109 wd 0.0500 time 0.3757 (0.3867) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 3.1264 (3.2830) grad_norm 1.7278 (1.6943) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][50/625] eta 0:03:41 lr 0.001109 wd 0.0500 time 0.3775 (0.3848) data time 0.0011 (0.0092) model time 0.0000 (0.0000) loss 2.5319 (3.2233) grad_norm 1.1931 (1.6183) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][60/625] eta 0:03:36 lr 0.001109 wd 0.0500 time 0.3667 (0.3838) data time 0.0008 (0.0078) model time 0.3659 (0.3773) loss 2.1676 (3.2477) grad_norm 1.2609 (1.6079) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][70/625] eta 0:03:32 lr 0.001109 wd 0.0500 time 0.3858 (0.3827) data time 0.0008 (0.0069) model time 0.3850 (0.3761) loss 2.2104 (3.2430) grad_norm 2.3762 (1.5889) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][80/625] eta 0:03:28 lr 0.001108 wd 0.0500 time 0.3817 (0.3822) data time 0.0011 (0.0062) model time 0.3806 (0.3767) loss 3.2621 (3.2085) grad_norm 1.3033 (1.5616) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][90/625] eta 0:03:24 lr 0.001108 wd 0.0500 time 0.3754 (0.3818) data time 0.0008 (0.0056) model time 0.3746 (0.3768) loss 3.7419 (3.2309) grad_norm 1.3055 (1.5240) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][100/625] eta 0:03:20 lr 0.001108 wd 0.0500 time 0.3869 (0.3817) data time 0.0008 (0.0052) model time 0.3861 (0.3774) loss 3.8374 (3.2274) grad_norm 1.1985 (1.5132) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][110/625] eta 0:03:16 lr 0.001108 wd 0.0500 time 0.3743 (0.3814) data time 0.0009 (0.0048) model time 0.3734 (0.3773) loss 2.4609 (3.2114) grad_norm 1.4613 (1.5111) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:48:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][120/625] eta 0:03:12 lr 0.001108 wd 0.0500 time 0.3742 (0.3809) data time 0.0011 (0.0045) model time 0.3730 (0.3769) loss 3.4207 (3.2180) grad_norm 1.2185 (1.5062) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][130/625] eta 0:03:08 lr 0.001108 wd 0.0500 time 0.3749 (0.3806) data time 0.0011 (0.0043) model time 0.3738 (0.3767) loss 2.9512 (3.2261) grad_norm 1.7765 (1.5085) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][140/625] eta 0:03:04 lr 0.001107 wd 0.0500 time 0.3674 (0.3802) data time 0.0009 (0.0041) model time 0.3665 (0.3764) loss 3.1031 (3.2230) grad_norm 1.2453 (1.4992) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][150/625] eta 0:03:01 lr 0.001107 wd 0.0500 time 0.3737 (0.3813) data time 0.0008 (0.0039) model time 0.3729 (0.3783) loss 3.3978 (3.2264) grad_norm 2.1092 (1.5067) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][160/625] eta 0:02:57 lr 0.001107 wd 0.0500 time 0.3697 (0.3811) data time 0.0011 (0.0037) model time 0.3686 (0.3782) loss 2.9013 (3.2364) grad_norm 1.2852 (1.5127) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][170/625] eta 0:02:54 lr 0.001107 wd 0.0500 time 0.3728 (0.3830) data time 0.0011 (0.0036) model time 0.3717 (0.3810) loss 3.4001 (3.2273) grad_norm 1.5787 (1.5248) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][180/625] eta 0:02:50 lr 0.001107 wd 0.0500 time 0.3796 (0.3836) data time 0.0009 (0.0034) model time 0.3787 (0.3819) loss 2.3010 (3.2077) grad_norm 1.5175 (1.5390) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][190/625] eta 0:02:46 lr 0.001106 wd 0.0500 time 0.3745 (0.3832) data time 0.0012 (0.0033) model time 0.3733 (0.3815) loss 1.8594 (3.2024) grad_norm 1.4071 (1.5553) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][200/625] eta 0:02:42 lr 0.001106 wd 0.0500 time 0.3774 (0.3830) data time 0.0008 (0.0032) model time 0.3766 (0.3812) loss 3.8283 (3.1990) grad_norm 1.4907 (1.5619) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][210/625] eta 0:02:38 lr 0.001106 wd 0.0500 time 0.3745 (0.3827) data time 0.0008 (0.0031) model time 0.3737 (0.3809) loss 3.4901 (3.2028) grad_norm 1.3526 (1.5671) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][220/625] eta 0:02:34 lr 0.001106 wd 0.0500 time 0.3791 (0.3825) data time 0.0012 (0.0030) model time 0.3779 (0.3806) loss 2.8070 (3.1987) grad_norm 1.3769 (1.5646) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][230/625] eta 0:02:31 lr 0.001106 wd 0.0500 time 0.3731 (0.3823) data time 0.0010 (0.0029) model time 0.3721 (0.3804) loss 3.1397 (3.1942) grad_norm 1.1970 (1.5634) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][240/625] eta 0:02:27 lr 0.001106 wd 0.0500 time 0.3757 (0.3821) data time 0.0009 (0.0029) model time 0.3748 (0.3801) loss 2.0171 (3.1852) grad_norm 1.8697 (1.5728) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][250/625] eta 0:02:23 lr 0.001105 wd 0.0500 time 0.3749 (0.3818) data time 0.0012 (0.0028) model time 0.3737 (0.3799) loss 3.1876 (3.1726) grad_norm 1.3733 (1.5752) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][260/625] eta 0:02:19 lr 0.001105 wd 0.0500 time 0.3737 (0.3817) data time 0.0011 (0.0027) model time 0.3726 (0.3797) loss 3.4210 (3.1680) grad_norm 1.3031 (1.5725) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][270/625] eta 0:02:15 lr 0.001105 wd 0.0500 time 0.3776 (0.3822) data time 0.0008 (0.0027) model time 0.3768 (0.3804) loss 2.9116 (3.1675) grad_norm 1.2137 (1.5628) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:49:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][280/625] eta 0:02:11 lr 0.001105 wd 0.0500 time 0.3716 (0.3820) data time 0.0014 (0.0026) model time 0.3702 (0.3802) loss 2.3712 (3.1567) grad_norm 1.2535 (1.5530) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][290/625] eta 0:02:08 lr 0.001105 wd 0.0500 time 0.3644 (0.3825) data time 0.0008 (0.0026) model time 0.3637 (0.3808) loss 2.4629 (3.1584) grad_norm 2.0259 (1.5499) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][300/625] eta 0:02:04 lr 0.001104 wd 0.0500 time 0.3684 (0.3829) data time 0.0009 (0.0025) model time 0.3676 (0.3813) loss 3.8290 (3.1534) grad_norm 1.2660 (1.5511) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][310/625] eta 0:02:00 lr 0.001104 wd 0.0500 time 0.3775 (0.3827) data time 0.0008 (0.0025) model time 0.3767 (0.3811) loss 2.1886 (3.1444) grad_norm 1.3740 (1.5454) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][320/625] eta 0:01:56 lr 0.001104 wd 0.0500 time 0.3812 (0.3825) data time 0.0011 (0.0024) model time 0.3801 (0.3809) loss 3.0542 (3.1479) grad_norm 2.0365 (1.5825) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][330/625] eta 0:01:52 lr 0.001104 wd 0.0500 time 0.3713 (0.3823) data time 0.0008 (0.0024) model time 0.3705 (0.3807) loss 3.7722 (3.1550) grad_norm 1.2845 (1.5786) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][340/625] eta 0:01:48 lr 0.001104 wd 0.0500 time 0.3726 (0.3821) data time 0.0009 (0.0024) model time 0.3717 (0.3805) loss 2.7761 (3.1617) grad_norm 1.4526 (1.5738) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][350/625] eta 0:01:45 lr 0.001104 wd 0.0500 time 0.3867 (0.3820) data time 0.0009 (0.0023) model time 0.3858 (0.3804) loss 2.1735 (3.1624) grad_norm 1.8144 (1.5752) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][360/625] eta 0:01:41 lr 0.001103 wd 0.0500 time 0.3832 (0.3818) data time 0.0012 (0.0023) model time 0.3820 (0.3802) loss 2.8178 (3.1679) grad_norm 1.5388 (1.5794) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][370/625] eta 0:01:37 lr 0.001103 wd 0.0500 time 0.3773 (0.3817) data time 0.0008 (0.0023) model time 0.3765 (0.3800) loss 2.2814 (3.1719) grad_norm 1.3114 (1.5817) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][380/625] eta 0:01:33 lr 0.001103 wd 0.0500 time 0.3855 (0.3815) data time 0.0009 (0.0022) model time 0.3846 (0.3798) loss 3.6505 (3.1738) grad_norm 1.5243 (1.5815) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][390/625] eta 0:01:29 lr 0.001103 wd 0.0500 time 0.3765 (0.3813) data time 0.0009 (0.0022) model time 0.3756 (0.3797) loss 3.8866 (3.1663) grad_norm 1.5970 (1.5833) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][400/625] eta 0:01:25 lr 0.001103 wd 0.0500 time 0.3808 (0.3812) data time 0.0011 (0.0022) model time 0.3796 (0.3795) loss 3.0064 (3.1725) grad_norm 1.5957 (1.5844) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][410/625] eta 0:01:21 lr 0.001103 wd 0.0500 time 0.3745 (0.3811) data time 0.0011 (0.0022) model time 0.3734 (0.3794) loss 4.4560 (3.1697) grad_norm 1.7635 (1.5797) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][420/625] eta 0:01:18 lr 0.001102 wd 0.0500 time 0.3769 (0.3810) data time 0.0008 (0.0021) model time 0.3761 (0.3793) loss 2.1442 (3.1752) grad_norm 1.7194 (1.5795) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][430/625] eta 0:01:14 lr 0.001102 wd 0.0500 time 0.3832 (0.3808) data time 0.0011 (0.0021) model time 0.3821 (0.3791) loss 3.2606 (3.1750) grad_norm 1.2270 (1.5733) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:50:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][440/625] eta 0:01:10 lr 0.001102 wd 0.0500 time 0.3778 (0.3807) data time 0.0012 (0.0021) model time 0.3766 (0.3790) loss 3.6533 (3.1761) grad_norm 2.2302 (1.5790) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][450/625] eta 0:01:06 lr 0.001102 wd 0.0500 time 0.3679 (0.3806) data time 0.0008 (0.0021) model time 0.3670 (0.3789) loss 3.5999 (3.1800) grad_norm 1.1767 (1.5787) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][460/625] eta 0:01:02 lr 0.001102 wd 0.0500 time 0.3759 (0.3805) data time 0.0008 (0.0020) model time 0.3750 (0.3788) loss 3.8092 (3.1853) grad_norm 1.7823 (1.5776) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][470/625] eta 0:00:58 lr 0.001101 wd 0.0500 time 0.3708 (0.3804) data time 0.0009 (0.0020) model time 0.3699 (0.3787) loss 3.2892 (3.1863) grad_norm 1.0637 (1.5788) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][480/625] eta 0:00:55 lr 0.001101 wd 0.0500 time 0.3733 (0.3806) data time 0.0010 (0.0020) model time 0.3723 (0.3789) loss 3.9221 (3.1886) grad_norm 1.0618 (1.5769) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][490/625] eta 0:00:51 lr 0.001101 wd 0.0500 time 0.3722 (0.3805) data time 0.0011 (0.0020) model time 0.3711 (0.3789) loss 3.4740 (3.1901) grad_norm 1.5486 (1.5728) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][500/625] eta 0:00:47 lr 0.001101 wd 0.0500 time 0.5777 (0.3813) data time 0.0013 (0.0020) model time 0.5764 (0.3797) loss 3.5727 (3.1917) grad_norm 1.4362 (1.5711) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][510/625] eta 0:00:43 lr 0.001101 wd 0.0500 time 0.3774 (0.3813) data time 0.0010 (0.0020) model time 0.3764 (0.3797) loss 2.3942 (3.1882) grad_norm 1.4599 (1.5661) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][520/625] eta 0:00:40 lr 0.001101 wd 0.0500 time 0.3839 (0.3812) data time 0.0011 (0.0020) model time 0.3828 (0.3796) loss 2.7244 (3.1935) grad_norm 1.4017 (1.5667) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][530/625] eta 0:00:36 lr 0.001100 wd 0.0500 time 0.3739 (0.3811) data time 0.0009 (0.0019) model time 0.3731 (0.3795) loss 3.7981 (3.1895) grad_norm 1.8385 (1.5705) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][540/625] eta 0:00:32 lr 0.001100 wd 0.0500 time 0.3845 (0.3810) data time 0.0011 (0.0019) model time 0.3833 (0.3795) loss 3.2618 (3.1891) grad_norm 1.5695 (1.5820) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][550/625] eta 0:00:28 lr 0.001100 wd 0.0500 time 0.3729 (0.3809) data time 0.0010 (0.0019) model time 0.3719 (0.3793) loss 3.2080 (3.1864) grad_norm 1.1308 (1.5780) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][560/625] eta 0:00:24 lr 0.001100 wd 0.0500 time 0.3745 (0.3808) data time 0.0009 (0.0019) model time 0.3736 (0.3792) loss 2.8889 (3.1835) grad_norm 1.4255 (1.5761) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][570/625] eta 0:00:20 lr 0.001100 wd 0.0500 time 0.3715 (0.3807) data time 0.0011 (0.0019) model time 0.3704 (0.3791) loss 3.1008 (3.1834) grad_norm 1.6687 (1.5741) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][580/625] eta 0:00:17 lr 0.001100 wd 0.0500 time 0.3713 (0.3807) data time 0.0010 (0.0019) model time 0.3703 (0.3791) loss 3.7330 (3.1832) grad_norm 1.2874 (1.5781) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][590/625] eta 0:00:13 lr 0.001099 wd 0.0500 time 0.3863 (0.3806) data time 0.0008 (0.0019) model time 0.3855 (0.3790) loss 3.9392 (3.1833) grad_norm 1.1806 (1.5807) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:51:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][600/625] eta 0:00:09 lr 0.001099 wd 0.0500 time 0.3727 (0.3805) data time 0.0008 (0.0019) model time 0.3719 (0.3789) loss 3.7700 (3.1865) grad_norm 3.2603 (1.5848) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][610/625] eta 0:00:05 lr 0.001099 wd 0.0500 time 0.3553 (0.3804) data time 0.0008 (0.0018) model time 0.3546 (0.3788) loss 2.2415 (3.1857) grad_norm 1.6404 (1.5842) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [151/300][620/625] eta 0:00:01 lr 0.001099 wd 0.0500 time 0.3610 (0.3801) data time 0.0009 (0.0018) model time 0.3600 (0.3785) loss 3.0559 (3.1860) grad_norm 1.4062 (1.5806) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:08 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 151 training takes 0:03:57 [2024-08-27 09:52:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:52:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.6235 (0.6235) Acc@1 86.572 (86.572) Acc@5 97.998 (97.998) Mem 11224MB [2024-08-27 09:52:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 0.9902 (0.7552) Acc@1 77.441 (83.669) Acc@5 94.189 (96.831) Mem 11224MB [2024-08-27 09:52:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.1211 (0.8963) Acc@1 72.314 (80.108) Acc@5 93.164 (95.224) Mem 11224MB [2024-08-27 09:52:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.930 Acc@5 95.230 [2024-08-27 09:52:12 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.9% [2024-08-27 09:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.865 (0.865) Loss 0.5137 (0.5137) Acc@1 88.623 (88.623) Acc@5 98.584 (98.584) Mem 11224MB [2024-08-27 09:52:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.182) Loss 0.8467 (0.6410) Acc@1 79.736 (85.471) Acc@5 95.654 (97.483) Mem 11224MB [2024-08-27 09:52:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.140) Loss 0.9761 (0.7675) Acc@1 75.684 (82.310) Acc@5 94.531 (96.196) Mem 11224MB [2024-08-27 09:52:15 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.052 Acc@5 96.199 [2024-08-27 09:52:15 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-08-27 09:52:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][0/625] eta 0:14:10 lr 0.001099 wd 0.0500 time 1.3602 (1.3602) data time 0.5411 (0.5411) model time 0.0000 (0.0000) loss 3.1971 (3.1971) grad_norm 1.2571 (1.2571) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][10/625] eta 0:04:56 lr 0.001099 wd 0.0500 time 0.3781 (0.4828) data time 0.0013 (0.0503) model time 0.0000 (0.0000) loss 3.9383 (3.1435) grad_norm 1.2628 (1.5546) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][20/625] eta 0:04:21 lr 0.001098 wd 0.0500 time 0.3787 (0.4324) data time 0.0010 (0.0269) model time 0.0000 (0.0000) loss 2.8980 (3.1997) grad_norm 2.9241 (1.6498) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][30/625] eta 0:04:06 lr 0.001098 wd 0.0500 time 0.3824 (0.4151) data time 0.0009 (0.0185) model time 0.0000 (0.0000) loss 2.9192 (3.2459) grad_norm 1.6481 (1.6626) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][40/625] eta 0:03:57 lr 0.001098 wd 0.0500 time 0.3738 (0.4054) data time 0.0008 (0.0143) model time 0.0000 (0.0000) loss 3.7565 (3.2421) grad_norm 1.2604 (1.6590) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][50/625] eta 0:03:49 lr 0.001098 wd 0.0500 time 0.3715 (0.3991) data time 0.0010 (0.0117) model time 0.0000 (0.0000) loss 2.8025 (3.2227) grad_norm 2.3722 (1.6175) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][60/625] eta 0:03:43 lr 0.001098 wd 0.0500 time 0.3770 (0.3952) data time 0.0013 (0.0100) model time 0.3758 (0.3740) loss 2.1562 (3.1896) grad_norm 1.5969 (1.6219) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][70/625] eta 0:03:41 lr 0.001098 wd 0.0500 time 0.3731 (0.3986) data time 0.0010 (0.0087) model time 0.3721 (0.3962) loss 2.7385 (3.1633) grad_norm 1.7114 (1.6179) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][80/625] eta 0:03:35 lr 0.001097 wd 0.0500 time 0.3781 (0.3960) data time 0.0010 (0.0078) model time 0.3771 (0.3897) loss 2.6863 (3.1804) grad_norm 1.2529 (1.5964) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][90/625] eta 0:03:30 lr 0.001097 wd 0.0500 time 0.3714 (0.3937) data time 0.0010 (0.0071) model time 0.3704 (0.3857) loss 3.6299 (3.1832) grad_norm 1.1879 (1.5811) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][100/625] eta 0:03:25 lr 0.001097 wd 0.0500 time 0.3748 (0.3919) data time 0.0009 (0.0065) model time 0.3739 (0.3834) loss 3.3659 (3.1684) grad_norm 1.5675 (1.5706) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:52:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][110/625] eta 0:03:21 lr 0.001097 wd 0.0500 time 0.3839 (0.3906) data time 0.0012 (0.0060) model time 0.3827 (0.3822) loss 2.9183 (3.1909) grad_norm 1.2771 (1.5699) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][120/625] eta 0:03:16 lr 0.001097 wd 0.0500 time 0.3762 (0.3895) data time 0.0011 (0.0056) model time 0.3751 (0.3814) loss 2.7733 (3.1551) grad_norm 2.3184 (1.5732) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][130/625] eta 0:03:12 lr 0.001096 wd 0.0500 time 0.3606 (0.3897) data time 0.0009 (0.0053) model time 0.3597 (0.3826) loss 3.6561 (3.1728) grad_norm 1.7075 (1.6032) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][140/625] eta 0:03:08 lr 0.001096 wd 0.0500 time 0.3787 (0.3888) data time 0.0009 (0.0050) model time 0.3779 (0.3818) loss 4.0717 (3.1802) grad_norm 1.3996 (1.5848) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][150/625] eta 0:03:04 lr 0.001096 wd 0.0500 time 0.3775 (0.3880) data time 0.0008 (0.0047) model time 0.3767 (0.3812) loss 3.7017 (3.1900) grad_norm 1.5154 (1.5798) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][160/625] eta 0:03:00 lr 0.001096 wd 0.0500 time 0.3774 (0.3873) data time 0.0010 (0.0045) model time 0.3764 (0.3806) loss 2.4802 (3.2052) grad_norm 1.4727 (1.5774) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][170/625] eta 0:02:55 lr 0.001096 wd 0.0500 time 0.3692 (0.3866) data time 0.0008 (0.0043) model time 0.3684 (0.3801) loss 3.5794 (3.1978) grad_norm 1.6983 (1.5707) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][180/625] eta 0:02:51 lr 0.001096 wd 0.0500 time 0.3806 (0.3860) data time 0.0012 (0.0041) model time 0.3793 (0.3797) loss 3.5617 (3.1974) grad_norm 1.5355 (1.5924) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][190/625] eta 0:02:47 lr 0.001095 wd 0.0500 time 0.3756 (0.3854) data time 0.0008 (0.0040) model time 0.3748 (0.3793) loss 3.5537 (3.1964) grad_norm 1.0657 (1.5979) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][200/625] eta 0:02:43 lr 0.001095 wd 0.0500 time 0.3821 (0.3850) data time 0.0011 (0.0038) model time 0.3810 (0.3791) loss 3.3385 (3.1931) grad_norm 1.6408 (1.6087) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][210/625] eta 0:02:39 lr 0.001095 wd 0.0500 time 0.3791 (0.3845) data time 0.0011 (0.0037) model time 0.3781 (0.3787) loss 3.4525 (3.1919) grad_norm 1.7737 (1.6142) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][220/625] eta 0:02:35 lr 0.001095 wd 0.0500 time 0.3750 (0.3840) data time 0.0012 (0.0036) model time 0.3738 (0.3784) loss 3.2542 (3.2026) grad_norm 1.3218 (1.6078) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][230/625] eta 0:02:31 lr 0.001095 wd 0.0500 time 0.3803 (0.3836) data time 0.0012 (0.0035) model time 0.3791 (0.3781) loss 3.3827 (3.2103) grad_norm 1.3946 (1.6041) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][240/625] eta 0:02:27 lr 0.001094 wd 0.0500 time 0.3772 (0.3833) data time 0.0009 (0.0034) model time 0.3763 (0.3779) loss 1.9378 (3.2025) grad_norm 2.3258 (1.6050) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][250/625] eta 0:02:24 lr 0.001094 wd 0.0500 time 0.3634 (0.3847) data time 0.0010 (0.0033) model time 0.3624 (0.3799) loss 3.6244 (3.2094) grad_norm 1.3910 (1.5996) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][260/625] eta 0:02:20 lr 0.001094 wd 0.0500 time 0.3802 (0.3852) data time 0.0008 (0.0032) model time 0.3794 (0.3806) loss 3.4452 (3.2166) grad_norm 1.3014 (1.5914) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][270/625] eta 0:02:16 lr 0.001094 wd 0.0500 time 0.3664 (0.3848) data time 0.0010 (0.0031) model time 0.3654 (0.3804) loss 3.0273 (3.2195) grad_norm 1.5753 (1.5898) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:54:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][280/625] eta 0:02:12 lr 0.001094 wd 0.0500 time 0.3746 (0.3845) data time 0.0008 (0.0031) model time 0.3738 (0.3801) loss 4.4984 (3.2235) grad_norm 1.5042 (1.5936) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:54:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][290/625] eta 0:02:08 lr 0.001094 wd 0.0500 time 0.3795 (0.3842) data time 0.0008 (0.0030) model time 0.3787 (0.3799) loss 2.7188 (3.2220) grad_norm 1.3953 (1.5857) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 09:54:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][300/625] eta 0:02:04 lr 0.001093 wd 0.0500 time 0.3835 (0.3839) data time 0.0009 (0.0029) model time 0.3826 (0.3796) loss 4.2256 (3.2238) grad_norm 1.3257 (1.5942) loss_scale 1024.0000 (523.9070) mem 11224MB [2024-08-27 09:54:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][310/625] eta 0:02:00 lr 0.001093 wd 0.0500 time 0.3725 (0.3836) data time 0.0009 (0.0029) model time 0.3716 (0.3794) loss 2.9556 (3.2177) grad_norm 2.5359 (1.5929) loss_scale 1024.0000 (539.9871) mem 11224MB [2024-08-27 09:54:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][320/625] eta 0:01:56 lr 0.001093 wd 0.0500 time 0.3729 (0.3834) data time 0.0011 (0.0028) model time 0.3718 (0.3792) loss 3.4897 (3.2166) grad_norm 1.9935 (1.5881) loss_scale 1024.0000 (555.0654) mem 11224MB [2024-08-27 09:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][330/625] eta 0:01:53 lr 0.001093 wd 0.0500 time 0.3828 (0.3831) data time 0.0009 (0.0028) model time 0.3819 (0.3791) loss 2.9430 (3.2172) grad_norm 1.4473 (1.5829) loss_scale 1024.0000 (569.2326) mem 11224MB [2024-08-27 09:54:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][340/625] eta 0:01:49 lr 0.001093 wd 0.0500 time 0.3807 (0.3834) data time 0.0012 (0.0027) model time 0.3796 (0.3795) loss 3.1692 (3.2183) grad_norm 1.8260 (1.5858) loss_scale 1024.0000 (582.5689) mem 11224MB [2024-08-27 09:54:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][350/625] eta 0:01:45 lr 0.001093 wd 0.0500 time 0.3727 (0.3832) data time 0.0009 (0.0027) model time 0.3719 (0.3794) loss 3.6213 (3.2249) grad_norm 1.6031 (1.5876) loss_scale 1024.0000 (595.1453) mem 11224MB [2024-08-27 09:54:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][360/625] eta 0:01:41 lr 0.001092 wd 0.0500 time 0.3800 (0.3831) data time 0.0010 (0.0026) model time 0.3789 (0.3793) loss 3.0378 (3.2196) grad_norm 1.5299 (1.5894) loss_scale 1024.0000 (607.0249) mem 11224MB [2024-08-27 09:54:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][370/625] eta 0:01:38 lr 0.001092 wd 0.0500 time 0.5852 (0.3846) data time 0.0010 (0.0026) model time 0.5842 (0.3812) loss 3.9197 (3.2231) grad_norm 1.5107 (1.5952) loss_scale 1024.0000 (618.2642) mem 11224MB [2024-08-27 09:54:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][380/625] eta 0:01:34 lr 0.001092 wd 0.0500 time 0.3722 (0.3844) data time 0.0009 (0.0026) model time 0.3713 (0.3810) loss 3.9619 (3.2269) grad_norm 1.3308 (1.5941) loss_scale 1024.0000 (628.9134) mem 11224MB [2024-08-27 09:54:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][390/625] eta 0:01:30 lr 0.001092 wd 0.0500 time 0.3782 (0.3842) data time 0.0014 (0.0025) model time 0.3769 (0.3808) loss 4.0281 (3.2297) grad_norm 1.5111 (1.5931) loss_scale 1024.0000 (639.0179) mem 11224MB [2024-08-27 09:54:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][400/625] eta 0:01:26 lr 0.001092 wd 0.0500 time 0.3656 (0.3839) data time 0.0008 (0.0025) model time 0.3648 (0.3806) loss 3.5596 (3.2318) grad_norm 1.2985 (1.5919) loss_scale 1024.0000 (648.6185) mem 11224MB [2024-08-27 09:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][410/625] eta 0:01:22 lr 0.001091 wd 0.0500 time 0.3768 (0.3838) data time 0.0014 (0.0025) model time 0.3754 (0.3804) loss 3.0221 (3.2364) grad_norm 1.7371 (1.5945) loss_scale 1024.0000 (657.7518) mem 11224MB [2024-08-27 09:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][420/625] eta 0:01:18 lr 0.001091 wd 0.0500 time 0.3726 (0.3836) data time 0.0013 (0.0024) model time 0.3713 (0.3803) loss 3.5825 (3.2389) grad_norm 1.3506 (1.5890) loss_scale 1024.0000 (666.4513) mem 11224MB [2024-08-27 09:55:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][430/625] eta 0:01:14 lr 0.001091 wd 0.0500 time 0.3824 (0.3835) data time 0.0010 (0.0024) model time 0.3814 (0.3802) loss 3.7804 (3.2439) grad_norm 1.1707 (1.5864) loss_scale 1024.0000 (674.7471) mem 11224MB [2024-08-27 09:55:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][440/625] eta 0:01:10 lr 0.001091 wd 0.0500 time 0.3829 (0.3833) data time 0.0011 (0.0024) model time 0.3818 (0.3801) loss 3.0637 (3.2382) grad_norm 0.9769 (1.5919) loss_scale 1024.0000 (682.6667) mem 11224MB [2024-08-27 09:55:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][450/625] eta 0:01:07 lr 0.001091 wd 0.0500 time 0.3730 (0.3832) data time 0.0011 (0.0024) model time 0.3718 (0.3800) loss 2.8100 (3.2343) grad_norm 1.0978 (1.5886) loss_scale 1024.0000 (690.2350) mem 11224MB [2024-08-27 09:55:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][460/625] eta 0:01:03 lr 0.001091 wd 0.0500 time 0.3740 (0.3830) data time 0.0013 (0.0023) model time 0.3727 (0.3799) loss 2.7242 (3.2309) grad_norm 1.2567 (1.5840) loss_scale 1024.0000 (697.4751) mem 11224MB [2024-08-27 09:55:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][470/625] eta 0:00:59 lr 0.001090 wd 0.0500 time 0.3761 (0.3829) data time 0.0009 (0.0023) model time 0.3753 (0.3797) loss 2.7169 (3.2368) grad_norm 1.1460 (1.5795) loss_scale 1024.0000 (704.4076) mem 11224MB [2024-08-27 09:55:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][480/625] eta 0:00:55 lr 0.001090 wd 0.0500 time 0.3836 (0.3828) data time 0.0011 (0.0023) model time 0.3824 (0.3796) loss 3.2675 (3.2377) grad_norm 2.0944 (1.5811) loss_scale 1024.0000 (711.0520) mem 11224MB [2024-08-27 09:55:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][490/625] eta 0:00:51 lr 0.001090 wd 0.0500 time 0.3749 (0.3826) data time 0.0011 (0.0023) model time 0.3738 (0.3795) loss 2.4728 (3.2347) grad_norm 1.5094 (1.5864) loss_scale 1024.0000 (717.4257) mem 11224MB [2024-08-27 09:55:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][500/625] eta 0:00:47 lr 0.001090 wd 0.0500 time 0.3769 (0.3825) data time 0.0011 (0.0022) model time 0.3758 (0.3795) loss 2.7418 (3.2319) grad_norm 1.0920 (1.5884) loss_scale 1024.0000 (723.5449) mem 11224MB [2024-08-27 09:55:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][510/625] eta 0:00:43 lr 0.001090 wd 0.0500 time 0.3765 (0.3825) data time 0.0012 (0.0022) model time 0.3753 (0.3794) loss 3.4396 (3.2322) grad_norm 1.8496 (1.5902) loss_scale 1024.0000 (729.4247) mem 11224MB [2024-08-27 09:55:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][520/625] eta 0:00:40 lr 0.001090 wd 0.0500 time 0.3761 (0.3826) data time 0.0013 (0.0022) model time 0.3748 (0.3796) loss 3.4104 (3.2321) grad_norm 1.3624 (1.5910) loss_scale 1024.0000 (735.0787) mem 11224MB [2024-08-27 09:55:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][530/625] eta 0:00:36 lr 0.001089 wd 0.0500 time 0.3759 (0.3825) data time 0.0011 (0.0022) model time 0.3748 (0.3796) loss 3.3693 (3.2320) grad_norm 2.2555 (1.6007) loss_scale 1024.0000 (740.5198) mem 11224MB [2024-08-27 09:55:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][540/625] eta 0:00:32 lr 0.001089 wd 0.0500 time 0.3745 (0.3824) data time 0.0010 (0.0022) model time 0.3735 (0.3795) loss 3.1548 (3.2295) grad_norm 1.0304 (1.5991) loss_scale 1024.0000 (745.7597) mem 11224MB [2024-08-27 09:55:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][550/625] eta 0:00:28 lr 0.001089 wd 0.0500 time 0.3655 (0.3823) data time 0.0009 (0.0021) model time 0.3646 (0.3795) loss 3.6344 (3.2265) grad_norm 1.5940 (1.6027) loss_scale 1024.0000 (750.8094) mem 11224MB [2024-08-27 09:55:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][560/625] eta 0:00:24 lr 0.001089 wd 0.0500 time 0.3748 (0.3823) data time 0.0008 (0.0021) model time 0.3740 (0.3794) loss 2.8241 (3.2201) grad_norm 1.3455 (1.6092) loss_scale 1024.0000 (755.6791) mem 11224MB [2024-08-27 09:55:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][570/625] eta 0:00:21 lr 0.001089 wd 0.0500 time 0.3772 (0.3822) data time 0.0008 (0.0021) model time 0.3764 (0.3794) loss 3.5693 (3.2164) grad_norm 1.5507 (1.6057) loss_scale 1024.0000 (760.3783) mem 11224MB [2024-08-27 09:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][580/625] eta 0:00:17 lr 0.001088 wd 0.0500 time 0.3878 (0.3829) data time 0.0011 (0.0021) model time 0.3867 (0.3802) loss 2.0199 (3.2147) grad_norm 1.3652 (1.6033) loss_scale 1024.0000 (764.9157) mem 11224MB [2024-08-27 09:56:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][590/625] eta 0:00:13 lr 0.001088 wd 0.0500 time 0.3858 (0.3831) data time 0.0012 (0.0021) model time 0.3846 (0.3805) loss 2.9803 (3.2111) grad_norm 1.6105 (1.6038) loss_scale 1024.0000 (769.2995) mem 11224MB [2024-08-27 09:56:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][600/625] eta 0:00:09 lr 0.001088 wd 0.0500 time 0.3818 (0.3830) data time 0.0010 (0.0021) model time 0.3808 (0.3803) loss 3.6061 (3.2137) grad_norm 1.9814 (1.6026) loss_scale 1024.0000 (773.5374) mem 11224MB [2024-08-27 09:56:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][610/625] eta 0:00:05 lr 0.001088 wd 0.0500 time 0.3664 (0.3829) data time 0.0006 (0.0021) model time 0.3659 (0.3802) loss 2.2985 (3.2102) grad_norm 1.4832 (1.6018) loss_scale 1024.0000 (777.6367) mem 11224MB [2024-08-27 09:56:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [152/300][620/625] eta 0:00:01 lr 0.001088 wd 0.0500 time 0.3626 (0.3825) data time 0.0005 (0.0020) model time 0.3621 (0.3799) loss 2.7904 (3.2124) grad_norm 1.8601 (1.5997) loss_scale 1024.0000 (781.6039) mem 11224MB [2024-08-27 09:56:14 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 152 training takes 0:03:59 [2024-08-27 09:56:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 09:56:15 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 09:56:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.5830 (0.5830) Acc@1 87.744 (87.744) Acc@5 97.949 (97.949) Mem 11224MB [2024-08-27 09:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.143) Loss 0.9731 (0.7209) Acc@1 76.807 (83.572) Acc@5 94.043 (96.804) Mem 11224MB [2024-08-27 09:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.0664 (0.8636) Acc@1 74.121 (80.185) Acc@5 93.311 (95.238) Mem 11224MB [2024-08-27 09:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.916 Acc@5 95.256 [2024-08-27 09:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.9% [2024-08-27 09:56:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.922 (0.922) Loss 0.5137 (0.5137) Acc@1 88.770 (88.770) Acc@5 98.633 (98.633) Mem 11224MB [2024-08-27 09:56:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.183) Loss 0.8452 (0.6401) Acc@1 79.688 (85.525) Acc@5 95.654 (97.488) Mem 11224MB [2024-08-27 09:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.141) Loss 0.9761 (0.7665) Acc@1 75.488 (82.331) Acc@5 94.482 (96.210) Mem 11224MB [2024-08-27 09:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.092 Acc@5 96.213 [2024-08-27 09:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-08-27 09:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.09% [2024-08-27 09:56:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 09:56:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 09:56:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][0/625] eta 0:07:33 lr 0.001088 wd 0.0500 time 0.7255 (0.7255) data time 0.3722 (0.3722) model time 0.0000 (0.0000) loss 2.1592 (2.1592) grad_norm 2.2870 (2.2870) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][10/625] eta 0:04:11 lr 0.001087 wd 0.0500 time 0.3829 (0.4085) data time 0.0009 (0.0349) model time 0.0000 (0.0000) loss 4.0350 (3.4047) grad_norm 1.1769 (1.5157) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][20/625] eta 0:03:58 lr 0.001087 wd 0.0500 time 0.3794 (0.3941) data time 0.0011 (0.0189) model time 0.0000 (0.0000) loss 2.6975 (3.3113) grad_norm 1.2392 (1.5019) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][30/625] eta 0:03:51 lr 0.001087 wd 0.0500 time 0.3781 (0.3887) data time 0.0008 (0.0132) model time 0.0000 (0.0000) loss 2.8352 (3.1773) grad_norm 1.9432 (1.5302) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][40/625] eta 0:03:45 lr 0.001087 wd 0.0500 time 0.3767 (0.3862) data time 0.0012 (0.0102) model time 0.0000 (0.0000) loss 3.7422 (3.1753) grad_norm 1.6051 (1.5722) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][50/625] eta 0:03:40 lr 0.001087 wd 0.0500 time 0.3755 (0.3837) data time 0.0009 (0.0085) model time 0.0000 (0.0000) loss 3.7243 (3.1852) grad_norm 1.1532 (1.5729) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][60/625] eta 0:03:36 lr 0.001087 wd 0.0500 time 0.3756 (0.3827) data time 0.0009 (0.0073) model time 0.3747 (0.3769) loss 3.9756 (3.1603) grad_norm 1.2205 (1.5812) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][70/625] eta 0:03:31 lr 0.001086 wd 0.0500 time 0.3748 (0.3819) data time 0.0010 (0.0064) model time 0.3738 (0.3762) loss 3.3164 (3.1918) grad_norm 2.6278 (1.5650) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][80/625] eta 0:03:27 lr 0.001086 wd 0.0500 time 0.3776 (0.3812) data time 0.0009 (0.0057) model time 0.3768 (0.3759) loss 2.6255 (3.1661) grad_norm 1.3001 (1.5705) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:56:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][90/625] eta 0:03:23 lr 0.001086 wd 0.0500 time 0.3722 (0.3801) data time 0.0013 (0.0052) model time 0.3709 (0.3745) loss 3.0842 (3.1663) grad_norm 1.4446 (1.5888) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][100/625] eta 0:03:19 lr 0.001086 wd 0.0500 time 0.3749 (0.3797) data time 0.0014 (0.0048) model time 0.3735 (0.3745) loss 3.6262 (3.1725) grad_norm 1.8661 (1.5889) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][110/625] eta 0:03:15 lr 0.001086 wd 0.0500 time 0.3680 (0.3790) data time 0.0010 (0.0045) model time 0.3671 (0.3739) loss 3.8135 (3.1659) grad_norm 1.3648 (1.5745) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][120/625] eta 0:03:11 lr 0.001086 wd 0.0500 time 0.3632 (0.3783) data time 0.0012 (0.0042) model time 0.3620 (0.3733) loss 4.0189 (3.1721) grad_norm 1.7995 (1.5534) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][130/625] eta 0:03:07 lr 0.001085 wd 0.0500 time 0.3715 (0.3782) data time 0.0011 (0.0040) model time 0.3704 (0.3735) loss 3.1748 (3.1731) grad_norm 1.1643 (1.5396) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][140/625] eta 0:03:03 lr 0.001085 wd 0.0500 time 0.3747 (0.3780) data time 0.0008 (0.0038) model time 0.3739 (0.3736) loss 3.0636 (3.1651) grad_norm 1.1998 (1.5384) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][150/625] eta 0:02:59 lr 0.001085 wd 0.0500 time 0.3746 (0.3778) data time 0.0011 (0.0036) model time 0.3735 (0.3736) loss 3.0894 (3.1656) grad_norm 1.1199 (1.5251) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][160/625] eta 0:02:57 lr 0.001085 wd 0.0500 time 0.3811 (0.3808) data time 0.0010 (0.0035) model time 0.3800 (0.3783) loss 3.7925 (3.1542) grad_norm 1.5148 (1.5255) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][170/625] eta 0:02:53 lr 0.001085 wd 0.0500 time 0.3801 (0.3805) data time 0.0009 (0.0033) model time 0.3793 (0.3779) loss 3.8215 (3.1638) grad_norm 1.7939 (1.5315) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][180/625] eta 0:02:49 lr 0.001084 wd 0.0500 time 0.3613 (0.3801) data time 0.0010 (0.0032) model time 0.3603 (0.3775) loss 2.6763 (3.1574) grad_norm 1.2455 (1.5340) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][190/625] eta 0:02:45 lr 0.001084 wd 0.0500 time 0.3857 (0.3809) data time 0.0010 (0.0031) model time 0.3847 (0.3787) loss 3.2588 (3.1603) grad_norm 1.7098 (1.5380) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][200/625] eta 0:02:41 lr 0.001084 wd 0.0500 time 0.3726 (0.3806) data time 0.0011 (0.0030) model time 0.3715 (0.3783) loss 2.9844 (3.1697) grad_norm 1.4586 (1.5278) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][210/625] eta 0:02:37 lr 0.001084 wd 0.0500 time 0.3761 (0.3804) data time 0.0008 (0.0029) model time 0.3753 (0.3782) loss 2.8515 (3.1762) grad_norm 1.4762 (1.5200) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][220/625] eta 0:02:33 lr 0.001084 wd 0.0500 time 0.3766 (0.3802) data time 0.0009 (0.0029) model time 0.3757 (0.3780) loss 3.7974 (3.1639) grad_norm 1.4326 (1.5211) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][230/625] eta 0:02:30 lr 0.001084 wd 0.0500 time 0.3741 (0.3800) data time 0.0008 (0.0028) model time 0.3733 (0.3778) loss 3.4385 (3.1680) grad_norm 1.8461 (1.5301) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][240/625] eta 0:02:26 lr 0.001083 wd 0.0500 time 0.3765 (0.3799) data time 0.0010 (0.0027) model time 0.3755 (0.3777) loss 3.1147 (3.1634) grad_norm 1.4195 (1.5283) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][250/625] eta 0:02:22 lr 0.001083 wd 0.0500 time 0.3787 (0.3797) data time 0.0013 (0.0027) model time 0.3774 (0.3775) loss 2.7932 (3.1639) grad_norm 1.1310 (1.5217) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][260/625] eta 0:02:18 lr 0.001083 wd 0.0500 time 0.3668 (0.3794) data time 0.0009 (0.0026) model time 0.3660 (0.3772) loss 2.6184 (3.1652) grad_norm 1.7618 (1.5202) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][270/625] eta 0:02:14 lr 0.001083 wd 0.0500 time 0.3679 (0.3793) data time 0.0011 (0.0025) model time 0.3668 (0.3771) loss 3.6752 (3.1642) grad_norm 1.4602 (1.5209) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][280/625] eta 0:02:10 lr 0.001083 wd 0.0500 time 0.3725 (0.3791) data time 0.0011 (0.0025) model time 0.3715 (0.3770) loss 3.3434 (3.1738) grad_norm 2.0220 (1.5270) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][290/625] eta 0:02:06 lr 0.001083 wd 0.0500 time 0.3740 (0.3790) data time 0.0011 (0.0024) model time 0.3729 (0.3768) loss 3.4668 (3.1810) grad_norm 1.1040 (1.5288) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][300/625] eta 0:02:03 lr 0.001082 wd 0.0500 time 0.3725 (0.3788) data time 0.0013 (0.0024) model time 0.3713 (0.3767) loss 3.0083 (3.1779) grad_norm 1.5439 (1.5291) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][310/625] eta 0:01:59 lr 0.001082 wd 0.0500 time 0.3787 (0.3787) data time 0.0009 (0.0024) model time 0.3778 (0.3766) loss 3.8712 (3.1894) grad_norm 1.0887 (1.5236) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][320/625] eta 0:01:55 lr 0.001082 wd 0.0500 time 0.3758 (0.3786) data time 0.0009 (0.0023) model time 0.3749 (0.3765) loss 4.0075 (3.2013) grad_norm 1.8999 (1.5192) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][330/625] eta 0:01:51 lr 0.001082 wd 0.0500 time 0.3707 (0.3785) data time 0.0008 (0.0023) model time 0.3699 (0.3764) loss 3.9183 (3.2001) grad_norm 1.6011 (1.5191) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][340/625] eta 0:01:48 lr 0.001082 wd 0.0500 time 0.3716 (0.3794) data time 0.0009 (0.0023) model time 0.3707 (0.3775) loss 3.3501 (3.2006) grad_norm 1.5251 (1.5216) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][350/625] eta 0:01:44 lr 0.001081 wd 0.0500 time 0.3734 (0.3799) data time 0.0011 (0.0022) model time 0.3723 (0.3781) loss 3.2003 (3.1991) grad_norm 1.3280 (1.5171) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][360/625] eta 0:01:40 lr 0.001081 wd 0.0500 time 0.3717 (0.3797) data time 0.0013 (0.0022) model time 0.3704 (0.3779) loss 3.3853 (3.1974) grad_norm 1.3016 (1.5183) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][370/625] eta 0:01:36 lr 0.001081 wd 0.0500 time 0.3715 (0.3800) data time 0.0014 (0.0022) model time 0.3701 (0.3782) loss 3.7345 (3.1986) grad_norm 1.2513 (1.5119) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][380/625] eta 0:01:33 lr 0.001081 wd 0.0500 time 0.3733 (0.3799) data time 0.0009 (0.0021) model time 0.3724 (0.3781) loss 2.5585 (3.1977) grad_norm 1.4211 (1.5089) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][390/625] eta 0:01:29 lr 0.001081 wd 0.0500 time 0.3738 (0.3798) data time 0.0008 (0.0021) model time 0.3730 (0.3781) loss 4.2764 (3.2068) grad_norm 1.4282 (1.5064) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][400/625] eta 0:01:25 lr 0.001081 wd 0.0500 time 0.3708 (0.3797) data time 0.0008 (0.0021) model time 0.3700 (0.3779) loss 2.1370 (3.2033) grad_norm 1.5464 (1.5016) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:58:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][410/625] eta 0:01:21 lr 0.001080 wd 0.0500 time 0.3741 (0.3795) data time 0.0010 (0.0021) model time 0.3732 (0.3778) loss 3.2040 (3.1987) grad_norm 1.2608 (1.5026) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][420/625] eta 0:01:17 lr 0.001080 wd 0.0500 time 0.3824 (0.3794) data time 0.0009 (0.0020) model time 0.3815 (0.3777) loss 3.9988 (3.1948) grad_norm 1.6223 (1.5019) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][430/625] eta 0:01:13 lr 0.001080 wd 0.0500 time 0.3755 (0.3793) data time 0.0012 (0.0020) model time 0.3743 (0.3776) loss 2.1420 (3.1942) grad_norm 1.1135 (1.4989) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][440/625] eta 0:01:10 lr 0.001080 wd 0.0500 time 0.3653 (0.3792) data time 0.0010 (0.0020) model time 0.3642 (0.3775) loss 2.4260 (3.1930) grad_norm 1.2310 (1.4985) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][450/625] eta 0:01:06 lr 0.001080 wd 0.0500 time 0.3764 (0.3791) data time 0.0012 (0.0020) model time 0.3752 (0.3774) loss 3.2763 (3.1938) grad_norm 1.2404 (1.4976) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][460/625] eta 0:01:02 lr 0.001080 wd 0.0500 time 0.3749 (0.3799) data time 0.0008 (0.0020) model time 0.3741 (0.3783) loss 3.8567 (3.1994) grad_norm 1.2427 (1.4981) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][470/625] eta 0:00:58 lr 0.001079 wd 0.0500 time 0.3750 (0.3803) data time 0.0011 (0.0020) model time 0.3739 (0.3787) loss 4.1409 (3.2037) grad_norm 2.3394 (1.4990) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][480/625] eta 0:00:55 lr 0.001079 wd 0.0500 time 0.3717 (0.3802) data time 0.0010 (0.0019) model time 0.3707 (0.3786) loss 3.9694 (3.1979) grad_norm 1.3552 (1.4994) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][490/625] eta 0:00:51 lr 0.001079 wd 0.0500 time 0.3717 (0.3804) data time 0.0008 (0.0019) model time 0.3709 (0.3789) loss 4.1759 (3.2002) grad_norm 1.9082 (1.5039) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][500/625] eta 0:00:47 lr 0.001079 wd 0.0500 time 0.3724 (0.3804) data time 0.0013 (0.0019) model time 0.3711 (0.3788) loss 3.2579 (3.2017) grad_norm 1.1374 (1.5048) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][510/625] eta 0:00:43 lr 0.001079 wd 0.0500 time 0.3772 (0.3803) data time 0.0012 (0.0019) model time 0.3760 (0.3788) loss 2.7971 (3.1981) grad_norm 1.3694 (1.5049) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][520/625] eta 0:00:39 lr 0.001078 wd 0.0500 time 0.3761 (0.3802) data time 0.0013 (0.0019) model time 0.3748 (0.3787) loss 2.4224 (3.1970) grad_norm 1.6915 (1.5097) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][530/625] eta 0:00:36 lr 0.001078 wd 0.0500 time 0.3824 (0.3802) data time 0.0008 (0.0019) model time 0.3816 (0.3786) loss 2.4568 (3.1958) grad_norm 1.4203 (1.5095) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][540/625] eta 0:00:32 lr 0.001078 wd 0.0500 time 0.3829 (0.3801) data time 0.0011 (0.0019) model time 0.3818 (0.3785) loss 2.1980 (3.1963) grad_norm 1.4777 (1.5082) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 09:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][550/625] eta 0:00:28 lr 0.001078 wd 0.0500 time 0.3796 (0.3800) data time 0.0010 (0.0018) model time 0.3786 (0.3785) loss 3.7410 (3.1927) grad_norm 2.1165 (inf) loss_scale 512.0000 (1014.7078) mem 11224MB [2024-08-27 09:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][560/625] eta 0:00:24 lr 0.001078 wd 0.0500 time 0.3754 (0.3799) data time 0.0008 (0.0018) model time 0.3746 (0.3783) loss 2.9774 (3.1914) grad_norm 1.2882 (inf) loss_scale 512.0000 (1005.7469) mem 11224MB [2024-08-27 10:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][570/625] eta 0:00:20 lr 0.001078 wd 0.0500 time 0.3806 (0.3798) data time 0.0008 (0.0018) model time 0.3798 (0.3783) loss 3.1851 (3.1958) grad_norm 1.3558 (inf) loss_scale 512.0000 (997.0998) mem 11224MB [2024-08-27 10:00:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][580/625] eta 0:00:17 lr 0.001077 wd 0.0500 time 0.3763 (0.3797) data time 0.0009 (0.0018) model time 0.3754 (0.3782) loss 4.1944 (3.1958) grad_norm 1.8862 (inf) loss_scale 512.0000 (988.7504) mem 11224MB [2024-08-27 10:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][590/625] eta 0:00:13 lr 0.001077 wd 0.0500 time 0.3736 (0.3797) data time 0.0010 (0.0018) model time 0.3726 (0.3781) loss 2.1628 (3.1941) grad_norm 1.1225 (inf) loss_scale 512.0000 (980.6836) mem 11224MB [2024-08-27 10:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][600/625] eta 0:00:09 lr 0.001077 wd 0.0500 time 0.3813 (0.3797) data time 0.0008 (0.0018) model time 0.3805 (0.3781) loss 4.0196 (3.1946) grad_norm 1.5696 (inf) loss_scale 512.0000 (972.8852) mem 11224MB [2024-08-27 10:00:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][610/625] eta 0:00:05 lr 0.001077 wd 0.0500 time 0.3725 (0.3796) data time 0.0006 (0.0018) model time 0.3719 (0.3781) loss 2.7116 (3.1938) grad_norm 2.1929 (inf) loss_scale 512.0000 (965.3421) mem 11224MB [2024-08-27 10:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [153/300][620/625] eta 0:00:01 lr 0.001077 wd 0.0500 time 0.3593 (0.3793) data time 0.0005 (0.0018) model time 0.3588 (0.3778) loss 3.0510 (3.1907) grad_norm 1.4514 (inf) loss_scale 512.0000 (958.0419) mem 11224MB [2024-08-27 10:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 153 training takes 0:03:57 [2024-08-27 10:00:20 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 10:00:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 10:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.498 (0.498) Loss 0.6118 (0.6118) Acc@1 87.354 (87.354) Acc@5 97.754 (97.754) Mem 11224MB [2024-08-27 10:00:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 1.0264 (0.7498) Acc@1 76.807 (83.447) Acc@5 94.092 (96.782) Mem 11224MB [2024-08-27 10:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 1.0947 (0.8912) Acc@1 74.268 (79.950) Acc@5 92.969 (95.196) Mem 11224MB [2024-08-27 10:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.750 Acc@5 95.188 [2024-08-27 10:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.8% [2024-08-27 10:00:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.861 (0.861) Loss 0.5137 (0.5137) Acc@1 88.867 (88.867) Acc@5 98.633 (98.633) Mem 11224MB [2024-08-27 10:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.182) Loss 0.8442 (0.6393) Acc@1 79.785 (85.556) Acc@5 95.752 (97.528) Mem 11224MB [2024-08-27 10:00:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.140) Loss 0.9736 (0.7653) Acc@1 75.586 (82.361) Acc@5 94.629 (96.254) Mem 11224MB [2024-08-27 10:00:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.116 Acc@5 96.253 [2024-08-27 10:00:27 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-08-27 10:00:27 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.12% [2024-08-27 10:00:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 10:00:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 10:00:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][0/625] eta 0:07:48 lr 0.001077 wd 0.0500 time 0.7503 (0.7503) data time 0.4088 (0.4088) model time 0.0000 (0.0000) loss 2.9412 (2.9412) grad_norm 1.2222 (1.2222) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:00:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][10/625] eta 0:04:12 lr 0.001076 wd 0.0500 time 0.3761 (0.4101) data time 0.0009 (0.0382) model time 0.0000 (0.0000) loss 4.0694 (3.3064) grad_norm 1.0284 (1.3765) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][20/625] eta 0:04:01 lr 0.001076 wd 0.0500 time 0.3755 (0.3992) data time 0.0008 (0.0205) model time 0.0000 (0.0000) loss 3.2431 (3.3023) grad_norm 1.6949 (1.3256) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][30/625] eta 0:04:04 lr 0.001076 wd 0.0500 time 0.3721 (0.4107) data time 0.0011 (0.0143) model time 0.0000 (0.0000) loss 3.2969 (3.2776) grad_norm 0.9667 (1.3793) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:00:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][40/625] eta 0:03:55 lr 0.001076 wd 0.0500 time 0.3643 (0.4022) data time 0.0012 (0.0111) model time 0.0000 (0.0000) loss 3.1238 (3.2854) grad_norm 1.1332 (1.4112) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][50/625] eta 0:03:48 lr 0.001076 wd 0.0500 time 0.3750 (0.3972) data time 0.0008 (0.0091) model time 0.0000 (0.0000) loss 3.4313 (3.2379) grad_norm 2.2559 (1.4953) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:00:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][60/625] eta 0:03:42 lr 0.001076 wd 0.0500 time 0.3685 (0.3936) data time 0.0008 (0.0078) model time 0.3677 (0.3744) loss 2.1159 (3.2111) grad_norm 1.2621 (1.5218) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][70/625] eta 0:03:36 lr 0.001075 wd 0.0500 time 0.3713 (0.3909) data time 0.0011 (0.0069) model time 0.3703 (0.3738) loss 2.6454 (3.2287) grad_norm 1.2787 (1.5601) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][80/625] eta 0:03:31 lr 0.001075 wd 0.0500 time 0.3647 (0.3887) data time 0.0009 (0.0062) model time 0.3638 (0.3732) loss 3.2094 (3.2656) grad_norm 1.3676 (1.5494) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][90/625] eta 0:03:27 lr 0.001075 wd 0.0500 time 0.3788 (0.3873) data time 0.0008 (0.0056) model time 0.3781 (0.3737) loss 1.9824 (3.2462) grad_norm 1.2003 (1.5379) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][100/625] eta 0:03:22 lr 0.001075 wd 0.0500 time 0.3679 (0.3859) data time 0.0010 (0.0052) model time 0.3669 (0.3733) loss 2.5462 (3.2441) grad_norm 1.5641 (1.5391) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][110/625] eta 0:03:18 lr 0.001075 wd 0.0500 time 0.3805 (0.3848) data time 0.0008 (0.0048) model time 0.3797 (0.3732) loss 2.4496 (3.2274) grad_norm 1.6794 (1.5290) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][120/625] eta 0:03:13 lr 0.001074 wd 0.0500 time 0.3842 (0.3842) data time 0.0007 (0.0045) model time 0.3835 (0.3735) loss 3.9227 (3.2328) grad_norm 1.2085 (1.5255) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][130/625] eta 0:03:09 lr 0.001074 wd 0.0500 time 0.3739 (0.3836) data time 0.0008 (0.0043) model time 0.3731 (0.3738) loss 3.1921 (3.2259) grad_norm 1.4246 (1.5152) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][140/625] eta 0:03:05 lr 0.001074 wd 0.0500 time 0.3724 (0.3830) data time 0.0012 (0.0040) model time 0.3712 (0.3738) loss 3.1210 (3.2342) grad_norm 1.5244 (1.5066) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][150/625] eta 0:03:01 lr 0.001074 wd 0.0500 time 0.3794 (0.3825) data time 0.0010 (0.0039) model time 0.3784 (0.3738) loss 3.3952 (3.2205) grad_norm 1.9430 (1.5044) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][160/625] eta 0:02:57 lr 0.001074 wd 0.0500 time 0.3735 (0.3820) data time 0.0010 (0.0037) model time 0.3724 (0.3738) loss 3.6886 (3.2248) grad_norm 1.2637 (1.5202) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][170/625] eta 0:02:53 lr 0.001074 wd 0.0500 time 0.3699 (0.3816) data time 0.0009 (0.0035) model time 0.3690 (0.3738) loss 3.3048 (3.2330) grad_norm 1.8871 (1.5269) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][180/625] eta 0:02:49 lr 0.001073 wd 0.0500 time 0.3807 (0.3814) data time 0.0008 (0.0034) model time 0.3799 (0.3740) loss 3.6048 (3.2307) grad_norm 1.2415 (1.5266) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][190/625] eta 0:02:45 lr 0.001073 wd 0.0500 time 0.3749 (0.3811) data time 0.0008 (0.0033) model time 0.3741 (0.3741) loss 3.9223 (3.2349) grad_norm 1.4844 (1.5248) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][200/625] eta 0:02:42 lr 0.001073 wd 0.0500 time 0.3744 (0.3815) data time 0.0010 (0.0032) model time 0.3734 (0.3751) loss 1.9732 (3.2342) grad_norm 1.5605 (1.5289) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][210/625] eta 0:02:38 lr 0.001073 wd 0.0500 time 0.3810 (0.3813) data time 0.0013 (0.0031) model time 0.3797 (0.3750) loss 3.3194 (3.2269) grad_norm 1.4248 (1.5376) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][220/625] eta 0:02:34 lr 0.001073 wd 0.0500 time 0.3722 (0.3811) data time 0.0011 (0.0030) model time 0.3711 (0.3751) loss 2.7410 (3.2341) grad_norm 1.2921 (1.5406) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:01:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][230/625] eta 0:02:30 lr 0.001073 wd 0.0500 time 0.3750 (0.3809) data time 0.0011 (0.0029) model time 0.3739 (0.3751) loss 3.2433 (3.2323) grad_norm 3.8727 (1.5587) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][240/625] eta 0:02:27 lr 0.001072 wd 0.0500 time 0.3842 (0.3827) data time 0.0008 (0.0029) model time 0.3833 (0.3776) loss 4.0329 (3.2325) grad_norm 1.9312 (1.5644) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][250/625] eta 0:02:23 lr 0.001072 wd 0.0500 time 0.3692 (0.3823) data time 0.0014 (0.0028) model time 0.3678 (0.3774) loss 3.1483 (3.2318) grad_norm 1.7566 (1.5750) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][260/625] eta 0:02:19 lr 0.001072 wd 0.0500 time 0.3755 (0.3820) data time 0.0007 (0.0027) model time 0.3747 (0.3772) loss 3.5185 (3.2255) grad_norm 1.7060 (1.5749) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][270/625] eta 0:02:15 lr 0.001072 wd 0.0500 time 0.3722 (0.3819) data time 0.0014 (0.0027) model time 0.3708 (0.3772) loss 3.4967 (3.2181) grad_norm 1.5440 (1.5795) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][280/625] eta 0:02:11 lr 0.001072 wd 0.0500 time 0.3739 (0.3818) data time 0.0009 (0.0026) model time 0.3731 (0.3772) loss 3.3575 (3.2233) grad_norm 1.0688 (1.5746) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][290/625] eta 0:02:07 lr 0.001071 wd 0.0500 time 0.3805 (0.3816) data time 0.0008 (0.0026) model time 0.3797 (0.3772) loss 3.1159 (3.2079) grad_norm 1.6820 (1.5696) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][300/625] eta 0:02:03 lr 0.001071 wd 0.0500 time 0.3718 (0.3815) data time 0.0011 (0.0025) model time 0.3707 (0.3771) loss 3.1348 (3.2048) grad_norm 1.6316 (1.5713) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][310/625] eta 0:02:00 lr 0.001071 wd 0.0500 time 0.3875 (0.3813) data time 0.0010 (0.0025) model time 0.3865 (0.3770) loss 3.5129 (3.2027) grad_norm 1.4431 (1.5701) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][320/625] eta 0:01:56 lr 0.001071 wd 0.0500 time 0.3658 (0.3816) data time 0.0010 (0.0024) model time 0.3648 (0.3775) loss 3.1049 (3.1979) grad_norm 1.5510 (1.5660) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][330/625] eta 0:01:52 lr 0.001071 wd 0.0500 time 0.3762 (0.3815) data time 0.0008 (0.0024) model time 0.3754 (0.3775) loss 3.2049 (3.1930) grad_norm 1.5165 (1.5684) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][340/625] eta 0:01:48 lr 0.001071 wd 0.0500 time 0.3781 (0.3814) data time 0.0014 (0.0023) model time 0.3766 (0.3774) loss 3.0085 (3.1894) grad_norm 1.1709 (1.5683) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][350/625] eta 0:01:44 lr 0.001070 wd 0.0500 time 0.3783 (0.3813) data time 0.0013 (0.0023) model time 0.3770 (0.3775) loss 2.4788 (3.1902) grad_norm 1.8656 (1.5681) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][360/625] eta 0:01:41 lr 0.001070 wd 0.0500 time 0.3760 (0.3812) data time 0.0014 (0.0023) model time 0.3746 (0.3774) loss 2.8030 (3.1873) grad_norm 2.7850 (1.6248) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][370/625] eta 0:01:37 lr 0.001070 wd 0.0500 time 0.3799 (0.3811) data time 0.0010 (0.0023) model time 0.3789 (0.3774) loss 2.1224 (3.1929) grad_norm 1.2460 (1.6290) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][380/625] eta 0:01:33 lr 0.001070 wd 0.0500 time 0.3771 (0.3810) data time 0.0008 (0.0022) model time 0.3763 (0.3773) loss 3.7744 (3.1899) grad_norm 1.3067 (1.6209) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:02:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][390/625] eta 0:01:29 lr 0.001070 wd 0.0500 time 0.3823 (0.3809) data time 0.0011 (0.0022) model time 0.3813 (0.3773) loss 2.6417 (3.1848) grad_norm 1.5204 (1.6157) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][400/625] eta 0:01:25 lr 0.001070 wd 0.0500 time 0.3711 (0.3808) data time 0.0009 (0.0022) model time 0.3702 (0.3772) loss 2.6985 (3.1802) grad_norm 1.4086 (1.6101) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][410/625] eta 0:01:21 lr 0.001069 wd 0.0500 time 0.3765 (0.3806) data time 0.0012 (0.0021) model time 0.3752 (0.3771) loss 3.1180 (3.1872) grad_norm 1.6294 (1.6073) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][420/625] eta 0:01:18 lr 0.001069 wd 0.0500 time 0.3759 (0.3821) data time 0.0008 (0.0021) model time 0.3751 (0.3789) loss 4.1429 (3.1937) grad_norm 1.4919 (1.6075) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][430/625] eta 0:01:14 lr 0.001069 wd 0.0500 time 0.3684 (0.3819) data time 0.0008 (0.0021) model time 0.3676 (0.3786) loss 3.8681 (3.1973) grad_norm 1.0618 (1.6048) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][440/625] eta 0:01:10 lr 0.001069 wd 0.0500 time 0.3692 (0.3817) data time 0.0011 (0.0021) model time 0.3681 (0.3785) loss 3.1188 (3.1968) grad_norm 1.7694 (1.6021) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][450/625] eta 0:01:06 lr 0.001069 wd 0.0500 time 0.3822 (0.3816) data time 0.0012 (0.0021) model time 0.3810 (0.3785) loss 3.5888 (3.1966) grad_norm 1.3796 (1.5990) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][460/625] eta 0:01:02 lr 0.001068 wd 0.0500 time 0.3679 (0.3815) data time 0.0012 (0.0020) model time 0.3667 (0.3784) loss 2.8070 (3.1924) grad_norm 1.0926 (1.5951) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][470/625] eta 0:00:59 lr 0.001068 wd 0.0500 time 0.3840 (0.3814) data time 0.0011 (0.0020) model time 0.3829 (0.3783) loss 3.0505 (3.1959) grad_norm 1.1931 (1.5894) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][480/625] eta 0:00:55 lr 0.001068 wd 0.0500 time 0.3653 (0.3813) data time 0.0008 (0.0020) model time 0.3644 (0.3783) loss 2.9631 (3.1961) grad_norm 1.1718 (1.5847) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][490/625] eta 0:00:51 lr 0.001068 wd 0.0500 time 0.3762 (0.3812) data time 0.0008 (0.0020) model time 0.3754 (0.3782) loss 2.9969 (3.1981) grad_norm 1.5143 (1.5821) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][500/625] eta 0:00:47 lr 0.001068 wd 0.0500 time 0.3697 (0.3811) data time 0.0009 (0.0020) model time 0.3688 (0.3782) loss 3.4412 (3.2035) grad_norm 1.1679 (1.5831) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][510/625] eta 0:00:43 lr 0.001068 wd 0.0500 time 0.3805 (0.3810) data time 0.0011 (0.0019) model time 0.3794 (0.3781) loss 2.9581 (3.2054) grad_norm 1.5546 (1.5829) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][520/625] eta 0:00:39 lr 0.001067 wd 0.0500 time 0.3698 (0.3809) data time 0.0008 (0.0019) model time 0.3690 (0.3780) loss 3.2031 (3.2019) grad_norm 1.1710 (1.5826) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][530/625] eta 0:00:36 lr 0.001067 wd 0.0500 time 0.3773 (0.3811) data time 0.0011 (0.0019) model time 0.3763 (0.3783) loss 2.5115 (3.1966) grad_norm 1.4123 (1.5846) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][540/625] eta 0:00:32 lr 0.001067 wd 0.0500 time 0.3762 (0.3822) data time 0.0009 (0.0019) model time 0.3754 (0.3795) loss 3.1699 (3.1934) grad_norm 1.3005 (1.5784) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:03:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][550/625] eta 0:00:28 lr 0.001067 wd 0.0500 time 0.3795 (0.3821) data time 0.0009 (0.0019) model time 0.3786 (0.3794) loss 2.8023 (3.1913) grad_norm 2.2142 (1.5779) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][560/625] eta 0:00:24 lr 0.001067 wd 0.0500 time 0.3768 (0.3821) data time 0.0010 (0.0019) model time 0.3758 (0.3794) loss 3.0426 (3.1875) grad_norm 1.8112 (1.5785) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][570/625] eta 0:00:21 lr 0.001066 wd 0.0500 time 0.3760 (0.3820) data time 0.0008 (0.0019) model time 0.3752 (0.3793) loss 3.3603 (3.1889) grad_norm 1.4236 (1.5854) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][580/625] eta 0:00:17 lr 0.001066 wd 0.0500 time 0.3847 (0.3819) data time 0.0008 (0.0019) model time 0.3839 (0.3793) loss 3.3569 (3.1893) grad_norm 1.5206 (1.5849) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][590/625] eta 0:00:13 lr 0.001066 wd 0.0500 time 0.3740 (0.3818) data time 0.0008 (0.0018) model time 0.3732 (0.3792) loss 3.9979 (3.1900) grad_norm 2.1015 (1.5833) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][600/625] eta 0:00:09 lr 0.001066 wd 0.0500 time 0.3805 (0.3817) data time 0.0007 (0.0018) model time 0.3798 (0.3792) loss 3.0621 (3.1883) grad_norm 1.5702 (1.5820) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][610/625] eta 0:00:05 lr 0.001066 wd 0.0500 time 0.3742 (0.3817) data time 0.0008 (0.0018) model time 0.3734 (0.3792) loss 3.2620 (3.1911) grad_norm 2.3949 (1.5864) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [154/300][620/625] eta 0:00:01 lr 0.001066 wd 0.0500 time 0.3652 (0.3815) data time 0.0006 (0.0018) model time 0.3646 (0.3789) loss 3.4705 (3.1919) grad_norm 1.5454 (1.5880) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 154 training takes 0:03:58 [2024-08-27 10:04:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 10:04:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 10:04:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.500 (0.500) Loss 0.6016 (0.6016) Acc@1 87.061 (87.061) Acc@5 98.096 (98.096) Mem 11224MB [2024-08-27 10:04:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.9707 (0.7562) Acc@1 77.246 (83.283) Acc@5 94.580 (96.871) Mem 11224MB [2024-08-27 10:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 1.1250 (0.9003) Acc@1 73.145 (79.925) Acc@5 93.311 (95.199) Mem 11224MB [2024-08-27 10:04:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.659 Acc@5 95.172 [2024-08-27 10:04:31 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 79.7% [2024-08-27 10:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.892 (0.892) Loss 0.5127 (0.5127) Acc@1 88.965 (88.965) Acc@5 98.633 (98.633) Mem 11224MB [2024-08-27 10:04:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.181) Loss 0.8433 (0.6383) Acc@1 79.883 (85.560) Acc@5 95.850 (97.532) Mem 11224MB [2024-08-27 10:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.139) Loss 0.9707 (0.7640) Acc@1 75.684 (82.387) Acc@5 94.531 (96.259) Mem 11224MB [2024-08-27 10:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.136 Acc@5 96.255 [2024-08-27 10:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-08-27 10:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.14% [2024-08-27 10:04:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 10:04:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 10:04:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][0/625] eta 0:08:24 lr 0.001066 wd 0.0500 time 0.8066 (0.8066) data time 0.4661 (0.4661) model time 0.0000 (0.0000) loss 3.9434 (3.9434) grad_norm 1.9407 (1.9407) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][10/625] eta 0:04:15 lr 0.001065 wd 0.0500 time 0.3814 (0.4154) data time 0.0010 (0.0435) model time 0.0000 (0.0000) loss 3.3656 (3.2770) grad_norm 1.7448 (1.5494) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][20/625] eta 0:03:59 lr 0.001065 wd 0.0500 time 0.3780 (0.3955) data time 0.0008 (0.0236) model time 0.0000 (0.0000) loss 2.9042 (3.0874) grad_norm 1.5619 (1.5160) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][30/625] eta 0:03:51 lr 0.001065 wd 0.0500 time 0.3831 (0.3893) data time 0.0011 (0.0163) model time 0.0000 (0.0000) loss 2.9778 (3.1374) grad_norm 2.0568 (1.5898) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][40/625] eta 0:03:48 lr 0.001065 wd 0.0500 time 0.3711 (0.3902) data time 0.0011 (0.0126) model time 0.0000 (0.0000) loss 3.1022 (3.1605) grad_norm 1.5844 (1.5337) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][50/625] eta 0:03:43 lr 0.001065 wd 0.0500 time 0.3765 (0.3880) data time 0.0010 (0.0104) model time 0.0000 (0.0000) loss 2.8048 (3.1847) grad_norm 1.2220 (1.5096) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][60/625] eta 0:03:38 lr 0.001064 wd 0.0500 time 0.3798 (0.3862) data time 0.0012 (0.0089) model time 0.3787 (0.3762) loss 3.0864 (3.1665) grad_norm 1.3984 (1.5793) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][70/625] eta 0:03:36 lr 0.001064 wd 0.0500 time 0.3686 (0.3900) data time 0.0010 (0.0078) model time 0.3675 (0.3940) loss 3.2759 (3.1738) grad_norm 1.4831 (1.5714) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][80/625] eta 0:03:32 lr 0.001064 wd 0.0500 time 0.3705 (0.3908) data time 0.0010 (0.0070) model time 0.3694 (0.3945) loss 3.4063 (3.1859) grad_norm 1.2722 (1.5935) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][90/625] eta 0:03:28 lr 0.001064 wd 0.0500 time 0.3738 (0.3892) data time 0.0009 (0.0063) model time 0.3729 (0.3897) loss 2.1251 (3.1727) grad_norm 1.6928 (1.5849) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][100/625] eta 0:03:23 lr 0.001064 wd 0.0500 time 0.3803 (0.3880) data time 0.0011 (0.0058) model time 0.3793 (0.3868) loss 3.6134 (3.1794) grad_norm 1.4050 (1.6054) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][110/625] eta 0:03:19 lr 0.001064 wd 0.0500 time 0.3765 (0.3872) data time 0.0009 (0.0054) model time 0.3756 (0.3854) loss 2.4865 (3.1635) grad_norm 1.7279 (1.6275) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][120/625] eta 0:03:15 lr 0.001063 wd 0.0500 time 0.3689 (0.3863) data time 0.0011 (0.0050) model time 0.3678 (0.3839) loss 3.6362 (3.1777) grad_norm 1.7333 (1.6334) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][130/625] eta 0:03:10 lr 0.001063 wd 0.0500 time 0.3769 (0.3856) data time 0.0011 (0.0047) model time 0.3759 (0.3830) loss 3.3384 (3.1643) grad_norm 1.6046 (1.6492) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][140/625] eta 0:03:06 lr 0.001063 wd 0.0500 time 0.3731 (0.3849) data time 0.0009 (0.0045) model time 0.3722 (0.3820) loss 2.2500 (3.1728) grad_norm 1.2783 (1.6391) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][150/625] eta 0:03:02 lr 0.001063 wd 0.0500 time 0.3798 (0.3843) data time 0.0011 (0.0043) model time 0.3786 (0.3812) loss 3.0706 (3.1852) grad_norm 2.3322 (1.6398) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][160/625] eta 0:02:58 lr 0.001063 wd 0.0500 time 0.3863 (0.3849) data time 0.0009 (0.0041) model time 0.3854 (0.3823) loss 2.0403 (3.1715) grad_norm 1.9386 (1.6391) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][170/625] eta 0:02:54 lr 0.001062 wd 0.0500 time 0.3795 (0.3844) data time 0.0008 (0.0039) model time 0.3787 (0.3817) loss 2.6215 (3.1592) grad_norm 1.6460 (1.6270) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][180/625] eta 0:02:50 lr 0.001062 wd 0.0500 time 0.3736 (0.3839) data time 0.0011 (0.0037) model time 0.3725 (0.3811) loss 3.3751 (3.1635) grad_norm 1.3767 (1.6145) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][190/625] eta 0:02:46 lr 0.001062 wd 0.0500 time 0.3811 (0.3835) data time 0.0010 (0.0036) model time 0.3801 (0.3807) loss 1.7786 (3.1579) grad_norm 1.4545 (1.6052) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][200/625] eta 0:02:42 lr 0.001062 wd 0.0500 time 0.3824 (0.3832) data time 0.0010 (0.0035) model time 0.3814 (0.3804) loss 3.2356 (3.1734) grad_norm 1.1740 (1.5944) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][210/625] eta 0:02:38 lr 0.001062 wd 0.0500 time 0.3756 (0.3829) data time 0.0008 (0.0034) model time 0.3748 (0.3801) loss 3.0076 (3.1645) grad_norm 1.3118 (1.5855) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][220/625] eta 0:02:34 lr 0.001062 wd 0.0500 time 0.3746 (0.3826) data time 0.0010 (0.0033) model time 0.3735 (0.3799) loss 2.7368 (3.1678) grad_norm 1.3692 (1.5864) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][230/625] eta 0:02:31 lr 0.001061 wd 0.0500 time 0.3752 (0.3823) data time 0.0010 (0.0032) model time 0.3742 (0.3796) loss 3.3263 (3.1758) grad_norm 1.8767 (1.5984) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][240/625] eta 0:02:27 lr 0.001061 wd 0.0500 time 0.3813 (0.3822) data time 0.0010 (0.0031) model time 0.3803 (0.3795) loss 2.9750 (3.1819) grad_norm 1.4083 (1.5949) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][250/625] eta 0:02:23 lr 0.001061 wd 0.0500 time 0.3826 (0.3819) data time 0.0008 (0.0030) model time 0.3817 (0.3793) loss 2.8709 (3.1789) grad_norm 1.2688 (1.5920) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][260/625] eta 0:02:19 lr 0.001061 wd 0.0500 time 0.3732 (0.3819) data time 0.0008 (0.0029) model time 0.3724 (0.3792) loss 3.5931 (3.1772) grad_norm 1.5232 (1.5917) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][270/625] eta 0:02:15 lr 0.001061 wd 0.0500 time 0.3680 (0.3816) data time 0.0011 (0.0029) model time 0.3669 (0.3790) loss 3.6168 (3.1773) grad_norm 1.7177 (1.5958) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][280/625] eta 0:02:12 lr 0.001061 wd 0.0500 time 0.5960 (0.3830) data time 0.0010 (0.0028) model time 0.5950 (0.3808) loss 2.8124 (3.1761) grad_norm 1.3235 (1.6058) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][290/625] eta 0:02:08 lr 0.001060 wd 0.0500 time 0.3694 (0.3835) data time 0.0009 (0.0028) model time 0.3685 (0.3814) loss 3.8858 (3.1768) grad_norm 1.1974 (1.5966) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][300/625] eta 0:02:04 lr 0.001060 wd 0.0500 time 0.3757 (0.3833) data time 0.0008 (0.0027) model time 0.3749 (0.3812) loss 3.9968 (3.1914) grad_norm 1.3233 (1.5863) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][310/625] eta 0:02:00 lr 0.001060 wd 0.0500 time 0.3711 (0.3830) data time 0.0008 (0.0027) model time 0.3702 (0.3809) loss 3.5795 (3.1941) grad_norm 1.2274 (1.5868) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][320/625] eta 0:01:56 lr 0.001060 wd 0.0500 time 0.3762 (0.3828) data time 0.0008 (0.0026) model time 0.3754 (0.3807) loss 4.1945 (3.1859) grad_norm 1.7488 (1.5907) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][330/625] eta 0:01:52 lr 0.001060 wd 0.0500 time 0.3761 (0.3827) data time 0.0011 (0.0026) model time 0.3750 (0.3806) loss 2.4400 (3.1848) grad_norm 2.1092 (1.5893) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][340/625] eta 0:01:49 lr 0.001059 wd 0.0500 time 0.3789 (0.3825) data time 0.0008 (0.0026) model time 0.3781 (0.3804) loss 2.8859 (3.1897) grad_norm 2.5584 (1.5872) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][350/625] eta 0:01:45 lr 0.001059 wd 0.0500 time 0.3720 (0.3823) data time 0.0011 (0.0025) model time 0.3709 (0.3802) loss 3.1675 (3.1905) grad_norm 1.2512 (1.5865) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][360/625] eta 0:01:41 lr 0.001059 wd 0.0500 time 0.3731 (0.3822) data time 0.0009 (0.0025) model time 0.3722 (0.3801) loss 3.7254 (3.1933) grad_norm 1.6973 (1.5793) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][370/625] eta 0:01:37 lr 0.001059 wd 0.0500 time 0.3877 (0.3826) data time 0.0010 (0.0024) model time 0.3867 (0.3805) loss 2.8789 (3.1933) grad_norm 1.0021 (1.5766) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][380/625] eta 0:01:33 lr 0.001059 wd 0.0500 time 0.3781 (0.3825) data time 0.0013 (0.0024) model time 0.3768 (0.3804) loss 2.9133 (3.1929) grad_norm 1.3439 (1.5809) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][390/625] eta 0:01:29 lr 0.001059 wd 0.0500 time 0.3747 (0.3824) data time 0.0008 (0.0024) model time 0.3739 (0.3803) loss 3.8728 (3.1981) grad_norm 1.6683 (1.5853) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][400/625] eta 0:01:25 lr 0.001058 wd 0.0500 time 0.3650 (0.3822) data time 0.0011 (0.0024) model time 0.3640 (0.3802) loss 3.3256 (3.1984) grad_norm 1.4876 (1.5823) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][410/625] eta 0:01:22 lr 0.001058 wd 0.0500 time 0.3843 (0.3821) data time 0.0011 (0.0023) model time 0.3832 (0.3801) loss 2.6689 (3.2028) grad_norm 1.8126 (1.5815) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][420/625] eta 0:01:18 lr 0.001058 wd 0.0500 time 0.3673 (0.3820) data time 0.0013 (0.0023) model time 0.3660 (0.3800) loss 3.2492 (3.2068) grad_norm 1.6075 (1.5860) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][430/625] eta 0:01:14 lr 0.001058 wd 0.0500 time 0.3824 (0.3819) data time 0.0010 (0.0023) model time 0.3814 (0.3799) loss 3.1830 (3.2087) grad_norm 1.3435 (1.5854) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][440/625] eta 0:01:10 lr 0.001058 wd 0.0500 time 0.3770 (0.3817) data time 0.0008 (0.0022) model time 0.3762 (0.3798) loss 3.8984 (3.2053) grad_norm 2.2761 (1.5859) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][450/625] eta 0:01:06 lr 0.001058 wd 0.0500 time 0.3731 (0.3816) data time 0.0013 (0.0022) model time 0.3718 (0.3796) loss 3.2124 (3.2077) grad_norm 1.3546 (1.5924) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][460/625] eta 0:01:03 lr 0.001057 wd 0.0500 time 0.5671 (0.3819) data time 0.0010 (0.0022) model time 0.5661 (0.3800) loss 3.2350 (3.2064) grad_norm 1.5578 (1.5872) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][470/625] eta 0:00:59 lr 0.001057 wd 0.0500 time 0.3787 (0.3830) data time 0.0008 (0.0022) model time 0.3778 (0.3812) loss 3.0657 (3.2084) grad_norm 1.2100 (1.5860) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][480/625] eta 0:00:55 lr 0.001057 wd 0.0500 time 0.3800 (0.3829) data time 0.0008 (0.0021) model time 0.3791 (0.3811) loss 3.6402 (3.2097) grad_norm 1.6981 (1.5845) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][490/625] eta 0:00:51 lr 0.001057 wd 0.0500 time 0.3782 (0.3828) data time 0.0008 (0.0021) model time 0.3773 (0.3810) loss 2.2439 (3.2136) grad_norm 1.6690 (1.5878) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][500/625] eta 0:00:47 lr 0.001057 wd 0.0500 time 0.3666 (0.3827) data time 0.0013 (0.0021) model time 0.3653 (0.3809) loss 3.8266 (3.2184) grad_norm 1.5689 (1.5872) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][510/625] eta 0:00:43 lr 0.001056 wd 0.0500 time 0.3757 (0.3826) data time 0.0008 (0.0021) model time 0.3748 (0.3808) loss 3.8954 (3.2182) grad_norm 1.7082 (1.5883) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][520/625] eta 0:00:40 lr 0.001056 wd 0.0500 time 0.3650 (0.3825) data time 0.0011 (0.0021) model time 0.3640 (0.3808) loss 3.4530 (3.2215) grad_norm 1.3061 (1.5837) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:07:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][530/625] eta 0:00:36 lr 0.001056 wd 0.0500 time 0.3727 (0.3824) data time 0.0010 (0.0021) model time 0.3716 (0.3807) loss 3.3383 (3.2267) grad_norm 1.4031 (1.5796) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][540/625] eta 0:00:32 lr 0.001056 wd 0.0500 time 0.3728 (0.3823) data time 0.0011 (0.0020) model time 0.3716 (0.3806) loss 3.3342 (3.2245) grad_norm 1.8669 (1.5793) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][550/625] eta 0:00:28 lr 0.001056 wd 0.0500 time 0.3358 (0.3825) data time 0.0009 (0.0020) model time 0.3349 (0.3807) loss 2.8638 (3.2251) grad_norm 1.9613 (1.5765) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][560/625] eta 0:00:24 lr 0.001056 wd 0.0500 time 0.3835 (0.3823) data time 0.0011 (0.0020) model time 0.3825 (0.3806) loss 2.8761 (3.2232) grad_norm 1.2285 (1.5742) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][570/625] eta 0:00:21 lr 0.001055 wd 0.0500 time 0.3697 (0.3822) data time 0.0009 (0.0020) model time 0.3689 (0.3805) loss 2.2710 (3.2237) grad_norm 1.7327 (1.5724) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][580/625] eta 0:00:17 lr 0.001055 wd 0.0500 time 0.5720 (0.3828) data time 0.0011 (0.0020) model time 0.5710 (0.3811) loss 3.3409 (3.2227) grad_norm 2.1341 (1.5714) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][590/625] eta 0:00:13 lr 0.001055 wd 0.0500 time 0.3737 (0.3830) data time 0.0009 (0.0020) model time 0.3727 (0.3814) loss 3.7528 (3.2233) grad_norm 1.2170 (1.5708) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][600/625] eta 0:00:09 lr 0.001055 wd 0.0500 time 0.3797 (0.3829) data time 0.0012 (0.0020) model time 0.3784 (0.3813) loss 3.1020 (3.2184) grad_norm 1.0923 (1.5677) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][610/625] eta 0:00:05 lr 0.001055 wd 0.0500 time 0.3678 (0.3827) data time 0.0008 (0.0019) model time 0.3670 (0.3811) loss 3.0570 (3.2165) grad_norm 1.1422 (1.5649) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [155/300][620/625] eta 0:00:01 lr 0.001055 wd 0.0500 time 0.3664 (0.3825) data time 0.0006 (0.0019) model time 0.3659 (0.3808) loss 3.5532 (3.2161) grad_norm 1.6525 (1.5644) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:34 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 155 training takes 0:03:58 [2024-08-27 10:08:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 10:08:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 10:08:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 0.5850 (0.5850) Acc@1 87.012 (87.012) Acc@5 98.340 (98.340) Mem 11224MB [2024-08-27 10:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.146) Loss 0.9946 (0.7321) Acc@1 76.562 (83.669) Acc@5 94.141 (96.942) Mem 11224MB [2024-08-27 10:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 1.0732 (0.8710) Acc@1 74.854 (80.301) Acc@5 93.555 (95.317) Mem 11224MB [2024-08-27 10:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.076 Acc@5 95.333 [2024-08-27 10:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.1% [2024-08-27 10:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 80.08% [2024-08-27 10:08:38 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 10:08:39 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 10:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.513 (0.513) Loss 0.5117 (0.5117) Acc@1 88.965 (88.965) Acc@5 98.633 (98.633) Mem 11224MB [2024-08-27 10:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.8433 (0.6383) Acc@1 79.736 (85.565) Acc@5 95.898 (97.550) Mem 11224MB [2024-08-27 10:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.9673 (0.7632) Acc@1 75.830 (82.424) Acc@5 94.629 (96.282) Mem 11224MB [2024-08-27 10:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.170 Acc@5 96.275 [2024-08-27 10:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.2% [2024-08-27 10:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.17% [2024-08-27 10:08:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 10:08:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 10:08:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][0/625] eta 0:07:32 lr 0.001054 wd 0.0500 time 0.7236 (0.7236) data time 0.3795 (0.3795) model time 0.0000 (0.0000) loss 3.5086 (3.5086) grad_norm 1.6081 (1.6081) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][10/625] eta 0:04:10 lr 0.001054 wd 0.0500 time 0.3827 (0.4065) data time 0.0008 (0.0355) model time 0.0000 (0.0000) loss 3.2268 (3.2403) grad_norm 1.4944 (1.5705) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][20/625] eta 0:03:56 lr 0.001054 wd 0.0500 time 0.3776 (0.3913) data time 0.0008 (0.0191) model time 0.0000 (0.0000) loss 2.5791 (3.2250) grad_norm 1.5470 (1.5662) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:08:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][30/625] eta 0:03:49 lr 0.001054 wd 0.0500 time 0.3773 (0.3862) data time 0.0008 (0.0133) model time 0.0000 (0.0000) loss 3.2448 (3.1141) grad_norm 1.4906 (1.5810) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][40/625] eta 0:03:44 lr 0.001054 wd 0.0500 time 0.3786 (0.3834) data time 0.0008 (0.0104) model time 0.0000 (0.0000) loss 2.6026 (3.1618) grad_norm 1.3579 (1.5572) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][50/625] eta 0:03:39 lr 0.001054 wd 0.0500 time 0.3836 (0.3822) data time 0.0008 (0.0086) model time 0.0000 (0.0000) loss 2.1250 (3.1576) grad_norm 1.5997 (1.5738) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][60/625] eta 0:03:35 lr 0.001053 wd 0.0500 time 0.3743 (0.3808) data time 0.0010 (0.0073) model time 0.3732 (0.3724) loss 3.0073 (3.1741) grad_norm 1.5317 (1.5395) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][70/625] eta 0:03:31 lr 0.001053 wd 0.0500 time 0.3789 (0.3803) data time 0.0009 (0.0065) model time 0.3780 (0.3743) loss 3.5905 (3.1343) grad_norm 1.8126 (1.5301) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][80/625] eta 0:03:26 lr 0.001053 wd 0.0500 time 0.3726 (0.3797) data time 0.0012 (0.0058) model time 0.3713 (0.3742) loss 3.5922 (3.1204) grad_norm 1.1400 (1.5222) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][90/625] eta 0:03:22 lr 0.001053 wd 0.0500 time 0.3664 (0.3794) data time 0.0011 (0.0053) model time 0.3653 (0.3747) loss 3.4387 (3.1267) grad_norm 1.4662 (1.5064) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][100/625] eta 0:03:21 lr 0.001053 wd 0.0500 time 0.3808 (0.3832) data time 0.0008 (0.0049) model time 0.3799 (0.3829) loss 3.2824 (3.1261) grad_norm 3.6692 (1.6093) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][110/625] eta 0:03:16 lr 0.001052 wd 0.0500 time 0.3672 (0.3820) data time 0.0010 (0.0046) model time 0.3662 (0.3807) loss 2.3778 (3.1377) grad_norm 2.0337 (1.6617) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][120/625] eta 0:03:12 lr 0.001052 wd 0.0500 time 0.3740 (0.3815) data time 0.0010 (0.0043) model time 0.3731 (0.3799) loss 3.2137 (3.1450) grad_norm 1.8813 (1.6500) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][130/625] eta 0:03:08 lr 0.001052 wd 0.0500 time 0.3736 (0.3808) data time 0.0011 (0.0040) model time 0.3725 (0.3788) loss 3.8900 (3.1327) grad_norm 1.5679 (1.6267) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][140/625] eta 0:03:04 lr 0.001052 wd 0.0500 time 0.3754 (0.3803) data time 0.0009 (0.0038) model time 0.3745 (0.3781) loss 3.6753 (3.1665) grad_norm 2.2063 (1.6277) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][150/625] eta 0:03:00 lr 0.001052 wd 0.0500 time 0.3768 (0.3801) data time 0.0011 (0.0036) model time 0.3757 (0.3779) loss 2.2367 (3.1724) grad_norm 1.7368 (1.6449) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][160/625] eta 0:02:56 lr 0.001052 wd 0.0500 time 0.3707 (0.3800) data time 0.0009 (0.0035) model time 0.3698 (0.3778) loss 4.2198 (3.1882) grad_norm 2.1520 (1.6493) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][170/625] eta 0:02:52 lr 0.001051 wd 0.0500 time 0.3772 (0.3797) data time 0.0011 (0.0033) model time 0.3761 (0.3775) loss 3.5550 (3.1684) grad_norm 1.6339 (1.6481) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][180/625] eta 0:02:49 lr 0.001051 wd 0.0500 time 0.3395 (0.3801) data time 0.0010 (0.0032) model time 0.3385 (0.3781) loss 3.4689 (3.1878) grad_norm 1.5057 (1.6437) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][190/625] eta 0:02:45 lr 0.001051 wd 0.0500 time 0.3858 (0.3800) data time 0.0010 (0.0031) model time 0.3847 (0.3780) loss 2.4401 (3.1940) grad_norm 1.8411 (1.6436) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][200/625] eta 0:02:41 lr 0.001051 wd 0.0500 time 0.3841 (0.3798) data time 0.0010 (0.0030) model time 0.3831 (0.3778) loss 3.7039 (3.1953) grad_norm 1.4203 (1.6374) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][210/625] eta 0:02:37 lr 0.001051 wd 0.0500 time 0.3753 (0.3797) data time 0.0011 (0.0029) model time 0.3743 (0.3778) loss 3.5205 (3.1889) grad_norm 1.2619 (1.6314) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][220/625] eta 0:02:33 lr 0.001051 wd 0.0500 time 0.3737 (0.3795) data time 0.0008 (0.0029) model time 0.3728 (0.3776) loss 4.0756 (3.2057) grad_norm 3.2008 (1.6359) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][230/625] eta 0:02:29 lr 0.001050 wd 0.0500 time 0.3723 (0.3793) data time 0.0007 (0.0028) model time 0.3715 (0.3774) loss 2.8465 (3.1882) grad_norm 1.0551 (1.6297) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][240/625] eta 0:02:25 lr 0.001050 wd 0.0500 time 0.3693 (0.3792) data time 0.0008 (0.0027) model time 0.3685 (0.3773) loss 3.8753 (3.1887) grad_norm 1.5533 (1.6223) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][250/625] eta 0:02:22 lr 0.001050 wd 0.0500 time 0.3825 (0.3792) data time 0.0011 (0.0026) model time 0.3814 (0.3773) loss 3.6868 (3.1888) grad_norm 1.3193 (1.6263) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][260/625] eta 0:02:18 lr 0.001050 wd 0.0500 time 0.3875 (0.3791) data time 0.0012 (0.0026) model time 0.3863 (0.3772) loss 3.1661 (3.1892) grad_norm 1.0027 (1.6211) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][270/625] eta 0:02:14 lr 0.001050 wd 0.0500 time 0.3775 (0.3791) data time 0.0010 (0.0025) model time 0.3764 (0.3772) loss 3.5094 (3.1961) grad_norm 1.3931 (1.6073) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][280/625] eta 0:02:10 lr 0.001049 wd 0.0500 time 0.3693 (0.3789) data time 0.0008 (0.0025) model time 0.3685 (0.3770) loss 2.1974 (3.1834) grad_norm 1.3525 (1.6121) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][290/625] eta 0:02:06 lr 0.001049 wd 0.0500 time 0.3854 (0.3788) data time 0.0009 (0.0024) model time 0.3845 (0.3770) loss 3.2632 (3.1845) grad_norm 1.9639 (1.6227) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][300/625] eta 0:02:03 lr 0.001049 wd 0.0500 time 0.3392 (0.3793) data time 0.0011 (0.0024) model time 0.3382 (0.3776) loss 2.1169 (3.1803) grad_norm 1.2900 (1.6181) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][310/625] eta 0:01:59 lr 0.001049 wd 0.0500 time 0.3784 (0.3807) data time 0.0010 (0.0024) model time 0.3774 (0.3792) loss 3.0681 (3.1843) grad_norm 1.3719 (1.6171) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][320/625] eta 0:01:56 lr 0.001049 wd 0.0500 time 0.3779 (0.3806) data time 0.0010 (0.0023) model time 0.3769 (0.3791) loss 2.8658 (3.1775) grad_norm 1.1925 (1.6231) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][330/625] eta 0:01:52 lr 0.001049 wd 0.0500 time 0.3724 (0.3804) data time 0.0010 (0.0023) model time 0.3713 (0.3789) loss 3.7005 (3.1812) grad_norm 1.9658 (1.6480) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][340/625] eta 0:01:48 lr 0.001048 wd 0.0500 time 0.3772 (0.3803) data time 0.0008 (0.0023) model time 0.3764 (0.3789) loss 3.4195 (3.1835) grad_norm 1.4962 (1.6439) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:10:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][350/625] eta 0:01:44 lr 0.001048 wd 0.0500 time 0.3757 (0.3802) data time 0.0012 (0.0022) model time 0.3745 (0.3787) loss 3.4656 (3.1839) grad_norm 1.0253 (1.6380) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][360/625] eta 0:01:40 lr 0.001048 wd 0.0500 time 0.3665 (0.3800) data time 0.0008 (0.0022) model time 0.3657 (0.3785) loss 2.9273 (3.1901) grad_norm 1.5519 (1.6348) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][370/625] eta 0:01:36 lr 0.001048 wd 0.0500 time 0.3685 (0.3799) data time 0.0013 (0.0022) model time 0.3672 (0.3784) loss 2.9978 (3.1940) grad_norm 1.5139 (1.6355) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][380/625] eta 0:01:33 lr 0.001048 wd 0.0500 time 0.3686 (0.3798) data time 0.0010 (0.0022) model time 0.3676 (0.3782) loss 3.3690 (3.1945) grad_norm 2.0292 (1.6327) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][390/625] eta 0:01:29 lr 0.001047 wd 0.0500 time 0.3805 (0.3796) data time 0.0012 (0.0021) model time 0.3792 (0.3781) loss 3.4099 (3.1957) grad_norm 1.8648 (1.6279) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][400/625] eta 0:01:25 lr 0.001047 wd 0.0500 time 0.3757 (0.3795) data time 0.0009 (0.0021) model time 0.3748 (0.3779) loss 2.4976 (3.1988) grad_norm 1.2436 (1.6206) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][410/625] eta 0:01:21 lr 0.001047 wd 0.0500 time 0.3731 (0.3794) data time 0.0012 (0.0021) model time 0.3719 (0.3778) loss 3.3563 (3.2042) grad_norm 1.3046 (1.6165) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][420/625] eta 0:01:17 lr 0.001047 wd 0.0500 time 0.3748 (0.3792) data time 0.0008 (0.0021) model time 0.3740 (0.3777) loss 3.5524 (3.2076) grad_norm 1.7675 (1.6159) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][430/625] eta 0:01:13 lr 0.001047 wd 0.0500 time 0.3834 (0.3791) data time 0.0011 (0.0020) model time 0.3824 (0.3776) loss 2.5470 (3.2031) grad_norm 2.3778 (1.6187) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][440/625] eta 0:01:10 lr 0.001047 wd 0.0500 time 0.3857 (0.3791) data time 0.0008 (0.0020) model time 0.3849 (0.3775) loss 3.5571 (3.2058) grad_norm 1.2224 (1.6168) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][450/625] eta 0:01:06 lr 0.001046 wd 0.0500 time 0.3786 (0.3791) data time 0.0010 (0.0020) model time 0.3776 (0.3775) loss 2.2964 (3.2023) grad_norm 2.8623 (1.6393) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][460/625] eta 0:01:02 lr 0.001046 wd 0.0500 time 0.3755 (0.3790) data time 0.0010 (0.0020) model time 0.3745 (0.3774) loss 3.0654 (3.2071) grad_norm 1.3195 (1.6426) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][470/625] eta 0:00:58 lr 0.001046 wd 0.0500 time 0.3809 (0.3790) data time 0.0009 (0.0020) model time 0.3801 (0.3774) loss 3.2567 (3.2097) grad_norm 1.3764 (1.6416) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][480/625] eta 0:00:54 lr 0.001046 wd 0.0500 time 0.3794 (0.3790) data time 0.0011 (0.0019) model time 0.3783 (0.3774) loss 3.4635 (3.2117) grad_norm 0.9257 (1.6387) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][490/625] eta 0:00:51 lr 0.001046 wd 0.0500 time 0.3845 (0.3801) data time 0.0008 (0.0019) model time 0.3838 (0.3787) loss 2.5783 (3.2141) grad_norm 1.4017 (1.6325) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][500/625] eta 0:00:47 lr 0.001046 wd 0.0500 time 0.3699 (0.3800) data time 0.0011 (0.0019) model time 0.3687 (0.3786) loss 3.2271 (3.2169) grad_norm 1.3269 (1.6304) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:11:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][510/625] eta 0:00:43 lr 0.001045 wd 0.0500 time 0.3826 (0.3803) data time 0.0011 (0.0019) model time 0.3815 (0.3789) loss 3.5180 (3.2178) grad_norm 1.9662 (1.6240) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][520/625] eta 0:00:39 lr 0.001045 wd 0.0500 time 0.3628 (0.3802) data time 0.0010 (0.0019) model time 0.3618 (0.3788) loss 3.3994 (3.2144) grad_norm 1.6657 (1.6299) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][530/625] eta 0:00:36 lr 0.001045 wd 0.0500 time 0.3766 (0.3801) data time 0.0008 (0.0019) model time 0.3757 (0.3788) loss 2.2998 (3.2107) grad_norm 1.2689 (1.6290) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][540/625] eta 0:00:32 lr 0.001045 wd 0.0500 time 0.3704 (0.3801) data time 0.0009 (0.0019) model time 0.3694 (0.3787) loss 3.4647 (3.2106) grad_norm 1.3168 (1.6271) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][550/625] eta 0:00:28 lr 0.001045 wd 0.0500 time 0.3846 (0.3801) data time 0.0009 (0.0018) model time 0.3837 (0.3787) loss 3.6233 (3.2142) grad_norm 1.4792 (1.6266) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][560/625] eta 0:00:24 lr 0.001044 wd 0.0500 time 0.3792 (0.3801) data time 0.0007 (0.0018) model time 0.3785 (0.3787) loss 3.2006 (3.2093) grad_norm 2.7472 (1.6335) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][570/625] eta 0:00:20 lr 0.001044 wd 0.0500 time 0.3859 (0.3800) data time 0.0008 (0.0018) model time 0.3850 (0.3786) loss 3.3204 (3.2073) grad_norm 2.3651 (1.6433) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][580/625] eta 0:00:17 lr 0.001044 wd 0.0500 time 0.3720 (0.3800) data time 0.0015 (0.0018) model time 0.3705 (0.3786) loss 3.7640 (3.2104) grad_norm 1.1472 (1.6427) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][590/625] eta 0:00:13 lr 0.001044 wd 0.0500 time 0.3757 (0.3799) data time 0.0008 (0.0018) model time 0.3749 (0.3785) loss 3.8128 (3.2070) grad_norm 3.0547 (1.6388) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][600/625] eta 0:00:09 lr 0.001044 wd 0.0500 time 0.3747 (0.3798) data time 0.0011 (0.0018) model time 0.3736 (0.3784) loss 3.3989 (3.2088) grad_norm 2.1927 (1.6403) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][610/625] eta 0:00:05 lr 0.001044 wd 0.0500 time 0.3650 (0.3804) data time 0.0008 (0.0018) model time 0.3641 (0.3791) loss 2.8107 (3.2073) grad_norm 2.1858 (1.6415) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [156/300][620/625] eta 0:00:01 lr 0.001043 wd 0.0500 time 0.3668 (0.3802) data time 0.0009 (0.0018) model time 0.3660 (0.3788) loss 3.3617 (3.2090) grad_norm 1.7466 (1.6388) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 156 training takes 0:03:57 [2024-08-27 10:12:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 10:12:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 10:12:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.485 (0.485) Loss 0.5942 (0.5942) Acc@1 87.354 (87.354) Acc@5 98.096 (98.096) Mem 11224MB [2024-08-27 10:12:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.142) Loss 0.9678 (0.7282) Acc@1 77.734 (83.900) Acc@5 94.189 (96.906) Mem 11224MB [2024-08-27 10:12:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 1.0605 (0.8602) Acc@1 74.414 (80.618) Acc@5 93.896 (95.433) Mem 11224MB [2024-08-27 10:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.334 Acc@5 95.379 [2024-08-27 10:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.3% [2024-08-27 10:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 80.33% [2024-08-27 10:12:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-08-27 10:12:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-08-27 10:12:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.503 (0.503) Loss 0.5088 (0.5088) Acc@1 88.916 (88.916) Acc@5 98.584 (98.584) Mem 11224MB [2024-08-27 10:12:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 0.8413 (0.6372) Acc@1 79.785 (85.591) Acc@5 95.850 (97.519) Mem 11224MB [2024-08-27 10:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 0.9653 (0.7620) Acc@1 75.830 (82.452) Acc@5 94.775 (96.275) Mem 11224MB [2024-08-27 10:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.192 Acc@5 96.269 [2024-08-27 10:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.2% [2024-08-27 10:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.19% [2024-08-27 10:12:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 10:12:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 10:12:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][0/625] eta 0:07:52 lr 0.001043 wd 0.0500 time 0.7565 (0.7565) data time 0.4088 (0.4088) model time 0.0000 (0.0000) loss 3.8251 (3.8251) grad_norm 1.3993 (1.3993) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][10/625] eta 0:04:17 lr 0.001043 wd 0.0500 time 0.3772 (0.4188) data time 0.0008 (0.0381) model time 0.0000 (0.0000) loss 2.8203 (3.2017) grad_norm 2.0658 (2.5709) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:12:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][20/625] eta 0:04:01 lr 0.001043 wd 0.0500 time 0.3823 (0.3992) data time 0.0014 (0.0205) model time 0.0000 (0.0000) loss 2.8162 (3.1742) grad_norm 1.2711 (2.1338) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:13:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][30/625] eta 0:03:53 lr 0.001043 wd 0.0500 time 0.3730 (0.3920) data time 0.0009 (0.0143) model time 0.0000 (0.0000) loss 2.6643 (3.2117) grad_norm 1.2742 (1.8711) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:13:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][40/625] eta 0:03:47 lr 0.001043 wd 0.0500 time 0.3801 (0.3886) data time 0.0009 (0.0111) model time 0.0000 (0.0000) loss 3.8691 (3.2207) grad_norm 1.7631 (1.7934) loss_scale 512.0000 (512.0000) mem 11224MB [2024-08-27 10:13:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][50/625] eta 0:03:41 lr 0.001042 wd 0.0500 time 0.3685 (0.3861) data time 0.0011 (0.0091) model time 0.0000 (0.0000) loss 3.2805 (3.1907) grad_norm 1.4667 (1.7573) loss_scale 1024.0000 (612.3922) mem 11224MB [2024-08-27 10:13:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][60/625] eta 0:03:36 lr 0.001042 wd 0.0500 time 0.3719 (0.3840) data time 0.0011 (0.0079) model time 0.3708 (0.3722) loss 2.3243 (3.1645) grad_norm 1.9229 (1.7024) loss_scale 1024.0000 (679.8689) mem 11224MB [2024-08-27 10:13:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][70/625] eta 0:03:32 lr 0.001042 wd 0.0500 time 0.3805 (0.3830) data time 0.0012 (0.0069) model time 0.3794 (0.3739) loss 3.3053 (3.1786) grad_norm 1.9926 (1.6389) loss_scale 1024.0000 (728.3380) mem 11224MB [2024-08-27 10:13:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][80/625] eta 0:03:28 lr 0.001042 wd 0.0500 time 0.3887 (0.3822) data time 0.0008 (0.0062) model time 0.3879 (0.3744) loss 3.0152 (3.1817) grad_norm 1.0679 (1.6203) loss_scale 1024.0000 (764.8395) mem 11224MB [2024-08-27 10:13:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][90/625] eta 0:03:24 lr 0.001042 wd 0.0500 time 0.3645 (0.3814) data time 0.0008 (0.0057) model time 0.3636 (0.3740) loss 2.2324 (3.1694) grad_norm 1.4852 (1.6123) loss_scale 1024.0000 (793.3187) mem 11224MB [2024-08-27 10:13:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][100/625] eta 0:03:20 lr 0.001042 wd 0.0500 time 0.3803 (0.3810) data time 0.0009 (0.0052) model time 0.3794 (0.3745) loss 3.3070 (3.1864) grad_norm 1.5346 (1.5910) loss_scale 1024.0000 (816.1584) mem 11224MB [2024-08-27 10:13:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][110/625] eta 0:03:15 lr 0.001041 wd 0.0500 time 0.3728 (0.3806) data time 0.0008 (0.0049) model time 0.3720 (0.3746) loss 2.0479 (3.1690) grad_norm 1.6317 (1.5774) loss_scale 1024.0000 (834.8829) mem 11224MB [2024-08-27 10:13:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][120/625] eta 0:03:12 lr 0.001041 wd 0.0500 time 0.3909 (0.3803) data time 0.0011 (0.0046) model time 0.3898 (0.3749) loss 3.2795 (3.1754) grad_norm 1.2228 (1.5585) loss_scale 1024.0000 (850.5124) mem 11224MB [2024-08-27 10:13:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][130/625] eta 0:03:08 lr 0.001041 wd 0.0500 time 0.3767 (0.3813) data time 0.0012 (0.0043) model time 0.3755 (0.3771) loss 2.2078 (3.1563) grad_norm 1.4126 (1.5438) loss_scale 1024.0000 (863.7557) mem 11224MB [2024-08-27 10:13:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][140/625] eta 0:03:04 lr 0.001041 wd 0.0500 time 0.3793 (0.3809) data time 0.0009 (0.0041) model time 0.3784 (0.3767) loss 3.2649 (3.1613) grad_norm 1.1650 (1.5341) loss_scale 1024.0000 (875.1206) mem 11224MB [2024-08-27 10:13:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][150/625] eta 0:03:00 lr 0.001041 wd 0.0500 time 0.3736 (0.3806) data time 0.0007 (0.0039) model time 0.3729 (0.3766) loss 2.9066 (3.1394) grad_norm 1.9731 (1.5414) loss_scale 1024.0000 (884.9801) mem 11224MB [2024-08-27 10:13:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][160/625] eta 0:02:56 lr 0.001040 wd 0.0500 time 0.3770 (0.3803) data time 0.0011 (0.0037) model time 0.3758 (0.3765) loss 3.1986 (3.1347) grad_norm 1.7780 (1.5477) loss_scale 1024.0000 (893.6149) mem 11224MB [2024-08-27 10:13:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][170/625] eta 0:02:54 lr 0.001040 wd 0.0500 time 0.3780 (0.3827) data time 0.0011 (0.0036) model time 0.3769 (0.3801) loss 3.0057 (3.1415) grad_norm 1.1948 (1.5749) loss_scale 1024.0000 (901.2398) mem 11224MB [2024-08-27 10:14:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][180/625] eta 0:02:50 lr 0.001040 wd 0.0500 time 0.3735 (0.3825) data time 0.0013 (0.0034) model time 0.3723 (0.3799) loss 2.0167 (3.1376) grad_norm 1.4109 (1.5656) loss_scale 1024.0000 (908.0221) mem 11224MB [2024-08-27 10:14:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][190/625] eta 0:02:46 lr 0.001040 wd 0.0500 time 0.3767 (0.3822) data time 0.0009 (0.0033) model time 0.3759 (0.3796) loss 2.6475 (3.1386) grad_norm 4.0414 (1.5996) loss_scale 1024.0000 (914.0942) mem 11224MB [2024-08-27 10:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][200/625] eta 0:02:42 lr 0.001040 wd 0.0500 time 0.3762 (0.3819) data time 0.0008 (0.0032) model time 0.3754 (0.3794) loss 2.6327 (3.1413) grad_norm 1.3652 (1.6075) loss_scale 1024.0000 (919.5622) mem 11224MB [2024-08-27 10:14:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][210/625] eta 0:02:38 lr 0.001040 wd 0.0500 time 0.3747 (0.3816) data time 0.0011 (0.0031) model time 0.3736 (0.3791) loss 3.2220 (3.1574) grad_norm 1.6384 (1.6063) loss_scale 1024.0000 (924.5118) mem 11224MB [2024-08-27 10:14:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][220/625] eta 0:02:34 lr 0.001039 wd 0.0500 time 0.3765 (0.3813) data time 0.0012 (0.0030) model time 0.3753 (0.3786) loss 2.4059 (3.1586) grad_norm 1.5962 (1.6029) loss_scale 1024.0000 (929.0136) mem 11224MB [2024-08-27 10:14:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][230/625] eta 0:02:30 lr 0.001039 wd 0.0500 time 0.3651 (0.3809) data time 0.0012 (0.0029) model time 0.3639 (0.3783) loss 3.4978 (3.1499) grad_norm 1.3995 (1.6010) loss_scale 1024.0000 (933.1255) mem 11224MB [2024-08-27 10:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][240/625] eta 0:02:26 lr 0.001039 wd 0.0500 time 0.3789 (0.3807) data time 0.0012 (0.0029) model time 0.3777 (0.3781) loss 2.4449 (3.1519) grad_norm 1.2989 (1.6046) loss_scale 1024.0000 (936.8963) mem 11224MB [2024-08-27 10:14:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][250/625] eta 0:02:22 lr 0.001039 wd 0.0500 time 0.3787 (0.3804) data time 0.0009 (0.0028) model time 0.3778 (0.3778) loss 2.8089 (3.1502) grad_norm 1.5878 (1.6049) loss_scale 1024.0000 (940.3665) mem 11224MB [2024-08-27 10:14:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][260/625] eta 0:02:18 lr 0.001039 wd 0.0500 time 0.3711 (0.3802) data time 0.0010 (0.0027) model time 0.3701 (0.3776) loss 3.4583 (3.1609) grad_norm 1.3802 (1.6099) loss_scale 1024.0000 (943.5709) mem 11224MB [2024-08-27 10:14:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][270/625] eta 0:02:14 lr 0.001039 wd 0.0500 time 0.3781 (0.3801) data time 0.0010 (0.0027) model time 0.3771 (0.3775) loss 3.3780 (3.1666) grad_norm 1.3252 (1.6103) loss_scale 1024.0000 (946.5387) mem 11224MB [2024-08-27 10:14:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][280/625] eta 0:02:11 lr 0.001038 wd 0.0500 time 0.3825 (0.3799) data time 0.0011 (0.0026) model time 0.3814 (0.3774) loss 3.3231 (3.1685) grad_norm 2.9110 (1.6241) loss_scale 1024.0000 (949.2954) mem 11224MB [2024-08-27 10:14:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][290/625] eta 0:02:07 lr 0.001038 wd 0.0500 time 0.3722 (0.3812) data time 0.0008 (0.0026) model time 0.3714 (0.3790) loss 2.9767 (3.1647) grad_norm 1.4984 (1.6351) loss_scale 1024.0000 (951.8625) mem 11224MB [2024-08-27 10:14:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][300/625] eta 0:02:03 lr 0.001038 wd 0.0500 time 0.3760 (0.3810) data time 0.0008 (0.0025) model time 0.3752 (0.3789) loss 2.4781 (3.1606) grad_norm 1.0320 (1.6310) loss_scale 1024.0000 (954.2591) mem 11224MB [2024-08-27 10:14:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][310/625] eta 0:02:00 lr 0.001038 wd 0.0500 time 0.3703 (0.3810) data time 0.0010 (0.0025) model time 0.3693 (0.3788) loss 3.6086 (3.1561) grad_norm 1.7308 (1.6308) loss_scale 1024.0000 (956.5016) mem 11224MB [2024-08-27 10:14:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][320/625] eta 0:01:56 lr 0.001038 wd 0.0500 time 0.3761 (0.3808) data time 0.0008 (0.0024) model time 0.3753 (0.3787) loss 2.5814 (3.1555) grad_norm 1.4177 (1.6339) loss_scale 1024.0000 (958.6044) mem 11224MB [2024-08-27 10:14:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][330/625] eta 0:01:52 lr 0.001037 wd 0.0500 time 0.3722 (0.3807) data time 0.0011 (0.0024) model time 0.3711 (0.3786) loss 3.0794 (3.1612) grad_norm 1.4920 (1.6277) loss_scale 1024.0000 (960.5801) mem 11224MB [2024-08-27 10:15:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][340/625] eta 0:01:48 lr 0.001037 wd 0.0500 time 0.3706 (0.3811) data time 0.0009 (0.0023) model time 0.3697 (0.3791) loss 3.3799 (3.1693) grad_norm 1.2724 (1.6219) loss_scale 1024.0000 (962.4399) mem 11224MB [2024-08-27 10:15:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][350/625] eta 0:01:44 lr 0.001037 wd 0.0500 time 0.3661 (0.3810) data time 0.0008 (0.0023) model time 0.3653 (0.3789) loss 3.9123 (3.1680) grad_norm 1.4867 (1.6219) loss_scale 1024.0000 (964.1937) mem 11224MB [2024-08-27 10:15:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][360/625] eta 0:01:40 lr 0.001037 wd 0.0500 time 0.3744 (0.3807) data time 0.0011 (0.0023) model time 0.3733 (0.3787) loss 3.4514 (3.1682) grad_norm 1.5146 (1.6213) loss_scale 1024.0000 (965.8504) mem 11224MB [2024-08-27 10:15:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][370/625] eta 0:01:37 lr 0.001037 wd 0.0500 time 0.3799 (0.3806) data time 0.0013 (0.0023) model time 0.3786 (0.3786) loss 2.9508 (3.1652) grad_norm 2.4052 (1.6261) loss_scale 1024.0000 (967.4178) mem 11224MB [2024-08-27 10:15:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][380/625] eta 0:01:33 lr 0.001037 wd 0.0500 time 0.3749 (0.3805) data time 0.0009 (0.0022) model time 0.3740 (0.3785) loss 2.7690 (3.1676) grad_norm 1.4610 (1.6301) loss_scale 1024.0000 (968.9029) mem 11224MB [2024-08-27 10:15:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][390/625] eta 0:01:29 lr 0.001036 wd 0.0500 time 0.3702 (0.3804) data time 0.0010 (0.0022) model time 0.3691 (0.3784) loss 3.0695 (3.1702) grad_norm 1.4965 (1.6300) loss_scale 1024.0000 (970.3120) mem 11224MB [2024-08-27 10:15:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][400/625] eta 0:01:25 lr 0.001036 wd 0.0500 time 0.3790 (0.3803) data time 0.0008 (0.0022) model time 0.3782 (0.3783) loss 3.8762 (3.1747) grad_norm 1.0615 (1.6232) loss_scale 1024.0000 (971.6509) mem 11224MB [2024-08-27 10:15:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][410/625] eta 0:01:21 lr 0.001036 wd 0.0500 time 0.3786 (0.3802) data time 0.0009 (0.0021) model time 0.3777 (0.3782) loss 2.8714 (3.1762) grad_norm 1.9207 (1.6214) loss_scale 1024.0000 (972.9246) mem 11224MB [2024-08-27 10:15:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][420/625] eta 0:01:17 lr 0.001036 wd 0.0500 time 0.3743 (0.3801) data time 0.0010 (0.0021) model time 0.3733 (0.3781) loss 3.1806 (3.1779) grad_norm 1.1344 (1.6167) loss_scale 1024.0000 (974.1378) mem 11224MB [2024-08-27 10:15:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][430/625] eta 0:01:14 lr 0.001036 wd 0.0500 time 0.3788 (0.3800) data time 0.0011 (0.0021) model time 0.3777 (0.3780) loss 3.2476 (3.1769) grad_norm 1.1001 (1.6132) loss_scale 1024.0000 (975.2947) mem 11224MB [2024-08-27 10:15:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][440/625] eta 0:01:10 lr 0.001036 wd 0.0500 time 0.3719 (0.3799) data time 0.0011 (0.0021) model time 0.3708 (0.3780) loss 3.3743 (3.1827) grad_norm 1.3427 (1.6089) loss_scale 1024.0000 (976.3991) mem 11224MB [2024-08-27 10:15:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][450/625] eta 0:01:06 lr 0.001035 wd 0.0500 time 0.3746 (0.3799) data time 0.0008 (0.0021) model time 0.3738 (0.3779) loss 3.7546 (3.1888) grad_norm 1.5808 (1.6094) loss_scale 1024.0000 (977.4545) mem 11224MB [2024-08-27 10:15:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][460/625] eta 0:01:02 lr 0.001035 wd 0.0500 time 0.3720 (0.3798) data time 0.0011 (0.0020) model time 0.3709 (0.3779) loss 3.4913 (3.1931) grad_norm 1.3692 (1.6089) loss_scale 1024.0000 (978.4642) mem 11224MB [2024-08-27 10:15:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][470/625] eta 0:00:58 lr 0.001035 wd 0.0500 time 0.3616 (0.3797) data time 0.0008 (0.0020) model time 0.3608 (0.3777) loss 2.3891 (3.1911) grad_norm 1.3841 (1.6079) loss_scale 1024.0000 (979.4310) mem 11224MB [2024-08-27 10:15:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][480/625] eta 0:00:55 lr 0.001035 wd 0.0500 time 0.3727 (0.3796) data time 0.0011 (0.0020) model time 0.3716 (0.3776) loss 3.4926 (3.1872) grad_norm 1.3723 (1.6026) loss_scale 1024.0000 (980.3576) mem 11224MB [2024-08-27 10:15:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][490/625] eta 0:00:51 lr 0.001035 wd 0.0500 time 0.3790 (0.3795) data time 0.0012 (0.0020) model time 0.3778 (0.3775) loss 3.5193 (3.1911) grad_norm 2.7351 (1.6056) loss_scale 1024.0000 (981.2464) mem 11224MB [2024-08-27 10:16:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][500/625] eta 0:00:47 lr 0.001034 wd 0.0500 time 0.3793 (0.3806) data time 0.0008 (0.0020) model time 0.3784 (0.3788) loss 2.1301 (3.1911) grad_norm 1.8102 (1.6057) loss_scale 1024.0000 (982.0998) mem 11224MB [2024-08-27 10:16:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][510/625] eta 0:00:43 lr 0.001034 wd 0.0500 time 0.3812 (0.3805) data time 0.0008 (0.0020) model time 0.3804 (0.3787) loss 1.9855 (3.1920) grad_norm 1.8697 (1.6101) loss_scale 1024.0000 (982.9198) mem 11224MB [2024-08-27 10:16:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][520/625] eta 0:00:39 lr 0.001034 wd 0.0500 time 0.3657 (0.3807) data time 0.0015 (0.0019) model time 0.3643 (0.3790) loss 3.8255 (3.1977) grad_norm 2.2237 (1.6236) loss_scale 1024.0000 (983.7083) mem 11224MB [2024-08-27 10:16:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][530/625] eta 0:00:36 lr 0.001034 wd 0.0500 time 0.3794 (0.3806) data time 0.0012 (0.0019) model time 0.3782 (0.3789) loss 3.3395 (3.1946) grad_norm 1.1821 (1.6233) loss_scale 1024.0000 (984.4670) mem 11224MB [2024-08-27 10:16:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][540/625] eta 0:00:32 lr 0.001034 wd 0.0500 time 0.3738 (0.3806) data time 0.0014 (0.0019) model time 0.3724 (0.3788) loss 3.1188 (3.1889) grad_norm 1.3968 (1.6196) loss_scale 1024.0000 (985.1978) mem 11224MB [2024-08-27 10:16:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][550/625] eta 0:00:28 lr 0.001034 wd 0.0500 time 0.3797 (0.3804) data time 0.0013 (0.0019) model time 0.3783 (0.3787) loss 3.1141 (3.1943) grad_norm 2.1854 (1.6201) loss_scale 1024.0000 (985.9020) mem 11224MB [2024-08-27 10:16:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][560/625] eta 0:00:24 lr 0.001033 wd 0.0500 time 0.3706 (0.3804) data time 0.0013 (0.0019) model time 0.3693 (0.3787) loss 3.1788 (3.1962) grad_norm 1.5826 (1.6214) loss_scale 1024.0000 (986.5811) mem 11224MB [2024-08-27 10:16:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][570/625] eta 0:00:20 lr 0.001033 wd 0.0500 time 0.3780 (0.3803) data time 0.0011 (0.0019) model time 0.3770 (0.3785) loss 2.6028 (3.1950) grad_norm 1.3760 (1.6166) loss_scale 1024.0000 (987.2364) mem 11224MB [2024-08-27 10:16:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][580/625] eta 0:00:17 lr 0.001033 wd 0.0500 time 0.3833 (0.3802) data time 0.0009 (0.0019) model time 0.3824 (0.3785) loss 2.9690 (3.1958) grad_norm 1.6575 (1.6166) loss_scale 1024.0000 (987.8692) mem 11224MB [2024-08-27 10:16:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][590/625] eta 0:00:13 lr 0.001033 wd 0.0500 time 0.3728 (0.3801) data time 0.0011 (0.0019) model time 0.3717 (0.3784) loss 3.3316 (3.1999) grad_norm 2.1737 (1.6300) loss_scale 1024.0000 (988.4805) mem 11224MB [2024-08-27 10:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][600/625] eta 0:00:09 lr 0.001033 wd 0.0500 time 0.3823 (0.3801) data time 0.0009 (0.0018) model time 0.3814 (0.3783) loss 3.5175 (3.2017) grad_norm 1.6514 (1.6270) loss_scale 1024.0000 (989.0715) mem 11224MB [2024-08-27 10:16:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][610/625] eta 0:00:05 lr 0.001032 wd 0.0500 time 0.3726 (0.3799) data time 0.0008 (0.0018) model time 0.3718 (0.3782) loss 3.1734 (3.1959) grad_norm 1.3815 (1.6260) loss_scale 1024.0000 (989.6432) mem 11224MB [2024-08-27 10:16:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [157/300][620/625] eta 0:00:01 lr 0.001032 wd 0.0500 time 0.3687 (0.3797) data time 0.0008 (0.0018) model time 0.3679 (0.3779) loss 2.8674 (3.1955) grad_norm 1.4320 (1.6228) loss_scale 1024.0000 (990.1965) mem 11224MB [2024-08-27 10:16:48 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 157 training takes 0:03:57 [2024-08-27 10:16:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 10:16:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-27 10:16:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.6035 (0.6035) Acc@1 87.012 (87.012) Acc@5 98.145 (98.145) Mem 11224MB [2024-08-27 10:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.146) Loss 0.9380 (0.7252) Acc@1 77.783 (83.856) Acc@5 94.482 (96.973) Mem 11224MB [2024-08-27 10:16:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 1.1133 (0.8757) Acc@1 73.486 (80.229) Acc@5 93.555 (95.294) Mem 11224MB [2024-08-27 10:16:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 79.950 Acc@5 95.270 [2024-08-27 10:16:52 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.0% [2024-08-27 10:16:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.879 (0.879) Loss 0.5068 (0.5068) Acc@1 89.062 (89.062) Acc@5 98.535 (98.535) Mem 11224MB [2024-08-27 10:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.180) Loss 0.8403 (0.6364) Acc@1 79.834 (85.574) Acc@5 95.996 (97.523) Mem 11224MB [2024-08-27 10:16:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.139) Loss 0.9648 (0.7607) Acc@1 75.928 (82.480) Acc@5 94.971 (96.303) Mem 11224MB [2024-08-27 10:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.218 Acc@5 96.299 [2024-08-27 10:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.2% [2024-08-27 10:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.22% [2024-08-27 10:16:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-27 10:16:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-27 10:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][0/625] eta 0:09:19 lr 0.001032 wd 0.0500 time 0.8956 (0.8956) data time 0.5588 (0.5588) model time 0.0000 (0.0000) loss 2.1977 (2.1977) grad_norm 1.6751 (1.6751) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][10/625] eta 0:04:19 lr 0.001032 wd 0.0500 time 0.3844 (0.4226) data time 0.0008 (0.0518) model time 0.0000 (0.0000) loss 2.5051 (2.9847) grad_norm 1.2477 (1.4214) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][20/625] eta 0:04:02 lr 0.001032 wd 0.0500 time 0.3795 (0.4013) data time 0.0009 (0.0277) model time 0.0000 (0.0000) loss 3.4407 (3.2216) grad_norm 1.3901 (1.4119) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][30/625] eta 0:03:53 lr 0.001032 wd 0.0500 time 0.3728 (0.3932) data time 0.0009 (0.0191) model time 0.0000 (0.0000) loss 3.7989 (3.2246) grad_norm 1.4305 (1.4249) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][40/625] eta 0:03:47 lr 0.001032 wd 0.0500 time 0.3732 (0.3889) data time 0.0009 (0.0148) model time 0.0000 (0.0000) loss 2.5658 (3.2120) grad_norm 1.4116 (1.4781) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][50/625] eta 0:03:42 lr 0.001031 wd 0.0500 time 0.3740 (0.3864) data time 0.0011 (0.0121) model time 0.0000 (0.0000) loss 2.5058 (3.1955) grad_norm 1.6393 (1.5113) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][60/625] eta 0:03:37 lr 0.001031 wd 0.0500 time 0.3753 (0.3849) data time 0.0010 (0.0103) model time 0.3743 (0.3760) loss 2.5350 (3.1579) grad_norm 1.5230 (1.5114) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][70/625] eta 0:03:37 lr 0.001031 wd 0.0500 time 0.3620 (0.3919) data time 0.0010 (0.0090) model time 0.3610 (0.4047) loss 3.6854 (3.1654) grad_norm 1.4029 (1.4799) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][80/625] eta 0:03:32 lr 0.001031 wd 0.0500 time 0.3682 (0.3899) data time 0.0009 (0.0080) model time 0.3673 (0.3947) loss 3.5196 (3.1577) grad_norm 2.1097 (1.5196) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][90/625] eta 0:03:27 lr 0.001031 wd 0.0500 time 0.3759 (0.3882) data time 0.0011 (0.0073) model time 0.3748 (0.3894) loss 2.8535 (3.1493) grad_norm 2.0994 (1.5118) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][100/625] eta 0:03:23 lr 0.001030 wd 0.0500 time 0.3728 (0.3869) data time 0.0008 (0.0067) model time 0.3721 (0.3862) loss 3.8101 (3.1462) grad_norm 1.6568 (1.4969) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][110/625] eta 0:03:18 lr 0.001030 wd 0.0500 time 0.3724 (0.3857) data time 0.0009 (0.0062) model time 0.3715 (0.3840) loss 2.1993 (3.1364) grad_norm 1.8989 (1.4877) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][120/625] eta 0:03:14 lr 0.001030 wd 0.0500 time 0.3751 (0.3849) data time 0.0012 (0.0058) model time 0.3739 (0.3826) loss 3.0678 (3.1515) grad_norm 1.1622 (1.4709) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][130/625] eta 0:03:10 lr 0.001030 wd 0.0500 time 0.3793 (0.3843) data time 0.0008 (0.0054) model time 0.3785 (0.3818) loss 2.9778 (3.1672) grad_norm 1.0610 (1.4683) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][140/625] eta 0:03:06 lr 0.001030 wd 0.0500 time 0.3797 (0.3839) data time 0.0008 (0.0051) model time 0.3789 (0.3813) loss 3.9419 (3.1696) grad_norm 1.6381 (1.5008) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][150/625] eta 0:03:02 lr 0.001030 wd 0.0500 time 0.3746 (0.3834) data time 0.0013 (0.0049) model time 0.3734 (0.3807) loss 2.9096 (3.1724) grad_norm 1.2868 (1.5088) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:17:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][160/625] eta 0:02:58 lr 0.001029 wd 0.0500 time 0.3700 (0.3829) data time 0.0011 (0.0046) model time 0.3690 (0.3801) loss 1.9761 (3.1565) grad_norm 1.5632 (1.5113) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][170/625] eta 0:02:54 lr 0.001029 wd 0.0500 time 0.3786 (0.3835) data time 0.0009 (0.0044) model time 0.3777 (0.3811) loss 2.9908 (3.1444) grad_norm 1.7289 (1.5199) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][180/625] eta 0:02:50 lr 0.001029 wd 0.0500 time 0.3783 (0.3831) data time 0.0010 (0.0043) model time 0.3773 (0.3806) loss 3.0775 (3.1409) grad_norm 1.0700 (1.5465) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][190/625] eta 0:02:46 lr 0.001029 wd 0.0500 time 0.3963 (0.3829) data time 0.0008 (0.0041) model time 0.3955 (0.3805) loss 3.0500 (3.1486) grad_norm 1.2006 (1.5568) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][200/625] eta 0:02:42 lr 0.001029 wd 0.0500 time 0.3747 (0.3825) data time 0.0008 (0.0039) model time 0.3740 (0.3800) loss 3.1510 (3.1545) grad_norm 1.5946 (1.5540) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][210/625] eta 0:02:38 lr 0.001028 wd 0.0500 time 0.3712 (0.3820) data time 0.0010 (0.0038) model time 0.3701 (0.3795) loss 3.0648 (3.1593) grad_norm 1.2404 (1.5409) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][220/625] eta 0:02:34 lr 0.001028 wd 0.0500 time 0.3715 (0.3818) data time 0.0011 (0.0037) model time 0.3704 (0.3793) loss 2.8573 (3.1693) grad_norm 1.1614 (1.5348) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][230/625] eta 0:02:30 lr 0.001028 wd 0.0500 time 0.3752 (0.3816) data time 0.0008 (0.0036) model time 0.3744 (0.3791) loss 2.2389 (3.1715) grad_norm 1.9457 (1.5333) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][240/625] eta 0:02:26 lr 0.001028 wd 0.0500 time 0.3777 (0.3815) data time 0.0011 (0.0035) model time 0.3766 (0.3790) loss 3.1772 (3.1762) grad_norm 2.2309 (1.5344) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][250/625] eta 0:02:23 lr 0.001028 wd 0.0500 time 0.5646 (0.3834) data time 0.0008 (0.0034) model time 0.5637 (0.3815) loss 3.5099 (3.1856) grad_norm 1.3404 (1.5375) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-27 10:18:37 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-27 10:18:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-27 10:18:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-30 11:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-30 11:19:44 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-30 11:21:13 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-30 11:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-30 11:21:27 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-30 11:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-30 11:21:37 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-30 11:21:39 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-30 11:21:41 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-30 11:21:41 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 158) [2024-08-30 11:21:41 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-30 11:21:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][260/625] eta 1:27:05 lr 0.001028 wd 0.0500 time 14.3165 (14.3165) data time 0.7342 (0.7342) model time 13.5823 (13.5823) loss 4.0986 (4.0986) grad_norm 1.3750 (1.3750) loss_scale 1024.0000 (1024.0000) mem 23875MB [2024-08-30 11:22:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][270/625] eta 0:09:39 lr 0.001027 wd 0.0500 time 0.3458 (1.6313) data time 0.0010 (0.0676) model time 0.3449 (1.5636) loss 2.6091 (3.5392) grad_norm 1.3316 (1.4493) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][280/625] eta 0:05:51 lr 0.001027 wd 0.0500 time 0.3477 (1.0199) data time 0.0009 (0.0359) model time 0.3468 (0.9841) loss 3.3285 (3.4886) grad_norm 1.3791 (1.4228) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][290/625] eta 0:04:31 lr 0.001027 wd 0.0500 time 0.3470 (0.8096) data time 0.0007 (0.0247) model time 0.3463 (0.7849) loss 2.5334 (3.4653) grad_norm 1.3283 (1.4405) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][300/625] eta 0:03:48 lr 0.001027 wd 0.0500 time 0.6089 (0.7034) data time 0.0009 (0.0189) model time 0.6081 (0.6845) loss 3.4790 (3.3662) grad_norm 2.1399 (1.4561) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][310/625] eta 0:03:19 lr 0.001027 wd 0.0500 time 0.3420 (0.6332) data time 0.0007 (0.0154) model time 0.3413 (0.6178) loss 3.8042 (3.3726) grad_norm 1.2748 (1.5157) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][320/625] eta 0:02:58 lr 0.001027 wd 0.0500 time 0.3440 (0.5860) data time 0.0009 (0.0130) model time 0.3431 (0.5730) loss 3.0781 (3.3489) grad_norm 1.7413 (1.5081) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][330/625] eta 0:02:42 lr 0.001026 wd 0.0500 time 0.3435 (0.5525) data time 0.0009 (0.0113) model time 0.3426 (0.5412) loss 2.7006 (3.2952) grad_norm 1.4922 (1.5123) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][340/625] eta 0:02:30 lr 0.001026 wd 0.0500 time 0.3501 (0.5273) data time 0.0009 (0.0100) model time 0.3492 (0.5172) loss 2.7524 (3.2864) grad_norm 1.5215 (1.5084) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][350/625] eta 0:02:19 lr 0.001026 wd 0.0500 time 0.3474 (0.5074) data time 0.0007 (0.0090) model time 0.3467 (0.4984) loss 4.0219 (3.2768) grad_norm 2.6987 (1.5858) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][360/625] eta 0:02:10 lr 0.001026 wd 0.0500 time 0.3411 (0.4915) data time 0.0006 (0.0082) model time 0.3404 (0.4833) loss 3.3454 (3.2841) grad_norm 2.2547 (1.6492) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][370/625] eta 0:02:02 lr 0.001026 wd 0.0500 time 0.3472 (0.4787) data time 0.0009 (0.0076) model time 0.3462 (0.4711) loss 2.9676 (3.2890) grad_norm 1.6529 (1.6209) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][380/625] eta 0:01:54 lr 0.001025 wd 0.0500 time 0.3427 (0.4678) data time 0.0006 (0.0070) model time 0.3421 (0.4607) loss 2.2412 (3.2813) grad_norm 2.0337 (1.6151) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][390/625] eta 0:01:47 lr 0.001025 wd 0.0500 time 0.3465 (0.4585) data time 0.0008 (0.0065) model time 0.3458 (0.4520) loss 3.6243 (3.2788) grad_norm 2.4687 (1.6269) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][400/625] eta 0:01:41 lr 0.001025 wd 0.0500 time 0.3422 (0.4508) data time 0.0007 (0.0061) model time 0.3415 (0.4446) loss 3.1389 (3.2666) grad_norm 1.3061 (1.6285) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][410/625] eta 0:01:35 lr 0.001025 wd 0.0500 time 0.3486 (0.4440) data time 0.0009 (0.0058) model time 0.3478 (0.4382) loss 2.3302 (3.2504) grad_norm 1.7660 (1.6269) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][420/625] eta 0:01:29 lr 0.001025 wd 0.0500 time 0.3401 (0.4381) data time 0.0008 (0.0055) model time 0.3393 (0.4326) loss 3.3283 (3.2490) grad_norm 1.7946 (1.6260) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:22:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][430/625] eta 0:01:24 lr 0.001025 wd 0.0500 time 0.3442 (0.4328) data time 0.0008 (0.0052) model time 0.3434 (0.4276) loss 3.2851 (3.2421) grad_norm 1.4540 (1.6262) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:23:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][440/625] eta 0:01:19 lr 0.001024 wd 0.0500 time 0.3527 (0.4281) data time 0.0009 (0.0050) model time 0.3517 (0.4231) loss 3.5346 (3.2261) grad_norm 1.5279 (1.6118) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:23:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][450/625] eta 0:01:14 lr 0.001024 wd 0.0500 time 0.3446 (0.4239) data time 0.0008 (0.0048) model time 0.3437 (0.4191) loss 2.8022 (3.2202) grad_norm 1.5795 (1.6049) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:23:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][460/625] eta 0:01:09 lr 0.001024 wd 0.0500 time 0.3531 (0.4202) data time 0.0009 (0.0046) model time 0.3522 (0.4156) loss 2.9519 (3.2069) grad_norm 2.4697 (1.6084) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:23:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][470/625] eta 0:01:04 lr 0.001024 wd 0.0500 time 0.3507 (0.4168) data time 0.0010 (0.0044) model time 0.3497 (0.4123) loss 3.6813 (3.2033) grad_norm 1.2091 (1.6196) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:23:16 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-30 11:23:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-30 11:23:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-30 11:28:47 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-30 11:28:48 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-30 11:29:00 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-30 11:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-30 11:29:12 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-30 11:29:14 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-30 11:29:16 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-30 11:29:16 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 158) [2024-08-30 11:29:16 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-08-30 11:29:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][480/625] eta 0:33:05 lr 0.001024 wd 0.0500 time 13.6947 (13.6947) data time 1.0089 (1.0089) model time 12.6858 (12.6858) loss 3.9038 (3.9038) grad_norm 1.5119 (1.5119) loss_scale 1024.0000 (1024.0000) mem 23875MB [2024-08-30 11:29:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][490/625] eta 0:03:38 lr 0.001024 wd 0.0500 time 0.3471 (1.6163) data time 0.0009 (0.0926) model time 0.3462 (1.5237) loss 2.7952 (3.5083) grad_norm 1.3801 (1.4518) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:29:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][500/625] eta 0:02:06 lr 0.001023 wd 0.0500 time 0.3458 (1.0119) data time 0.0009 (0.0489) model time 0.3449 (0.9629) loss 3.4894 (3.4485) grad_norm 1.4652 (1.5190) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][510/625] eta 0:01:32 lr 0.001023 wd 0.0500 time 0.3472 (0.8038) data time 0.0006 (0.0334) model time 0.3466 (0.7703) loss 2.7033 (3.4606) grad_norm 1.9656 (1.5314) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:29:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][520/625] eta 0:01:13 lr 0.001023 wd 0.0500 time 0.6156 (0.6986) data time 0.0008 (0.0255) model time 0.6148 (0.6731) loss 3.0786 (3.4131) grad_norm 1.4354 (1.5054) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:29:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][530/625] eta 0:00:59 lr 0.001023 wd 0.0500 time 0.3416 (0.6297) data time 0.0007 (0.0207) model time 0.3409 (0.6090) loss 3.4174 (3.3840) grad_norm 1.6720 (1.5241) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:29:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][540/625] eta 0:00:49 lr 0.001023 wd 0.0500 time 0.3474 (0.5832) data time 0.0010 (0.0174) model time 0.3464 (0.5657) loss 3.3464 (3.3611) grad_norm 2.3680 (1.6389) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:29:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][550/625] eta 0:00:41 lr 0.001022 wd 0.0500 time 0.3491 (0.5498) data time 0.0008 (0.0151) model time 0.3484 (0.5347) loss 3.0114 (3.3312) grad_norm 1.3793 (1.6479) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:30:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][560/625] eta 0:00:34 lr 0.001022 wd 0.0500 time 0.3449 (0.5250) data time 0.0008 (0.0134) model time 0.3441 (0.5117) loss 2.9601 (3.3219) grad_norm 1.6208 (1.6434) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:30:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][570/625] eta 0:00:27 lr 0.001022 wd 0.0500 time 0.3431 (0.5056) data time 0.0008 (0.0120) model time 0.3423 (0.4936) loss 3.6728 (3.3008) grad_norm 1.9693 (1.6744) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:30:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][580/625] eta 0:00:22 lr 0.001022 wd 0.0500 time 0.3615 (0.4906) data time 0.0007 (0.0110) model time 0.3608 (0.4796) loss 3.8429 (3.3049) grad_norm 1.9317 (1.7026) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:30:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][590/625] eta 0:00:16 lr 0.001022 wd 0.0500 time 0.3449 (0.4779) data time 0.0008 (0.0101) model time 0.3441 (0.4677) loss 2.6574 (3.3043) grad_norm 1.5715 (1.7036) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][600/625] eta 0:00:11 lr 0.001022 wd 0.0500 time 0.3433 (0.4669) data time 0.0007 (0.0094) model time 0.3426 (0.4576) loss 1.8393 (3.3014) grad_norm 1.3932 (1.6834) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][610/625] eta 0:00:06 lr 0.001021 wd 0.0500 time 0.3475 (0.4579) data time 0.0006 (0.0087) model time 0.3469 (0.4491) loss 3.5091 (3.2832) grad_norm 2.0075 (1.6652) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:30:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [158/300][620/625] eta 0:00:02 lr 0.001021 wd 0.0500 time 0.3338 (0.4496) data time 0.0004 (0.0082) model time 0.3334 (0.4414) loss 3.2647 (3.2658) grad_norm 2.0120 (1.6616) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-08-30 11:30:25 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 158 training takes 0:01:04 [2024-08-30 11:30:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-30 11:30:32 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-30 11:30:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.432 (0.432) Loss 0.5791 (0.5791) Acc@1 87.012 (87.012) Acc@5 98.389 (98.389) Mem 11219MB [2024-08-30 11:30:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.127) Loss 0.9531 (0.7192) Acc@1 77.930 (83.896) Acc@5 94.385 (96.888) Mem 11219MB [2024-08-30 11:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.109) Loss 1.0312 (0.8660) Acc@1 76.465 (80.404) Acc@5 93.799 (95.287) Mem 11219MB [2024-08-30 11:30:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.168 Acc@5 95.286 [2024-08-30 11:30:36 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.2% [2024-08-30 11:30:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.939 (0.939) Loss 0.5049 (0.5049) Acc@1 88.965 (88.965) Acc@5 98.535 (98.535) Mem 11219MB [2024-08-30 11:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.175) Loss 0.8408 (0.6359) Acc@1 79.785 (85.596) Acc@5 95.898 (97.532) Mem 11219MB [2024-08-30 11:30:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.089 (0.135) Loss 0.9634 (0.7601) Acc@1 76.221 (82.478) Acc@5 95.117 (96.322) Mem 11219MB [2024-08-30 11:30:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.212 Acc@5 96.313 [2024-08-30 11:30:40 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.2% [2024-08-30 11:30:40 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.21% [2024-08-30 11:30:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-08-30 11:30:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-08-30 11:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][0/625] eta 0:07:37 lr 0.001021 wd 0.0500 time 0.7322 (0.7322) data time 0.3677 (0.3677) model time 0.0000 (0.0000) loss 2.6829 (2.6829) grad_norm 1.5009 (1.5009) loss_scale 1024.0000 (1024.0000) mem 11228MB [2024-08-30 11:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][10/625] eta 0:03:56 lr 0.001021 wd 0.0500 time 0.3482 (0.3839) data time 0.0009 (0.0343) model time 0.0000 (0.0000) loss 3.0604 (3.0634) grad_norm 1.2996 (1.4728) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:30:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][20/625] eta 0:03:41 lr 0.001021 wd 0.0500 time 0.3468 (0.3665) data time 0.0009 (0.0187) model time 0.0000 (0.0000) loss 3.0157 (3.1620) grad_norm 1.4955 (1.3903) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:30:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][30/625] eta 0:03:36 lr 0.001021 wd 0.0500 time 0.3500 (0.3633) data time 0.0008 (0.0130) model time 0.0000 (0.0000) loss 2.7771 (3.1206) grad_norm 1.4386 (1.4008) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:30:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][40/625] eta 0:03:30 lr 0.001020 wd 0.0500 time 0.3496 (0.3598) data time 0.0009 (0.0100) model time 0.0000 (0.0000) loss 2.5306 (3.0983) grad_norm 1.4940 (1.4325) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:30:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][50/625] eta 0:03:25 lr 0.001020 wd 0.0500 time 0.3461 (0.3574) data time 0.0007 (0.0084) model time 0.0000 (0.0000) loss 2.5909 (3.0974) grad_norm 1.6368 (1.6175) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][60/625] eta 0:03:21 lr 0.001020 wd 0.0500 time 0.3582 (0.3562) data time 0.0006 (0.0071) model time 0.3576 (0.3495) loss 2.2888 (3.0703) grad_norm 1.4342 (1.5867) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][70/625] eta 0:03:17 lr 0.001020 wd 0.0500 time 0.3421 (0.3552) data time 0.0007 (0.0063) model time 0.3414 (0.3486) loss 2.1038 (3.0634) grad_norm 1.3696 (1.5635) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][80/625] eta 0:03:13 lr 0.001020 wd 0.0500 time 0.3456 (0.3544) data time 0.0006 (0.0056) model time 0.3451 (0.3484) loss 3.3978 (3.0979) grad_norm 3.2176 (1.6268) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][90/625] eta 0:03:09 lr 0.001020 wd 0.0500 time 0.3559 (0.3538) data time 0.0007 (0.0051) model time 0.3551 (0.3482) loss 3.6562 (3.0933) grad_norm 0.9399 (1.6539) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][100/625] eta 0:03:06 lr 0.001019 wd 0.0500 time 0.3552 (0.3556) data time 0.0006 (0.0047) model time 0.3545 (0.3527) loss 2.1204 (3.0948) grad_norm 1.2918 (1.6526) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][110/625] eta 0:03:02 lr 0.001019 wd 0.0500 time 0.3430 (0.3549) data time 0.0010 (0.0044) model time 0.3420 (0.3517) loss 2.2869 (3.0786) grad_norm 1.9301 (1.6309) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][120/625] eta 0:02:58 lr 0.001019 wd 0.0500 time 0.3490 (0.3542) data time 0.0006 (0.0041) model time 0.3483 (0.3508) loss 2.8530 (3.0669) grad_norm 1.6286 (1.6358) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][130/625] eta 0:02:55 lr 0.001019 wd 0.0500 time 0.3447 (0.3536) data time 0.0009 (0.0039) model time 0.3438 (0.3501) loss 3.3467 (3.0839) grad_norm 1.4249 (1.6242) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][140/625] eta 0:02:51 lr 0.001019 wd 0.0500 time 0.3516 (0.3533) data time 0.0009 (0.0037) model time 0.3507 (0.3499) loss 2.5425 (3.0828) grad_norm 1.2407 (1.6084) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][150/625] eta 0:02:47 lr 0.001018 wd 0.0500 time 0.3940 (0.3536) data time 0.0006 (0.0035) model time 0.3934 (0.3506) loss 2.1738 (3.0727) grad_norm 1.2079 (1.5846) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][160/625] eta 0:02:44 lr 0.001018 wd 0.0500 time 0.3504 (0.3537) data time 0.0008 (0.0034) model time 0.3497 (0.3509) loss 1.8271 (3.0619) grad_norm 1.3565 (1.5794) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][170/625] eta 0:02:40 lr 0.001018 wd 0.0500 time 0.3468 (0.3534) data time 0.0006 (0.0033) model time 0.3462 (0.3506) loss 3.5315 (3.0832) grad_norm 1.9665 (1.5887) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][180/625] eta 0:02:37 lr 0.001018 wd 0.0500 time 0.3419 (0.3533) data time 0.0008 (0.0032) model time 0.3410 (0.3506) loss 3.2939 (3.1021) grad_norm 1.8061 (1.5919) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][190/625] eta 0:02:33 lr 0.001018 wd 0.0500 time 0.3487 (0.3530) data time 0.0009 (0.0030) model time 0.3478 (0.3503) loss 3.0950 (3.0999) grad_norm 1.3054 (1.5916) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-08-30 11:31:49 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-08-30 11:31:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-08-30 11:31:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-08-30 11:33:42 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-08-30 11:33:43 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-08-30 11:33:57 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-08-30 11:34:12 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-08-30 11:34:12 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-08-30 11:34:15 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-08-30 11:34:17 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-08-30 11:34:17 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 159) [2024-08-30 11:34:17 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 05:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 05:59:31 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 05:59:44 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 05:59:53 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 05:59:53 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 05:59:55 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 05:59:57 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 05:59:57 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 159) [2024-09-01 05:59:57 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 06:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][200/625] eta 0:16:14 lr 0.001018 wd 0.0500 time 0.3425 (2.2921) data time 0.0007 (0.0788) model time 0.3418 (2.2133) loss 3.4558 (3.4866) grad_norm 1.3230 (1.6962) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][210/625] eta 0:08:46 lr 0.001017 wd 0.0500 time 0.3461 (1.2680) data time 0.0010 (0.0378) model time 0.3451 (1.2302) loss 3.5626 (3.3869) grad_norm 1.2952 (1.5914) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][220/625] eta 0:06:27 lr 0.001017 wd 0.0500 time 0.3454 (0.9574) data time 0.0007 (0.0251) model time 0.3447 (0.9324) loss 3.8944 (3.4611) grad_norm 1.2082 (1.5219) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][230/625] eta 0:05:16 lr 0.001017 wd 0.0500 time 0.3459 (0.8012) data time 0.0008 (0.0189) model time 0.3451 (0.7824) loss 2.9574 (3.3626) grad_norm 1.8203 (1.5646) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][240/625] eta 0:04:34 lr 0.001017 wd 0.0500 time 0.3451 (0.7140) data time 0.0008 (0.0152) model time 0.3443 (0.6988) loss 3.3528 (3.3299) grad_norm 1.3718 (1.5656) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][250/625] eta 0:04:04 lr 0.001017 wd 0.0500 time 0.3546 (0.6519) data time 0.0006 (0.0128) model time 0.3540 (0.6391) loss 2.4907 (3.3037) grad_norm 1.4284 (1.5858) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][260/625] eta 0:03:41 lr 0.001016 wd 0.0500 time 0.3458 (0.6080) data time 0.0009 (0.0110) model time 0.3450 (0.5970) loss 3.1897 (3.2750) grad_norm 1.3908 (1.5525) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][270/625] eta 0:03:24 lr 0.001016 wd 0.0500 time 0.3388 (0.5752) data time 0.0009 (0.0097) model time 0.3379 (0.5654) loss 3.3305 (3.2656) grad_norm 1.6559 (1.5556) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][280/625] eta 0:03:09 lr 0.001016 wd 0.0500 time 0.3443 (0.5497) data time 0.0007 (0.0088) model time 0.3436 (0.5409) loss 3.4065 (3.2412) grad_norm 1.4094 (1.5658) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][290/625] eta 0:02:57 lr 0.001016 wd 0.0500 time 0.3478 (0.5293) data time 0.0009 (0.0080) model time 0.3469 (0.5213) loss 2.9678 (3.2538) grad_norm 1.6839 (1.5638) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:00:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][300/625] eta 0:02:46 lr 0.001016 wd 0.0500 time 0.3478 (0.5126) data time 0.0006 (0.0073) model time 0.3472 (0.5053) loss 3.8596 (3.2668) grad_norm 2.4405 (1.5692) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][310/625] eta 0:02:37 lr 0.001016 wd 0.0500 time 0.3428 (0.4987) data time 0.0008 (0.0068) model time 0.3420 (0.4919) loss 3.5348 (3.2661) grad_norm 1.0979 (1.5852) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][320/625] eta 0:02:28 lr 0.001015 wd 0.0500 time 0.3421 (0.4870) data time 0.0006 (0.0063) model time 0.3414 (0.4807) loss 3.1207 (3.2550) grad_norm 1.7834 (1.5962) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][330/625] eta 0:02:20 lr 0.001015 wd 0.0500 time 0.3491 (0.4771) data time 0.0007 (0.0059) model time 0.3484 (0.4712) loss 3.6723 (3.2415) grad_norm 1.3765 (1.6324) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][340/625] eta 0:02:13 lr 0.001015 wd 0.0500 time 0.3477 (0.4684) data time 0.0006 (0.0056) model time 0.3471 (0.4628) loss 2.9516 (3.2272) grad_norm 1.9586 (1.6273) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][350/625] eta 0:02:06 lr 0.001015 wd 0.0500 time 0.3441 (0.4608) data time 0.0007 (0.0053) model time 0.3434 (0.4555) loss 3.7907 (3.2229) grad_norm 1.3956 (1.6279) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][360/625] eta 0:02:00 lr 0.001015 wd 0.0500 time 0.3439 (0.4541) data time 0.0008 (0.0050) model time 0.3431 (0.4491) loss 3.3608 (3.2270) grad_norm 1.2652 (1.6204) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][370/625] eta 0:01:54 lr 0.001015 wd 0.0500 time 0.3588 (0.4481) data time 0.0007 (0.0048) model time 0.3581 (0.4434) loss 3.4097 (3.2159) grad_norm 1.4453 (1.6131) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][380/625] eta 0:01:48 lr 0.001014 wd 0.0500 time 0.3450 (0.4428) data time 0.0007 (0.0046) model time 0.3444 (0.4382) loss 4.1132 (3.2196) grad_norm 2.4983 (1.6296) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][390/625] eta 0:01:42 lr 0.001014 wd 0.0500 time 0.3467 (0.4379) data time 0.0009 (0.0044) model time 0.3458 (0.4336) loss 2.2175 (3.2033) grad_norm 1.2323 (1.6413) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][400/625] eta 0:01:37 lr 0.001014 wd 0.0500 time 0.3553 (0.4338) data time 0.0006 (0.0042) model time 0.3547 (0.4295) loss 3.2523 (3.1950) grad_norm 1.3408 (1.6282) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][410/625] eta 0:01:32 lr 0.001014 wd 0.0500 time 0.3477 (0.4299) data time 0.0007 (0.0041) model time 0.3470 (0.4258) loss 3.8605 (3.1938) grad_norm 1.2194 (1.6118) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][420/625] eta 0:01:27 lr 0.001014 wd 0.0500 time 0.3473 (0.4282) data time 0.0007 (0.0039) model time 0.3466 (0.4242) loss 2.5276 (3.1962) grad_norm 1.6229 (1.6397) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][430/625] eta 0:01:22 lr 0.001013 wd 0.0500 time 0.3579 (0.4256) data time 0.0006 (0.0038) model time 0.3573 (0.4218) loss 2.4245 (3.1936) grad_norm 1.6248 (1.6304) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][440/625] eta 0:01:18 lr 0.001013 wd 0.0500 time 0.3439 (0.4224) data time 0.0007 (0.0037) model time 0.3431 (0.4187) loss 3.0089 (3.1946) grad_norm 1.7688 (1.6297) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][450/625] eta 0:01:13 lr 0.001013 wd 0.0500 time 0.3448 (0.4195) data time 0.0010 (0.0036) model time 0.3438 (0.4159) loss 3.5051 (3.1881) grad_norm 1.4104 (1.6340) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][460/625] eta 0:01:08 lr 0.001013 wd 0.0500 time 0.3444 (0.4168) data time 0.0007 (0.0035) model time 0.3436 (0.4134) loss 1.9429 (3.1784) grad_norm 1.5554 (1.6459) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][470/625] eta 0:01:04 lr 0.001013 wd 0.0500 time 0.3462 (0.4143) data time 0.0009 (0.0034) model time 0.3453 (0.4110) loss 3.3234 (3.1859) grad_norm 1.9158 (1.6928) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][480/625] eta 0:00:59 lr 0.001013 wd 0.0500 time 0.3466 (0.4120) data time 0.0008 (0.0033) model time 0.3458 (0.4087) loss 3.0453 (3.1849) grad_norm 1.0656 (1.6868) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][490/625] eta 0:00:55 lr 0.001012 wd 0.0500 time 0.3470 (0.4098) data time 0.0010 (0.0032) model time 0.3460 (0.4066) loss 3.2315 (3.1691) grad_norm 1.1897 (1.6784) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][500/625] eta 0:00:50 lr 0.001012 wd 0.0500 time 0.3486 (0.4078) data time 0.0008 (0.0031) model time 0.3478 (0.4047) loss 3.2282 (3.1639) grad_norm 1.2820 (1.6742) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][510/625] eta 0:00:46 lr 0.001012 wd 0.0500 time 0.3502 (0.4059) data time 0.0006 (0.0031) model time 0.3496 (0.4028) loss 4.1403 (3.1750) grad_norm 1.4039 (1.6716) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][520/625] eta 0:00:42 lr 0.001012 wd 0.0500 time 0.3431 (0.4042) data time 0.0011 (0.0030) model time 0.3420 (0.4012) loss 2.0195 (3.1784) grad_norm 1.7518 (1.6744) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][530/625] eta 0:00:38 lr 0.001012 wd 0.0500 time 0.3494 (0.4026) data time 0.0009 (0.0029) model time 0.3485 (0.3997) loss 3.2164 (3.1800) grad_norm 1.6048 (1.6660) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][540/625] eta 0:00:34 lr 0.001012 wd 0.0500 time 0.3448 (0.4022) data time 0.0007 (0.0029) model time 0.3441 (0.3994) loss 3.2779 (3.1850) grad_norm 2.3969 (1.6913) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][550/625] eta 0:00:30 lr 0.001011 wd 0.0500 time 0.3461 (0.4008) data time 0.0007 (0.0028) model time 0.3454 (0.3980) loss 2.5565 (3.1822) grad_norm 1.4060 (1.6850) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][560/625] eta 0:00:25 lr 0.001011 wd 0.0500 time 0.3451 (0.3994) data time 0.0008 (0.0028) model time 0.3442 (0.3966) loss 3.4438 (3.1814) grad_norm 2.4833 (1.6841) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][570/625] eta 0:00:21 lr 0.001011 wd 0.0500 time 0.3443 (0.3980) data time 0.0007 (0.0027) model time 0.3437 (0.3953) loss 3.4763 (3.1818) grad_norm 1.5388 (1.6896) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][580/625] eta 0:00:17 lr 0.001011 wd 0.0500 time 0.3452 (0.3967) data time 0.0006 (0.0027) model time 0.3446 (0.3941) loss 3.6917 (3.1766) grad_norm 1.4035 (1.6931) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][590/625] eta 0:00:13 lr 0.001011 wd 0.0500 time 0.3442 (0.3955) data time 0.0007 (0.0026) model time 0.3435 (0.3929) loss 3.6648 (3.1819) grad_norm 1.8458 (1.6896) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][600/625] eta 0:00:09 lr 0.001010 wd 0.0500 time 0.3478 (0.3944) data time 0.0009 (0.0026) model time 0.3469 (0.3918) loss 2.7297 (3.1844) grad_norm 1.3929 (1.6890) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][610/625] eta 0:00:05 lr 0.001010 wd 0.0500 time 0.3415 (0.3937) data time 0.0004 (0.0025) model time 0.3410 (0.3911) loss 2.5466 (3.1821) grad_norm 1.1920 (1.6815) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [159/300][620/625] eta 0:00:01 lr 0.001010 wd 0.0500 time 0.3410 (0.3925) data time 0.0004 (0.0025) model time 0.3405 (0.3900) loss 3.5207 (3.1896) grad_norm 1.8976 (1.6775) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 06:02:51 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 159 training takes 0:02:49 [2024-09-01 06:02:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:02:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:02:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.431 (0.431) Loss 0.5405 (0.5405) Acc@1 87.500 (87.500) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 06:02:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.127) Loss 0.9961 (0.7148) Acc@1 75.977 (83.851) Acc@5 94.580 (97.061) Mem 11219MB [2024-09-01 06:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 1.0664 (0.8586) Acc@1 74.023 (80.436) Acc@5 93.506 (95.482) Mem 11219MB [2024-09-01 06:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.214 Acc@5 95.455 [2024-09-01 06:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.2% [2024-09-01 06:03:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.858 (0.858) Loss 0.5049 (0.5049) Acc@1 88.965 (88.965) Acc@5 98.633 (98.633) Mem 11219MB [2024-09-01 06:03:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.171) Loss 0.8389 (0.6349) Acc@1 79.980 (85.618) Acc@5 95.850 (97.528) Mem 11219MB [2024-09-01 06:03:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.133) Loss 0.9609 (0.7587) Acc@1 76.318 (82.494) Acc@5 95.117 (96.310) Mem 11219MB [2024-09-01 06:03:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.244 Acc@5 96.313 [2024-09-01 06:03:03 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.2% [2024-09-01 06:03:03 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.24% [2024-09-01 06:03:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:03:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:03:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][0/625] eta 0:09:01 lr 0.001010 wd 0.0500 time 0.8657 (0.8657) data time 0.3923 (0.3923) model time 0.0000 (0.0000) loss 3.4377 (3.4377) grad_norm 1.9374 (1.9374) loss_scale 1024.0000 (1024.0000) mem 11228MB [2024-09-01 06:03:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][10/625] eta 0:04:03 lr 0.001010 wd 0.0500 time 0.3399 (0.3961) data time 0.0008 (0.0365) model time 0.0000 (0.0000) loss 3.4831 (3.1880) grad_norm 1.1449 (1.5891) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][20/625] eta 0:03:45 lr 0.001010 wd 0.0500 time 0.3445 (0.3729) data time 0.0006 (0.0196) model time 0.0000 (0.0000) loss 4.0270 (3.1101) grad_norm 1.8711 (1.5924) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][30/625] eta 0:03:37 lr 0.001009 wd 0.0500 time 0.3443 (0.3657) data time 0.0006 (0.0136) model time 0.0000 (0.0000) loss 3.7117 (3.0792) grad_norm 1.1106 (1.6105) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][40/625] eta 0:03:31 lr 0.001009 wd 0.0500 time 0.3443 (0.3615) data time 0.0009 (0.0105) model time 0.0000 (0.0000) loss 3.3806 (3.0071) grad_norm 1.4935 (1.6001) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][50/625] eta 0:03:26 lr 0.001009 wd 0.0500 time 0.3496 (0.3590) data time 0.0011 (0.0086) model time 0.0000 (0.0000) loss 3.0914 (3.0500) grad_norm 1.4219 (1.6000) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][60/625] eta 0:03:22 lr 0.001009 wd 0.0500 time 0.3513 (0.3578) data time 0.0008 (0.0073) model time 0.3505 (0.3504) loss 3.3708 (3.0840) grad_norm 2.2829 (1.5878) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][70/625] eta 0:03:20 lr 0.001009 wd 0.0500 time 0.3437 (0.3617) data time 0.0009 (0.0064) model time 0.3429 (0.3677) loss 3.3511 (3.0927) grad_norm 1.2690 (1.5815) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][80/625] eta 0:03:16 lr 0.001009 wd 0.0500 time 0.3490 (0.3603) data time 0.0008 (0.0057) model time 0.3482 (0.3615) loss 3.3831 (3.1537) grad_norm 1.0997 (1.5446) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][90/625] eta 0:03:12 lr 0.001008 wd 0.0500 time 0.3460 (0.3589) data time 0.0006 (0.0052) model time 0.3454 (0.3580) loss 2.7029 (3.1306) grad_norm 1.5017 (1.5829) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][100/625] eta 0:03:07 lr 0.001008 wd 0.0500 time 0.3454 (0.3580) data time 0.0006 (0.0047) model time 0.3447 (0.3561) loss 2.4474 (3.1254) grad_norm 1.2903 (1.5852) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][110/625] eta 0:03:04 lr 0.001008 wd 0.0500 time 0.3487 (0.3575) data time 0.0006 (0.0044) model time 0.3481 (0.3553) loss 3.8457 (3.1285) grad_norm 3.1064 (1.6171) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][120/625] eta 0:03:00 lr 0.001008 wd 0.0500 time 0.5123 (0.3581) data time 0.0006 (0.0041) model time 0.5117 (0.3566) loss 3.2168 (3.1452) grad_norm 2.3362 (1.6761) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][130/625] eta 0:02:56 lr 0.001008 wd 0.0500 time 0.3524 (0.3575) data time 0.0008 (0.0038) model time 0.3516 (0.3557) loss 3.4607 (3.1548) grad_norm 1.6694 (1.6929) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][140/625] eta 0:02:53 lr 0.001008 wd 0.0500 time 0.3488 (0.3571) data time 0.0007 (0.0036) model time 0.3482 (0.3551) loss 3.3065 (3.1589) grad_norm 1.8787 (1.6816) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:03:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][150/625] eta 0:02:49 lr 0.001007 wd 0.0500 time 0.3557 (0.3565) data time 0.0006 (0.0034) model time 0.3552 (0.3544) loss 2.4677 (3.1700) grad_norm 1.1253 (1.6605) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:04:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][160/625] eta 0:02:45 lr 0.001007 wd 0.0500 time 0.3526 (0.3561) data time 0.0006 (0.0033) model time 0.3520 (0.3539) loss 3.1398 (3.1752) grad_norm 2.6797 (1.6747) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][170/625] eta 0:02:41 lr 0.001007 wd 0.0500 time 0.3464 (0.3558) data time 0.0010 (0.0031) model time 0.3454 (0.3535) loss 3.0883 (3.1617) grad_norm 1.8000 (1.6763) loss_scale 2048.0000 (1053.9415) mem 11224MB [2024-09-01 06:04:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][180/625] eta 0:02:38 lr 0.001007 wd 0.0500 time 0.3479 (0.3555) data time 0.0006 (0.0030) model time 0.3473 (0.3533) loss 3.2590 (3.1673) grad_norm 1.1450 (1.6679) loss_scale 2048.0000 (1108.8619) mem 11224MB [2024-09-01 06:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][190/625] eta 0:02:34 lr 0.001007 wd 0.0500 time 0.3492 (0.3552) data time 0.0007 (0.0029) model time 0.3486 (0.3529) loss 3.4638 (3.1837) grad_norm 1.5157 (1.6562) loss_scale 2048.0000 (1158.0314) mem 11224MB [2024-09-01 06:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][200/625] eta 0:02:30 lr 0.001006 wd 0.0500 time 0.3469 (0.3548) data time 0.0009 (0.0028) model time 0.3460 (0.3526) loss 2.9716 (3.1836) grad_norm 1.2908 (1.6788) loss_scale 2048.0000 (1202.3085) mem 11224MB [2024-09-01 06:04:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][210/625] eta 0:02:27 lr 0.001006 wd 0.0500 time 0.3523 (0.3546) data time 0.0006 (0.0027) model time 0.3517 (0.3524) loss 2.7448 (3.1737) grad_norm 1.2855 (1.6802) loss_scale 2048.0000 (1242.3886) mem 11224MB [2024-09-01 06:04:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][220/625] eta 0:02:23 lr 0.001006 wd 0.0500 time 0.3441 (0.3543) data time 0.0006 (0.0026) model time 0.3435 (0.3521) loss 3.4830 (3.1739) grad_norm 1.0292 (1.6797) loss_scale 2048.0000 (1278.8416) mem 11224MB [2024-09-01 06:04:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][230/625] eta 0:02:19 lr 0.001006 wd 0.0500 time 0.3469 (0.3541) data time 0.0009 (0.0026) model time 0.3460 (0.3518) loss 3.6271 (3.1609) grad_norm 1.8668 (1.6744) loss_scale 2048.0000 (1312.1385) mem 11224MB [2024-09-01 06:04:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][240/625] eta 0:02:16 lr 0.001006 wd 0.0500 time 0.3483 (0.3538) data time 0.0007 (0.0025) model time 0.3476 (0.3515) loss 3.4004 (3.1759) grad_norm 1.4001 (1.6701) loss_scale 2048.0000 (1342.6722) mem 11224MB [2024-09-01 06:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][250/625] eta 0:02:12 lr 0.001006 wd 0.0500 time 0.3489 (0.3536) data time 0.0006 (0.0024) model time 0.3483 (0.3514) loss 3.3894 (3.1775) grad_norm 1.7729 (1.6684) loss_scale 2048.0000 (1370.7729) mem 11224MB [2024-09-01 06:04:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][260/625] eta 0:02:08 lr 0.001005 wd 0.0500 time 0.3491 (0.3534) data time 0.0009 (0.0024) model time 0.3482 (0.3512) loss 3.4664 (3.1731) grad_norm 1.2877 (1.6649) loss_scale 2048.0000 (1396.7203) mem 11224MB [2024-09-01 06:04:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][270/625] eta 0:02:05 lr 0.001005 wd 0.0500 time 0.3460 (0.3532) data time 0.0008 (0.0023) model time 0.3452 (0.3510) loss 3.7236 (3.1671) grad_norm 1.1974 (1.6620) loss_scale 2048.0000 (1420.7528) mem 11224MB [2024-09-01 06:04:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][280/625] eta 0:02:02 lr 0.001005 wd 0.0500 time 0.3435 (0.3545) data time 0.0007 (0.0022) model time 0.3429 (0.3526) loss 2.5927 (3.1597) grad_norm 1.2683 (1.6548) loss_scale 2048.0000 (1443.0747) mem 11224MB [2024-09-01 06:04:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][290/625] eta 0:01:58 lr 0.001005 wd 0.0500 time 0.3503 (0.3543) data time 0.0007 (0.0022) model time 0.3496 (0.3525) loss 3.2305 (3.1594) grad_norm 1.0585 (1.6473) loss_scale 2048.0000 (1463.8625) mem 11224MB [2024-09-01 06:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][300/625] eta 0:01:55 lr 0.001005 wd 0.0500 time 0.3491 (0.3541) data time 0.0007 (0.0022) model time 0.3484 (0.3523) loss 3.2152 (3.1687) grad_norm 1.1448 (1.6390) loss_scale 2048.0000 (1483.2691) mem 11224MB [2024-09-01 06:04:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][310/625] eta 0:01:51 lr 0.001004 wd 0.0500 time 0.3469 (0.3545) data time 0.0006 (0.0021) model time 0.3463 (0.3527) loss 2.2721 (3.1766) grad_norm 1.1533 (1.6287) loss_scale 2048.0000 (1501.4277) mem 11224MB [2024-09-01 06:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][320/625] eta 0:01:48 lr 0.001004 wd 0.0500 time 0.3465 (0.3542) data time 0.0009 (0.0021) model time 0.3456 (0.3525) loss 2.1207 (3.1740) grad_norm 1.4191 (1.6207) loss_scale 2048.0000 (1518.4548) mem 11224MB [2024-09-01 06:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][330/625] eta 0:01:44 lr 0.001004 wd 0.0500 time 0.3475 (0.3540) data time 0.0008 (0.0020) model time 0.3467 (0.3523) loss 2.8201 (3.1778) grad_norm 1.5574 (1.6135) loss_scale 2048.0000 (1534.4532) mem 11224MB [2024-09-01 06:05:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][340/625] eta 0:01:40 lr 0.001004 wd 0.0500 time 0.3433 (0.3538) data time 0.0008 (0.0020) model time 0.3425 (0.3520) loss 3.0089 (3.1813) grad_norm 1.7661 (1.6139) loss_scale 2048.0000 (1549.5132) mem 11224MB [2024-09-01 06:05:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][350/625] eta 0:01:37 lr 0.001004 wd 0.0500 time 0.3548 (0.3537) data time 0.0006 (0.0020) model time 0.3542 (0.3519) loss 2.7808 (3.1815) grad_norm 2.1441 (1.6143) loss_scale 2048.0000 (1563.7151) mem 11224MB [2024-09-01 06:05:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][360/625] eta 0:01:33 lr 0.001004 wd 0.0500 time 0.3454 (0.3537) data time 0.0009 (0.0019) model time 0.3445 (0.3519) loss 3.6385 (3.1830) grad_norm 1.5652 (1.6102) loss_scale 2048.0000 (1577.1302) mem 11224MB [2024-09-01 06:05:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][370/625] eta 0:01:30 lr 0.001003 wd 0.0500 time 0.3536 (0.3536) data time 0.0009 (0.0019) model time 0.3527 (0.3519) loss 2.5608 (3.1760) grad_norm 1.3574 (1.6063) loss_scale 2048.0000 (1589.8221) mem 11224MB [2024-09-01 06:05:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][380/625] eta 0:01:26 lr 0.001003 wd 0.0500 time 0.3553 (0.3534) data time 0.0009 (0.0019) model time 0.3544 (0.3517) loss 3.4563 (3.1653) grad_norm 2.0852 (1.6089) loss_scale 2048.0000 (1601.8478) mem 11224MB [2024-09-01 06:05:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][390/625] eta 0:01:23 lr 0.001003 wd 0.0500 time 0.3563 (0.3534) data time 0.0009 (0.0018) model time 0.3554 (0.3517) loss 3.4202 (3.1665) grad_norm 1.5636 (1.6151) loss_scale 2048.0000 (1613.2583) mem 11224MB [2024-09-01 06:05:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][400/625] eta 0:01:19 lr 0.001003 wd 0.0500 time 0.3409 (0.3533) data time 0.0007 (0.0018) model time 0.3402 (0.3516) loss 2.6468 (3.1602) grad_norm 1.9121 (1.6125) loss_scale 2048.0000 (1624.0998) mem 11224MB [2024-09-01 06:05:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][410/625] eta 0:01:15 lr 0.001003 wd 0.0500 time 0.3590 (0.3532) data time 0.0007 (0.0018) model time 0.3583 (0.3516) loss 2.1465 (3.1553) grad_norm 1.3722 (1.6093) loss_scale 2048.0000 (1634.4136) mem 11224MB [2024-09-01 06:05:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][420/625] eta 0:01:12 lr 0.001003 wd 0.0500 time 0.3451 (0.3532) data time 0.0008 (0.0018) model time 0.3444 (0.3515) loss 3.4336 (3.1542) grad_norm 2.1132 (1.6049) loss_scale 2048.0000 (1644.2375) mem 11224MB [2024-09-01 06:05:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][430/625] eta 0:01:08 lr 0.001002 wd 0.0500 time 0.3471 (0.3534) data time 0.0008 (0.0018) model time 0.3462 (0.3518) loss 3.6698 (3.1541) grad_norm 1.2730 (1.6038) loss_scale 2048.0000 (1653.6056) mem 11224MB [2024-09-01 06:05:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][440/625] eta 0:01:05 lr 0.001002 wd 0.0500 time 0.3459 (0.3533) data time 0.0008 (0.0017) model time 0.3451 (0.3517) loss 3.5763 (3.1562) grad_norm 1.8423 (1.6075) loss_scale 2048.0000 (1662.5488) mem 11224MB [2024-09-01 06:05:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][450/625] eta 0:01:01 lr 0.001002 wd 0.0500 time 0.3539 (0.3533) data time 0.0007 (0.0017) model time 0.3532 (0.3517) loss 4.0420 (3.1520) grad_norm 1.9786 (1.6144) loss_scale 2048.0000 (1671.0953) mem 11224MB [2024-09-01 06:05:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][460/625] eta 0:00:58 lr 0.001002 wd 0.0500 time 0.4920 (0.3535) data time 0.0010 (0.0017) model time 0.4910 (0.3519) loss 3.6150 (3.1509) grad_norm 1.6531 (1.6163) loss_scale 2048.0000 (1679.2711) mem 11224MB [2024-09-01 06:05:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][470/625] eta 0:00:54 lr 0.001002 wd 0.0500 time 0.3499 (0.3542) data time 0.0007 (0.0017) model time 0.3492 (0.3528) loss 2.9582 (3.1518) grad_norm 1.2098 (1.6121) loss_scale 2048.0000 (1687.0998) mem 11224MB [2024-09-01 06:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][480/625] eta 0:00:51 lr 0.001001 wd 0.0500 time 0.3522 (0.3541) data time 0.0007 (0.0017) model time 0.3515 (0.3527) loss 2.7080 (3.1485) grad_norm 1.7460 (1.6088) loss_scale 2048.0000 (1694.6029) mem 11224MB [2024-09-01 06:05:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][490/625] eta 0:00:47 lr 0.001001 wd 0.0500 time 0.3407 (0.3540) data time 0.0006 (0.0016) model time 0.3401 (0.3526) loss 3.5892 (3.1539) grad_norm 1.5844 (1.6061) loss_scale 2048.0000 (1701.8004) mem 11224MB [2024-09-01 06:06:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][500/625] eta 0:00:44 lr 0.001001 wd 0.0500 time 0.3446 (0.3539) data time 0.0010 (0.0016) model time 0.3436 (0.3524) loss 2.8110 (3.1567) grad_norm 1.2681 (1.6038) loss_scale 2048.0000 (1708.7106) mem 11224MB [2024-09-01 06:06:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][510/625] eta 0:00:40 lr 0.001001 wd 0.0500 time 0.3450 (0.3538) data time 0.0008 (0.0016) model time 0.3442 (0.3524) loss 2.6138 (3.1507) grad_norm 1.3634 (1.6016) loss_scale 2048.0000 (1715.3503) mem 11224MB [2024-09-01 06:06:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][520/625] eta 0:00:37 lr 0.001001 wd 0.0500 time 0.3424 (0.3537) data time 0.0008 (0.0016) model time 0.3416 (0.3522) loss 2.5969 (3.1462) grad_norm 1.1977 (1.6006) loss_scale 2048.0000 (1721.7351) mem 11224MB [2024-09-01 06:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][530/625] eta 0:00:33 lr 0.001001 wd 0.0500 time 0.3444 (0.3536) data time 0.0006 (0.0016) model time 0.3438 (0.3521) loss 2.9112 (3.1460) grad_norm 1.5365 (1.6002) loss_scale 2048.0000 (1727.8795) mem 11224MB [2024-09-01 06:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][540/625] eta 0:00:30 lr 0.001000 wd 0.0500 time 0.3466 (0.3535) data time 0.0009 (0.0016) model time 0.3456 (0.3520) loss 3.3457 (3.1486) grad_norm 1.1039 (1.5969) loss_scale 2048.0000 (1733.7967) mem 11224MB [2024-09-01 06:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][550/625] eta 0:00:26 lr 0.001000 wd 0.0500 time 0.3499 (0.3534) data time 0.0009 (0.0016) model time 0.3491 (0.3519) loss 3.7865 (3.1464) grad_norm 1.1927 (1.5939) loss_scale 2048.0000 (1739.4991) mem 11224MB [2024-09-01 06:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][560/625] eta 0:00:22 lr 0.001000 wd 0.0500 time 0.3457 (0.3533) data time 0.0008 (0.0016) model time 0.3448 (0.3518) loss 2.6088 (3.1445) grad_norm 1.3151 (1.5908) loss_scale 2048.0000 (1744.9982) mem 11224MB [2024-09-01 06:06:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][570/625] eta 0:00:19 lr 0.001000 wd 0.0500 time 0.3503 (0.3532) data time 0.0007 (0.0015) model time 0.3495 (0.3517) loss 3.5143 (3.1462) grad_norm 2.3737 (1.6034) loss_scale 2048.0000 (1750.3047) mem 11224MB [2024-09-01 06:06:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][580/625] eta 0:00:15 lr 0.001000 wd 0.0500 time 0.5589 (0.3535) data time 0.0008 (0.0015) model time 0.5581 (0.3521) loss 3.5177 (3.1527) grad_norm 1.7582 (1.6094) loss_scale 2048.0000 (1755.4286) mem 11224MB [2024-09-01 06:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][590/625] eta 0:00:12 lr 0.001000 wd 0.0500 time 0.3509 (0.3538) data time 0.0008 (0.0015) model time 0.3500 (0.3525) loss 2.2016 (3.1497) grad_norm 1.7948 (1.6121) loss_scale 2048.0000 (1760.3790) mem 11224MB [2024-09-01 06:06:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][600/625] eta 0:00:08 lr 0.000999 wd 0.0500 time 0.3420 (0.3537) data time 0.0009 (0.0015) model time 0.3411 (0.3524) loss 3.3549 (3.1453) grad_norm 1.3578 (1.6211) loss_scale 2048.0000 (1765.1647) mem 11224MB [2024-09-01 06:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][610/625] eta 0:00:05 lr 0.000999 wd 0.0500 time 0.3387 (0.3536) data time 0.0006 (0.0015) model time 0.3381 (0.3523) loss 2.3633 (3.1501) grad_norm 1.4390 (1.6169) loss_scale 2048.0000 (1769.7938) mem 11224MB [2024-09-01 06:06:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [160/300][620/625] eta 0:00:01 lr 0.000999 wd 0.0500 time 0.3489 (0.3535) data time 0.0007 (0.0015) model time 0.3482 (0.3521) loss 3.3198 (3.1499) grad_norm 1.6367 (1.6152) loss_scale 2048.0000 (1774.2738) mem 11224MB [2024-09-01 06:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 160 training takes 0:03:40 [2024-09-01 06:06:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:06:46 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 2.525 (2.525) Loss 0.5850 (0.5850) Acc@1 87.012 (87.012) Acc@5 98.340 (98.340) Mem 11224MB [2024-09-01 06:06:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.321) Loss 0.9839 (0.7182) Acc@1 76.318 (83.691) Acc@5 95.020 (97.026) Mem 11224MB [2024-09-01 06:06:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.212) Loss 1.0244 (0.8511) Acc@1 75.244 (80.487) Acc@5 93.750 (95.564) Mem 11224MB [2024-09-01 06:06:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.194 Acc@5 95.515 [2024-09-01 06:06:50 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.2% [2024-09-01 06:06:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.870 (0.870) Loss 0.5029 (0.5029) Acc@1 88.916 (88.916) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 06:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.171) Loss 0.8384 (0.6344) Acc@1 80.176 (85.636) Acc@5 95.996 (97.528) Mem 11224MB [2024-09-01 06:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.133) Loss 0.9590 (0.7579) Acc@1 76.465 (82.522) Acc@5 94.873 (96.291) Mem 11224MB [2024-09-01 06:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.278 Acc@5 96.295 [2024-09-01 06:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.3% [2024-09-01 06:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.28% [2024-09-01 06:06:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:06:54 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][0/625] eta 0:07:08 lr 0.000999 wd 0.0500 time 0.6864 (0.6864) data time 0.3608 (0.3608) model time 0.0000 (0.0000) loss 3.2189 (3.2189) grad_norm 1.5742 (1.5742) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][10/625] eta 0:03:54 lr 0.000999 wd 0.0500 time 0.3477 (0.3809) data time 0.0008 (0.0335) model time 0.0000 (0.0000) loss 1.8683 (2.9613) grad_norm 1.3271 (1.5845) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][20/625] eta 0:03:41 lr 0.000999 wd 0.0500 time 0.3435 (0.3666) data time 0.0007 (0.0180) model time 0.0000 (0.0000) loss 3.8889 (3.0875) grad_norm 1.5800 (1.5222) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][30/625] eta 0:03:35 lr 0.000998 wd 0.0500 time 0.3514 (0.3619) data time 0.0006 (0.0124) model time 0.0000 (0.0000) loss 3.4823 (3.0724) grad_norm 1.5998 (1.5343) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][40/625] eta 0:03:30 lr 0.000998 wd 0.0500 time 0.3475 (0.3592) data time 0.0006 (0.0096) model time 0.0000 (0.0000) loss 2.3201 (3.0655) grad_norm 1.3194 (1.5545) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][50/625] eta 0:03:25 lr 0.000998 wd 0.0500 time 0.3482 (0.3574) data time 0.0006 (0.0079) model time 0.0000 (0.0000) loss 2.1653 (3.0809) grad_norm 1.5353 (1.5405) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][60/625] eta 0:03:21 lr 0.000998 wd 0.0500 time 0.3458 (0.3558) data time 0.0008 (0.0068) model time 0.3450 (0.3466) loss 3.3354 (3.1265) grad_norm 1.5375 (1.5316) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][70/625] eta 0:03:17 lr 0.000998 wd 0.0500 time 0.3556 (0.3554) data time 0.0008 (0.0059) model time 0.3548 (0.3493) loss 3.1671 (3.1300) grad_norm 1.4971 (1.5287) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][80/625] eta 0:03:13 lr 0.000997 wd 0.0500 time 0.3398 (0.3549) data time 0.0006 (0.0054) model time 0.3392 (0.3493) loss 3.1679 (3.1157) grad_norm 2.1805 (1.5433) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][90/625] eta 0:03:09 lr 0.000997 wd 0.0500 time 0.3673 (0.3546) data time 0.0009 (0.0049) model time 0.3664 (0.3498) loss 2.4748 (3.1130) grad_norm 1.4470 (1.5584) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][100/625] eta 0:03:07 lr 0.000997 wd 0.0500 time 0.3471 (0.3580) data time 0.0008 (0.0045) model time 0.3463 (0.3575) loss 3.3813 (3.1120) grad_norm 1.3554 (1.5446) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][110/625] eta 0:03:03 lr 0.000997 wd 0.0500 time 0.3470 (0.3571) data time 0.0009 (0.0042) model time 0.3462 (0.3558) loss 3.0740 (3.1027) grad_norm 1.7558 (1.5481) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][120/625] eta 0:03:00 lr 0.000997 wd 0.0500 time 0.3474 (0.3565) data time 0.0008 (0.0039) model time 0.3465 (0.3547) loss 3.4476 (3.1330) grad_norm 2.8495 (1.6005) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][130/625] eta 0:02:56 lr 0.000997 wd 0.0500 time 0.3615 (0.3560) data time 0.0009 (0.0037) model time 0.3607 (0.3541) loss 3.4442 (3.1568) grad_norm 1.7971 (1.6114) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][140/625] eta 0:02:52 lr 0.000996 wd 0.0500 time 0.3483 (0.3556) data time 0.0008 (0.0035) model time 0.3475 (0.3535) loss 3.3507 (3.1597) grad_norm 1.3205 (1.6076) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][150/625] eta 0:02:49 lr 0.000996 wd 0.0500 time 0.3534 (0.3565) data time 0.0005 (0.0033) model time 0.3528 (0.3551) loss 3.1011 (3.1645) grad_norm 1.8238 (1.6036) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][160/625] eta 0:02:45 lr 0.000996 wd 0.0500 time 0.3592 (0.3561) data time 0.0008 (0.0031) model time 0.3584 (0.3545) loss 1.8541 (3.1600) grad_norm 1.5826 (1.5942) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][170/625] eta 0:02:41 lr 0.000996 wd 0.0500 time 0.3456 (0.3557) data time 0.0006 (0.0030) model time 0.3450 (0.3540) loss 3.2619 (3.1465) grad_norm 1.2921 (1.5914) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:07:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][180/625] eta 0:02:38 lr 0.000996 wd 0.0500 time 0.3516 (0.3555) data time 0.0008 (0.0029) model time 0.3508 (0.3538) loss 3.2600 (3.1471) grad_norm 2.7464 (1.6032) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][190/625] eta 0:02:34 lr 0.000996 wd 0.0500 time 0.3488 (0.3553) data time 0.0008 (0.0028) model time 0.3480 (0.3536) loss 2.0896 (3.1455) grad_norm 1.1562 (1.6060) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][200/625] eta 0:02:30 lr 0.000995 wd 0.0500 time 0.3513 (0.3550) data time 0.0009 (0.0027) model time 0.3505 (0.3533) loss 3.6187 (3.1480) grad_norm 2.2210 (1.6061) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][210/625] eta 0:02:27 lr 0.000995 wd 0.0500 time 0.3515 (0.3549) data time 0.0009 (0.0026) model time 0.3507 (0.3532) loss 3.3461 (3.1575) grad_norm 1.4546 (1.6101) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][220/625] eta 0:02:23 lr 0.000995 wd 0.0500 time 0.3456 (0.3548) data time 0.0007 (0.0025) model time 0.3449 (0.3531) loss 2.9322 (3.1643) grad_norm 1.6246 (1.6088) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][230/625] eta 0:02:20 lr 0.000995 wd 0.0500 time 0.3438 (0.3547) data time 0.0009 (0.0025) model time 0.3429 (0.3530) loss 3.1806 (3.1751) grad_norm 1.5298 (1.6126) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][240/625] eta 0:02:16 lr 0.000995 wd 0.0500 time 0.3534 (0.3546) data time 0.0009 (0.0024) model time 0.3525 (0.3530) loss 3.8425 (3.1806) grad_norm 1.4324 (1.6144) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][250/625] eta 0:02:12 lr 0.000994 wd 0.0500 time 0.3592 (0.3546) data time 0.0006 (0.0023) model time 0.3586 (0.3529) loss 2.9125 (3.1735) grad_norm 1.9153 (1.6102) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][260/625] eta 0:02:09 lr 0.000994 wd 0.0500 time 0.3433 (0.3544) data time 0.0008 (0.0023) model time 0.3424 (0.3527) loss 3.0403 (3.1594) grad_norm 1.2082 (1.6046) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][270/625] eta 0:02:05 lr 0.000994 wd 0.0500 time 0.3513 (0.3549) data time 0.0008 (0.0022) model time 0.3505 (0.3534) loss 2.9364 (3.1602) grad_norm 1.4895 (1.6227) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][280/625] eta 0:02:02 lr 0.000994 wd 0.0500 time 0.3515 (0.3548) data time 0.0007 (0.0022) model time 0.3509 (0.3532) loss 3.7134 (3.1649) grad_norm 8.1442 (1.6482) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][290/625] eta 0:01:58 lr 0.000994 wd 0.0500 time 0.3490 (0.3547) data time 0.0007 (0.0021) model time 0.3483 (0.3532) loss 3.5662 (3.1680) grad_norm 1.4613 (1.6595) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][300/625] eta 0:01:55 lr 0.000994 wd 0.0500 time 0.3542 (0.3546) data time 0.0009 (0.0021) model time 0.3533 (0.3530) loss 2.5165 (3.1736) grad_norm 1.3131 (1.6599) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][310/625] eta 0:01:52 lr 0.000993 wd 0.0500 time 0.3540 (0.3558) data time 0.0008 (0.0021) model time 0.3532 (0.3545) loss 2.6359 (3.1786) grad_norm 1.4316 (1.6511) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][320/625] eta 0:01:48 lr 0.000993 wd 0.0500 time 0.3449 (0.3556) data time 0.0008 (0.0021) model time 0.3440 (0.3543) loss 3.2264 (3.1801) grad_norm 2.0808 (1.6516) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][330/625] eta 0:01:44 lr 0.000993 wd 0.0500 time 0.3504 (0.3555) data time 0.0006 (0.0020) model time 0.3498 (0.3541) loss 3.8953 (3.1818) grad_norm 2.3837 (1.6518) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][340/625] eta 0:01:41 lr 0.000993 wd 0.0500 time 0.3535 (0.3553) data time 0.0006 (0.0020) model time 0.3528 (0.3540) loss 3.3399 (3.1833) grad_norm 1.4817 (1.6546) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][350/625] eta 0:01:37 lr 0.000993 wd 0.0500 time 0.3520 (0.3552) data time 0.0006 (0.0020) model time 0.3514 (0.3538) loss 3.1338 (3.1882) grad_norm 1.5687 (1.6521) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][360/625] eta 0:01:34 lr 0.000992 wd 0.0500 time 0.3529 (0.3551) data time 0.0009 (0.0019) model time 0.3520 (0.3537) loss 3.2733 (3.1866) grad_norm 1.0454 (1.6471) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][370/625] eta 0:01:30 lr 0.000992 wd 0.0500 time 0.3521 (0.3550) data time 0.0006 (0.0019) model time 0.3515 (0.3536) loss 3.2111 (3.1821) grad_norm 1.4198 (1.6387) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][380/625] eta 0:01:26 lr 0.000992 wd 0.0500 time 0.3512 (0.3549) data time 0.0008 (0.0019) model time 0.3504 (0.3536) loss 2.4610 (3.1822) grad_norm 1.2598 (1.6336) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][390/625] eta 0:01:23 lr 0.000992 wd 0.0500 time 0.3504 (0.3549) data time 0.0007 (0.0018) model time 0.3497 (0.3536) loss 2.4051 (3.1814) grad_norm 1.5678 (1.6291) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][400/625] eta 0:01:19 lr 0.000992 wd 0.0500 time 0.3523 (0.3548) data time 0.0008 (0.0018) model time 0.3514 (0.3535) loss 2.4368 (3.1841) grad_norm 1.3132 (1.6301) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][410/625] eta 0:01:16 lr 0.000992 wd 0.0500 time 0.3514 (0.3548) data time 0.0010 (0.0018) model time 0.3504 (0.3535) loss 3.1166 (3.1858) grad_norm 2.1775 (1.6303) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][420/625] eta 0:01:12 lr 0.000991 wd 0.0500 time 0.3573 (0.3547) data time 0.0006 (0.0018) model time 0.3567 (0.3534) loss 2.8638 (3.1793) grad_norm 1.7067 (1.6302) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][430/625] eta 0:01:09 lr 0.000991 wd 0.0500 time 0.3510 (0.3547) data time 0.0007 (0.0018) model time 0.3503 (0.3534) loss 3.4551 (3.1725) grad_norm 1.6867 (1.6326) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][440/625] eta 0:01:05 lr 0.000991 wd 0.0500 time 0.3477 (0.3546) data time 0.0008 (0.0017) model time 0.3470 (0.3533) loss 3.3199 (3.1646) grad_norm 1.2892 (1.6316) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][450/625] eta 0:01:02 lr 0.000991 wd 0.0500 time 0.3550 (0.3546) data time 0.0008 (0.0017) model time 0.3542 (0.3533) loss 2.8972 (3.1685) grad_norm 1.6778 (1.6327) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][460/625] eta 0:00:58 lr 0.000991 wd 0.0500 time 0.3602 (0.3546) data time 0.0008 (0.0017) model time 0.3594 (0.3533) loss 3.3956 (3.1699) grad_norm 2.2081 (1.6485) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][470/625] eta 0:00:54 lr 0.000991 wd 0.0500 time 0.3596 (0.3545) data time 0.0008 (0.0017) model time 0.3589 (0.3532) loss 3.3554 (3.1705) grad_norm 3.3222 (1.6596) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][480/625] eta 0:00:51 lr 0.000990 wd 0.0500 time 0.3472 (0.3548) data time 0.0009 (0.0017) model time 0.3463 (0.3536) loss 3.0172 (3.1697) grad_norm 1.6075 (1.6580) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][490/625] eta 0:00:48 lr 0.000990 wd 0.0500 time 0.3407 (0.3559) data time 0.0009 (0.0017) model time 0.3398 (0.3548) loss 2.4105 (3.1648) grad_norm 1.3243 (1.6557) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][500/625] eta 0:00:44 lr 0.000990 wd 0.0500 time 0.3520 (0.3558) data time 0.0008 (0.0016) model time 0.3512 (0.3547) loss 3.3950 (3.1661) grad_norm 1.5410 (1.6524) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][510/625] eta 0:00:40 lr 0.000990 wd 0.0500 time 0.3517 (0.3557) data time 0.0009 (0.0016) model time 0.3509 (0.3545) loss 3.5517 (3.1580) grad_norm 1.7591 (1.6603) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][520/625] eta 0:00:37 lr 0.000990 wd 0.0500 time 0.3480 (0.3555) data time 0.0009 (0.0016) model time 0.3472 (0.3544) loss 3.4461 (3.1578) grad_norm 1.2979 (1.6594) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][530/625] eta 0:00:33 lr 0.000989 wd 0.0500 time 0.3563 (0.3555) data time 0.0007 (0.0016) model time 0.3557 (0.3543) loss 3.5481 (3.1599) grad_norm 2.6898 (1.6663) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][540/625] eta 0:00:30 lr 0.000989 wd 0.0500 time 0.3507 (0.3554) data time 0.0008 (0.0016) model time 0.3499 (0.3543) loss 3.3052 (3.1637) grad_norm 1.5095 (1.6695) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][550/625] eta 0:00:26 lr 0.000989 wd 0.0500 time 0.3501 (0.3554) data time 0.0008 (0.0016) model time 0.3493 (0.3543) loss 3.3599 (3.1691) grad_norm 1.1821 (1.6678) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][560/625] eta 0:00:23 lr 0.000989 wd 0.0500 time 0.3450 (0.3554) data time 0.0009 (0.0016) model time 0.3441 (0.3542) loss 2.4914 (3.1704) grad_norm 1.3159 (1.6675) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][570/625] eta 0:00:19 lr 0.000989 wd 0.0500 time 0.3550 (0.3553) data time 0.0008 (0.0015) model time 0.3542 (0.3542) loss 3.4219 (3.1691) grad_norm 1.3337 (1.6632) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][580/625] eta 0:00:15 lr 0.000989 wd 0.0500 time 0.3574 (0.3552) data time 0.0006 (0.0015) model time 0.3568 (0.3541) loss 2.5592 (3.1711) grad_norm 1.6578 (1.6590) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][590/625] eta 0:00:12 lr 0.000988 wd 0.0500 time 0.3509 (0.3552) data time 0.0006 (0.0015) model time 0.3502 (0.3541) loss 3.9329 (3.1632) grad_norm 1.2406 (1.6600) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][600/625] eta 0:00:08 lr 0.000988 wd 0.0500 time 0.3509 (0.3551) data time 0.0006 (0.0015) model time 0.3503 (0.3540) loss 3.1442 (3.1554) grad_norm 1.0637 (1.6570) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][610/625] eta 0:00:05 lr 0.000988 wd 0.0500 time 0.3455 (0.3557) data time 0.0004 (0.0015) model time 0.3451 (0.3546) loss 3.7159 (3.1548) grad_norm 2.2409 (1.6575) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [161/300][620/625] eta 0:00:01 lr 0.000988 wd 0.0500 time 0.3354 (0.3554) data time 0.0004 (0.0015) model time 0.3350 (0.3543) loss 3.1563 (3.1556) grad_norm 1.1640 (1.6546) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 161 training takes 0:03:42 [2024-09-01 06:10:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:10:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:10:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.447 (0.447) Loss 0.5791 (0.5791) Acc@1 87.549 (87.549) Acc@5 98.145 (98.145) Mem 11224MB [2024-09-01 06:10:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.9302 (0.7119) Acc@1 77.832 (84.126) Acc@5 94.434 (97.013) Mem 11224MB [2024-09-01 06:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.112) Loss 1.0967 (0.8474) Acc@1 73.291 (80.671) Acc@5 93.604 (95.522) Mem 11224MB [2024-09-01 06:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.456 Acc@5 95.469 [2024-09-01 06:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.5% [2024-09-01 06:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 80.46% [2024-09-01 06:10:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 06:10:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 06:10:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.437 (0.437) Loss 0.5024 (0.5024) Acc@1 89.014 (89.014) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 06:10:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.128) Loss 0.8359 (0.6337) Acc@1 80.225 (85.653) Acc@5 95.947 (97.554) Mem 11224MB [2024-09-01 06:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 0.9556 (0.7566) Acc@1 76.758 (82.573) Acc@5 95.020 (96.324) Mem 11224MB [2024-09-01 06:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.346 Acc@5 96.327 [2024-09-01 06:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.3% [2024-09-01 06:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.35% [2024-09-01 06:10:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:10:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:10:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][0/625] eta 0:07:13 lr 0.000988 wd 0.0500 time 0.6930 (0.6930) data time 0.3834 (0.3834) model time 0.0000 (0.0000) loss 3.3751 (3.3751) grad_norm 0.9758 (0.9758) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][10/625] eta 0:03:55 lr 0.000988 wd 0.0500 time 0.3472 (0.3833) data time 0.0007 (0.0356) model time 0.0000 (0.0000) loss 3.2003 (3.1803) grad_norm 1.7621 (1.3821) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][20/625] eta 0:03:43 lr 0.000987 wd 0.0500 time 0.3500 (0.3695) data time 0.0008 (0.0190) model time 0.0000 (0.0000) loss 2.9870 (3.3334) grad_norm 1.5902 (1.4055) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][30/625] eta 0:03:35 lr 0.000987 wd 0.0500 time 0.3506 (0.3627) data time 0.0006 (0.0132) model time 0.0000 (0.0000) loss 2.8996 (3.2530) grad_norm 1.9584 (1.4253) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][40/625] eta 0:03:30 lr 0.000987 wd 0.0500 time 0.3454 (0.3599) data time 0.0008 (0.0102) model time 0.0000 (0.0000) loss 2.6371 (3.1960) grad_norm 1.2327 (1.4687) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][50/625] eta 0:03:25 lr 0.000987 wd 0.0500 time 0.3408 (0.3583) data time 0.0007 (0.0083) model time 0.0000 (0.0000) loss 2.4974 (3.2081) grad_norm 1.2240 (1.4803) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][60/625] eta 0:03:21 lr 0.000987 wd 0.0500 time 0.3449 (0.3570) data time 0.0006 (0.0071) model time 0.3443 (0.3498) loss 2.3540 (3.1682) grad_norm 1.6427 (1.4769) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][70/625] eta 0:03:17 lr 0.000987 wd 0.0500 time 0.3505 (0.3563) data time 0.0009 (0.0062) model time 0.3496 (0.3503) loss 3.4091 (3.1315) grad_norm 1.3778 (1.4759) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][80/625] eta 0:03:13 lr 0.000986 wd 0.0500 time 0.3465 (0.3556) data time 0.0007 (0.0056) model time 0.3458 (0.3503) loss 4.0822 (3.1684) grad_norm 1.3516 (1.4738) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][90/625] eta 0:03:09 lr 0.000986 wd 0.0500 time 0.3487 (0.3551) data time 0.0006 (0.0050) model time 0.3481 (0.3501) loss 3.8034 (3.1678) grad_norm 1.4464 (1.4741) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][100/625] eta 0:03:06 lr 0.000986 wd 0.0500 time 0.3483 (0.3546) data time 0.0006 (0.0046) model time 0.3477 (0.3501) loss 2.1595 (3.1586) grad_norm 2.5990 (1.5189) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][110/625] eta 0:03:02 lr 0.000986 wd 0.0500 time 0.3444 (0.3540) data time 0.0006 (0.0043) model time 0.3438 (0.3496) loss 3.9576 (3.1740) grad_norm 2.2775 (1.5614) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][120/625] eta 0:02:58 lr 0.000986 wd 0.0500 time 0.3480 (0.3539) data time 0.0009 (0.0040) model time 0.3471 (0.3499) loss 2.6650 (3.1817) grad_norm 1.1185 (1.5616) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][130/625] eta 0:02:55 lr 0.000985 wd 0.0500 time 0.3556 (0.3536) data time 0.0006 (0.0038) model time 0.3550 (0.3497) loss 2.9845 (3.1770) grad_norm 1.9726 (1.5532) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][140/625] eta 0:02:51 lr 0.000985 wd 0.0500 time 0.3461 (0.3533) data time 0.0006 (0.0036) model time 0.3455 (0.3496) loss 3.4247 (3.1812) grad_norm 1.3921 (1.5987) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][150/625] eta 0:02:48 lr 0.000985 wd 0.0500 time 0.3480 (0.3543) data time 0.0008 (0.0034) model time 0.3471 (0.3514) loss 3.0580 (3.1931) grad_norm 1.1662 (1.6089) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][160/625] eta 0:02:44 lr 0.000985 wd 0.0500 time 0.3519 (0.3541) data time 0.0008 (0.0032) model time 0.3511 (0.3513) loss 3.4154 (3.1802) grad_norm 2.1703 (1.6007) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][170/625] eta 0:02:42 lr 0.000985 wd 0.0500 time 0.3510 (0.3569) data time 0.0007 (0.0031) model time 0.3504 (0.3555) loss 2.2426 (3.1642) grad_norm 2.1380 (1.6021) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][180/625] eta 0:02:38 lr 0.000985 wd 0.0500 time 0.3479 (0.3564) data time 0.0010 (0.0030) model time 0.3470 (0.3549) loss 2.5711 (3.1600) grad_norm 1.1882 (1.5963) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][190/625] eta 0:02:34 lr 0.000984 wd 0.0500 time 0.3463 (0.3560) data time 0.0007 (0.0028) model time 0.3456 (0.3544) loss 3.4995 (3.1587) grad_norm 1.9865 (1.6029) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:11:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][200/625] eta 0:02:31 lr 0.000984 wd 0.0500 time 0.3507 (0.3555) data time 0.0005 (0.0027) model time 0.3502 (0.3538) loss 3.7034 (3.1624) grad_norm 1.2729 (1.5954) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][210/625] eta 0:02:27 lr 0.000984 wd 0.0500 time 0.3459 (0.3553) data time 0.0008 (0.0026) model time 0.3451 (0.3535) loss 2.4297 (3.1560) grad_norm 2.2533 (1.5910) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][220/625] eta 0:02:23 lr 0.000984 wd 0.0500 time 0.3513 (0.3551) data time 0.0008 (0.0026) model time 0.3505 (0.3533) loss 3.8053 (3.1547) grad_norm 1.4914 (1.5846) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][230/625] eta 0:02:20 lr 0.000984 wd 0.0500 time 0.3550 (0.3550) data time 0.0008 (0.0025) model time 0.3542 (0.3532) loss 2.1206 (3.1457) grad_norm 1.5094 (1.5801) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][240/625] eta 0:02:16 lr 0.000984 wd 0.0500 time 0.3460 (0.3548) data time 0.0009 (0.0024) model time 0.3451 (0.3531) loss 2.4295 (3.1330) grad_norm 1.2688 (1.5827) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][250/625] eta 0:02:13 lr 0.000983 wd 0.0500 time 0.3494 (0.3547) data time 0.0009 (0.0024) model time 0.3485 (0.3530) loss 3.1805 (3.1349) grad_norm 2.0571 (1.6040) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][260/625] eta 0:02:09 lr 0.000983 wd 0.0500 time 0.3496 (0.3547) data time 0.0006 (0.0023) model time 0.3490 (0.3529) loss 3.0486 (3.1323) grad_norm 1.6857 (1.6058) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][270/625] eta 0:02:06 lr 0.000983 wd 0.0500 time 0.3480 (0.3551) data time 0.0009 (0.0022) model time 0.3472 (0.3535) loss 2.2552 (3.1208) grad_norm 1.4300 (1.6123) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][280/625] eta 0:02:02 lr 0.000983 wd 0.0500 time 0.3600 (0.3550) data time 0.0006 (0.0022) model time 0.3595 (0.3534) loss 2.6587 (3.1200) grad_norm 2.2986 (1.6136) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][290/625] eta 0:01:59 lr 0.000983 wd 0.0500 time 0.3396 (0.3561) data time 0.0007 (0.0021) model time 0.3389 (0.3548) loss 3.7219 (3.1174) grad_norm 1.2969 (1.6025) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][300/625] eta 0:01:55 lr 0.000982 wd 0.0500 time 0.3511 (0.3559) data time 0.0006 (0.0021) model time 0.3504 (0.3546) loss 1.9714 (3.1030) grad_norm 1.6585 (1.6053) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][310/625] eta 0:01:52 lr 0.000982 wd 0.0500 time 0.3464 (0.3557) data time 0.0009 (0.0021) model time 0.3455 (0.3543) loss 3.0156 (3.1026) grad_norm 1.5491 (1.6043) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][320/625] eta 0:01:48 lr 0.000982 wd 0.0500 time 0.3482 (0.3554) data time 0.0007 (0.0020) model time 0.3475 (0.3540) loss 3.3526 (3.1032) grad_norm 1.3891 (1.5991) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][330/625] eta 0:01:44 lr 0.000982 wd 0.0500 time 0.3506 (0.3552) data time 0.0006 (0.0020) model time 0.3499 (0.3538) loss 2.9700 (3.1105) grad_norm 1.6321 (1.5950) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][340/625] eta 0:01:41 lr 0.000982 wd 0.0500 time 0.3458 (0.3551) data time 0.0006 (0.0020) model time 0.3452 (0.3536) loss 2.5181 (3.1080) grad_norm 5.8843 (1.6050) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][350/625] eta 0:01:37 lr 0.000982 wd 0.0500 time 0.3538 (0.3550) data time 0.0008 (0.0020) model time 0.3530 (0.3535) loss 3.1739 (3.1175) grad_norm 1.2774 (1.6130) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][360/625] eta 0:01:34 lr 0.000981 wd 0.0500 time 0.3520 (0.3548) data time 0.0006 (0.0019) model time 0.3513 (0.3534) loss 2.5010 (3.1219) grad_norm 1.5491 (1.6223) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][370/625] eta 0:01:30 lr 0.000981 wd 0.0500 time 0.3542 (0.3547) data time 0.0006 (0.0019) model time 0.3536 (0.3533) loss 3.9990 (3.1272) grad_norm 1.3391 (1.6212) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][380/625] eta 0:01:26 lr 0.000981 wd 0.0500 time 0.3481 (0.3545) data time 0.0006 (0.0019) model time 0.3475 (0.3531) loss 3.8950 (3.1252) grad_norm 1.5400 (1.6146) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][390/625] eta 0:01:23 lr 0.000981 wd 0.0500 time 0.3456 (0.3544) data time 0.0008 (0.0018) model time 0.3448 (0.3529) loss 3.1464 (3.1280) grad_norm 1.6181 (1.6123) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][400/625] eta 0:01:19 lr 0.000981 wd 0.0500 time 0.3503 (0.3542) data time 0.0006 (0.0018) model time 0.3496 (0.3527) loss 3.6810 (3.1253) grad_norm 1.7540 (1.6137) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][410/625] eta 0:01:16 lr 0.000981 wd 0.0500 time 0.3496 (0.3541) data time 0.0007 (0.0018) model time 0.3489 (0.3526) loss 2.0610 (3.1289) grad_norm 0.8673 (1.6150) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][420/625] eta 0:01:12 lr 0.000980 wd 0.0500 time 0.3457 (0.3539) data time 0.0008 (0.0018) model time 0.3449 (0.3525) loss 2.7492 (3.1264) grad_norm 1.4354 (1.6130) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][430/625] eta 0:01:08 lr 0.000980 wd 0.0500 time 0.3512 (0.3538) data time 0.0006 (0.0017) model time 0.3506 (0.3524) loss 3.7264 (3.1292) grad_norm 1.0487 (1.6059) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][440/625] eta 0:01:05 lr 0.000980 wd 0.0500 time 0.3453 (0.3537) data time 0.0006 (0.0017) model time 0.3447 (0.3522) loss 3.0324 (3.1299) grad_norm 1.5373 (1.6061) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][450/625] eta 0:01:01 lr 0.000980 wd 0.0500 time 0.3471 (0.3536) data time 0.0006 (0.0017) model time 0.3465 (0.3521) loss 3.6876 (3.1316) grad_norm 1.6059 (1.6038) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][460/625] eta 0:00:58 lr 0.000980 wd 0.0500 time 0.3503 (0.3535) data time 0.0006 (0.0017) model time 0.3497 (0.3520) loss 3.3931 (3.1330) grad_norm 0.9115 (1.6041) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][470/625] eta 0:00:54 lr 0.000979 wd 0.0500 time 0.3496 (0.3534) data time 0.0006 (0.0017) model time 0.3490 (0.3519) loss 3.4971 (3.1305) grad_norm 1.2245 (1.5971) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][480/625] eta 0:00:51 lr 0.000979 wd 0.0500 time 0.3459 (0.3537) data time 0.0008 (0.0017) model time 0.3451 (0.3523) loss 3.4872 (3.1336) grad_norm 2.2385 (1.5993) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][490/625] eta 0:00:47 lr 0.000979 wd 0.0500 time 0.3491 (0.3536) data time 0.0008 (0.0016) model time 0.3483 (0.3521) loss 3.3062 (3.1361) grad_norm 1.8087 (1.5998) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][500/625] eta 0:00:44 lr 0.000979 wd 0.0500 time 0.3437 (0.3543) data time 0.0006 (0.0016) model time 0.3431 (0.3529) loss 2.3090 (3.1344) grad_norm 1.1997 (1.5971) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][510/625] eta 0:00:40 lr 0.000979 wd 0.0500 time 0.3395 (0.3541) data time 0.0008 (0.0016) model time 0.3387 (0.3528) loss 2.7056 (3.1392) grad_norm 1.7201 (1.5970) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][520/625] eta 0:00:37 lr 0.000979 wd 0.0500 time 0.3532 (0.3541) data time 0.0008 (0.0016) model time 0.3525 (0.3527) loss 3.5448 (3.1364) grad_norm 1.4159 (1.5956) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][530/625] eta 0:00:33 lr 0.000978 wd 0.0500 time 0.3459 (0.3540) data time 0.0008 (0.0016) model time 0.3451 (0.3527) loss 3.1643 (3.1356) grad_norm 2.1795 (1.5978) loss_scale 2048.0000 (2048.0000) mem 11224MB [2024-09-01 06:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][540/625] eta 0:00:30 lr 0.000978 wd 0.0500 time 0.3443 (0.3539) data time 0.0008 (0.0016) model time 0.3434 (0.3525) loss 2.8362 (3.1325) grad_norm 1.6882 (nan) loss_scale 1024.0000 (2042.3216) mem 11224MB [2024-09-01 06:14:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][550/625] eta 0:00:26 lr 0.000978 wd 0.0500 time 0.3487 (0.3537) data time 0.0005 (0.0015) model time 0.3482 (0.3524) loss 2.3455 (3.1282) grad_norm 1.5439 (nan) loss_scale 1024.0000 (2023.8403) mem 11224MB [2024-09-01 06:14:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][560/625] eta 0:00:22 lr 0.000978 wd 0.0500 time 0.3439 (0.3536) data time 0.0008 (0.0015) model time 0.3430 (0.3523) loss 2.8910 (3.1274) grad_norm 1.6126 (nan) loss_scale 1024.0000 (2006.0178) mem 11224MB [2024-09-01 06:14:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][570/625] eta 0:00:19 lr 0.000978 wd 0.0500 time 0.3478 (0.3535) data time 0.0009 (0.0015) model time 0.3469 (0.3522) loss 3.3773 (3.1278) grad_norm 1.4685 (nan) loss_scale 1024.0000 (1988.8196) mem 11224MB [2024-09-01 06:14:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][580/625] eta 0:00:15 lr 0.000977 wd 0.0500 time 0.3452 (0.3534) data time 0.0006 (0.0015) model time 0.3446 (0.3521) loss 4.3435 (3.1269) grad_norm 1.8067 (nan) loss_scale 1024.0000 (1972.2134) mem 11224MB [2024-09-01 06:14:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][590/625] eta 0:00:12 lr 0.000977 wd 0.0500 time 0.3452 (0.3533) data time 0.0006 (0.0015) model time 0.3446 (0.3520) loss 3.5042 (3.1294) grad_norm 1.8049 (nan) loss_scale 1024.0000 (1956.1692) mem 11224MB [2024-09-01 06:14:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][600/625] eta 0:00:08 lr 0.000977 wd 0.0500 time 0.3430 (0.3532) data time 0.0009 (0.0015) model time 0.3421 (0.3519) loss 2.4813 (3.1293) grad_norm 1.3490 (nan) loss_scale 1024.0000 (1940.6589) mem 11224MB [2024-09-01 06:14:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][610/625] eta 0:00:05 lr 0.000977 wd 0.0500 time 0.3444 (0.3531) data time 0.0006 (0.0015) model time 0.3438 (0.3518) loss 3.0611 (3.1305) grad_norm 1.9598 (nan) loss_scale 1024.0000 (1925.6563) mem 11224MB [2024-09-01 06:14:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [162/300][620/625] eta 0:00:01 lr 0.000977 wd 0.0500 time 0.3395 (0.3529) data time 0.0006 (0.0015) model time 0.3389 (0.3516) loss 3.2485 (3.1296) grad_norm 1.1637 (nan) loss_scale 1024.0000 (1911.1369) mem 11224MB [2024-09-01 06:14:26 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 162 training takes 0:03:40 [2024-09-01 06:14:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:14:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:14:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.432 (0.432) Loss 0.5908 (0.5908) Acc@1 87.256 (87.256) Acc@5 98.047 (98.047) Mem 11224MB [2024-09-01 06:14:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.9751 (0.7174) Acc@1 76.758 (83.767) Acc@5 94.824 (96.990) Mem 11224MB [2024-09-01 06:14:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 1.0654 (0.8523) Acc@1 73.730 (80.545) Acc@5 93.652 (95.459) Mem 11224MB [2024-09-01 06:14:29 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.348 Acc@5 95.483 [2024-09-01 06:14:29 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.3% [2024-09-01 06:14:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.817 (0.817) Loss 0.5015 (0.5015) Acc@1 89.014 (89.014) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 06:14:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.168) Loss 0.8340 (0.6328) Acc@1 80.078 (85.636) Acc@5 96.045 (97.563) Mem 11224MB [2024-09-01 06:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.132) Loss 0.9541 (0.7553) Acc@1 76.562 (82.561) Acc@5 95.117 (96.333) Mem 11224MB [2024-09-01 06:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.330 Acc@5 96.347 [2024-09-01 06:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.3% [2024-09-01 06:14:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][0/625] eta 0:12:14 lr 0.000977 wd 0.0500 time 1.1749 (1.1749) data time 0.5378 (0.5378) model time 0.0000 (0.0000) loss 3.1487 (3.1487) grad_norm 1.6861 (1.6861) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:14:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][10/625] eta 0:04:28 lr 0.000977 wd 0.0500 time 0.3472 (0.4368) data time 0.0009 (0.0497) model time 0.0000 (0.0000) loss 3.3948 (3.3066) grad_norm 1.8484 (1.9272) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:14:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][20/625] eta 0:03:58 lr 0.000976 wd 0.0500 time 0.3459 (0.3949) data time 0.0006 (0.0264) model time 0.0000 (0.0000) loss 3.0910 (3.3097) grad_norm 1.3942 (1.8913) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:14:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][30/625] eta 0:03:46 lr 0.000976 wd 0.0500 time 0.3474 (0.3806) data time 0.0007 (0.0182) model time 0.0000 (0.0000) loss 3.8954 (3.3157) grad_norm 1.1988 (1.7023) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:14:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][40/625] eta 0:03:38 lr 0.000976 wd 0.0500 time 0.3522 (0.3728) data time 0.0008 (0.0140) model time 0.0000 (0.0000) loss 2.8619 (3.2671) grad_norm 2.1153 (1.6784) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:14:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][50/625] eta 0:03:31 lr 0.000976 wd 0.0500 time 0.3503 (0.3681) data time 0.0008 (0.0115) model time 0.0000 (0.0000) loss 2.1034 (3.2165) grad_norm 1.6210 (1.6729) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:14:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][60/625] eta 0:03:27 lr 0.000976 wd 0.0500 time 0.5571 (0.3680) data time 0.0007 (0.0097) model time 0.5564 (0.3664) loss 2.3138 (3.1786) grad_norm 1.5926 (1.6618) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:14:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][70/625] eta 0:03:24 lr 0.000975 wd 0.0500 time 0.3477 (0.3682) data time 0.0007 (0.0085) model time 0.3470 (0.3676) loss 2.7584 (3.1653) grad_norm 1.3621 (1.6680) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][80/625] eta 0:03:19 lr 0.000975 wd 0.0500 time 0.3491 (0.3657) data time 0.0008 (0.0075) model time 0.3483 (0.3606) loss 3.3987 (3.1688) grad_norm 1.6034 (1.6547) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][90/625] eta 0:03:14 lr 0.000975 wd 0.0500 time 0.3507 (0.3638) data time 0.0007 (0.0068) model time 0.3499 (0.3574) loss 3.9697 (3.1526) grad_norm 2.5941 (1.6543) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][100/625] eta 0:03:10 lr 0.000975 wd 0.0500 time 0.3457 (0.3622) data time 0.0009 (0.0062) model time 0.3449 (0.3553) loss 2.8449 (3.1739) grad_norm 1.9287 (1.6488) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][110/625] eta 0:03:05 lr 0.000975 wd 0.0500 time 0.3484 (0.3608) data time 0.0009 (0.0057) model time 0.3474 (0.3538) loss 2.6549 (3.1327) grad_norm 1.4421 (1.6308) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][120/625] eta 0:03:01 lr 0.000975 wd 0.0500 time 0.3504 (0.3597) data time 0.0006 (0.0053) model time 0.3498 (0.3527) loss 3.2878 (3.1373) grad_norm 1.2148 (1.6271) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][130/625] eta 0:02:58 lr 0.000974 wd 0.0500 time 0.3397 (0.3599) data time 0.0006 (0.0050) model time 0.3391 (0.3538) loss 3.3886 (3.1340) grad_norm 1.9346 (1.6596) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][140/625] eta 0:02:54 lr 0.000974 wd 0.0500 time 0.3438 (0.3590) data time 0.0007 (0.0047) model time 0.3431 (0.3530) loss 4.0486 (3.1471) grad_norm 1.3648 (1.6436) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][150/625] eta 0:02:50 lr 0.000974 wd 0.0500 time 0.3503 (0.3584) data time 0.0009 (0.0044) model time 0.3494 (0.3525) loss 2.0945 (3.1545) grad_norm 1.7882 (1.6514) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][160/625] eta 0:02:46 lr 0.000974 wd 0.0500 time 0.3465 (0.3577) data time 0.0006 (0.0042) model time 0.3459 (0.3521) loss 3.0934 (3.1463) grad_norm 1.2270 (1.6457) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][170/625] eta 0:02:42 lr 0.000974 wd 0.0500 time 0.3482 (0.3572) data time 0.0009 (0.0040) model time 0.3474 (0.3517) loss 3.8878 (3.1530) grad_norm 1.6736 (1.6359) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][180/625] eta 0:02:38 lr 0.000973 wd 0.0500 time 0.3492 (0.3567) data time 0.0006 (0.0038) model time 0.3486 (0.3514) loss 3.2313 (3.1500) grad_norm 2.8123 (1.6283) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][190/625] eta 0:02:34 lr 0.000973 wd 0.0500 time 0.3492 (0.3562) data time 0.0010 (0.0037) model time 0.3482 (0.3510) loss 3.6789 (3.1515) grad_norm 1.3459 (1.6226) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][200/625] eta 0:02:31 lr 0.000973 wd 0.0500 time 0.3543 (0.3558) data time 0.0008 (0.0035) model time 0.3535 (0.3507) loss 3.5062 (3.1517) grad_norm 1.1978 (1.6206) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][210/625] eta 0:02:27 lr 0.000973 wd 0.0500 time 0.3464 (0.3553) data time 0.0008 (0.0034) model time 0.3455 (0.3504) loss 3.3723 (3.1589) grad_norm 1.4433 (1.6106) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][220/625] eta 0:02:23 lr 0.000973 wd 0.0500 time 0.3491 (0.3551) data time 0.0008 (0.0033) model time 0.3484 (0.3504) loss 3.2101 (3.1632) grad_norm 2.0626 (1.6168) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][230/625] eta 0:02:20 lr 0.000973 wd 0.0500 time 0.3469 (0.3548) data time 0.0008 (0.0032) model time 0.3461 (0.3502) loss 2.2980 (3.1603) grad_norm 1.5188 (1.6151) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:15:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][240/625] eta 0:02:16 lr 0.000972 wd 0.0500 time 0.3654 (0.3546) data time 0.0008 (0.0031) model time 0.3646 (0.3501) loss 3.4729 (3.1655) grad_norm 1.4186 (1.6090) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][250/625] eta 0:02:13 lr 0.000972 wd 0.0500 time 0.3422 (0.3565) data time 0.0006 (0.0030) model time 0.3416 (0.3527) loss 3.5300 (3.1785) grad_norm 1.1476 (1.6129) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][260/625] eta 0:02:10 lr 0.000972 wd 0.0500 time 0.3514 (0.3562) data time 0.0008 (0.0029) model time 0.3506 (0.3525) loss 3.2084 (3.1849) grad_norm 1.2478 (1.6064) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][270/625] eta 0:02:06 lr 0.000972 wd 0.0500 time 0.3478 (0.3559) data time 0.0006 (0.0028) model time 0.3472 (0.3523) loss 3.7312 (3.1853) grad_norm 2.2333 (1.6190) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][280/625] eta 0:02:02 lr 0.000972 wd 0.0500 time 0.3506 (0.3557) data time 0.0006 (0.0028) model time 0.3499 (0.3521) loss 2.8709 (3.1819) grad_norm 1.7046 (1.6241) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][290/625] eta 0:01:59 lr 0.000972 wd 0.0500 time 0.3513 (0.3555) data time 0.0006 (0.0027) model time 0.3507 (0.3520) loss 4.0525 (3.1819) grad_norm 1.4966 (1.6251) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][300/625] eta 0:01:55 lr 0.000971 wd 0.0500 time 0.3443 (0.3553) data time 0.0007 (0.0026) model time 0.3436 (0.3519) loss 2.8272 (3.1740) grad_norm 1.8813 (1.6671) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][310/625] eta 0:01:51 lr 0.000971 wd 0.0500 time 0.3474 (0.3550) data time 0.0009 (0.0026) model time 0.3465 (0.3516) loss 3.6612 (3.1753) grad_norm 1.3663 (1.6807) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][320/625] eta 0:01:48 lr 0.000971 wd 0.0500 time 0.3570 (0.3548) data time 0.0006 (0.0025) model time 0.3564 (0.3515) loss 2.9127 (3.1776) grad_norm 1.5767 (1.6776) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][330/625] eta 0:01:44 lr 0.000971 wd 0.0500 time 0.3505 (0.3546) data time 0.0008 (0.0025) model time 0.3497 (0.3513) loss 3.2113 (3.1801) grad_norm 2.8681 (1.6745) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][340/625] eta 0:01:41 lr 0.000971 wd 0.0500 time 0.3490 (0.3550) data time 0.0008 (0.0024) model time 0.3482 (0.3518) loss 3.5345 (3.1906) grad_norm 1.6192 (1.6679) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][350/625] eta 0:01:37 lr 0.000970 wd 0.0500 time 0.3473 (0.3548) data time 0.0009 (0.0024) model time 0.3465 (0.3517) loss 3.0222 (3.1842) grad_norm 1.4204 (1.6671) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][360/625] eta 0:01:33 lr 0.000970 wd 0.0500 time 0.3463 (0.3546) data time 0.0008 (0.0023) model time 0.3455 (0.3515) loss 3.3726 (3.1855) grad_norm 1.4766 (1.6649) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][370/625] eta 0:01:30 lr 0.000970 wd 0.0500 time 0.3459 (0.3554) data time 0.0007 (0.0023) model time 0.3452 (0.3526) loss 3.9925 (3.1909) grad_norm 1.4031 (1.6697) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][380/625] eta 0:01:27 lr 0.000970 wd 0.0500 time 0.3500 (0.3553) data time 0.0008 (0.0023) model time 0.3492 (0.3525) loss 3.1257 (3.1932) grad_norm 1.1956 (1.6646) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][390/625] eta 0:01:23 lr 0.000970 wd 0.0500 time 0.3462 (0.3551) data time 0.0006 (0.0022) model time 0.3456 (0.3523) loss 3.8508 (3.1956) grad_norm 1.1543 (1.6610) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][400/625] eta 0:01:19 lr 0.000970 wd 0.0500 time 0.3443 (0.3549) data time 0.0009 (0.0022) model time 0.3434 (0.3522) loss 2.7590 (3.1991) grad_norm 1.4998 (1.6513) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][410/625] eta 0:01:16 lr 0.000969 wd 0.0500 time 0.3504 (0.3548) data time 0.0009 (0.0022) model time 0.3495 (0.3520) loss 3.4384 (3.2007) grad_norm 1.1102 (1.6431) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][420/625] eta 0:01:12 lr 0.000969 wd 0.0500 time 0.3522 (0.3546) data time 0.0005 (0.0021) model time 0.3516 (0.3519) loss 3.2274 (3.2020) grad_norm 1.3947 (1.6492) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][430/625] eta 0:01:09 lr 0.000969 wd 0.0500 time 0.3492 (0.3545) data time 0.0009 (0.0021) model time 0.3484 (0.3518) loss 2.9123 (3.1935) grad_norm 1.9041 (1.6481) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][440/625] eta 0:01:05 lr 0.000969 wd 0.0500 time 0.3529 (0.3544) data time 0.0009 (0.0021) model time 0.3520 (0.3517) loss 2.7904 (3.1924) grad_norm 1.7329 (1.6444) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][450/625] eta 0:01:01 lr 0.000969 wd 0.0500 time 0.3518 (0.3542) data time 0.0008 (0.0020) model time 0.3510 (0.3516) loss 2.2560 (3.1875) grad_norm 1.5872 (1.6417) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][460/625] eta 0:00:58 lr 0.000969 wd 0.0500 time 0.3475 (0.3541) data time 0.0006 (0.0020) model time 0.3469 (0.3515) loss 2.4109 (3.1898) grad_norm 1.6968 (1.6500) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][470/625] eta 0:00:54 lr 0.000968 wd 0.0500 time 0.3490 (0.3541) data time 0.0008 (0.0020) model time 0.3482 (0.3515) loss 2.8579 (3.1910) grad_norm 1.6288 (1.6486) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][480/625] eta 0:00:51 lr 0.000968 wd 0.0500 time 0.3495 (0.3539) data time 0.0008 (0.0020) model time 0.3487 (0.3514) loss 2.3131 (3.1860) grad_norm 1.9786 (1.6463) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][490/625] eta 0:00:47 lr 0.000968 wd 0.0500 time 0.3448 (0.3538) data time 0.0008 (0.0020) model time 0.3440 (0.3513) loss 3.1832 (3.1855) grad_norm 1.4931 (1.6460) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][500/625] eta 0:00:44 lr 0.000968 wd 0.0500 time 0.3496 (0.3537) data time 0.0009 (0.0019) model time 0.3487 (0.3512) loss 3.1662 (3.1837) grad_norm 1.1922 (1.6397) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][510/625] eta 0:00:40 lr 0.000968 wd 0.0500 time 0.3638 (0.3536) data time 0.0008 (0.0019) model time 0.3630 (0.3512) loss 3.1800 (3.1841) grad_norm 1.2633 (1.6369) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][520/625] eta 0:00:37 lr 0.000967 wd 0.0500 time 0.3479 (0.3535) data time 0.0009 (0.0019) model time 0.3470 (0.3511) loss 3.2427 (3.1824) grad_norm 1.3265 (1.6363) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][530/625] eta 0:00:33 lr 0.000967 wd 0.0500 time 0.3443 (0.3537) data time 0.0007 (0.0019) model time 0.3436 (0.3513) loss 2.7348 (3.1793) grad_norm 1.1303 (1.6317) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][540/625] eta 0:00:30 lr 0.000967 wd 0.0500 time 0.3528 (0.3536) data time 0.0006 (0.0018) model time 0.3522 (0.3512) loss 3.4008 (3.1769) grad_norm 1.7289 (1.6668) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][550/625] eta 0:00:26 lr 0.000967 wd 0.0500 time 0.3443 (0.3535) data time 0.0007 (0.0018) model time 0.3436 (0.3512) loss 2.2841 (3.1700) grad_norm 1.2982 (1.6623) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][560/625] eta 0:00:22 lr 0.000967 wd 0.0500 time 0.3491 (0.3534) data time 0.0007 (0.0018) model time 0.3484 (0.3511) loss 3.4363 (3.1656) grad_norm 1.6585 (1.6599) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][570/625] eta 0:00:19 lr 0.000967 wd 0.0500 time 0.3498 (0.3534) data time 0.0008 (0.0018) model time 0.3490 (0.3510) loss 2.1659 (3.1628) grad_norm 1.6859 (1.6562) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:17:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][580/625] eta 0:00:15 lr 0.000966 wd 0.0500 time 0.3433 (0.3540) data time 0.0008 (0.0018) model time 0.3425 (0.3518) loss 3.2372 (3.1603) grad_norm 1.7004 (1.6560) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][590/625] eta 0:00:12 lr 0.000966 wd 0.0500 time 0.3480 (0.3539) data time 0.0008 (0.0018) model time 0.3472 (0.3517) loss 3.3112 (3.1632) grad_norm 1.1686 (1.6532) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][600/625] eta 0:00:08 lr 0.000966 wd 0.0500 time 0.3472 (0.3538) data time 0.0007 (0.0018) model time 0.3465 (0.3516) loss 2.0675 (3.1598) grad_norm 1.2286 (1.6504) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][610/625] eta 0:00:05 lr 0.000966 wd 0.0500 time 0.3437 (0.3537) data time 0.0004 (0.0018) model time 0.3434 (0.3515) loss 2.6039 (3.1619) grad_norm 0.9721 (1.6484) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [163/300][620/625] eta 0:00:01 lr 0.000966 wd 0.0500 time 0.3375 (0.3535) data time 0.0004 (0.0017) model time 0.3371 (0.3513) loss 4.0136 (3.1625) grad_norm 1.6205 (1.6469) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 163 training takes 0:03:40 [2024-09-01 06:18:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:18:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.440 (0.440) Loss 0.5566 (0.5566) Acc@1 87.744 (87.744) Acc@5 98.096 (98.096) Mem 11224MB [2024-09-01 06:18:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.9390 (0.7034) Acc@1 77.295 (84.020) Acc@5 94.678 (96.982) Mem 11224MB [2024-09-01 06:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.098 (0.111) Loss 1.0508 (0.8371) Acc@1 74.268 (80.752) Acc@5 93.604 (95.543) Mem 11224MB [2024-09-01 06:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.418 Acc@5 95.531 [2024-09-01 06:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.4% [2024-09-01 06:18:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.747 (0.747) Loss 0.5020 (0.5020) Acc@1 88.916 (88.916) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 06:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.163) Loss 0.8330 (0.6319) Acc@1 80.078 (85.693) Acc@5 96.094 (97.594) Mem 11224MB [2024-09-01 06:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.129) Loss 0.9517 (0.7540) Acc@1 76.709 (82.606) Acc@5 94.971 (96.347) Mem 11224MB [2024-09-01 06:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.366 Acc@5 96.359 [2024-09-01 06:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.4% [2024-09-01 06:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.37% [2024-09-01 06:18:20 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:18:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:18:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][0/625] eta 0:07:14 lr 0.000966 wd 0.0500 time 0.6948 (0.6948) data time 0.3826 (0.3826) model time 0.0000 (0.0000) loss 3.7654 (3.7654) grad_norm 1.7292 (1.7292) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][10/625] eta 0:03:53 lr 0.000965 wd 0.0500 time 0.3527 (0.3793) data time 0.0009 (0.0356) model time 0.0000 (0.0000) loss 2.5681 (3.2672) grad_norm 1.6408 (1.7033) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][20/625] eta 0:03:40 lr 0.000965 wd 0.0500 time 0.3494 (0.3652) data time 0.0007 (0.0190) model time 0.0000 (0.0000) loss 2.3200 (3.0510) grad_norm 1.4731 (1.6418) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][30/625] eta 0:03:33 lr 0.000965 wd 0.0500 time 0.3441 (0.3592) data time 0.0011 (0.0132) model time 0.0000 (0.0000) loss 3.6388 (3.1186) grad_norm 1.1953 (1.6253) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][40/625] eta 0:03:28 lr 0.000965 wd 0.0500 time 0.3474 (0.3565) data time 0.0006 (0.0102) model time 0.0000 (0.0000) loss 3.9165 (3.1628) grad_norm 2.0897 (1.6286) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][50/625] eta 0:03:23 lr 0.000965 wd 0.0500 time 0.3459 (0.3546) data time 0.0006 (0.0083) model time 0.0000 (0.0000) loss 3.6550 (3.1322) grad_norm 1.6970 (1.6521) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][60/625] eta 0:03:19 lr 0.000965 wd 0.0500 time 0.3492 (0.3534) data time 0.0010 (0.0071) model time 0.3482 (0.3465) loss 3.7971 (3.1714) grad_norm 1.6454 (1.6375) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][70/625] eta 0:03:15 lr 0.000964 wd 0.0500 time 0.3449 (0.3527) data time 0.0007 (0.0062) model time 0.3442 (0.3469) loss 2.7344 (3.1430) grad_norm 2.1006 (1.6255) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][80/625] eta 0:03:11 lr 0.000964 wd 0.0500 time 0.3481 (0.3522) data time 0.0009 (0.0056) model time 0.3472 (0.3473) loss 3.2117 (3.1546) grad_norm 1.9931 (1.6241) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][90/625] eta 0:03:08 lr 0.000964 wd 0.0500 time 0.3528 (0.3519) data time 0.0008 (0.0050) model time 0.3520 (0.3476) loss 3.0681 (3.1508) grad_norm 1.5131 (1.5877) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:18:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][100/625] eta 0:03:04 lr 0.000964 wd 0.0500 time 0.3557 (0.3515) data time 0.0006 (0.0046) model time 0.3551 (0.3476) loss 3.8243 (3.1504) grad_norm 2.6072 (1.5972) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][110/625] eta 0:03:01 lr 0.000964 wd 0.0500 time 0.3438 (0.3526) data time 0.0008 (0.0043) model time 0.3430 (0.3500) loss 3.5508 (3.1522) grad_norm 1.4567 (1.5847) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][120/625] eta 0:02:57 lr 0.000963 wd 0.0500 time 0.3435 (0.3522) data time 0.0008 (0.0040) model time 0.3427 (0.3497) loss 3.4833 (3.1668) grad_norm 1.2778 (1.5695) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][130/625] eta 0:02:54 lr 0.000963 wd 0.0500 time 0.3492 (0.3519) data time 0.0007 (0.0038) model time 0.3484 (0.3494) loss 2.7962 (3.1547) grad_norm 1.8338 (1.5798) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][140/625] eta 0:02:50 lr 0.000963 wd 0.0500 time 0.3510 (0.3517) data time 0.0008 (0.0036) model time 0.3501 (0.3492) loss 3.7274 (3.1544) grad_norm 1.6337 (1.5824) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][150/625] eta 0:02:47 lr 0.000963 wd 0.0500 time 0.5606 (0.3528) data time 0.0008 (0.0034) model time 0.5599 (0.3511) loss 3.6233 (3.1376) grad_norm 1.5353 (1.6062) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][160/625] eta 0:02:45 lr 0.000963 wd 0.0500 time 0.3488 (0.3551) data time 0.0006 (0.0032) model time 0.3482 (0.3545) loss 3.5583 (3.1421) grad_norm 1.5665 (1.6021) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][170/625] eta 0:02:41 lr 0.000963 wd 0.0500 time 0.3458 (0.3546) data time 0.0006 (0.0031) model time 0.3452 (0.3538) loss 2.2391 (3.1370) grad_norm 1.4782 (1.5975) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][180/625] eta 0:02:37 lr 0.000962 wd 0.0500 time 0.3505 (0.3542) data time 0.0009 (0.0030) model time 0.3497 (0.3532) loss 2.9245 (3.1382) grad_norm 1.2100 (1.5905) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][190/625] eta 0:02:33 lr 0.000962 wd 0.0500 time 0.3448 (0.3538) data time 0.0009 (0.0029) model time 0.3439 (0.3527) loss 2.9462 (3.1471) grad_norm 1.3864 (1.6173) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][200/625] eta 0:02:30 lr 0.000962 wd 0.0500 time 0.3470 (0.3537) data time 0.0006 (0.0027) model time 0.3464 (0.3525) loss 2.9597 (3.1507) grad_norm 1.7294 (1.6142) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][210/625] eta 0:02:26 lr 0.000962 wd 0.0500 time 0.3531 (0.3535) data time 0.0006 (0.0027) model time 0.3524 (0.3523) loss 3.7511 (3.1314) grad_norm 1.5682 (1.6137) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][220/625] eta 0:02:23 lr 0.000962 wd 0.0500 time 0.3479 (0.3532) data time 0.0006 (0.0026) model time 0.3473 (0.3519) loss 3.6974 (3.1337) grad_norm 1.3400 (1.6085) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][230/625] eta 0:02:19 lr 0.000962 wd 0.0500 time 0.3428 (0.3528) data time 0.0009 (0.0025) model time 0.3419 (0.3515) loss 3.3105 (3.1285) grad_norm 3.3532 (1.6143) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][240/625] eta 0:02:15 lr 0.000961 wd 0.0500 time 0.3444 (0.3526) data time 0.0008 (0.0024) model time 0.3435 (0.3513) loss 2.9958 (3.1273) grad_norm 1.4755 (1.6230) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][250/625] eta 0:02:12 lr 0.000961 wd 0.0500 time 0.3484 (0.3525) data time 0.0007 (0.0024) model time 0.3478 (0.3511) loss 2.9289 (3.1284) grad_norm 1.2192 (1.6186) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][260/625] eta 0:02:08 lr 0.000961 wd 0.0500 time 0.3485 (0.3524) data time 0.0008 (0.0023) model time 0.3477 (0.3510) loss 3.6494 (3.1279) grad_norm 1.5222 (1.6146) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:19:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][270/625] eta 0:02:05 lr 0.000961 wd 0.0500 time 0.3485 (0.3523) data time 0.0008 (0.0023) model time 0.3476 (0.3509) loss 3.1897 (3.1341) grad_norm 1.5741 (1.6123) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][280/625] eta 0:02:01 lr 0.000961 wd 0.0500 time 0.3487 (0.3522) data time 0.0006 (0.0022) model time 0.3481 (0.3509) loss 3.4038 (3.1397) grad_norm 2.3311 (1.6197) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][290/625] eta 0:01:57 lr 0.000960 wd 0.0500 time 0.3470 (0.3522) data time 0.0009 (0.0022) model time 0.3462 (0.3508) loss 3.2546 (3.1365) grad_norm 2.2264 (1.6268) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][300/625] eta 0:01:54 lr 0.000960 wd 0.0500 time 0.3500 (0.3520) data time 0.0006 (0.0021) model time 0.3494 (0.3507) loss 4.0984 (3.1489) grad_norm 2.9592 (1.6257) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][310/625] eta 0:01:50 lr 0.000960 wd 0.0500 time 0.3525 (0.3519) data time 0.0006 (0.0021) model time 0.3518 (0.3505) loss 3.6889 (3.1591) grad_norm 1.6565 (1.6273) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][320/625] eta 0:01:47 lr 0.000960 wd 0.0500 time 0.3522 (0.3523) data time 0.0007 (0.0020) model time 0.3515 (0.3510) loss 3.8510 (3.1611) grad_norm 1.9962 (1.6194) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][330/625] eta 0:01:43 lr 0.000960 wd 0.0500 time 0.3432 (0.3522) data time 0.0007 (0.0020) model time 0.3425 (0.3510) loss 2.8599 (3.1624) grad_norm 1.6107 (1.6205) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][340/625] eta 0:01:40 lr 0.000960 wd 0.0500 time 0.3434 (0.3536) data time 0.0009 (0.0020) model time 0.3425 (0.3526) loss 3.3230 (3.1632) grad_norm 1.7471 (1.6180) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][350/625] eta 0:01:37 lr 0.000959 wd 0.0500 time 0.3505 (0.3545) data time 0.0006 (0.0019) model time 0.3499 (0.3536) loss 2.8945 (3.1591) grad_norm 1.3624 (1.6251) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][360/625] eta 0:01:33 lr 0.000959 wd 0.0500 time 0.3508 (0.3543) data time 0.0009 (0.0019) model time 0.3498 (0.3534) loss 3.4258 (3.1600) grad_norm 2.1790 (1.6293) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][370/625] eta 0:01:30 lr 0.000959 wd 0.0500 time 0.3473 (0.3541) data time 0.0006 (0.0019) model time 0.3466 (0.3532) loss 2.4644 (3.1588) grad_norm 1.4215 (1.6266) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][380/625] eta 0:01:26 lr 0.000959 wd 0.0500 time 0.3497 (0.3540) data time 0.0006 (0.0018) model time 0.3491 (0.3530) loss 3.7002 (3.1638) grad_norm 2.3191 (1.6245) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][390/625] eta 0:01:23 lr 0.000959 wd 0.0500 time 0.3500 (0.3539) data time 0.0006 (0.0018) model time 0.3493 (0.3529) loss 1.8857 (3.1573) grad_norm 1.2740 (1.6216) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][400/625] eta 0:01:19 lr 0.000958 wd 0.0500 time 0.3466 (0.3537) data time 0.0007 (0.0018) model time 0.3460 (0.3527) loss 3.1626 (3.1564) grad_norm 1.3299 (1.6239) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][410/625] eta 0:01:16 lr 0.000958 wd 0.0500 time 0.3531 (0.3537) data time 0.0007 (0.0018) model time 0.3524 (0.3527) loss 4.0182 (3.1544) grad_norm 1.9899 (1.6278) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][420/625] eta 0:01:12 lr 0.000958 wd 0.0500 time 0.3456 (0.3536) data time 0.0008 (0.0018) model time 0.3447 (0.3526) loss 2.4376 (3.1531) grad_norm 1.1307 (1.6255) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][430/625] eta 0:01:08 lr 0.000958 wd 0.0500 time 0.3532 (0.3535) data time 0.0009 (0.0017) model time 0.3523 (0.3525) loss 2.2365 (3.1508) grad_norm 1.0649 (1.6209) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:20:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][440/625] eta 0:01:05 lr 0.000958 wd 0.0500 time 0.3494 (0.3533) data time 0.0008 (0.0017) model time 0.3486 (0.3523) loss 3.2743 (3.1490) grad_norm 1.4976 (1.6212) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][450/625] eta 0:01:01 lr 0.000958 wd 0.0500 time 0.3452 (0.3532) data time 0.0006 (0.0017) model time 0.3446 (0.3522) loss 3.6905 (3.1541) grad_norm 2.8356 (1.6238) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][460/625] eta 0:00:58 lr 0.000957 wd 0.0500 time 0.3401 (0.3540) data time 0.0006 (0.0017) model time 0.3395 (0.3531) loss 3.6278 (3.1552) grad_norm 1.4112 (1.6242) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][470/625] eta 0:00:54 lr 0.000957 wd 0.0500 time 0.3434 (0.3544) data time 0.0007 (0.0017) model time 0.3427 (0.3535) loss 3.9872 (3.1515) grad_norm 1.6388 (1.6247) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][480/625] eta 0:00:51 lr 0.000957 wd 0.0500 time 0.3496 (0.3542) data time 0.0006 (0.0016) model time 0.3489 (0.3533) loss 4.0530 (3.1528) grad_norm 1.4641 (1.6274) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][490/625] eta 0:00:47 lr 0.000957 wd 0.0500 time 0.3445 (0.3541) data time 0.0008 (0.0016) model time 0.3437 (0.3532) loss 2.9834 (3.1538) grad_norm 1.0685 (1.6314) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][500/625] eta 0:00:44 lr 0.000957 wd 0.0500 time 0.3562 (0.3543) data time 0.0008 (0.0016) model time 0.3553 (0.3535) loss 2.6716 (3.1486) grad_norm 2.4176 (1.6318) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][510/625] eta 0:00:40 lr 0.000957 wd 0.0500 time 0.3499 (0.3542) data time 0.0009 (0.0016) model time 0.3490 (0.3533) loss 2.4050 (3.1484) grad_norm 1.4345 (1.6292) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][520/625] eta 0:00:37 lr 0.000956 wd 0.0500 time 0.3418 (0.3541) data time 0.0006 (0.0016) model time 0.3411 (0.3532) loss 2.3018 (3.1469) grad_norm 1.8401 (1.6364) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][530/625] eta 0:00:33 lr 0.000956 wd 0.0500 time 0.3437 (0.3540) data time 0.0008 (0.0016) model time 0.3429 (0.3531) loss 2.2904 (3.1488) grad_norm 1.6269 (1.6352) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][540/625] eta 0:00:30 lr 0.000956 wd 0.0500 time 0.3455 (0.3539) data time 0.0007 (0.0016) model time 0.3448 (0.3529) loss 3.2965 (3.1459) grad_norm 1.8457 (1.6372) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][550/625] eta 0:00:26 lr 0.000956 wd 0.0500 time 0.3365 (0.3538) data time 0.0006 (0.0015) model time 0.3358 (0.3528) loss 2.7942 (3.1444) grad_norm 1.2508 (1.6360) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][560/625] eta 0:00:22 lr 0.000956 wd 0.0500 time 0.3446 (0.3537) data time 0.0008 (0.0015) model time 0.3438 (0.3527) loss 3.0603 (3.1451) grad_norm 1.6049 (1.6336) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][570/625] eta 0:00:19 lr 0.000955 wd 0.0500 time 0.3512 (0.3536) data time 0.0007 (0.0015) model time 0.3505 (0.3527) loss 3.7469 (3.1460) grad_norm 1.5180 (1.6339) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][580/625] eta 0:00:15 lr 0.000955 wd 0.0500 time 0.3509 (0.3536) data time 0.0009 (0.0015) model time 0.3500 (0.3526) loss 2.2001 (3.1444) grad_norm 1.7801 (1.6343) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][590/625] eta 0:00:12 lr 0.000955 wd 0.0500 time 0.3465 (0.3535) data time 0.0007 (0.0015) model time 0.3458 (0.3526) loss 3.6367 (3.1446) grad_norm 1.1029 (1.6288) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][600/625] eta 0:00:08 lr 0.000955 wd 0.0500 time 0.3516 (0.3534) data time 0.0007 (0.0015) model time 0.3510 (0.3525) loss 2.7029 (3.1448) grad_norm 1.5461 (1.6305) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:21:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][610/625] eta 0:00:05 lr 0.000955 wd 0.0500 time 0.3382 (0.3533) data time 0.0004 (0.0015) model time 0.3378 (0.3524) loss 3.4166 (3.1439) grad_norm 1.4691 (1.6355) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [164/300][620/625] eta 0:00:01 lr 0.000955 wd 0.0500 time 0.3411 (0.3534) data time 0.0006 (0.0015) model time 0.3405 (0.3524) loss 1.9238 (3.1409) grad_norm 2.8240 (1.6403) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:02 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 164 training takes 0:03:40 [2024-09-01 06:22:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:22:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:22:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.463 (0.463) Loss 0.5801 (0.5801) Acc@1 87.158 (87.158) Acc@5 97.852 (97.852) Mem 11224MB [2024-09-01 06:22:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 0.9893 (0.7101) Acc@1 76.172 (84.184) Acc@5 94.678 (97.088) Mem 11224MB [2024-09-01 06:22:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.113) Loss 1.0693 (0.8482) Acc@1 74.658 (80.818) Acc@5 93.848 (95.580) Mem 11224MB [2024-09-01 06:22:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.524 Acc@5 95.527 [2024-09-01 06:22:06 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.5% [2024-09-01 06:22:06 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 80.52% [2024-09-01 06:22:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 06:22:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 06:22:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.428 (0.428) Loss 0.5015 (0.5015) Acc@1 89.014 (89.014) Acc@5 98.730 (98.730) Mem 11224MB [2024-09-01 06:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.129) Loss 0.8320 (0.6311) Acc@1 80.029 (85.760) Acc@5 96.094 (97.607) Mem 11224MB [2024-09-01 06:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 0.9502 (0.7528) Acc@1 76.660 (82.652) Acc@5 94.873 (96.359) Mem 11224MB [2024-09-01 06:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.394 Acc@5 96.371 [2024-09-01 06:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.4% [2024-09-01 06:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.39% [2024-09-01 06:22:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:22:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:22:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][0/625] eta 0:07:29 lr 0.000954 wd 0.0500 time 0.7186 (0.7186) data time 0.3932 (0.3932) model time 0.0000 (0.0000) loss 3.7708 (3.7708) grad_norm 2.4717 (2.4717) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][10/625] eta 0:03:54 lr 0.000954 wd 0.0500 time 0.3462 (0.3816) data time 0.0007 (0.0365) model time 0.0000 (0.0000) loss 3.7499 (3.3303) grad_norm 1.4581 (1.6954) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][20/625] eta 0:03:41 lr 0.000954 wd 0.0500 time 0.3458 (0.3653) data time 0.0009 (0.0195) model time 0.0000 (0.0000) loss 3.4101 (3.2978) grad_norm 1.6871 (1.7654) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][30/625] eta 0:03:49 lr 0.000954 wd 0.0500 time 0.3558 (0.3855) data time 0.0009 (0.0135) model time 0.0000 (0.0000) loss 3.1949 (3.3126) grad_norm 2.6207 (1.8342) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][40/625] eta 0:03:40 lr 0.000954 wd 0.0500 time 0.3415 (0.3766) data time 0.0007 (0.0104) model time 0.0000 (0.0000) loss 4.1216 (3.2531) grad_norm 1.1339 (1.7427) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][50/625] eta 0:03:33 lr 0.000954 wd 0.0500 time 0.3460 (0.3715) data time 0.0007 (0.0085) model time 0.0000 (0.0000) loss 1.6969 (3.1892) grad_norm 1.4002 (1.6712) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][60/625] eta 0:03:28 lr 0.000953 wd 0.0500 time 0.3514 (0.3683) data time 0.0009 (0.0073) model time 0.3505 (0.3513) loss 2.6390 (3.1898) grad_norm 1.7194 (1.7464) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][70/625] eta 0:03:22 lr 0.000953 wd 0.0500 time 0.3503 (0.3657) data time 0.0006 (0.0064) model time 0.3497 (0.3499) loss 3.5595 (3.2164) grad_norm 1.2289 (1.7103) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][80/625] eta 0:03:18 lr 0.000953 wd 0.0500 time 0.3438 (0.3634) data time 0.0007 (0.0057) model time 0.3431 (0.3488) loss 2.4368 (3.2148) grad_norm 1.1924 (1.6808) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][90/625] eta 0:03:13 lr 0.000953 wd 0.0500 time 0.3533 (0.3617) data time 0.0009 (0.0052) model time 0.3524 (0.3484) loss 2.5106 (3.2248) grad_norm 1.4472 (1.6844) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][100/625] eta 0:03:10 lr 0.000953 wd 0.0500 time 0.3510 (0.3621) data time 0.0007 (0.0047) model time 0.3503 (0.3516) loss 2.2922 (3.2162) grad_norm 1.1923 (1.6547) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][110/625] eta 0:03:05 lr 0.000953 wd 0.0500 time 0.3444 (0.3608) data time 0.0007 (0.0044) model time 0.3437 (0.3508) loss 3.6212 (3.2189) grad_norm 1.7064 (1.6413) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][120/625] eta 0:03:01 lr 0.000952 wd 0.0500 time 0.3496 (0.3600) data time 0.0007 (0.0041) model time 0.3489 (0.3507) loss 3.2857 (3.2102) grad_norm 1.4686 (1.6263) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:22:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][130/625] eta 0:02:57 lr 0.000952 wd 0.0500 time 0.3231 (0.3591) data time 0.0009 (0.0039) model time 0.3222 (0.3503) loss 3.0911 (3.2147) grad_norm 1.6671 (1.6333) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][140/625] eta 0:02:53 lr 0.000952 wd 0.0500 time 0.3469 (0.3583) data time 0.0009 (0.0037) model time 0.3460 (0.3500) loss 3.5228 (3.1946) grad_norm 1.3869 (1.6236) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][150/625] eta 0:02:49 lr 0.000952 wd 0.0500 time 0.3494 (0.3578) data time 0.0007 (0.0035) model time 0.3487 (0.3499) loss 3.8389 (3.1969) grad_norm 1.7755 (1.6454) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][160/625] eta 0:02:46 lr 0.000952 wd 0.0500 time 0.3447 (0.3571) data time 0.0006 (0.0033) model time 0.3441 (0.3496) loss 3.6781 (3.2037) grad_norm 1.8118 (1.6708) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][170/625] eta 0:02:42 lr 0.000951 wd 0.0500 time 0.3474 (0.3567) data time 0.0007 (0.0032) model time 0.3468 (0.3496) loss 3.3093 (3.2058) grad_norm 1.4993 (1.6707) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][180/625] eta 0:02:38 lr 0.000951 wd 0.0500 time 0.3434 (0.3562) data time 0.0006 (0.0030) model time 0.3428 (0.3493) loss 3.8536 (3.2041) grad_norm 2.4192 (1.6679) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][190/625] eta 0:02:34 lr 0.000951 wd 0.0500 time 0.3638 (0.3558) data time 0.0007 (0.0029) model time 0.3631 (0.3492) loss 2.3489 (3.2006) grad_norm 3.1264 (1.6753) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][200/625] eta 0:02:31 lr 0.000951 wd 0.0500 time 0.3453 (0.3554) data time 0.0009 (0.0028) model time 0.3445 (0.3490) loss 3.5386 (3.1944) grad_norm 1.1663 (1.6718) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][210/625] eta 0:02:27 lr 0.000951 wd 0.0500 time 0.3519 (0.3550) data time 0.0009 (0.0027) model time 0.3510 (0.3489) loss 2.5430 (3.1950) grad_norm 1.4261 (1.6541) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][220/625] eta 0:02:24 lr 0.000951 wd 0.0500 time 0.3513 (0.3556) data time 0.0008 (0.0026) model time 0.3505 (0.3499) loss 3.1222 (3.1919) grad_norm 1.2882 (1.6455) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][230/625] eta 0:02:20 lr 0.000950 wd 0.0500 time 0.3538 (0.3553) data time 0.0006 (0.0026) model time 0.3531 (0.3498) loss 4.0266 (3.1992) grad_norm 3.0714 (1.6475) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][240/625] eta 0:02:17 lr 0.000950 wd 0.0500 time 0.3471 (0.3576) data time 0.0008 (0.0025) model time 0.3463 (0.3530) loss 3.3667 (3.2007) grad_norm 2.0814 (1.6448) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][250/625] eta 0:02:13 lr 0.000950 wd 0.0500 time 0.3459 (0.3571) data time 0.0006 (0.0024) model time 0.3453 (0.3526) loss 3.3779 (3.1943) grad_norm 1.3712 (1.6342) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][260/625] eta 0:02:10 lr 0.000950 wd 0.0500 time 0.3437 (0.3567) data time 0.0008 (0.0024) model time 0.3429 (0.3522) loss 3.6007 (3.1864) grad_norm 1.5062 (1.6253) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][270/625] eta 0:02:06 lr 0.000950 wd 0.0500 time 0.3437 (0.3563) data time 0.0006 (0.0023) model time 0.3431 (0.3520) loss 3.5850 (3.1936) grad_norm 1.6194 (1.6430) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][280/625] eta 0:02:02 lr 0.000950 wd 0.0500 time 0.3443 (0.3561) data time 0.0007 (0.0023) model time 0.3435 (0.3518) loss 3.3921 (3.1787) grad_norm 1.3613 (1.6410) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][290/625] eta 0:01:59 lr 0.000949 wd 0.0500 time 0.3459 (0.3558) data time 0.0009 (0.0022) model time 0.3451 (0.3516) loss 3.1199 (3.1808) grad_norm 1.5791 (1.6408) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:23:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][300/625] eta 0:01:55 lr 0.000949 wd 0.0500 time 0.3447 (0.3555) data time 0.0009 (0.0022) model time 0.3438 (0.3514) loss 3.4690 (3.1791) grad_norm 1.4111 (1.6436) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][310/625] eta 0:01:51 lr 0.000949 wd 0.0500 time 0.3468 (0.3553) data time 0.0008 (0.0021) model time 0.3461 (0.3512) loss 3.1404 (3.1747) grad_norm 1.8909 (1.6438) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][320/625] eta 0:01:48 lr 0.000949 wd 0.0500 time 0.3401 (0.3550) data time 0.0008 (0.0021) model time 0.3393 (0.3511) loss 3.0872 (3.1706) grad_norm 1.6799 (1.6469) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][330/625] eta 0:01:44 lr 0.000949 wd 0.0500 time 0.3472 (0.3548) data time 0.0009 (0.0020) model time 0.3463 (0.3509) loss 3.1875 (3.1692) grad_norm 1.5044 (1.6421) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][340/625] eta 0:01:41 lr 0.000948 wd 0.0500 time 0.3429 (0.3546) data time 0.0008 (0.0020) model time 0.3421 (0.3508) loss 2.0527 (3.1672) grad_norm 1.3791 (1.6407) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][350/625] eta 0:01:37 lr 0.000948 wd 0.0500 time 0.3501 (0.3545) data time 0.0009 (0.0020) model time 0.3492 (0.3507) loss 2.7921 (3.1641) grad_norm 1.4800 (1.6343) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][360/625] eta 0:01:33 lr 0.000948 wd 0.0500 time 0.3489 (0.3543) data time 0.0006 (0.0019) model time 0.3483 (0.3506) loss 2.1041 (3.1693) grad_norm 1.3662 (1.6361) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][370/625] eta 0:01:30 lr 0.000948 wd 0.0500 time 0.3444 (0.3542) data time 0.0007 (0.0019) model time 0.3438 (0.3506) loss 3.2170 (3.1660) grad_norm 1.6032 (1.6319) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][380/625] eta 0:01:26 lr 0.000948 wd 0.0500 time 0.3460 (0.3540) data time 0.0008 (0.0019) model time 0.3452 (0.3505) loss 2.6034 (3.1615) grad_norm 1.7678 (1.6370) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][390/625] eta 0:01:23 lr 0.000948 wd 0.0500 time 0.3484 (0.3539) data time 0.0006 (0.0019) model time 0.3478 (0.3504) loss 2.6749 (3.1602) grad_norm 1.4937 (1.6302) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][400/625] eta 0:01:19 lr 0.000947 wd 0.0500 time 0.3452 (0.3537) data time 0.0008 (0.0018) model time 0.3444 (0.3503) loss 2.7484 (3.1642) grad_norm 1.8020 (1.6272) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][410/625] eta 0:01:16 lr 0.000947 wd 0.0500 time 0.3508 (0.3536) data time 0.0007 (0.0018) model time 0.3501 (0.3502) loss 3.9668 (3.1695) grad_norm 1.8256 (1.6265) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][420/625] eta 0:01:12 lr 0.000947 wd 0.0500 time 0.3477 (0.3546) data time 0.0006 (0.0018) model time 0.3471 (0.3515) loss 3.7116 (3.1725) grad_norm 1.4940 (1.6311) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][430/625] eta 0:01:09 lr 0.000947 wd 0.0500 time 0.3475 (0.3553) data time 0.0009 (0.0018) model time 0.3465 (0.3523) loss 2.5699 (3.1699) grad_norm 1.9084 (1.6310) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][440/625] eta 0:01:05 lr 0.000947 wd 0.0500 time 0.3497 (0.3552) data time 0.0009 (0.0017) model time 0.3488 (0.3522) loss 3.3131 (3.1684) grad_norm 1.6596 (1.6335) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][450/625] eta 0:01:02 lr 0.000947 wd 0.0500 time 0.3536 (0.3550) data time 0.0008 (0.0017) model time 0.3528 (0.3520) loss 2.3880 (3.1637) grad_norm 1.5212 (1.6354) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][460/625] eta 0:00:58 lr 0.000946 wd 0.0500 time 0.3566 (0.3549) data time 0.0008 (0.0017) model time 0.3558 (0.3519) loss 3.3182 (3.1651) grad_norm 2.3716 (1.6439) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:24:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][470/625] eta 0:00:54 lr 0.000946 wd 0.0500 time 0.3590 (0.3548) data time 0.0007 (0.0017) model time 0.3583 (0.3519) loss 3.2164 (3.1643) grad_norm 1.3443 (1.6440) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][480/625] eta 0:00:51 lr 0.000946 wd 0.0500 time 0.3466 (0.3547) data time 0.0006 (0.0017) model time 0.3459 (0.3518) loss 3.3560 (3.1663) grad_norm 1.4154 (1.6394) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][490/625] eta 0:00:47 lr 0.000946 wd 0.0500 time 0.3520 (0.3545) data time 0.0007 (0.0016) model time 0.3513 (0.3517) loss 3.5610 (3.1705) grad_norm 1.5641 (1.6399) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][500/625] eta 0:00:44 lr 0.000946 wd 0.0500 time 0.3510 (0.3544) data time 0.0009 (0.0016) model time 0.3500 (0.3516) loss 3.1322 (3.1734) grad_norm 1.4961 (1.6380) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][510/625] eta 0:00:40 lr 0.000945 wd 0.0500 time 0.3423 (0.3543) data time 0.0006 (0.0016) model time 0.3417 (0.3515) loss 2.8308 (3.1680) grad_norm 1.3245 (1.6328) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][520/625] eta 0:00:37 lr 0.000945 wd 0.0500 time 0.3466 (0.3542) data time 0.0009 (0.0016) model time 0.3457 (0.3515) loss 2.2734 (3.1637) grad_norm 1.5315 (1.6329) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][530/625] eta 0:00:33 lr 0.000945 wd 0.0500 time 0.3530 (0.3541) data time 0.0006 (0.0016) model time 0.3524 (0.3514) loss 3.6301 (3.1598) grad_norm 1.6245 (1.6330) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][540/625] eta 0:00:30 lr 0.000945 wd 0.0500 time 0.5689 (0.3551) data time 0.0007 (0.0016) model time 0.5682 (0.3526) loss 2.6761 (3.1586) grad_norm 1.3134 (1.6324) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][550/625] eta 0:00:26 lr 0.000945 wd 0.0500 time 0.3502 (0.3550) data time 0.0009 (0.0016) model time 0.3494 (0.3525) loss 3.1156 (3.1537) grad_norm 1.7776 (1.6346) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][560/625] eta 0:00:23 lr 0.000945 wd 0.0500 time 0.3511 (0.3549) data time 0.0007 (0.0015) model time 0.3504 (0.3524) loss 3.3173 (3.1554) grad_norm 1.5879 (1.6330) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][570/625] eta 0:00:19 lr 0.000944 wd 0.0500 time 0.3551 (0.3549) data time 0.0006 (0.0015) model time 0.3544 (0.3524) loss 3.1989 (3.1563) grad_norm 1.2399 (1.6325) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][580/625] eta 0:00:15 lr 0.000944 wd 0.0500 time 0.3499 (0.3548) data time 0.0007 (0.0015) model time 0.3492 (0.3524) loss 3.7206 (3.1538) grad_norm 1.5810 (1.6341) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][590/625] eta 0:00:12 lr 0.000944 wd 0.0500 time 0.3443 (0.3547) data time 0.0006 (0.0015) model time 0.3437 (0.3522) loss 3.3675 (3.1495) grad_norm 1.8338 (1.6321) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][600/625] eta 0:00:08 lr 0.000944 wd 0.0500 time 0.3478 (0.3546) data time 0.0009 (0.0015) model time 0.3469 (0.3522) loss 3.4939 (3.1509) grad_norm 2.6082 (1.6342) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][610/625] eta 0:00:05 lr 0.000944 wd 0.0500 time 0.4727 (0.3547) data time 0.0004 (0.0015) model time 0.4724 (0.3523) loss 3.2853 (3.1538) grad_norm 1.2003 (1.6298) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [165/300][620/625] eta 0:00:01 lr 0.000943 wd 0.0500 time 0.3422 (0.3544) data time 0.0005 (0.0015) model time 0.3417 (0.3520) loss 2.9562 (3.1580) grad_norm 1.5292 (1.6270) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:25:52 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 165 training takes 0:03:41 [2024-09-01 06:25:52 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:25:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:25:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.454 (0.454) Loss 0.5728 (0.5728) Acc@1 88.086 (88.086) Acc@5 98.291 (98.291) Mem 11224MB [2024-09-01 06:25:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 0.9297 (0.7057) Acc@1 78.125 (84.109) Acc@5 95.068 (97.030) Mem 11224MB [2024-09-01 06:25:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.113) Loss 1.0176 (0.8392) Acc@1 75.244 (80.757) Acc@5 94.043 (95.482) Mem 11224MB [2024-09-01 06:25:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.494 Acc@5 95.483 [2024-09-01 06:25:56 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.5% [2024-09-01 06:25:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.787 (0.787) Loss 0.5010 (0.5010) Acc@1 89.062 (89.062) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 06:25:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.163) Loss 0.8320 (0.6303) Acc@1 80.029 (85.764) Acc@5 96.094 (97.616) Mem 11224MB [2024-09-01 06:25:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.129) Loss 0.9482 (0.7517) Acc@1 76.709 (82.701) Acc@5 94.971 (96.405) Mem 11224MB [2024-09-01 06:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.452 Acc@5 96.409 [2024-09-01 06:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 06:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.45% [2024-09-01 06:25:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:26:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:26:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][0/625] eta 0:07:29 lr 0.000943 wd 0.0500 time 0.7185 (0.7185) data time 0.4097 (0.4097) model time 0.0000 (0.0000) loss 3.4601 (3.4601) grad_norm 1.5049 (1.5049) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:26:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][10/625] eta 0:03:55 lr 0.000943 wd 0.0500 time 0.3747 (0.3836) data time 0.0007 (0.0380) model time 0.0000 (0.0000) loss 2.9415 (3.0007) grad_norm 1.4046 (1.8731) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:26:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][20/625] eta 0:03:41 lr 0.000943 wd 0.0500 time 0.3500 (0.3663) data time 0.0009 (0.0203) model time 0.0000 (0.0000) loss 3.0211 (3.1256) grad_norm 1.7317 (1.7973) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:26:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][30/625] eta 0:03:34 lr 0.000943 wd 0.0500 time 0.3469 (0.3603) data time 0.0009 (0.0140) model time 0.0000 (0.0000) loss 3.6518 (3.1609) grad_norm 1.3621 (1.7310) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:26:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][40/625] eta 0:03:28 lr 0.000943 wd 0.0500 time 0.3501 (0.3571) data time 0.0008 (0.0108) model time 0.0000 (0.0000) loss 2.5614 (3.1404) grad_norm 1.2526 (1.6725) loss_scale 2048.0000 (1098.9268) mem 11224MB [2024-09-01 06:26:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][50/625] eta 0:03:24 lr 0.000943 wd 0.0500 time 0.3469 (0.3555) data time 0.0008 (0.0089) model time 0.0000 (0.0000) loss 3.0776 (3.0997) grad_norm 1.1567 (1.6214) loss_scale 2048.0000 (1285.0196) mem 11224MB [2024-09-01 06:26:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][60/625] eta 0:03:20 lr 0.000942 wd 0.0500 time 0.3522 (0.3543) data time 0.0009 (0.0075) model time 0.3513 (0.3473) loss 3.4175 (3.1198) grad_norm 2.4119 (inf) loss_scale 1024.0000 (1242.2295) mem 11224MB [2024-09-01 06:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][70/625] eta 0:03:21 lr 0.000942 wd 0.0500 time 0.5779 (0.3624) data time 0.0008 (0.0067) model time 0.5771 (0.3787) loss 3.5046 (3.1422) grad_norm 4.3496 (inf) loss_scale 1024.0000 (1211.4930) mem 11224MB [2024-09-01 06:26:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][80/625] eta 0:03:16 lr 0.000942 wd 0.0500 time 0.3477 (0.3607) data time 0.0006 (0.0060) model time 0.3471 (0.3685) loss 1.9617 (3.1181) grad_norm 1.4554 (inf) loss_scale 1024.0000 (1188.3457) mem 11224MB [2024-09-01 06:26:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][90/625] eta 0:03:12 lr 0.000942 wd 0.0500 time 0.3561 (0.3595) data time 0.0009 (0.0054) model time 0.3553 (0.3636) loss 3.6126 (3.1137) grad_norm 1.0967 (inf) loss_scale 1024.0000 (1170.2857) mem 11224MB [2024-09-01 06:26:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][100/625] eta 0:03:08 lr 0.000942 wd 0.0500 time 0.3476 (0.3584) data time 0.0006 (0.0049) model time 0.3470 (0.3603) loss 2.3240 (3.0918) grad_norm 1.7281 (inf) loss_scale 1024.0000 (1155.8020) mem 11224MB [2024-09-01 06:26:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][110/625] eta 0:03:04 lr 0.000941 wd 0.0500 time 0.3528 (0.3577) data time 0.0008 (0.0046) model time 0.3520 (0.3585) loss 3.6359 (3.0996) grad_norm 2.0777 (inf) loss_scale 1024.0000 (1143.9279) mem 11224MB [2024-09-01 06:26:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][120/625] eta 0:03:00 lr 0.000941 wd 0.0500 time 0.3536 (0.3570) data time 0.0008 (0.0043) model time 0.3528 (0.3571) loss 3.0564 (3.0932) grad_norm 1.6267 (inf) loss_scale 1024.0000 (1134.0165) mem 11224MB [2024-09-01 06:26:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][130/625] eta 0:02:57 lr 0.000941 wd 0.0500 time 0.3450 (0.3577) data time 0.0007 (0.0040) model time 0.3443 (0.3582) loss 2.2265 (3.1021) grad_norm 1.3036 (inf) loss_scale 1024.0000 (1125.6183) mem 11224MB [2024-09-01 06:26:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][140/625] eta 0:02:53 lr 0.000941 wd 0.0500 time 0.3437 (0.3570) data time 0.0010 (0.0038) model time 0.3427 (0.3569) loss 2.9292 (3.1117) grad_norm 1.1520 (inf) loss_scale 1024.0000 (1118.4113) mem 11224MB [2024-09-01 06:26:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][150/625] eta 0:02:49 lr 0.000941 wd 0.0500 time 0.3489 (0.3565) data time 0.0006 (0.0036) model time 0.3483 (0.3560) loss 2.3666 (3.1081) grad_norm 1.9753 (inf) loss_scale 1024.0000 (1112.1589) mem 11224MB [2024-09-01 06:26:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][160/625] eta 0:02:45 lr 0.000941 wd 0.0500 time 0.3417 (0.3559) data time 0.0006 (0.0034) model time 0.3411 (0.3552) loss 2.7275 (3.0966) grad_norm 1.2700 (inf) loss_scale 1024.0000 (1106.6832) mem 11224MB [2024-09-01 06:27:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][170/625] eta 0:02:41 lr 0.000940 wd 0.0500 time 0.3435 (0.3555) data time 0.0009 (0.0033) model time 0.3427 (0.3545) loss 3.2416 (3.0977) grad_norm 1.1561 (inf) loss_scale 1024.0000 (1101.8480) mem 11224MB [2024-09-01 06:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][180/625] eta 0:02:38 lr 0.000940 wd 0.0500 time 0.3564 (0.3552) data time 0.0008 (0.0031) model time 0.3556 (0.3542) loss 1.7892 (3.0922) grad_norm 1.3242 (inf) loss_scale 1024.0000 (1097.5470) mem 11224MB [2024-09-01 06:27:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][190/625] eta 0:02:34 lr 0.000940 wd 0.0500 time 0.3538 (0.3549) data time 0.0008 (0.0030) model time 0.3529 (0.3539) loss 2.7744 (3.0997) grad_norm 1.8895 (inf) loss_scale 1024.0000 (1093.6963) mem 11224MB [2024-09-01 06:27:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][200/625] eta 0:02:30 lr 0.000940 wd 0.0500 time 0.3433 (0.3546) data time 0.0006 (0.0029) model time 0.3427 (0.3534) loss 2.8843 (3.0927) grad_norm 1.8031 (inf) loss_scale 1024.0000 (1090.2289) mem 11224MB [2024-09-01 06:27:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][210/625] eta 0:02:26 lr 0.000940 wd 0.0500 time 0.3488 (0.3542) data time 0.0009 (0.0028) model time 0.3480 (0.3529) loss 2.3040 (3.0966) grad_norm 1.3605 (inf) loss_scale 1024.0000 (1087.0900) mem 11224MB [2024-09-01 06:27:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][220/625] eta 0:02:23 lr 0.000940 wd 0.0500 time 0.3491 (0.3539) data time 0.0008 (0.0027) model time 0.3484 (0.3526) loss 3.3203 (3.1063) grad_norm 1.2538 (inf) loss_scale 1024.0000 (1084.2353) mem 11224MB [2024-09-01 06:27:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][230/625] eta 0:02:19 lr 0.000939 wd 0.0500 time 0.3463 (0.3536) data time 0.0009 (0.0026) model time 0.3454 (0.3522) loss 2.8909 (3.1082) grad_norm 1.7593 (inf) loss_scale 1024.0000 (1081.6277) mem 11224MB [2024-09-01 06:27:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][240/625] eta 0:02:16 lr 0.000939 wd 0.0500 time 0.3438 (0.3534) data time 0.0007 (0.0026) model time 0.3431 (0.3520) loss 2.5875 (3.1044) grad_norm 1.2232 (inf) loss_scale 1024.0000 (1079.2365) mem 11224MB [2024-09-01 06:27:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][250/625] eta 0:02:12 lr 0.000939 wd 0.0500 time 0.3536 (0.3532) data time 0.0006 (0.0025) model time 0.3530 (0.3518) loss 3.6474 (3.1065) grad_norm 1.5390 (inf) loss_scale 1024.0000 (1077.0359) mem 11224MB [2024-09-01 06:27:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][260/625] eta 0:02:08 lr 0.000939 wd 0.0500 time 0.3456 (0.3531) data time 0.0008 (0.0024) model time 0.3448 (0.3516) loss 3.3391 (3.1165) grad_norm 1.9555 (inf) loss_scale 1024.0000 (1075.0038) mem 11224MB [2024-09-01 06:27:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][270/625] eta 0:02:05 lr 0.000939 wd 0.0500 time 0.3467 (0.3529) data time 0.0008 (0.0024) model time 0.3460 (0.3514) loss 2.5153 (3.1167) grad_norm 1.2751 (inf) loss_scale 1024.0000 (1073.1218) mem 11224MB [2024-09-01 06:27:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][280/625] eta 0:02:02 lr 0.000938 wd 0.0500 time 0.3466 (0.3543) data time 0.0006 (0.0023) model time 0.3460 (0.3532) loss 3.8623 (3.1173) grad_norm 1.4406 (inf) loss_scale 1024.0000 (1071.3737) mem 11224MB [2024-09-01 06:27:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][290/625] eta 0:01:58 lr 0.000938 wd 0.0500 time 0.3506 (0.3542) data time 0.0006 (0.0023) model time 0.3500 (0.3530) loss 3.8259 (3.1310) grad_norm 1.7014 (inf) loss_scale 1024.0000 (1069.7457) mem 11224MB [2024-09-01 06:27:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][300/625] eta 0:01:55 lr 0.000938 wd 0.0500 time 0.3574 (0.3541) data time 0.0006 (0.0022) model time 0.3567 (0.3529) loss 3.1176 (3.1313) grad_norm 1.5565 (inf) loss_scale 1024.0000 (1068.2259) mem 11224MB [2024-09-01 06:27:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][310/625] eta 0:01:51 lr 0.000938 wd 0.0500 time 0.5553 (0.3546) data time 0.0007 (0.0022) model time 0.5546 (0.3535) loss 3.6147 (3.1182) grad_norm 1.5729 (inf) loss_scale 1024.0000 (1066.8039) mem 11224MB [2024-09-01 06:27:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][320/625] eta 0:01:48 lr 0.000938 wd 0.0500 time 0.3507 (0.3543) data time 0.0006 (0.0021) model time 0.3501 (0.3532) loss 2.2462 (3.1191) grad_norm 1.4263 (inf) loss_scale 1024.0000 (1065.4704) mem 11224MB [2024-09-01 06:27:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][330/625] eta 0:01:44 lr 0.000938 wd 0.0500 time 0.3495 (0.3541) data time 0.0007 (0.0021) model time 0.3489 (0.3530) loss 3.0113 (3.1215) grad_norm 2.1740 (inf) loss_scale 1024.0000 (1064.2175) mem 11224MB [2024-09-01 06:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][340/625] eta 0:01:40 lr 0.000937 wd 0.0500 time 0.3446 (0.3540) data time 0.0008 (0.0021) model time 0.3438 (0.3528) loss 3.2271 (3.1210) grad_norm 1.3602 (inf) loss_scale 1024.0000 (1063.0381) mem 11224MB [2024-09-01 06:28:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][350/625] eta 0:01:37 lr 0.000937 wd 0.0500 time 0.3482 (0.3538) data time 0.0007 (0.0020) model time 0.3475 (0.3526) loss 3.0004 (3.1196) grad_norm 1.7101 (inf) loss_scale 1024.0000 (1061.9259) mem 11224MB [2024-09-01 06:28:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][360/625] eta 0:01:33 lr 0.000937 wd 0.0500 time 0.3454 (0.3537) data time 0.0008 (0.0020) model time 0.3446 (0.3525) loss 3.3902 (3.1179) grad_norm 1.9697 (inf) loss_scale 1024.0000 (1060.8753) mem 11224MB [2024-09-01 06:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][370/625] eta 0:01:30 lr 0.000937 wd 0.0500 time 0.3608 (0.3536) data time 0.0008 (0.0020) model time 0.3599 (0.3524) loss 3.3168 (3.1195) grad_norm 1.4955 (inf) loss_scale 1024.0000 (1059.8814) mem 11224MB [2024-09-01 06:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][380/625] eta 0:01:26 lr 0.000937 wd 0.0500 time 0.3492 (0.3534) data time 0.0006 (0.0019) model time 0.3486 (0.3522) loss 3.9935 (3.1262) grad_norm 2.2665 (inf) loss_scale 1024.0000 (1058.9396) mem 11224MB [2024-09-01 06:28:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][390/625] eta 0:01:23 lr 0.000936 wd 0.0500 time 0.3512 (0.3533) data time 0.0008 (0.0019) model time 0.3504 (0.3521) loss 3.6962 (3.1259) grad_norm 1.2230 (inf) loss_scale 1024.0000 (1058.0460) mem 11224MB [2024-09-01 06:28:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][400/625] eta 0:01:19 lr 0.000936 wd 0.0500 time 0.3427 (0.3532) data time 0.0010 (0.0019) model time 0.3418 (0.3519) loss 2.5726 (3.1263) grad_norm 1.2776 (inf) loss_scale 1024.0000 (1057.1970) mem 11224MB [2024-09-01 06:28:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][410/625] eta 0:01:15 lr 0.000936 wd 0.0500 time 0.3432 (0.3530) data time 0.0009 (0.0019) model time 0.3424 (0.3518) loss 3.2360 (3.1298) grad_norm 1.5071 (inf) loss_scale 1024.0000 (1056.3893) mem 11224MB [2024-09-01 06:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][420/625] eta 0:01:12 lr 0.000936 wd 0.0500 time 0.3494 (0.3529) data time 0.0007 (0.0018) model time 0.3487 (0.3516) loss 3.4328 (3.1302) grad_norm 1.1908 (inf) loss_scale 1024.0000 (1055.6200) mem 11224MB [2024-09-01 06:28:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][430/625] eta 0:01:08 lr 0.000936 wd 0.0500 time 0.3105 (0.3532) data time 0.0008 (0.0018) model time 0.3097 (0.3520) loss 3.7331 (3.1276) grad_norm 1.5096 (inf) loss_scale 1024.0000 (1054.8863) mem 11224MB [2024-09-01 06:28:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][440/625] eta 0:01:05 lr 0.000936 wd 0.0500 time 0.3501 (0.3531) data time 0.0008 (0.0018) model time 0.3494 (0.3519) loss 3.4098 (3.1348) grad_norm 1.8590 (inf) loss_scale 1024.0000 (1054.1859) mem 11224MB [2024-09-01 06:28:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][450/625] eta 0:01:01 lr 0.000935 wd 0.0500 time 0.3528 (0.3530) data time 0.0007 (0.0018) model time 0.3520 (0.3518) loss 3.8363 (3.1357) grad_norm 1.5140 (inf) loss_scale 1024.0000 (1053.5166) mem 11224MB [2024-09-01 06:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][460/625] eta 0:00:58 lr 0.000935 wd 0.0500 time 0.3481 (0.3529) data time 0.0006 (0.0018) model time 0.3475 (0.3517) loss 3.1620 (3.1413) grad_norm 1.1287 (inf) loss_scale 1024.0000 (1052.8764) mem 11224MB [2024-09-01 06:28:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][470/625] eta 0:00:54 lr 0.000935 wd 0.0500 time 0.3511 (0.3540) data time 0.0007 (0.0017) model time 0.3504 (0.3529) loss 3.7603 (3.1412) grad_norm 2.4951 (inf) loss_scale 1024.0000 (1052.2633) mem 11224MB [2024-09-01 06:28:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][480/625] eta 0:00:51 lr 0.000935 wd 0.0500 time 0.3437 (0.3539) data time 0.0006 (0.0017) model time 0.3431 (0.3528) loss 1.8740 (3.1438) grad_norm 1.6217 (inf) loss_scale 1024.0000 (1051.6757) mem 11224MB [2024-09-01 06:28:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][490/625] eta 0:00:47 lr 0.000935 wd 0.0500 time 0.3541 (0.3538) data time 0.0009 (0.0017) model time 0.3532 (0.3527) loss 3.4615 (3.1476) grad_norm 1.5470 (inf) loss_scale 1024.0000 (1051.1120) mem 11224MB [2024-09-01 06:28:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][500/625] eta 0:00:44 lr 0.000935 wd 0.0500 time 0.3438 (0.3537) data time 0.0007 (0.0017) model time 0.3431 (0.3527) loss 3.8378 (3.1482) grad_norm 1.4313 (inf) loss_scale 1024.0000 (1050.5709) mem 11224MB [2024-09-01 06:29:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][510/625] eta 0:00:40 lr 0.000934 wd 0.0500 time 0.3585 (0.3537) data time 0.0015 (0.0017) model time 0.3570 (0.3526) loss 2.9073 (3.1512) grad_norm 1.8334 (inf) loss_scale 1024.0000 (1050.0509) mem 11224MB [2024-09-01 06:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][520/625] eta 0:00:37 lr 0.000934 wd 0.0500 time 0.3524 (0.3536) data time 0.0008 (0.0017) model time 0.3516 (0.3525) loss 3.3697 (3.1569) grad_norm 1.4422 (inf) loss_scale 1024.0000 (1049.5509) mem 11224MB [2024-09-01 06:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][530/625] eta 0:00:33 lr 0.000934 wd 0.0500 time 0.3445 (0.3535) data time 0.0008 (0.0016) model time 0.3438 (0.3524) loss 3.7325 (3.1555) grad_norm 1.6198 (inf) loss_scale 1024.0000 (1049.0697) mem 11224MB [2024-09-01 06:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][540/625] eta 0:00:30 lr 0.000934 wd 0.0500 time 0.3478 (0.3534) data time 0.0007 (0.0016) model time 0.3470 (0.3523) loss 2.7354 (3.1559) grad_norm 1.2091 (inf) loss_scale 1024.0000 (1048.6063) mem 11224MB [2024-09-01 06:29:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][550/625] eta 0:00:26 lr 0.000934 wd 0.0500 time 0.3499 (0.3533) data time 0.0009 (0.0016) model time 0.3490 (0.3522) loss 3.0608 (3.1529) grad_norm 2.4867 (inf) loss_scale 1024.0000 (1048.1597) mem 11224MB [2024-09-01 06:29:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][560/625] eta 0:00:22 lr 0.000933 wd 0.0500 time 0.3436 (0.3532) data time 0.0006 (0.0016) model time 0.3430 (0.3521) loss 2.2713 (3.1538) grad_norm 1.6527 (inf) loss_scale 1024.0000 (1047.7291) mem 11224MB [2024-09-01 06:29:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][570/625] eta 0:00:19 lr 0.000933 wd 0.0500 time 0.3496 (0.3532) data time 0.0008 (0.0016) model time 0.3488 (0.3521) loss 2.8909 (3.1527) grad_norm 4.8529 (inf) loss_scale 1024.0000 (1047.3135) mem 11224MB [2024-09-01 06:29:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][580/625] eta 0:00:15 lr 0.000933 wd 0.0500 time 0.3517 (0.3531) data time 0.0006 (0.0016) model time 0.3511 (0.3520) loss 3.6361 (3.1528) grad_norm 1.3536 (inf) loss_scale 1024.0000 (1046.9122) mem 11224MB [2024-09-01 06:29:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][590/625] eta 0:00:12 lr 0.000933 wd 0.0500 time 0.3484 (0.3537) data time 0.0008 (0.0016) model time 0.3476 (0.3526) loss 3.1593 (3.1493) grad_norm 1.5624 (inf) loss_scale 1024.0000 (1046.5245) mem 11224MB [2024-09-01 06:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][600/625] eta 0:00:08 lr 0.000933 wd 0.0500 time 0.3437 (0.3536) data time 0.0007 (0.0015) model time 0.3430 (0.3525) loss 3.0400 (3.1493) grad_norm 2.3451 (inf) loss_scale 1024.0000 (1046.1498) mem 11224MB [2024-09-01 06:29:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][610/625] eta 0:00:05 lr 0.000933 wd 0.0500 time 0.3467 (0.3535) data time 0.0004 (0.0015) model time 0.3463 (0.3524) loss 3.5666 (3.1511) grad_norm 1.3202 (inf) loss_scale 1024.0000 (1045.7872) mem 11224MB [2024-09-01 06:29:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [166/300][620/625] eta 0:00:01 lr 0.000932 wd 0.0500 time 0.3401 (0.3533) data time 0.0006 (0.0015) model time 0.3395 (0.3523) loss 3.7093 (3.1522) grad_norm 1.3426 (inf) loss_scale 1024.0000 (1045.4364) mem 11224MB [2024-09-01 06:29:41 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 166 training takes 0:03:40 [2024-09-01 06:29:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:29:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:29:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.432 (0.432) Loss 0.5996 (0.5996) Acc@1 87.500 (87.500) Acc@5 98.389 (98.389) Mem 11224MB [2024-09-01 06:29:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.128) Loss 0.9971 (0.7301) Acc@1 76.416 (84.073) Acc@5 94.727 (97.097) Mem 11224MB [2024-09-01 06:29:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.111) Loss 1.0352 (0.8735) Acc@1 76.318 (80.685) Acc@5 93.896 (95.480) Mem 11224MB [2024-09-01 06:29:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.494 Acc@5 95.487 [2024-09-01 06:29:44 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.5% [2024-09-01 06:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.717 (0.717) Loss 0.5005 (0.5005) Acc@1 88.965 (88.965) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 06:29:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.160) Loss 0.8311 (0.6295) Acc@1 79.980 (85.795) Acc@5 96.045 (97.630) Mem 11224MB [2024-09-01 06:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.128) Loss 0.9473 (0.7503) Acc@1 76.758 (82.692) Acc@5 95.020 (96.429) Mem 11224MB [2024-09-01 06:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.440 Acc@5 96.431 [2024-09-01 06:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.4% [2024-09-01 06:29:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][0/625] eta 0:12:59 lr 0.000932 wd 0.0500 time 1.2473 (1.2473) data time 0.5041 (0.5041) model time 0.0000 (0.0000) loss 3.1518 (3.1518) grad_norm 1.7333 (1.7333) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:29:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][10/625] eta 0:04:24 lr 0.000932 wd 0.0500 time 0.3470 (0.4308) data time 0.0007 (0.0466) model time 0.0000 (0.0000) loss 2.1767 (3.2197) grad_norm 1.1323 (1.3332) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:29:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][20/625] eta 0:03:56 lr 0.000932 wd 0.0500 time 0.3402 (0.3908) data time 0.0006 (0.0247) model time 0.0000 (0.0000) loss 3.5736 (3.0806) grad_norm 1.1600 (1.4761) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:29:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][30/625] eta 0:03:44 lr 0.000932 wd 0.0500 time 0.3473 (0.3773) data time 0.0005 (0.0170) model time 0.0000 (0.0000) loss 2.6639 (3.1070) grad_norm 3.2304 (1.6304) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][40/625] eta 0:03:36 lr 0.000932 wd 0.0500 time 0.3459 (0.3699) data time 0.0007 (0.0131) model time 0.0000 (0.0000) loss 1.8842 (3.0464) grad_norm 1.1802 (1.6352) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][50/625] eta 0:03:30 lr 0.000931 wd 0.0500 time 0.3512 (0.3654) data time 0.0008 (0.0107) model time 0.0000 (0.0000) loss 2.4289 (3.0510) grad_norm 1.2734 (1.6146) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][60/625] eta 0:03:24 lr 0.000931 wd 0.0500 time 0.3480 (0.3626) data time 0.0008 (0.0091) model time 0.3472 (0.3474) loss 4.0869 (3.0351) grad_norm 1.5234 (1.6070) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][70/625] eta 0:03:20 lr 0.000931 wd 0.0500 time 0.3468 (0.3608) data time 0.0009 (0.0079) model time 0.3459 (0.3484) loss 3.1893 (3.0273) grad_norm 1.6252 (1.6009) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][80/625] eta 0:03:15 lr 0.000931 wd 0.0500 time 0.3454 (0.3592) data time 0.0008 (0.0070) model time 0.3446 (0.3477) loss 3.0402 (3.0215) grad_norm 1.3508 (1.6249) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][90/625] eta 0:03:11 lr 0.000931 wd 0.0500 time 0.3431 (0.3577) data time 0.0006 (0.0063) model time 0.3425 (0.3470) loss 3.6058 (3.0202) grad_norm 1.4976 (1.6285) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][100/625] eta 0:03:09 lr 0.000931 wd 0.0500 time 0.3404 (0.3604) data time 0.0009 (0.0058) model time 0.3395 (0.3545) loss 2.3788 (3.0452) grad_norm 1.6324 (1.6222) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][110/625] eta 0:03:05 lr 0.000930 wd 0.0500 time 0.3620 (0.3595) data time 0.0008 (0.0053) model time 0.3612 (0.3537) loss 3.3717 (3.0576) grad_norm 1.5505 (1.6432) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][120/625] eta 0:03:01 lr 0.000930 wd 0.0500 time 0.3490 (0.3585) data time 0.0008 (0.0050) model time 0.3482 (0.3526) loss 3.3464 (3.0491) grad_norm 1.5033 (1.6416) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][130/625] eta 0:02:56 lr 0.000930 wd 0.0500 time 0.3446 (0.3576) data time 0.0007 (0.0047) model time 0.3440 (0.3517) loss 3.6573 (3.0802) grad_norm 1.6288 (1.6379) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][140/625] eta 0:02:53 lr 0.000930 wd 0.0500 time 0.3467 (0.3570) data time 0.0007 (0.0044) model time 0.3460 (0.3513) loss 2.1271 (3.0774) grad_norm 2.3564 (1.6430) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][150/625] eta 0:02:49 lr 0.000930 wd 0.0500 time 0.3486 (0.3564) data time 0.0005 (0.0042) model time 0.3481 (0.3509) loss 3.7364 (3.0914) grad_norm 1.4191 (1.6750) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][160/625] eta 0:02:45 lr 0.000929 wd 0.0500 time 0.3523 (0.3557) data time 0.0009 (0.0039) model time 0.3514 (0.3504) loss 3.3109 (3.0851) grad_norm 1.6201 (1.6587) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][170/625] eta 0:02:41 lr 0.000929 wd 0.0500 time 0.3386 (0.3552) data time 0.0009 (0.0038) model time 0.3377 (0.3500) loss 3.2910 (3.1113) grad_norm 1.6147 (1.6644) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][180/625] eta 0:02:38 lr 0.000929 wd 0.0500 time 0.3471 (0.3559) data time 0.0008 (0.0036) model time 0.3463 (0.3513) loss 2.3161 (3.1239) grad_norm 1.6777 (1.6485) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][190/625] eta 0:02:34 lr 0.000929 wd 0.0500 time 0.3441 (0.3556) data time 0.0008 (0.0035) model time 0.3434 (0.3511) loss 3.0832 (3.1288) grad_norm 1.6363 (1.6605) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:30:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][200/625] eta 0:02:30 lr 0.000929 wd 0.0500 time 0.3505 (0.3552) data time 0.0009 (0.0034) model time 0.3496 (0.3509) loss 3.1001 (3.1221) grad_norm 1.4776 (1.6538) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][210/625] eta 0:02:27 lr 0.000929 wd 0.0500 time 0.3469 (0.3550) data time 0.0006 (0.0032) model time 0.3462 (0.3508) loss 3.5286 (3.1344) grad_norm 1.8703 (1.6481) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][220/625] eta 0:02:23 lr 0.000928 wd 0.0500 time 0.3483 (0.3546) data time 0.0006 (0.0031) model time 0.3476 (0.3506) loss 2.7149 (3.1174) grad_norm 1.9347 (1.6546) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][230/625] eta 0:02:19 lr 0.000928 wd 0.0500 time 0.3455 (0.3543) data time 0.0006 (0.0030) model time 0.3449 (0.3503) loss 3.6209 (3.1183) grad_norm 1.6597 (1.6525) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][240/625] eta 0:02:16 lr 0.000928 wd 0.0500 time 0.3492 (0.3541) data time 0.0006 (0.0029) model time 0.3486 (0.3502) loss 3.7916 (3.1230) grad_norm 1.9261 (1.6536) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][250/625] eta 0:02:12 lr 0.000928 wd 0.0500 time 0.3597 (0.3539) data time 0.0009 (0.0028) model time 0.3589 (0.3501) loss 3.3597 (3.1225) grad_norm 1.3180 (1.6492) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][260/625] eta 0:02:09 lr 0.000928 wd 0.0500 time 0.3503 (0.3537) data time 0.0010 (0.0028) model time 0.3494 (0.3500) loss 3.5059 (3.1248) grad_norm 1.4100 (1.6415) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][270/625] eta 0:02:05 lr 0.000928 wd 0.0500 time 0.3472 (0.3534) data time 0.0006 (0.0027) model time 0.3466 (0.3498) loss 2.1935 (3.1155) grad_norm 1.0537 (1.6405) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][280/625] eta 0:02:01 lr 0.000927 wd 0.0500 time 0.3512 (0.3533) data time 0.0008 (0.0026) model time 0.3504 (0.3498) loss 3.5722 (3.1177) grad_norm 2.1764 (1.6452) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][290/625] eta 0:01:58 lr 0.000927 wd 0.0500 time 0.3508 (0.3531) data time 0.0009 (0.0026) model time 0.3500 (0.3497) loss 2.1776 (3.1093) grad_norm 1.4115 (1.6410) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][300/625] eta 0:01:54 lr 0.000927 wd 0.0500 time 0.3506 (0.3535) data time 0.0008 (0.0025) model time 0.3499 (0.3502) loss 2.5090 (3.1189) grad_norm 1.5097 (1.6357) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:31:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][310/625] eta 0:01:51 lr 0.000927 wd 0.0500 time 0.3442 (0.3540) data time 0.0007 (0.0025) model time 0.3434 (0.3509) loss 2.8242 (3.1117) grad_norm 2.6343 (inf) loss_scale 512.0000 (1020.7074) mem 11224MB [2024-09-01 06:31:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][320/625] eta 0:01:47 lr 0.000927 wd 0.0500 time 0.3463 (0.3538) data time 0.0006 (0.0024) model time 0.3457 (0.3507) loss 3.5453 (3.1146) grad_norm 1.3419 (inf) loss_scale 512.0000 (1004.8598) mem 11224MB [2024-09-01 06:31:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][330/625] eta 0:01:44 lr 0.000926 wd 0.0500 time 0.3418 (0.3537) data time 0.0007 (0.0024) model time 0.3411 (0.3507) loss 3.3952 (3.1216) grad_norm 1.7749 (inf) loss_scale 512.0000 (989.9698) mem 11224MB [2024-09-01 06:31:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][340/625] eta 0:01:40 lr 0.000926 wd 0.0500 time 0.3469 (0.3535) data time 0.0007 (0.0023) model time 0.3462 (0.3506) loss 3.7398 (3.1225) grad_norm 1.6178 (inf) loss_scale 512.0000 (975.9531) mem 11224MB [2024-09-01 06:31:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][350/625] eta 0:01:37 lr 0.000926 wd 0.0500 time 0.3451 (0.3533) data time 0.0005 (0.0023) model time 0.3446 (0.3504) loss 3.0620 (3.1265) grad_norm 1.2217 (inf) loss_scale 512.0000 (962.7350) mem 11224MB [2024-09-01 06:31:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][360/625] eta 0:01:33 lr 0.000926 wd 0.0500 time 0.3516 (0.3532) data time 0.0009 (0.0022) model time 0.3507 (0.3503) loss 3.1392 (3.1296) grad_norm 1.3983 (inf) loss_scale 512.0000 (950.2493) mem 11224MB [2024-09-01 06:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][370/625] eta 0:01:30 lr 0.000926 wd 0.0500 time 0.3482 (0.3531) data time 0.0009 (0.0022) model time 0.3473 (0.3503) loss 3.5218 (3.1295) grad_norm 1.3041 (inf) loss_scale 512.0000 (938.4367) mem 11224MB [2024-09-01 06:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][380/625] eta 0:01:26 lr 0.000926 wd 0.0500 time 0.3580 (0.3530) data time 0.0008 (0.0022) model time 0.3572 (0.3502) loss 3.3458 (3.1268) grad_norm 1.5999 (inf) loss_scale 512.0000 (927.2441) mem 11224MB [2024-09-01 06:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][390/625] eta 0:01:22 lr 0.000925 wd 0.0500 time 0.3477 (0.3529) data time 0.0007 (0.0021) model time 0.3470 (0.3501) loss 2.8792 (3.1256) grad_norm 2.0183 (inf) loss_scale 512.0000 (916.6240) mem 11224MB [2024-09-01 06:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][400/625] eta 0:01:19 lr 0.000925 wd 0.0500 time 0.3493 (0.3527) data time 0.0009 (0.0021) model time 0.3484 (0.3500) loss 3.3950 (3.1319) grad_norm 1.6184 (inf) loss_scale 512.0000 (906.5337) mem 11224MB [2024-09-01 06:32:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][410/625] eta 0:01:15 lr 0.000925 wd 0.0500 time 0.3441 (0.3526) data time 0.0007 (0.0021) model time 0.3434 (0.3499) loss 3.5929 (3.1353) grad_norm 1.0492 (inf) loss_scale 512.0000 (896.9343) mem 11224MB [2024-09-01 06:32:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][420/625] eta 0:01:12 lr 0.000925 wd 0.0500 time 0.3469 (0.3525) data time 0.0009 (0.0020) model time 0.3460 (0.3499) loss 2.4896 (3.1327) grad_norm 1.3415 (inf) loss_scale 512.0000 (887.7910) mem 11224MB [2024-09-01 06:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][430/625] eta 0:01:08 lr 0.000925 wd 0.0500 time 0.3460 (0.3524) data time 0.0006 (0.0020) model time 0.3454 (0.3498) loss 3.4484 (3.1392) grad_norm 1.0476 (inf) loss_scale 512.0000 (879.0719) mem 11224MB [2024-09-01 06:32:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][440/625] eta 0:01:05 lr 0.000925 wd 0.0500 time 0.3467 (0.3523) data time 0.0008 (0.0020) model time 0.3459 (0.3497) loss 2.4044 (3.1393) grad_norm 2.0093 (inf) loss_scale 512.0000 (870.7483) mem 11224MB [2024-09-01 06:32:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][450/625] eta 0:01:01 lr 0.000924 wd 0.0500 time 0.3479 (0.3522) data time 0.0009 (0.0020) model time 0.3470 (0.3496) loss 3.4556 (3.1431) grad_norm 1.3963 (inf) loss_scale 512.0000 (862.7938) mem 11224MB [2024-09-01 06:32:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][460/625] eta 0:00:58 lr 0.000924 wd 0.0500 time 0.3470 (0.3521) data time 0.0006 (0.0019) model time 0.3464 (0.3496) loss 3.2736 (3.1453) grad_norm 1.4587 (inf) loss_scale 512.0000 (855.1844) mem 11224MB [2024-09-01 06:32:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][470/625] eta 0:00:54 lr 0.000924 wd 0.0500 time 0.3445 (0.3520) data time 0.0009 (0.0019) model time 0.3436 (0.3495) loss 3.6867 (3.1474) grad_norm 1.6963 (inf) loss_scale 512.0000 (847.8981) mem 11224MB [2024-09-01 06:32:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][480/625] eta 0:00:51 lr 0.000924 wd 0.0500 time 0.3471 (0.3519) data time 0.0007 (0.0019) model time 0.3464 (0.3494) loss 2.4166 (3.1523) grad_norm 1.5444 (inf) loss_scale 512.0000 (840.9148) mem 11224MB [2024-09-01 06:32:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][490/625] eta 0:00:47 lr 0.000924 wd 0.0500 time 0.3468 (0.3527) data time 0.0008 (0.0019) model time 0.3460 (0.3503) loss 3.4303 (3.1542) grad_norm 1.7441 (inf) loss_scale 512.0000 (834.2159) mem 11224MB [2024-09-01 06:32:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][500/625] eta 0:00:44 lr 0.000923 wd 0.0500 time 0.3514 (0.3526) data time 0.0009 (0.0019) model time 0.3506 (0.3503) loss 3.2227 (3.1564) grad_norm 2.0311 (inf) loss_scale 512.0000 (827.7844) mem 11224MB [2024-09-01 06:32:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][510/625] eta 0:00:40 lr 0.000923 wd 0.0500 time 0.3463 (0.3529) data time 0.0008 (0.0018) model time 0.3455 (0.3506) loss 3.4649 (3.1525) grad_norm 1.3844 (inf) loss_scale 512.0000 (821.6047) mem 11224MB [2024-09-01 06:32:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][520/625] eta 0:00:37 lr 0.000923 wd 0.0500 time 0.3489 (0.3527) data time 0.0006 (0.0018) model time 0.3483 (0.3505) loss 1.9215 (3.1504) grad_norm 1.2918 (inf) loss_scale 512.0000 (815.6622) mem 11224MB [2024-09-01 06:32:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][530/625] eta 0:00:33 lr 0.000923 wd 0.0500 time 0.3492 (0.3527) data time 0.0006 (0.0018) model time 0.3486 (0.3505) loss 3.3561 (3.1497) grad_norm 1.7621 (inf) loss_scale 512.0000 (809.9435) mem 11224MB [2024-09-01 06:32:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][540/625] eta 0:00:29 lr 0.000923 wd 0.0500 time 0.3530 (0.3526) data time 0.0007 (0.0018) model time 0.3524 (0.3504) loss 3.8538 (3.1529) grad_norm 1.8898 (inf) loss_scale 512.0000 (804.4362) mem 11224MB [2024-09-01 06:33:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][550/625] eta 0:00:26 lr 0.000923 wd 0.0500 time 0.3488 (0.3526) data time 0.0006 (0.0018) model time 0.3482 (0.3504) loss 3.2750 (3.1485) grad_norm 1.1184 (inf) loss_scale 512.0000 (799.1289) mem 11224MB [2024-09-01 06:33:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][560/625] eta 0:00:22 lr 0.000922 wd 0.0500 time 0.3510 (0.3526) data time 0.0006 (0.0018) model time 0.3504 (0.3504) loss 3.5277 (3.1480) grad_norm 1.2974 (inf) loss_scale 512.0000 (794.0107) mem 11224MB [2024-09-01 06:33:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][570/625] eta 0:00:19 lr 0.000922 wd 0.0500 time 0.3550 (0.3525) data time 0.0009 (0.0017) model time 0.3541 (0.3504) loss 3.7737 (3.1491) grad_norm 1.3944 (inf) loss_scale 512.0000 (789.0718) mem 11224MB [2024-09-01 06:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][580/625] eta 0:00:15 lr 0.000922 wd 0.0500 time 0.3414 (0.3525) data time 0.0007 (0.0017) model time 0.3407 (0.3503) loss 3.3165 (3.1443) grad_norm 2.2825 (inf) loss_scale 512.0000 (784.3029) mem 11224MB [2024-09-01 06:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][590/625] eta 0:00:12 lr 0.000922 wd 0.0500 time 0.3427 (0.3524) data time 0.0009 (0.0017) model time 0.3418 (0.3503) loss 3.3015 (3.1470) grad_norm 1.4649 (inf) loss_scale 512.0000 (779.6954) mem 11224MB [2024-09-01 06:33:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][600/625] eta 0:00:08 lr 0.000922 wd 0.0500 time 0.3487 (0.3523) data time 0.0009 (0.0017) model time 0.3478 (0.3503) loss 2.9575 (3.1488) grad_norm 1.4585 (inf) loss_scale 512.0000 (775.2413) mem 11224MB [2024-09-01 06:33:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][610/625] eta 0:00:05 lr 0.000922 wd 0.0500 time 0.3366 (0.3529) data time 0.0006 (0.0017) model time 0.3360 (0.3509) loss 3.4697 (3.1504) grad_norm 1.8998 (inf) loss_scale 512.0000 (770.9329) mem 11224MB [2024-09-01 06:33:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [167/300][620/625] eta 0:00:01 lr 0.000921 wd 0.0500 time 0.3504 (0.3527) data time 0.0004 (0.0017) model time 0.3500 (0.3507) loss 2.8815 (3.1459) grad_norm 1.5502 (inf) loss_scale 512.0000 (766.7633) mem 11224MB [2024-09-01 06:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 167 training takes 0:03:40 [2024-09-01 06:33:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:33:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:33:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.465 (0.465) Loss 0.5845 (0.5845) Acc@1 88.330 (88.330) Acc@5 97.852 (97.852) Mem 11224MB [2024-09-01 06:33:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.132) Loss 0.9517 (0.7168) Acc@1 77.783 (84.295) Acc@5 95.264 (97.115) Mem 11224MB [2024-09-01 06:33:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 1.0352 (0.8456) Acc@1 74.951 (81.055) Acc@5 93.701 (95.573) Mem 11224MB [2024-09-01 06:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.826 Acc@5 95.579 [2024-09-01 06:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.8% [2024-09-01 06:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 80.83% [2024-09-01 06:33:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 06:33:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 06:33:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.483 (0.483) Loss 0.5015 (0.5015) Acc@1 89.014 (89.014) Acc@5 98.730 (98.730) Mem 11224MB [2024-09-01 06:33:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.133) Loss 0.8315 (0.6292) Acc@1 79.834 (85.840) Acc@5 95.947 (97.621) Mem 11224MB [2024-09-01 06:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 0.9492 (0.7496) Acc@1 76.758 (82.726) Acc@5 95.020 (96.438) Mem 11224MB [2024-09-01 06:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.486 Acc@5 96.441 [2024-09-01 06:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 06:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.49% [2024-09-01 06:33:35 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:33:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:33:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][0/625] eta 0:10:13 lr 0.000921 wd 0.0500 time 0.9808 (0.9808) data time 0.6654 (0.6654) model time 0.0000 (0.0000) loss 2.6322 (2.6322) grad_norm 1.6380 (1.6380) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:33:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][10/625] eta 0:04:09 lr 0.000921 wd 0.0500 time 0.3497 (0.4065) data time 0.0009 (0.0613) model time 0.0000 (0.0000) loss 3.2926 (3.0021) grad_norm 1.5008 (1.4944) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:33:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][20/625] eta 0:03:49 lr 0.000921 wd 0.0500 time 0.3460 (0.3789) data time 0.0006 (0.0325) model time 0.0000 (0.0000) loss 2.8990 (3.0524) grad_norm 1.7959 (1.5034) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:33:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][30/625] eta 0:03:40 lr 0.000921 wd 0.0500 time 0.3506 (0.3703) data time 0.0006 (0.0223) model time 0.0000 (0.0000) loss 3.7399 (3.1362) grad_norm 2.7801 (1.6523) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:33:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][40/625] eta 0:03:33 lr 0.000921 wd 0.0500 time 0.3489 (0.3646) data time 0.0009 (0.0170) model time 0.0000 (0.0000) loss 3.3368 (3.1160) grad_norm 1.8056 (1.6454) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:33:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][50/625] eta 0:03:27 lr 0.000920 wd 0.0500 time 0.3511 (0.3609) data time 0.0009 (0.0139) model time 0.0000 (0.0000) loss 2.4707 (3.0877) grad_norm 1.1217 (1.5786) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:33:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][60/625] eta 0:03:22 lr 0.000920 wd 0.0500 time 0.3556 (0.3593) data time 0.0008 (0.0117) model time 0.3548 (0.3499) loss 3.4499 (3.0868) grad_norm 2.8959 (1.6341) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][70/625] eta 0:03:18 lr 0.000920 wd 0.0500 time 0.3475 (0.3578) data time 0.0006 (0.0102) model time 0.3469 (0.3490) loss 2.4885 (3.0828) grad_norm 1.5605 (1.6638) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][80/625] eta 0:03:14 lr 0.000920 wd 0.0500 time 0.3432 (0.3563) data time 0.0006 (0.0090) model time 0.3426 (0.3476) loss 1.9967 (3.0756) grad_norm 1.7667 (1.6787) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][90/625] eta 0:03:10 lr 0.000920 wd 0.0500 time 0.3476 (0.3553) data time 0.0007 (0.0081) model time 0.3468 (0.3472) loss 3.0011 (3.1089) grad_norm 2.2618 (1.7081) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][100/625] eta 0:03:06 lr 0.000919 wd 0.0500 time 0.3503 (0.3546) data time 0.0007 (0.0074) model time 0.3496 (0.3473) loss 2.1237 (3.0944) grad_norm 1.7610 (1.6996) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][110/625] eta 0:03:02 lr 0.000919 wd 0.0500 time 0.3486 (0.3539) data time 0.0009 (0.0068) model time 0.3478 (0.3471) loss 3.4813 (3.0829) grad_norm 1.7964 (1.6816) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][120/625] eta 0:02:58 lr 0.000919 wd 0.0500 time 0.3435 (0.3534) data time 0.0008 (0.0063) model time 0.3427 (0.3470) loss 2.5160 (3.0662) grad_norm 2.2574 (1.7112) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][130/625] eta 0:02:54 lr 0.000919 wd 0.0500 time 0.3610 (0.3531) data time 0.0007 (0.0059) model time 0.3603 (0.3473) loss 4.1596 (3.0881) grad_norm 1.4253 (1.7119) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][140/625] eta 0:02:51 lr 0.000919 wd 0.0500 time 0.3363 (0.3528) data time 0.0007 (0.0055) model time 0.3356 (0.3473) loss 3.1750 (3.0701) grad_norm 1.3302 (1.6892) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][150/625] eta 0:02:47 lr 0.000919 wd 0.0500 time 0.3473 (0.3524) data time 0.0009 (0.0052) model time 0.3464 (0.3472) loss 2.7002 (3.0608) grad_norm 1.4791 (1.6699) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][160/625] eta 0:02:43 lr 0.000918 wd 0.0500 time 0.3464 (0.3520) data time 0.0008 (0.0050) model time 0.3456 (0.3470) loss 3.1049 (3.0770) grad_norm 2.0115 (1.6677) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][170/625] eta 0:02:41 lr 0.000918 wd 0.0500 time 0.5273 (0.3539) data time 0.0009 (0.0047) model time 0.5264 (0.3501) loss 2.1608 (3.0743) grad_norm 1.3335 (1.6685) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][180/625] eta 0:02:37 lr 0.000918 wd 0.0500 time 0.3487 (0.3536) data time 0.0006 (0.0045) model time 0.3481 (0.3499) loss 2.2660 (3.0729) grad_norm 1.3754 (1.6714) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][190/625] eta 0:02:33 lr 0.000918 wd 0.0500 time 0.3558 (0.3536) data time 0.0006 (0.0043) model time 0.3551 (0.3500) loss 2.6229 (3.0684) grad_norm 1.7402 (1.6667) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][200/625] eta 0:02:30 lr 0.000918 wd 0.0500 time 0.3476 (0.3534) data time 0.0008 (0.0041) model time 0.3468 (0.3500) loss 3.2724 (3.0870) grad_norm 1.2866 (1.6617) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][210/625] eta 0:02:26 lr 0.000918 wd 0.0500 time 0.3471 (0.3537) data time 0.0009 (0.0040) model time 0.3463 (0.3506) loss 2.1456 (3.0918) grad_norm 1.1903 (1.6520) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][220/625] eta 0:02:23 lr 0.000917 wd 0.0500 time 0.3453 (0.3535) data time 0.0008 (0.0038) model time 0.3445 (0.3505) loss 3.4602 (3.0842) grad_norm 1.8401 (1.6487) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:34:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][230/625] eta 0:02:19 lr 0.000917 wd 0.0500 time 0.3449 (0.3533) data time 0.0008 (0.0037) model time 0.3441 (0.3504) loss 2.4471 (3.0851) grad_norm 1.6596 (1.6520) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][240/625] eta 0:02:15 lr 0.000917 wd 0.0500 time 0.3515 (0.3532) data time 0.0007 (0.0036) model time 0.3508 (0.3503) loss 2.8477 (3.0861) grad_norm 1.4607 (1.6532) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][250/625] eta 0:02:12 lr 0.000917 wd 0.0500 time 0.3427 (0.3531) data time 0.0009 (0.0035) model time 0.3417 (0.3503) loss 3.2560 (3.0962) grad_norm 1.4426 (1.6478) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][260/625] eta 0:02:08 lr 0.000917 wd 0.0500 time 0.3423 (0.3530) data time 0.0008 (0.0034) model time 0.3415 (0.3502) loss 3.3422 (3.1043) grad_norm 2.7919 (1.6544) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][270/625] eta 0:02:05 lr 0.000916 wd 0.0500 time 0.3446 (0.3528) data time 0.0009 (0.0033) model time 0.3437 (0.3501) loss 3.2652 (3.1041) grad_norm 1.6176 (1.6444) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][280/625] eta 0:02:01 lr 0.000916 wd 0.0500 time 0.3623 (0.3527) data time 0.0007 (0.0032) model time 0.3617 (0.3500) loss 2.1839 (3.0997) grad_norm 1.9571 (1.6468) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][290/625] eta 0:01:58 lr 0.000916 wd 0.0500 time 0.5651 (0.3540) data time 0.0006 (0.0031) model time 0.5645 (0.3517) loss 2.6300 (3.0981) grad_norm 2.0166 (1.6597) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][300/625] eta 0:01:54 lr 0.000916 wd 0.0500 time 0.3553 (0.3538) data time 0.0008 (0.0030) model time 0.3544 (0.3515) loss 3.4603 (3.0941) grad_norm 1.1630 (1.6512) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][310/625] eta 0:01:51 lr 0.000916 wd 0.0500 time 0.3479 (0.3537) data time 0.0007 (0.0030) model time 0.3472 (0.3515) loss 2.2567 (3.0944) grad_norm 1.6523 (1.6505) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][320/625] eta 0:01:47 lr 0.000916 wd 0.0500 time 0.3445 (0.3536) data time 0.0009 (0.0029) model time 0.3436 (0.3514) loss 3.1860 (3.1008) grad_norm 1.1683 (1.6431) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][330/625] eta 0:01:44 lr 0.000915 wd 0.0500 time 0.3498 (0.3535) data time 0.0007 (0.0028) model time 0.3491 (0.3513) loss 3.2689 (3.1056) grad_norm 1.6751 (1.6380) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][340/625] eta 0:01:40 lr 0.000915 wd 0.0500 time 0.3496 (0.3533) data time 0.0006 (0.0028) model time 0.3490 (0.3511) loss 3.3011 (3.1007) grad_norm 1.2228 (1.6333) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][350/625] eta 0:01:37 lr 0.000915 wd 0.0500 time 0.3485 (0.3532) data time 0.0009 (0.0027) model time 0.3476 (0.3510) loss 3.2810 (3.0968) grad_norm 1.7794 (1.6326) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][360/625] eta 0:01:33 lr 0.000915 wd 0.0500 time 0.3482 (0.3531) data time 0.0008 (0.0027) model time 0.3474 (0.3509) loss 2.8429 (3.0926) grad_norm 1.4863 (1.6342) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][370/625] eta 0:01:30 lr 0.000915 wd 0.0500 time 0.3441 (0.3530) data time 0.0006 (0.0026) model time 0.3435 (0.3509) loss 2.7016 (3.0962) grad_norm 1.5293 (1.6359) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][380/625] eta 0:01:26 lr 0.000915 wd 0.0500 time 0.3498 (0.3528) data time 0.0008 (0.0026) model time 0.3490 (0.3508) loss 2.6972 (3.0989) grad_norm 1.5381 (1.6311) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][390/625] eta 0:01:22 lr 0.000914 wd 0.0500 time 0.5094 (0.3531) data time 0.0007 (0.0025) model time 0.5087 (0.3511) loss 3.6481 (3.1023) grad_norm 1.5287 (1.6247) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:35:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][400/625] eta 0:01:19 lr 0.000914 wd 0.0500 time 0.3436 (0.3529) data time 0.0006 (0.0025) model time 0.3430 (0.3508) loss 2.7497 (3.1056) grad_norm 1.4698 (1.6230) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][410/625] eta 0:01:15 lr 0.000914 wd 0.0500 time 0.3512 (0.3527) data time 0.0008 (0.0025) model time 0.3505 (0.3507) loss 3.2538 (3.1061) grad_norm 1.6708 (1.6230) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][420/625] eta 0:01:12 lr 0.000914 wd 0.0500 time 0.3497 (0.3526) data time 0.0008 (0.0024) model time 0.3489 (0.3506) loss 3.1169 (3.1054) grad_norm 2.5583 (1.6273) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][430/625] eta 0:01:08 lr 0.000914 wd 0.0500 time 0.3482 (0.3525) data time 0.0009 (0.0024) model time 0.3474 (0.3505) loss 3.2349 (3.1138) grad_norm 1.8380 (1.6316) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][440/625] eta 0:01:05 lr 0.000913 wd 0.0500 time 0.3472 (0.3524) data time 0.0007 (0.0024) model time 0.3465 (0.3504) loss 3.7968 (3.1230) grad_norm 3.6532 (1.6387) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][450/625] eta 0:01:01 lr 0.000913 wd 0.0500 time 0.3443 (0.3523) data time 0.0008 (0.0023) model time 0.3435 (0.3503) loss 3.2538 (3.1238) grad_norm 1.7677 (1.6469) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][460/625] eta 0:00:58 lr 0.000913 wd 0.0500 time 0.3446 (0.3522) data time 0.0007 (0.0023) model time 0.3439 (0.3503) loss 2.0586 (3.1239) grad_norm 1.1987 (1.6485) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][470/625] eta 0:00:54 lr 0.000913 wd 0.0500 time 0.3525 (0.3521) data time 0.0009 (0.0023) model time 0.3516 (0.3502) loss 3.3868 (3.1198) grad_norm 1.1795 (1.6545) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][480/625] eta 0:00:51 lr 0.000913 wd 0.0500 time 0.3415 (0.3520) data time 0.0008 (0.0022) model time 0.3407 (0.3501) loss 3.5163 (3.1242) grad_norm 1.9182 (1.6537) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][490/625] eta 0:00:47 lr 0.000913 wd 0.0500 time 0.3500 (0.3519) data time 0.0007 (0.0022) model time 0.3493 (0.3500) loss 1.9243 (3.1238) grad_norm 2.8786 (1.6589) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][500/625] eta 0:00:44 lr 0.000912 wd 0.0500 time 0.3467 (0.3527) data time 0.0007 (0.0022) model time 0.3460 (0.3509) loss 2.0073 (3.1242) grad_norm 1.8971 (1.6593) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][510/625] eta 0:00:40 lr 0.000912 wd 0.0500 time 0.5669 (0.3530) data time 0.0008 (0.0021) model time 0.5661 (0.3513) loss 3.3519 (3.1314) grad_norm 1.9091 (1.6651) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][520/625] eta 0:00:37 lr 0.000912 wd 0.0500 time 0.3482 (0.3529) data time 0.0009 (0.0021) model time 0.3474 (0.3511) loss 3.3293 (3.1287) grad_norm 2.4772 (1.6623) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][530/625] eta 0:00:33 lr 0.000912 wd 0.0500 time 0.3450 (0.3527) data time 0.0008 (0.0021) model time 0.3442 (0.3510) loss 3.6779 (3.1262) grad_norm 1.7308 (1.6603) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][540/625] eta 0:00:29 lr 0.000912 wd 0.0500 time 0.3457 (0.3526) data time 0.0008 (0.0021) model time 0.3449 (0.3509) loss 3.6595 (3.1315) grad_norm 1.5942 (1.6580) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][550/625] eta 0:00:26 lr 0.000912 wd 0.0500 time 0.3492 (0.3526) data time 0.0008 (0.0021) model time 0.3483 (0.3508) loss 2.9536 (3.1332) grad_norm 1.6514 (1.6653) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][560/625] eta 0:00:22 lr 0.000911 wd 0.0500 time 0.3365 (0.3525) data time 0.0009 (0.0020) model time 0.3357 (0.3508) loss 2.5979 (3.1315) grad_norm 1.8718 (1.6670) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:36:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][570/625] eta 0:00:19 lr 0.000911 wd 0.0500 time 0.3449 (0.3524) data time 0.0006 (0.0020) model time 0.3442 (0.3507) loss 3.1486 (3.1299) grad_norm 1.0485 (1.6671) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][580/625] eta 0:00:15 lr 0.000911 wd 0.0500 time 0.3466 (0.3523) data time 0.0009 (0.0020) model time 0.3457 (0.3506) loss 3.3729 (3.1346) grad_norm 1.5449 (1.6675) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][590/625] eta 0:00:12 lr 0.000911 wd 0.0500 time 0.3484 (0.3522) data time 0.0007 (0.0020) model time 0.3477 (0.3506) loss 3.6084 (3.1324) grad_norm 1.9542 (1.6666) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][600/625] eta 0:00:08 lr 0.000911 wd 0.0500 time 0.3458 (0.3522) data time 0.0008 (0.0020) model time 0.3449 (0.3505) loss 3.2948 (3.1269) grad_norm 1.8979 (1.6694) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][610/625] eta 0:00:05 lr 0.000910 wd 0.0500 time 0.3434 (0.3521) data time 0.0006 (0.0019) model time 0.3428 (0.3504) loss 3.2837 (3.1279) grad_norm 2.0078 (1.6698) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [168/300][620/625] eta 0:00:01 lr 0.000910 wd 0.0500 time 0.3336 (0.3519) data time 0.0004 (0.0019) model time 0.3331 (0.3503) loss 2.9643 (3.1288) grad_norm 1.8953 (1.6733) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:16 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 168 training takes 0:03:39 [2024-09-01 06:37:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:37:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:37:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.487 (0.487) Loss 0.6123 (0.6123) Acc@1 87.012 (87.012) Acc@5 98.193 (98.193) Mem 11224MB [2024-09-01 06:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.133) Loss 0.9429 (0.7271) Acc@1 77.539 (83.873) Acc@5 95.068 (96.906) Mem 11224MB [2024-09-01 06:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.114) Loss 1.0703 (0.8531) Acc@1 74.365 (80.692) Acc@5 93.555 (95.454) Mem 11224MB [2024-09-01 06:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.484 Acc@5 95.471 [2024-09-01 06:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.5% [2024-09-01 06:37:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.779 (0.779) Loss 0.5020 (0.5020) Acc@1 89.014 (89.014) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 06:37:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.163) Loss 0.8291 (0.6284) Acc@1 79.834 (85.884) Acc@5 95.996 (97.638) Mem 11224MB [2024-09-01 06:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.129) Loss 0.9473 (0.7484) Acc@1 76.709 (82.780) Acc@5 94.922 (96.463) Mem 11224MB [2024-09-01 06:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.528 Acc@5 96.461 [2024-09-01 06:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 06:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.53% [2024-09-01 06:37:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:37:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][0/625] eta 0:07:27 lr 0.000910 wd 0.0500 time 0.7163 (0.7163) data time 0.3964 (0.3964) model time 0.0000 (0.0000) loss 2.6378 (2.6378) grad_norm 1.8851 (1.8851) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][10/625] eta 0:03:54 lr 0.000910 wd 0.0500 time 0.3533 (0.3816) data time 0.0006 (0.0367) model time 0.0000 (0.0000) loss 3.4299 (3.3590) grad_norm 1.4114 (1.8075) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][20/625] eta 0:03:41 lr 0.000910 wd 0.0500 time 0.3450 (0.3654) data time 0.0006 (0.0197) model time 0.0000 (0.0000) loss 3.4355 (3.2899) grad_norm 2.1609 (1.7559) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][30/625] eta 0:03:33 lr 0.000910 wd 0.0500 time 0.3481 (0.3595) data time 0.0006 (0.0136) model time 0.0000 (0.0000) loss 2.4211 (3.2290) grad_norm 1.5098 (1.7198) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][40/625] eta 0:03:31 lr 0.000909 wd 0.0500 time 0.5068 (0.3611) data time 0.0008 (0.0105) model time 0.0000 (0.0000) loss 2.3205 (3.1983) grad_norm 1.2041 (1.6796) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][50/625] eta 0:03:25 lr 0.000909 wd 0.0500 time 0.3446 (0.3576) data time 0.0009 (0.0086) model time 0.0000 (0.0000) loss 2.4235 (3.1637) grad_norm 1.2355 (1.6698) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][60/625] eta 0:03:21 lr 0.000909 wd 0.0500 time 0.3455 (0.3560) data time 0.0006 (0.0074) model time 0.3449 (0.3467) loss 3.9872 (3.1576) grad_norm 2.1877 (1.6660) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][70/625] eta 0:03:21 lr 0.000909 wd 0.0500 time 0.3478 (0.3637) data time 0.0006 (0.0064) model time 0.3472 (0.3782) loss 3.5674 (3.1390) grad_norm 1.1054 (1.6676) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][80/625] eta 0:03:17 lr 0.000909 wd 0.0500 time 0.3494 (0.3616) data time 0.0008 (0.0057) model time 0.3486 (0.3675) loss 2.9005 (3.1225) grad_norm 2.4987 (1.6889) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:37:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][90/625] eta 0:03:12 lr 0.000909 wd 0.0500 time 0.3502 (0.3602) data time 0.0007 (0.0052) model time 0.3495 (0.3627) loss 3.8829 (3.1023) grad_norm 0.9683 (1.7066) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][100/625] eta 0:03:08 lr 0.000908 wd 0.0500 time 0.3569 (0.3590) data time 0.0006 (0.0048) model time 0.3563 (0.3596) loss 2.5778 (3.0994) grad_norm 1.7450 (1.7503) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][110/625] eta 0:03:04 lr 0.000908 wd 0.0500 time 0.3393 (0.3582) data time 0.0009 (0.0044) model time 0.3384 (0.3578) loss 2.8664 (3.1075) grad_norm 1.7984 (1.7257) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][120/625] eta 0:03:00 lr 0.000908 wd 0.0500 time 0.3426 (0.3574) data time 0.0007 (0.0041) model time 0.3420 (0.3563) loss 2.6482 (3.1290) grad_norm 1.1873 (1.7036) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][130/625] eta 0:02:56 lr 0.000908 wd 0.0500 time 0.3506 (0.3567) data time 0.0006 (0.0039) model time 0.3500 (0.3552) loss 4.0650 (3.1381) grad_norm 1.7187 (1.6884) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][140/625] eta 0:02:52 lr 0.000908 wd 0.0500 time 0.3509 (0.3565) data time 0.0010 (0.0037) model time 0.3500 (0.3550) loss 3.9757 (3.1455) grad_norm 1.5978 (1.6851) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][150/625] eta 0:02:49 lr 0.000908 wd 0.0500 time 0.3557 (0.3562) data time 0.0007 (0.0035) model time 0.3550 (0.3546) loss 2.1392 (3.1260) grad_norm 1.7095 (1.6693) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][160/625] eta 0:02:45 lr 0.000907 wd 0.0500 time 0.3483 (0.3558) data time 0.0007 (0.0033) model time 0.3476 (0.3541) loss 3.2744 (3.1133) grad_norm 1.5515 (1.6829) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][170/625] eta 0:02:41 lr 0.000907 wd 0.0500 time 0.3473 (0.3556) data time 0.0009 (0.0032) model time 0.3464 (0.3538) loss 3.0037 (3.1035) grad_norm 1.5348 (1.6732) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][180/625] eta 0:02:38 lr 0.000907 wd 0.0500 time 0.3571 (0.3553) data time 0.0007 (0.0030) model time 0.3564 (0.3535) loss 3.5653 (3.1169) grad_norm 1.3470 (1.6564) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][190/625] eta 0:02:34 lr 0.000907 wd 0.0500 time 0.3619 (0.3550) data time 0.0006 (0.0029) model time 0.3613 (0.3533) loss 3.4988 (3.1307) grad_norm 2.4291 (1.6794) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][200/625] eta 0:02:30 lr 0.000907 wd 0.0500 time 0.3478 (0.3548) data time 0.0008 (0.0028) model time 0.3470 (0.3530) loss 2.9495 (3.1356) grad_norm 1.2958 (1.6832) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][210/625] eta 0:02:27 lr 0.000906 wd 0.0500 time 0.3493 (0.3544) data time 0.0008 (0.0027) model time 0.3485 (0.3526) loss 2.4961 (3.1426) grad_norm 1.8187 (1.6944) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][220/625] eta 0:02:23 lr 0.000906 wd 0.0500 time 0.3446 (0.3542) data time 0.0007 (0.0026) model time 0.3440 (0.3523) loss 1.8364 (3.1389) grad_norm 1.3480 (1.6816) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][230/625] eta 0:02:19 lr 0.000906 wd 0.0500 time 0.3416 (0.3543) data time 0.0009 (0.0026) model time 0.3407 (0.3526) loss 3.5595 (3.1466) grad_norm 1.0403 (1.6693) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][240/625] eta 0:02:16 lr 0.000906 wd 0.0500 time 0.3494 (0.3541) data time 0.0006 (0.0025) model time 0.3488 (0.3523) loss 3.9317 (3.1595) grad_norm 1.3768 (1.6682) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][250/625] eta 0:02:14 lr 0.000906 wd 0.0500 time 0.5714 (0.3575) data time 0.0008 (0.0024) model time 0.5706 (0.3566) loss 3.8575 (3.1735) grad_norm 1.8312 (1.6627) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][260/625] eta 0:02:10 lr 0.000906 wd 0.0500 time 0.3498 (0.3572) data time 0.0007 (0.0024) model time 0.3490 (0.3562) loss 3.3293 (3.1782) grad_norm 1.2859 (1.6581) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][270/625] eta 0:02:06 lr 0.000905 wd 0.0500 time 0.3472 (0.3569) data time 0.0007 (0.0023) model time 0.3464 (0.3558) loss 2.8380 (3.1741) grad_norm 1.5637 (1.6559) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][280/625] eta 0:02:03 lr 0.000905 wd 0.0500 time 0.3439 (0.3566) data time 0.0007 (0.0023) model time 0.3432 (0.3555) loss 4.0401 (3.1744) grad_norm 2.4608 (1.6945) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][290/625] eta 0:01:59 lr 0.000905 wd 0.0500 time 0.3451 (0.3564) data time 0.0008 (0.0022) model time 0.3443 (0.3553) loss 3.6314 (3.1747) grad_norm 1.8226 (1.7058) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][300/625] eta 0:01:55 lr 0.000905 wd 0.0500 time 0.3534 (0.3561) data time 0.0008 (0.0022) model time 0.3526 (0.3550) loss 3.3966 (3.1625) grad_norm 2.0973 (1.7075) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][310/625] eta 0:01:52 lr 0.000905 wd 0.0500 time 0.3507 (0.3559) data time 0.0008 (0.0021) model time 0.3499 (0.3547) loss 2.6496 (3.1650) grad_norm 1.2203 (1.7093) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][320/625] eta 0:01:48 lr 0.000905 wd 0.0500 time 0.3609 (0.3558) data time 0.0006 (0.0021) model time 0.3603 (0.3546) loss 3.5387 (3.1712) grad_norm 2.0654 (1.7138) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][330/625] eta 0:01:44 lr 0.000904 wd 0.0500 time 0.3525 (0.3556) data time 0.0008 (0.0020) model time 0.3517 (0.3544) loss 2.9113 (3.1651) grad_norm 1.5944 (1.7139) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][340/625] eta 0:01:41 lr 0.000904 wd 0.0500 time 0.3572 (0.3554) data time 0.0007 (0.0020) model time 0.3565 (0.3542) loss 3.4803 (3.1709) grad_norm 1.2163 (1.7062) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][350/625] eta 0:01:37 lr 0.000904 wd 0.0500 time 0.3472 (0.3559) data time 0.0005 (0.0020) model time 0.3466 (0.3547) loss 2.7169 (3.1705) grad_norm 1.1161 (1.6978) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][360/625] eta 0:01:34 lr 0.000904 wd 0.0500 time 0.3475 (0.3556) data time 0.0009 (0.0020) model time 0.3466 (0.3544) loss 3.3328 (3.1756) grad_norm 1.4271 (1.6979) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][370/625] eta 0:01:31 lr 0.000904 wd 0.0500 time 0.3418 (0.3575) data time 0.0006 (0.0019) model time 0.3411 (0.3566) loss 2.2260 (3.1742) grad_norm 1.5279 (1.6926) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][380/625] eta 0:01:27 lr 0.000903 wd 0.0500 time 0.3454 (0.3572) data time 0.0009 (0.0019) model time 0.3445 (0.3563) loss 3.3612 (3.1702) grad_norm 1.4101 (1.6844) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][390/625] eta 0:01:23 lr 0.000903 wd 0.0500 time 0.3456 (0.3571) data time 0.0008 (0.0019) model time 0.3448 (0.3561) loss 3.1230 (3.1642) grad_norm 1.4478 (1.6827) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][400/625] eta 0:01:20 lr 0.000903 wd 0.0500 time 0.3464 (0.3568) data time 0.0007 (0.0018) model time 0.3457 (0.3559) loss 2.9960 (3.1624) grad_norm 1.5355 (1.6866) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][410/625] eta 0:01:16 lr 0.000903 wd 0.0500 time 0.3457 (0.3566) data time 0.0007 (0.0018) model time 0.3450 (0.3556) loss 3.3251 (3.1594) grad_norm 2.4024 (1.6916) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][420/625] eta 0:01:13 lr 0.000903 wd 0.0500 time 0.3494 (0.3564) data time 0.0009 (0.0018) model time 0.3485 (0.3553) loss 2.8866 (3.1506) grad_norm 1.1872 (1.6879) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:39:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][430/625] eta 0:01:09 lr 0.000903 wd 0.0500 time 0.3476 (0.3562) data time 0.0006 (0.0018) model time 0.3471 (0.3552) loss 2.9032 (3.1500) grad_norm 1.6585 (1.6868) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][440/625] eta 0:01:05 lr 0.000902 wd 0.0500 time 0.3424 (0.3560) data time 0.0008 (0.0018) model time 0.3416 (0.3549) loss 2.5105 (3.1460) grad_norm 1.8469 (1.6835) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][450/625] eta 0:01:02 lr 0.000902 wd 0.0500 time 0.3483 (0.3559) data time 0.0008 (0.0017) model time 0.3475 (0.3548) loss 3.2629 (3.1475) grad_norm 2.0025 (1.6931) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][460/625] eta 0:00:58 lr 0.000902 wd 0.0500 time 0.3518 (0.3558) data time 0.0008 (0.0017) model time 0.3511 (0.3547) loss 3.3653 (3.1499) grad_norm 1.4920 (1.6955) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][470/625] eta 0:00:55 lr 0.000902 wd 0.0500 time 0.3536 (0.3556) data time 0.0008 (0.0017) model time 0.3528 (0.3545) loss 2.9183 (3.1500) grad_norm 1.7450 (1.6931) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][480/625] eta 0:00:51 lr 0.000902 wd 0.0500 time 0.3410 (0.3554) data time 0.0008 (0.0017) model time 0.3401 (0.3542) loss 3.0232 (3.1494) grad_norm 1.6074 (1.6876) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][490/625] eta 0:00:47 lr 0.000902 wd 0.0500 time 0.3544 (0.3553) data time 0.0006 (0.0017) model time 0.3539 (0.3541) loss 3.1504 (3.1547) grad_norm 1.7462 (1.6865) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][500/625] eta 0:00:44 lr 0.000901 wd 0.0500 time 0.3509 (0.3551) data time 0.0009 (0.0017) model time 0.3500 (0.3539) loss 2.6097 (3.1536) grad_norm 1.1690 (1.6836) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][510/625] eta 0:00:40 lr 0.000901 wd 0.0500 time 0.3544 (0.3550) data time 0.0009 (0.0017) model time 0.3535 (0.3538) loss 2.9666 (3.1567) grad_norm 1.1657 (1.6818) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][520/625] eta 0:00:37 lr 0.000901 wd 0.0500 time 0.3500 (0.3549) data time 0.0008 (0.0017) model time 0.3492 (0.3537) loss 3.1199 (3.1528) grad_norm 1.5307 (1.6827) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][530/625] eta 0:00:33 lr 0.000901 wd 0.0500 time 0.3519 (0.3548) data time 0.0008 (0.0016) model time 0.3511 (0.3536) loss 3.0935 (3.1533) grad_norm 1.4192 (1.6805) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][540/625] eta 0:00:30 lr 0.000901 wd 0.0500 time 0.3501 (0.3547) data time 0.0009 (0.0016) model time 0.3492 (0.3534) loss 2.8596 (3.1530) grad_norm 1.4563 (1.6771) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][550/625] eta 0:00:26 lr 0.000900 wd 0.0500 time 0.3542 (0.3546) data time 0.0006 (0.0016) model time 0.3536 (0.3533) loss 1.8560 (3.1460) grad_norm 1.3368 (1.6760) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][560/625] eta 0:00:23 lr 0.000900 wd 0.0500 time 0.3477 (0.3548) data time 0.0008 (0.0016) model time 0.3469 (0.3536) loss 2.8695 (3.1463) grad_norm 1.7835 (1.6733) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][570/625] eta 0:00:19 lr 0.000900 wd 0.0500 time 0.3474 (0.3547) data time 0.0009 (0.0016) model time 0.3465 (0.3536) loss 2.0179 (3.1496) grad_norm 1.2834 (1.6683) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][580/625] eta 0:00:16 lr 0.000900 wd 0.0500 time 0.5590 (0.3560) data time 0.0009 (0.0016) model time 0.5581 (0.3549) loss 2.9588 (3.1473) grad_norm 1.9572 (1.6660) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][590/625] eta 0:00:12 lr 0.000900 wd 0.0500 time 0.3538 (0.3559) data time 0.0005 (0.0016) model time 0.3532 (0.3548) loss 2.4521 (3.1453) grad_norm 2.3930 (1.6677) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:40:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][600/625] eta 0:00:08 lr 0.000900 wd 0.0500 time 0.3569 (0.3558) data time 0.0009 (0.0016) model time 0.3560 (0.3547) loss 3.1622 (3.1450) grad_norm 1.0934 (1.6669) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][610/625] eta 0:00:05 lr 0.000899 wd 0.0500 time 0.3444 (0.3557) data time 0.0006 (0.0015) model time 0.3438 (0.3546) loss 2.5931 (3.1405) grad_norm 1.1667 (1.6642) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [169/300][620/625] eta 0:00:01 lr 0.000899 wd 0.0500 time 0.3370 (0.3555) data time 0.0004 (0.0015) model time 0.3366 (0.3544) loss 2.7025 (3.1368) grad_norm 1.4067 (1.6608) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:06 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 169 training takes 0:03:42 [2024-09-01 06:41:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:41:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:41:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.484 (0.484) Loss 0.6147 (0.6147) Acc@1 87.256 (87.256) Acc@5 98.096 (98.096) Mem 11224MB [2024-09-01 06:41:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.133) Loss 0.9678 (0.7281) Acc@1 77.051 (83.984) Acc@5 95.068 (97.026) Mem 11224MB [2024-09-01 06:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.114) Loss 1.0908 (0.8606) Acc@1 73.975 (80.636) Acc@5 93.555 (95.536) Mem 11224MB [2024-09-01 06:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.424 Acc@5 95.549 [2024-09-01 06:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.4% [2024-09-01 06:41:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.793 (0.793) Loss 0.5020 (0.5020) Acc@1 88.965 (88.965) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 06:41:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.164) Loss 0.8296 (0.6277) Acc@1 79.834 (85.871) Acc@5 96.045 (97.630) Mem 11224MB [2024-09-01 06:41:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.129) Loss 0.9468 (0.7472) Acc@1 76.855 (82.789) Acc@5 94.824 (96.454) Mem 11224MB [2024-09-01 06:41:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.538 Acc@5 96.445 [2024-09-01 06:41:13 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 06:41:13 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.54% [2024-09-01 06:41:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:41:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][0/625] eta 0:06:57 lr 0.000899 wd 0.0500 time 0.6678 (0.6678) data time 0.3470 (0.3470) model time 0.0000 (0.0000) loss 2.6155 (2.6155) grad_norm 1.9805 (1.9805) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][10/625] eta 0:03:52 lr 0.000899 wd 0.0500 time 0.3507 (0.3774) data time 0.0008 (0.0323) model time 0.0000 (0.0000) loss 2.9928 (2.9384) grad_norm 2.2034 (1.8877) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][20/625] eta 0:03:39 lr 0.000899 wd 0.0500 time 0.3487 (0.3635) data time 0.0008 (0.0173) model time 0.0000 (0.0000) loss 3.2357 (3.0698) grad_norm 1.7976 (1.7374) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][30/625] eta 0:03:33 lr 0.000899 wd 0.0500 time 0.3491 (0.3588) data time 0.0008 (0.0120) model time 0.0000 (0.0000) loss 3.0759 (3.1023) grad_norm 0.8916 (1.6165) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][40/625] eta 0:03:28 lr 0.000898 wd 0.0500 time 0.3475 (0.3563) data time 0.0008 (0.0093) model time 0.0000 (0.0000) loss 2.7896 (3.1736) grad_norm 1.7557 (1.5703) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][50/625] eta 0:03:23 lr 0.000898 wd 0.0500 time 0.3455 (0.3546) data time 0.0008 (0.0077) model time 0.0000 (0.0000) loss 2.8629 (3.1580) grad_norm 1.8952 (1.5906) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][60/625] eta 0:03:21 lr 0.000898 wd 0.0500 time 0.3484 (0.3566) data time 0.0010 (0.0065) model time 0.3474 (0.3657) loss 2.3858 (3.1106) grad_norm 2.4716 (1.6180) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][70/625] eta 0:03:17 lr 0.000898 wd 0.0500 time 0.3460 (0.3554) data time 0.0008 (0.0057) model time 0.3451 (0.3564) loss 3.3079 (3.0942) grad_norm 1.3930 (1.6448) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][80/625] eta 0:03:13 lr 0.000898 wd 0.0500 time 0.3520 (0.3544) data time 0.0009 (0.0051) model time 0.3511 (0.3532) loss 2.5809 (3.1020) grad_norm 2.2691 (1.6459) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][90/625] eta 0:03:09 lr 0.000898 wd 0.0500 time 0.3554 (0.3535) data time 0.0006 (0.0047) model time 0.3547 (0.3513) loss 4.1651 (3.1024) grad_norm 1.6300 (1.6480) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][100/625] eta 0:03:05 lr 0.000897 wd 0.0500 time 0.3562 (0.3530) data time 0.0007 (0.0043) model time 0.3555 (0.3506) loss 3.6754 (3.1082) grad_norm 2.3568 (1.6577) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][110/625] eta 0:03:01 lr 0.000897 wd 0.0500 time 0.3461 (0.3524) data time 0.0008 (0.0040) model time 0.3453 (0.3497) loss 3.3443 (3.1247) grad_norm 2.5650 (1.6809) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:41:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][120/625] eta 0:02:57 lr 0.000897 wd 0.0500 time 0.3460 (0.3519) data time 0.0009 (0.0037) model time 0.3452 (0.3491) loss 2.3604 (3.1093) grad_norm 1.9960 (1.6938) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][130/625] eta 0:02:54 lr 0.000897 wd 0.0500 time 0.3499 (0.3515) data time 0.0006 (0.0035) model time 0.3493 (0.3487) loss 3.8723 (3.1221) grad_norm 1.2396 (1.6777) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][140/625] eta 0:02:50 lr 0.000897 wd 0.0500 time 0.3479 (0.3514) data time 0.0008 (0.0033) model time 0.3472 (0.3488) loss 3.1481 (3.1191) grad_norm 1.9911 (1.6862) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][150/625] eta 0:02:46 lr 0.000897 wd 0.0500 time 0.3452 (0.3511) data time 0.0009 (0.0031) model time 0.3443 (0.3485) loss 3.3069 (3.1183) grad_norm 1.4370 (1.6817) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][160/625] eta 0:02:45 lr 0.000896 wd 0.0500 time 0.5799 (0.3551) data time 0.0006 (0.0030) model time 0.5792 (0.3544) loss 3.5904 (3.1213) grad_norm 1.5018 (1.6808) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][170/625] eta 0:02:41 lr 0.000896 wd 0.0500 time 0.3567 (0.3548) data time 0.0006 (0.0029) model time 0.3561 (0.3540) loss 3.1576 (3.1005) grad_norm 1.1709 (1.6758) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][180/625] eta 0:02:38 lr 0.000896 wd 0.0500 time 0.3489 (0.3554) data time 0.0008 (0.0028) model time 0.3481 (0.3549) loss 2.9764 (3.1037) grad_norm 1.8699 (1.6786) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][190/625] eta 0:02:34 lr 0.000896 wd 0.0500 time 0.3596 (0.3551) data time 0.0006 (0.0027) model time 0.3590 (0.3544) loss 3.3411 (3.1090) grad_norm 1.5023 (1.6781) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][200/625] eta 0:02:30 lr 0.000896 wd 0.0500 time 0.3478 (0.3548) data time 0.0009 (0.0026) model time 0.3469 (0.3541) loss 3.0561 (3.1103) grad_norm 1.3487 (1.6788) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][210/625] eta 0:02:27 lr 0.000895 wd 0.0500 time 0.3474 (0.3545) data time 0.0006 (0.0025) model time 0.3469 (0.3536) loss 1.9072 (3.1025) grad_norm 1.4870 (1.6985) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][220/625] eta 0:02:23 lr 0.000895 wd 0.0500 time 0.3618 (0.3544) data time 0.0008 (0.0024) model time 0.3610 (0.3535) loss 2.5226 (3.0992) grad_norm 1.4408 (1.6990) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][230/625] eta 0:02:19 lr 0.000895 wd 0.0500 time 0.3503 (0.3541) data time 0.0006 (0.0023) model time 0.3497 (0.3531) loss 3.7205 (3.0930) grad_norm 1.8569 (1.7204) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][240/625] eta 0:02:16 lr 0.000895 wd 0.0500 time 0.3518 (0.3538) data time 0.0006 (0.0023) model time 0.3512 (0.3528) loss 3.5847 (3.0971) grad_norm 1.7935 (1.7187) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][250/625] eta 0:02:12 lr 0.000895 wd 0.0500 time 0.3489 (0.3537) data time 0.0007 (0.0022) model time 0.3481 (0.3526) loss 3.0867 (3.1015) grad_norm 1.5128 (1.7224) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][260/625] eta 0:02:09 lr 0.000895 wd 0.0500 time 0.3562 (0.3535) data time 0.0007 (0.0022) model time 0.3555 (0.3524) loss 3.5630 (3.1004) grad_norm 1.9690 (1.7526) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][270/625] eta 0:02:05 lr 0.000894 wd 0.0500 time 0.3502 (0.3534) data time 0.0007 (0.0021) model time 0.3496 (0.3523) loss 2.2732 (3.0914) grad_norm 4.5414 (1.7665) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][280/625] eta 0:02:01 lr 0.000894 wd 0.0500 time 0.3443 (0.3532) data time 0.0008 (0.0021) model time 0.3436 (0.3521) loss 3.2705 (3.0939) grad_norm 2.3431 (1.7937) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:42:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][290/625] eta 0:01:58 lr 0.000894 wd 0.0500 time 0.3461 (0.3531) data time 0.0006 (0.0020) model time 0.3455 (0.3519) loss 3.1839 (3.0937) grad_norm 1.8172 (1.7962) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][300/625] eta 0:01:54 lr 0.000894 wd 0.0500 time 0.3480 (0.3529) data time 0.0008 (0.0020) model time 0.3472 (0.3518) loss 2.1934 (3.0943) grad_norm 1.6218 (1.7885) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][310/625] eta 0:01:51 lr 0.000894 wd 0.0500 time 0.3445 (0.3528) data time 0.0008 (0.0020) model time 0.3436 (0.3516) loss 2.8595 (3.0971) grad_norm 1.3579 (1.7783) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][320/625] eta 0:01:47 lr 0.000894 wd 0.0500 time 0.3497 (0.3527) data time 0.0006 (0.0019) model time 0.3491 (0.3515) loss 2.2364 (3.0873) grad_norm 1.4246 (1.7689) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][330/625] eta 0:01:43 lr 0.000893 wd 0.0500 time 0.3439 (0.3525) data time 0.0008 (0.0019) model time 0.3431 (0.3513) loss 3.1084 (3.0869) grad_norm 1.6598 (1.7670) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][340/625] eta 0:01:41 lr 0.000893 wd 0.0500 time 0.5496 (0.3546) data time 0.0006 (0.0019) model time 0.5490 (0.3538) loss 2.9370 (3.0889) grad_norm 1.2677 (1.7695) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][350/625] eta 0:01:37 lr 0.000893 wd 0.0500 time 0.3520 (0.3556) data time 0.0008 (0.0018) model time 0.3512 (0.3549) loss 2.7308 (3.0875) grad_norm 1.4711 (1.7688) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][360/625] eta 0:01:34 lr 0.000893 wd 0.0500 time 0.3452 (0.3553) data time 0.0007 (0.0018) model time 0.3445 (0.3546) loss 3.4900 (3.0815) grad_norm 1.9166 (1.7617) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][370/625] eta 0:01:30 lr 0.000893 wd 0.0500 time 0.3456 (0.3551) data time 0.0008 (0.0018) model time 0.3448 (0.3544) loss 3.8111 (3.0867) grad_norm 1.9661 (1.7673) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][380/625] eta 0:01:26 lr 0.000892 wd 0.0500 time 0.3465 (0.3549) data time 0.0007 (0.0017) model time 0.3458 (0.3542) loss 3.4894 (3.0910) grad_norm 1.8098 (1.7665) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][390/625] eta 0:01:23 lr 0.000892 wd 0.0500 time 0.3486 (0.3552) data time 0.0010 (0.0017) model time 0.3477 (0.3544) loss 2.4356 (3.0895) grad_norm 1.3467 (1.7594) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][400/625] eta 0:01:19 lr 0.000892 wd 0.0500 time 0.3459 (0.3550) data time 0.0008 (0.0017) model time 0.3450 (0.3542) loss 3.8558 (3.0828) grad_norm 5.1362 (1.7633) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][410/625] eta 0:01:16 lr 0.000892 wd 0.0500 time 0.3564 (0.3548) data time 0.0008 (0.0017) model time 0.3556 (0.3540) loss 3.1993 (3.0912) grad_norm 1.4943 (1.7599) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][420/625] eta 0:01:12 lr 0.000892 wd 0.0500 time 0.3468 (0.3547) data time 0.0009 (0.0017) model time 0.3459 (0.3539) loss 2.5061 (3.0876) grad_norm 1.5128 (1.7551) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][430/625] eta 0:01:09 lr 0.000892 wd 0.0500 time 0.3532 (0.3546) data time 0.0008 (0.0016) model time 0.3524 (0.3537) loss 3.2418 (3.0854) grad_norm 1.2977 (1.7506) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:43:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][440/625] eta 0:01:05 lr 0.000891 wd 0.0500 time 0.3417 (0.3544) data time 0.0007 (0.0016) model time 0.3410 (0.3535) loss 3.3631 (3.0856) grad_norm 1.9262 (1.7440) loss_scale 1024.0000 (520.1270) mem 11224MB [2024-09-01 06:43:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][450/625] eta 0:01:01 lr 0.000891 wd 0.0500 time 0.3465 (0.3542) data time 0.0009 (0.0016) model time 0.3456 (0.3533) loss 2.9190 (3.0898) grad_norm 1.8727 (1.7377) loss_scale 1024.0000 (531.2993) mem 11224MB [2024-09-01 06:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][460/625] eta 0:00:58 lr 0.000891 wd 0.0500 time 0.3445 (0.3558) data time 0.0006 (0.0016) model time 0.3439 (0.3551) loss 3.1061 (3.0887) grad_norm 1.3013 (1.7301) loss_scale 1024.0000 (541.9870) mem 11224MB [2024-09-01 06:44:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][470/625] eta 0:00:55 lr 0.000891 wd 0.0500 time 0.3478 (0.3564) data time 0.0006 (0.0016) model time 0.3471 (0.3558) loss 3.9476 (3.1019) grad_norm 1.6691 (1.7313) loss_scale 1024.0000 (552.2208) mem 11224MB [2024-09-01 06:44:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][480/625] eta 0:00:51 lr 0.000891 wd 0.0500 time 0.3460 (0.3562) data time 0.0008 (0.0016) model time 0.3452 (0.3556) loss 2.9395 (3.1101) grad_norm 1.4280 (1.7305) loss_scale 1024.0000 (562.0291) mem 11224MB [2024-09-01 06:44:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][490/625] eta 0:00:48 lr 0.000891 wd 0.0500 time 0.3397 (0.3561) data time 0.0006 (0.0015) model time 0.3391 (0.3554) loss 3.7076 (3.1106) grad_norm 1.4244 (1.7303) loss_scale 1024.0000 (571.4379) mem 11224MB [2024-09-01 06:44:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][500/625] eta 0:00:44 lr 0.000890 wd 0.0500 time 0.3532 (0.3559) data time 0.0007 (0.0015) model time 0.3526 (0.3552) loss 3.1795 (3.1071) grad_norm 1.6564 (1.7251) loss_scale 1024.0000 (580.4711) mem 11224MB [2024-09-01 06:44:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][510/625] eta 0:00:40 lr 0.000890 wd 0.0500 time 0.3410 (0.3557) data time 0.0009 (0.0015) model time 0.3401 (0.3550) loss 2.9693 (3.1013) grad_norm 1.4825 (1.7230) loss_scale 1024.0000 (589.1507) mem 11224MB [2024-09-01 06:44:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][520/625] eta 0:00:37 lr 0.000890 wd 0.0500 time 0.3456 (0.3556) data time 0.0008 (0.0015) model time 0.3449 (0.3549) loss 3.1024 (3.1034) grad_norm 1.6022 (1.7219) loss_scale 1024.0000 (597.4971) mem 11224MB [2024-09-01 06:44:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][530/625] eta 0:00:33 lr 0.000890 wd 0.0500 time 0.3483 (0.3555) data time 0.0006 (0.0015) model time 0.3476 (0.3547) loss 3.0558 (3.1003) grad_norm 1.4477 (1.7179) loss_scale 1024.0000 (605.5292) mem 11224MB [2024-09-01 06:44:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][540/625] eta 0:00:30 lr 0.000890 wd 0.0500 time 0.3513 (0.3553) data time 0.0006 (0.0015) model time 0.3507 (0.3546) loss 3.2003 (3.1057) grad_norm 2.0854 (1.7107) loss_scale 1024.0000 (613.2643) mem 11224MB [2024-09-01 06:44:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][550/625] eta 0:00:26 lr 0.000889 wd 0.0500 time 0.3466 (0.3552) data time 0.0006 (0.0015) model time 0.3461 (0.3544) loss 3.8434 (3.1045) grad_norm 1.3449 (1.7170) loss_scale 1024.0000 (620.7187) mem 11224MB [2024-09-01 06:44:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][560/625] eta 0:00:23 lr 0.000889 wd 0.0500 time 0.3502 (0.3551) data time 0.0008 (0.0014) model time 0.3494 (0.3543) loss 3.0939 (3.1080) grad_norm 1.9225 (1.7149) loss_scale 1024.0000 (627.9073) mem 11224MB [2024-09-01 06:44:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][570/625] eta 0:00:19 lr 0.000889 wd 0.0500 time 0.3113 (0.3551) data time 0.0009 (0.0014) model time 0.3104 (0.3544) loss 3.3579 (3.1086) grad_norm 1.3683 (1.7103) loss_scale 1024.0000 (634.8441) mem 11224MB [2024-09-01 06:44:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][580/625] eta 0:00:15 lr 0.000889 wd 0.0500 time 0.3458 (0.3550) data time 0.0007 (0.0014) model time 0.3451 (0.3542) loss 3.8101 (3.1150) grad_norm 1.6381 (1.7070) loss_scale 1024.0000 (641.5422) mem 11224MB [2024-09-01 06:44:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][590/625] eta 0:00:12 lr 0.000889 wd 0.0500 time 0.3522 (0.3549) data time 0.0008 (0.0014) model time 0.3514 (0.3541) loss 2.6413 (3.1164) grad_norm 1.6491 (1.7059) loss_scale 1024.0000 (648.0135) mem 11224MB [2024-09-01 06:44:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][600/625] eta 0:00:08 lr 0.000889 wd 0.0500 time 0.3527 (0.3548) data time 0.0008 (0.0014) model time 0.3518 (0.3540) loss 2.4421 (3.1128) grad_norm 2.3581 (1.7058) loss_scale 1024.0000 (654.2696) mem 11224MB [2024-09-01 06:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][610/625] eta 0:00:05 lr 0.000888 wd 0.0500 time 0.3369 (0.3547) data time 0.0006 (0.0014) model time 0.3363 (0.3539) loss 3.5721 (3.1165) grad_norm 3.6032 (1.7294) loss_scale 1024.0000 (660.3208) mem 11224MB [2024-09-01 06:44:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [170/300][620/625] eta 0:00:01 lr 0.000888 wd 0.0500 time 0.3422 (0.3545) data time 0.0006 (0.0014) model time 0.3415 (0.3536) loss 3.7077 (3.1179) grad_norm 2.0291 (1.7342) loss_scale 1024.0000 (666.1771) mem 11224MB [2024-09-01 06:44:56 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 170 training takes 0:03:41 [2024-09-01 06:44:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:44:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:44:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.449 (0.449) Loss 0.5571 (0.5571) Acc@1 87.988 (87.988) Acc@5 98.193 (98.193) Mem 11224MB [2024-09-01 06:44:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 0.9395 (0.7019) Acc@1 76.758 (84.091) Acc@5 94.678 (97.106) Mem 11224MB [2024-09-01 06:44:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.0576 (0.8368) Acc@1 74.072 (80.790) Acc@5 93.359 (95.568) Mem 11224MB [2024-09-01 06:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.608 Acc@5 95.539 [2024-09-01 06:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.6% [2024-09-01 06:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.817 (0.817) Loss 0.5024 (0.5024) Acc@1 88.965 (88.965) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 06:45:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.170) Loss 0.8257 (0.6266) Acc@1 80.078 (85.955) Acc@5 96.143 (97.661) Mem 11224MB [2024-09-01 06:45:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.133) Loss 0.9458 (0.7461) Acc@1 76.855 (82.836) Acc@5 94.775 (96.477) Mem 11224MB [2024-09-01 06:45:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.584 Acc@5 96.475 [2024-09-01 06:45:03 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.6% [2024-09-01 06:45:03 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.58% [2024-09-01 06:45:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:45:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:45:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][0/625] eta 0:07:11 lr 0.000888 wd 0.0500 time 0.6908 (0.6908) data time 0.3810 (0.3810) model time 0.0000 (0.0000) loss 3.1703 (3.1703) grad_norm 1.8854 (1.8854) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][10/625] eta 0:03:55 lr 0.000888 wd 0.0500 time 0.3619 (0.3832) data time 0.0006 (0.0354) model time 0.0000 (0.0000) loss 3.8335 (3.1058) grad_norm 1.1585 (1.5576) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][20/625] eta 0:03:42 lr 0.000888 wd 0.0500 time 0.3469 (0.3671) data time 0.0009 (0.0189) model time 0.0000 (0.0000) loss 2.8061 (3.0351) grad_norm 2.0553 (1.6879) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][30/625] eta 0:03:57 lr 0.000888 wd 0.0500 time 0.3513 (0.3989) data time 0.0009 (0.0131) model time 0.0000 (0.0000) loss 2.9175 (3.0545) grad_norm 1.6252 (1.6437) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][40/625] eta 0:03:48 lr 0.000887 wd 0.0500 time 0.3500 (0.3907) data time 0.0006 (0.0101) model time 0.0000 (0.0000) loss 2.7097 (3.1284) grad_norm 1.5382 (1.7832) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][50/625] eta 0:03:39 lr 0.000887 wd 0.0500 time 0.3432 (0.3823) data time 0.0006 (0.0083) model time 0.0000 (0.0000) loss 3.8029 (3.1304) grad_norm 1.2942 (1.8210) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][60/625] eta 0:03:32 lr 0.000887 wd 0.0500 time 0.3473 (0.3767) data time 0.0008 (0.0071) model time 0.3464 (0.3475) loss 2.6403 (3.1345) grad_norm 2.1116 (1.8782) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][70/625] eta 0:03:26 lr 0.000887 wd 0.0500 time 0.3472 (0.3728) data time 0.0007 (0.0062) model time 0.3465 (0.3478) loss 2.1492 (3.1273) grad_norm 1.8898 (1.8591) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][80/625] eta 0:03:21 lr 0.000887 wd 0.0500 time 0.3557 (0.3696) data time 0.0008 (0.0055) model time 0.3549 (0.3473) loss 3.2794 (3.1498) grad_norm 1.5948 (1.8186) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][90/625] eta 0:03:16 lr 0.000887 wd 0.0500 time 0.3483 (0.3672) data time 0.0008 (0.0050) model time 0.3475 (0.3472) loss 2.4090 (3.1206) grad_norm 1.2234 (1.8007) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][100/625] eta 0:03:11 lr 0.000886 wd 0.0500 time 0.3477 (0.3652) data time 0.0006 (0.0046) model time 0.3471 (0.3469) loss 2.5277 (3.1164) grad_norm 1.3624 (1.7903) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][110/625] eta 0:03:07 lr 0.000886 wd 0.0500 time 0.3475 (0.3637) data time 0.0007 (0.0043) model time 0.3468 (0.3470) loss 3.3308 (3.1179) grad_norm 1.5937 (1.7963) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][120/625] eta 0:03:03 lr 0.000886 wd 0.0500 time 0.3484 (0.3625) data time 0.0008 (0.0040) model time 0.3476 (0.3472) loss 2.2659 (3.1281) grad_norm 1.8472 (1.7908) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][130/625] eta 0:02:58 lr 0.000886 wd 0.0500 time 0.3420 (0.3615) data time 0.0009 (0.0038) model time 0.3412 (0.3474) loss 3.1229 (3.1193) grad_norm 1.2261 (1.7546) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][140/625] eta 0:02:54 lr 0.000886 wd 0.0500 time 0.3453 (0.3604) data time 0.0009 (0.0035) model time 0.3444 (0.3472) loss 2.4582 (3.1028) grad_norm 2.2328 (1.7591) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:45:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][150/625] eta 0:02:50 lr 0.000885 wd 0.0500 time 0.3537 (0.3596) data time 0.0008 (0.0034) model time 0.3529 (0.3473) loss 3.5968 (3.1019) grad_norm 1.3625 (1.7744) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][160/625] eta 0:02:46 lr 0.000885 wd 0.0500 time 0.3377 (0.3589) data time 0.0006 (0.0032) model time 0.3371 (0.3472) loss 3.9276 (3.1241) grad_norm 1.0366 (1.7893) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][170/625] eta 0:02:42 lr 0.000885 wd 0.0500 time 0.3450 (0.3582) data time 0.0009 (0.0031) model time 0.3441 (0.3472) loss 3.6046 (3.1150) grad_norm 1.3115 (1.7703) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][180/625] eta 0:02:39 lr 0.000885 wd 0.0500 time 0.3465 (0.3577) data time 0.0008 (0.0029) model time 0.3457 (0.3472) loss 3.5175 (3.1111) grad_norm 1.8012 (1.7936) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][190/625] eta 0:02:35 lr 0.000885 wd 0.0500 time 0.3473 (0.3571) data time 0.0008 (0.0028) model time 0.3465 (0.3472) loss 3.3199 (3.1110) grad_norm 1.2023 (1.7868) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][200/625] eta 0:02:31 lr 0.000885 wd 0.0500 time 0.3510 (0.3567) data time 0.0008 (0.0027) model time 0.3502 (0.3472) loss 3.1454 (3.1093) grad_norm 3.7445 (1.7885) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][210/625] eta 0:02:27 lr 0.000884 wd 0.0500 time 0.3471 (0.3563) data time 0.0006 (0.0026) model time 0.3465 (0.3471) loss 3.3139 (3.1084) grad_norm 1.6496 (1.7896) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][220/625] eta 0:02:24 lr 0.000884 wd 0.0500 time 0.3501 (0.3559) data time 0.0008 (0.0026) model time 0.3493 (0.3471) loss 3.5626 (3.1104) grad_norm 1.1645 (1.7773) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][230/625] eta 0:02:20 lr 0.000884 wd 0.0500 time 0.5472 (0.3563) data time 0.0010 (0.0025) model time 0.5462 (0.3481) loss 3.2134 (3.1008) grad_norm 2.4711 (1.7704) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][240/625] eta 0:02:18 lr 0.000884 wd 0.0500 time 0.3449 (0.3594) data time 0.0006 (0.0024) model time 0.3443 (0.3524) loss 3.4424 (3.1114) grad_norm 1.5808 (1.7671) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][250/625] eta 0:02:14 lr 0.000884 wd 0.0500 time 0.3485 (0.3597) data time 0.0009 (0.0024) model time 0.3476 (0.3531) loss 3.4703 (3.1051) grad_norm 1.5826 (1.7581) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][260/625] eta 0:02:11 lr 0.000884 wd 0.0500 time 0.3530 (0.3592) data time 0.0008 (0.0023) model time 0.3522 (0.3528) loss 3.9410 (3.0962) grad_norm 2.4281 (1.7610) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][270/625] eta 0:02:07 lr 0.000883 wd 0.0500 time 0.3484 (0.3588) data time 0.0008 (0.0022) model time 0.3477 (0.3525) loss 3.1125 (3.0973) grad_norm 1.6885 (1.7609) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][280/625] eta 0:02:03 lr 0.000883 wd 0.0500 time 0.3460 (0.3584) data time 0.0008 (0.0022) model time 0.3452 (0.3522) loss 2.4541 (3.1051) grad_norm 1.2432 (1.7558) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][290/625] eta 0:01:59 lr 0.000883 wd 0.0500 time 0.3633 (0.3580) data time 0.0006 (0.0021) model time 0.3627 (0.3521) loss 3.6697 (3.1055) grad_norm 1.5941 (1.7550) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][300/625] eta 0:01:56 lr 0.000883 wd 0.0500 time 0.3469 (0.3577) data time 0.0006 (0.0021) model time 0.3463 (0.3518) loss 3.0159 (3.1096) grad_norm 2.3819 (1.7572) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][310/625] eta 0:01:52 lr 0.000883 wd 0.0500 time 0.3491 (0.3573) data time 0.0007 (0.0021) model time 0.3484 (0.3516) loss 3.6894 (3.1049) grad_norm 1.7863 (1.7494) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][320/625] eta 0:01:48 lr 0.000882 wd 0.0500 time 0.3423 (0.3570) data time 0.0008 (0.0020) model time 0.3415 (0.3514) loss 3.1761 (3.1055) grad_norm 2.7151 (1.7468) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][330/625] eta 0:01:45 lr 0.000882 wd 0.0500 time 0.3451 (0.3567) data time 0.0009 (0.0020) model time 0.3442 (0.3512) loss 3.4647 (3.1063) grad_norm 1.2000 (1.7426) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][340/625] eta 0:01:41 lr 0.000882 wd 0.0500 time 0.3414 (0.3565) data time 0.0008 (0.0019) model time 0.3405 (0.3511) loss 3.3761 (3.1086) grad_norm 1.3164 (1.7363) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][350/625] eta 0:01:37 lr 0.000882 wd 0.0500 time 0.3446 (0.3562) data time 0.0009 (0.0019) model time 0.3437 (0.3510) loss 3.4031 (3.1096) grad_norm 1.3704 (1.7316) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][360/625] eta 0:01:34 lr 0.000882 wd 0.0500 time 0.3448 (0.3560) data time 0.0009 (0.0019) model time 0.3439 (0.3508) loss 3.0011 (3.1119) grad_norm 1.3596 (1.7269) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][370/625] eta 0:01:30 lr 0.000882 wd 0.0500 time 0.3450 (0.3558) data time 0.0009 (0.0019) model time 0.3441 (0.3507) loss 3.6801 (3.1090) grad_norm 1.9162 (1.7296) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][380/625] eta 0:01:27 lr 0.000881 wd 0.0500 time 0.3483 (0.3555) data time 0.0006 (0.0018) model time 0.3477 (0.3506) loss 3.2695 (3.1070) grad_norm 4.3167 (1.7337) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][390/625] eta 0:01:23 lr 0.000881 wd 0.0500 time 0.3463 (0.3553) data time 0.0007 (0.0018) model time 0.3456 (0.3504) loss 2.9876 (3.1094) grad_norm 2.0576 (1.7469) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][400/625] eta 0:01:19 lr 0.000881 wd 0.0500 time 0.3492 (0.3551) data time 0.0008 (0.0018) model time 0.3485 (0.3503) loss 3.3514 (3.1156) grad_norm 1.6938 (1.7434) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][410/625] eta 0:01:16 lr 0.000881 wd 0.0500 time 0.3487 (0.3550) data time 0.0009 (0.0018) model time 0.3479 (0.3503) loss 2.2078 (3.1099) grad_norm 1.1173 (1.7408) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][420/625] eta 0:01:13 lr 0.000881 wd 0.0500 time 0.5271 (0.3571) data time 0.0008 (0.0017) model time 0.5263 (0.3528) loss 3.3075 (3.1055) grad_norm 2.0947 (1.7366) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][430/625] eta 0:01:09 lr 0.000881 wd 0.0500 time 0.3511 (0.3582) data time 0.0009 (0.0017) model time 0.3502 (0.3542) loss 3.2417 (3.1056) grad_norm 1.4475 (1.7342) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][440/625] eta 0:01:06 lr 0.000880 wd 0.0500 time 0.3479 (0.3580) data time 0.0009 (0.0017) model time 0.3470 (0.3540) loss 2.8677 (3.1044) grad_norm 1.8037 (1.7338) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][450/625] eta 0:01:02 lr 0.000880 wd 0.0500 time 0.3492 (0.3577) data time 0.0006 (0.0017) model time 0.3485 (0.3538) loss 3.2602 (3.1035) grad_norm 1.5543 (1.7554) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][460/625] eta 0:00:58 lr 0.000880 wd 0.0500 time 0.3478 (0.3575) data time 0.0008 (0.0017) model time 0.3469 (0.3536) loss 3.1976 (3.1019) grad_norm 1.3787 (1.7615) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][470/625] eta 0:00:55 lr 0.000880 wd 0.0500 time 0.3470 (0.3573) data time 0.0009 (0.0016) model time 0.3461 (0.3534) loss 2.5818 (3.1020) grad_norm 1.5664 (1.7567) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][480/625] eta 0:00:51 lr 0.000880 wd 0.0500 time 0.3478 (0.3571) data time 0.0006 (0.0016) model time 0.3472 (0.3533) loss 3.5059 (3.1008) grad_norm 1.7113 (1.7593) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][490/625] eta 0:00:48 lr 0.000879 wd 0.0500 time 0.3509 (0.3570) data time 0.0007 (0.0016) model time 0.3502 (0.3532) loss 2.1059 (3.1009) grad_norm 1.0774 (1.7570) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][500/625] eta 0:00:44 lr 0.000879 wd 0.0500 time 0.3406 (0.3569) data time 0.0008 (0.0016) model time 0.3398 (0.3532) loss 2.1741 (3.1032) grad_norm 2.0095 (1.7551) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][510/625] eta 0:00:41 lr 0.000879 wd 0.0500 time 0.3502 (0.3567) data time 0.0008 (0.0016) model time 0.3494 (0.3530) loss 3.4505 (3.1070) grad_norm 1.4789 (1.7517) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][520/625] eta 0:00:37 lr 0.000879 wd 0.0500 time 0.3471 (0.3566) data time 0.0007 (0.0016) model time 0.3463 (0.3529) loss 3.7318 (3.1064) grad_norm 2.1273 (1.7530) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][530/625] eta 0:00:33 lr 0.000879 wd 0.0500 time 0.3499 (0.3564) data time 0.0006 (0.0015) model time 0.3493 (0.3528) loss 2.4590 (3.1072) grad_norm 1.3457 (1.7543) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][540/625] eta 0:00:30 lr 0.000879 wd 0.0500 time 0.5617 (0.3577) data time 0.0009 (0.0015) model time 0.5607 (0.3543) loss 3.6927 (3.1139) grad_norm 1.7232 (1.7517) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][550/625] eta 0:00:26 lr 0.000878 wd 0.0500 time 0.3476 (0.3582) data time 0.0006 (0.0015) model time 0.3470 (0.3549) loss 3.7830 (3.1099) grad_norm 1.7252 (1.7484) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][560/625] eta 0:00:23 lr 0.000878 wd 0.0500 time 0.3502 (0.3580) data time 0.0006 (0.0015) model time 0.3496 (0.3548) loss 3.7674 (3.1170) grad_norm 1.1709 (1.7550) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][570/625] eta 0:00:19 lr 0.000878 wd 0.0500 time 0.3477 (0.3579) data time 0.0008 (0.0015) model time 0.3469 (0.3547) loss 2.7586 (3.1139) grad_norm 1.8541 (1.7528) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][580/625] eta 0:00:16 lr 0.000878 wd 0.0500 time 0.3541 (0.3577) data time 0.0008 (0.0015) model time 0.3532 (0.3545) loss 3.1398 (3.1127) grad_norm 2.0785 (1.7480) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][590/625] eta 0:00:12 lr 0.000878 wd 0.0500 time 0.3530 (0.3576) data time 0.0007 (0.0015) model time 0.3523 (0.3544) loss 3.8929 (3.1148) grad_norm 2.1220 (1.7442) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][600/625] eta 0:00:08 lr 0.000878 wd 0.0500 time 0.3445 (0.3574) data time 0.0010 (0.0015) model time 0.3435 (0.3543) loss 2.6321 (3.1174) grad_norm 1.4848 (1.7499) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][610/625] eta 0:00:05 lr 0.000877 wd 0.0500 time 0.3409 (0.3573) data time 0.0006 (0.0015) model time 0.3403 (0.3541) loss 3.2547 (3.1155) grad_norm 1.2933 (1.7492) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [171/300][620/625] eta 0:00:01 lr 0.000877 wd 0.0500 time 0.3389 (0.3570) data time 0.0004 (0.0014) model time 0.3384 (0.3539) loss 3.8249 (3.1162) grad_norm 1.8354 (1.7487) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 171 training takes 0:03:43 [2024-09-01 06:48:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:48:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:48:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.453 (0.453) Loss 0.5762 (0.5762) Acc@1 87.402 (87.402) Acc@5 98.389 (98.389) Mem 11224MB [2024-09-01 06:48:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 0.9419 (0.7033) Acc@1 78.174 (84.353) Acc@5 94.873 (97.097) Mem 11224MB [2024-09-01 06:48:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 1.0410 (0.8307) Acc@1 74.414 (81.076) Acc@5 93.750 (95.619) Mem 11224MB [2024-09-01 06:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.758 Acc@5 95.631 [2024-09-01 06:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.8% [2024-09-01 06:48:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.937 (0.937) Loss 0.5024 (0.5024) Acc@1 88.818 (88.818) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 06:48:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.176) Loss 0.8257 (0.6268) Acc@1 80.176 (85.946) Acc@5 96.094 (97.661) Mem 11224MB [2024-09-01 06:48:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.135) Loss 0.9424 (0.7455) Acc@1 76.953 (82.850) Acc@5 94.727 (96.496) Mem 11224MB [2024-09-01 06:48:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.580 Acc@5 96.495 [2024-09-01 06:48:54 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.6% [2024-09-01 06:48:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][0/625] eta 0:14:45 lr 0.000877 wd 0.0500 time 1.4170 (1.4170) data time 0.3885 (0.3885) model time 0.0000 (0.0000) loss 3.2156 (3.2156) grad_norm 1.4091 (1.4091) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:48:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][10/625] eta 0:04:33 lr 0.000877 wd 0.0500 time 0.3442 (0.4451) data time 0.0007 (0.0361) model time 0.0000 (0.0000) loss 4.0141 (3.0750) grad_norm 1.8768 (1.5962) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][20/625] eta 0:04:02 lr 0.000877 wd 0.0500 time 0.3522 (0.4000) data time 0.0006 (0.0193) model time 0.0000 (0.0000) loss 3.1545 (3.0043) grad_norm 2.4450 (1.7304) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][30/625] eta 0:03:47 lr 0.000877 wd 0.0500 time 0.3478 (0.3830) data time 0.0006 (0.0133) model time 0.0000 (0.0000) loss 2.0985 (3.0199) grad_norm 1.5883 (1.6424) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][40/625] eta 0:03:39 lr 0.000876 wd 0.0500 time 0.3502 (0.3749) data time 0.0007 (0.0103) model time 0.0000 (0.0000) loss 2.4119 (2.9490) grad_norm 1.3421 (1.6748) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][50/625] eta 0:03:32 lr 0.000876 wd 0.0500 time 0.3461 (0.3694) data time 0.0009 (0.0084) model time 0.0000 (0.0000) loss 3.4194 (3.0045) grad_norm 1.4356 (1.6466) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][60/625] eta 0:03:26 lr 0.000876 wd 0.0500 time 0.3430 (0.3659) data time 0.0007 (0.0072) model time 0.3422 (0.3468) loss 3.8213 (2.9991) grad_norm 2.5277 (1.6310) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][70/625] eta 0:03:25 lr 0.000876 wd 0.0500 time 0.5210 (0.3708) data time 0.0008 (0.0063) model time 0.5202 (0.3735) loss 2.8715 (3.0134) grad_norm 1.6569 (1.6321) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][80/625] eta 0:03:22 lr 0.000876 wd 0.0500 time 0.3505 (0.3707) data time 0.0006 (0.0056) model time 0.3499 (0.3720) loss 3.2706 (3.0204) grad_norm 1.6494 (1.6250) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][90/625] eta 0:03:17 lr 0.000876 wd 0.0500 time 0.3523 (0.3683) data time 0.0008 (0.0051) model time 0.3515 (0.3660) loss 2.9549 (3.0175) grad_norm 1.6978 (1.6499) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][100/625] eta 0:03:13 lr 0.000875 wd 0.0500 time 0.3443 (0.3680) data time 0.0008 (0.0047) model time 0.3434 (0.3658) loss 3.2008 (3.0219) grad_norm 1.2192 (1.6408) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][110/625] eta 0:03:08 lr 0.000875 wd 0.0500 time 0.3474 (0.3661) data time 0.0009 (0.0043) model time 0.3465 (0.3624) loss 3.3742 (3.0258) grad_norm 2.0779 (1.6525) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][120/625] eta 0:03:04 lr 0.000875 wd 0.0500 time 0.3471 (0.3646) data time 0.0009 (0.0040) model time 0.3463 (0.3603) loss 3.2211 (3.0057) grad_norm 1.0635 (1.6364) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][130/625] eta 0:02:59 lr 0.000875 wd 0.0500 time 0.3424 (0.3633) data time 0.0008 (0.0038) model time 0.3416 (0.3586) loss 3.1019 (3.0167) grad_norm 1.4324 (1.6436) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][140/625] eta 0:02:55 lr 0.000875 wd 0.0500 time 0.3448 (0.3622) data time 0.0009 (0.0036) model time 0.3439 (0.3572) loss 3.8516 (3.0274) grad_norm 1.4590 (1.6321) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][150/625] eta 0:02:51 lr 0.000874 wd 0.0500 time 0.3471 (0.3613) data time 0.0009 (0.0034) model time 0.3462 (0.3563) loss 3.2751 (3.0272) grad_norm 1.4013 (1.6209) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][160/625] eta 0:02:47 lr 0.000874 wd 0.0500 time 0.3533 (0.3604) data time 0.0007 (0.0032) model time 0.3526 (0.3554) loss 3.4213 (3.0431) grad_norm 1.5740 (1.6668) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][170/625] eta 0:02:43 lr 0.000874 wd 0.0500 time 0.3498 (0.3597) data time 0.0009 (0.0031) model time 0.3489 (0.3548) loss 3.4056 (3.0333) grad_norm 1.8621 (1.6723) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:49:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][180/625] eta 0:02:39 lr 0.000874 wd 0.0500 time 0.3444 (0.3592) data time 0.0008 (0.0030) model time 0.3436 (0.3543) loss 2.4535 (3.0381) grad_norm 1.4714 (1.6651) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][190/625] eta 0:02:35 lr 0.000874 wd 0.0500 time 0.3496 (0.3586) data time 0.0007 (0.0029) model time 0.3489 (0.3538) loss 3.4950 (3.0448) grad_norm 1.4522 (1.6502) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][200/625] eta 0:02:32 lr 0.000874 wd 0.0500 time 0.3480 (0.3580) data time 0.0008 (0.0028) model time 0.3472 (0.3533) loss 3.3475 (3.0487) grad_norm 1.3261 (1.6435) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][210/625] eta 0:02:28 lr 0.000873 wd 0.0500 time 0.3439 (0.3575) data time 0.0007 (0.0027) model time 0.3432 (0.3529) loss 3.2008 (3.0530) grad_norm 1.9209 (1.6624) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][220/625] eta 0:02:24 lr 0.000873 wd 0.0500 time 0.3463 (0.3570) data time 0.0008 (0.0026) model time 0.3454 (0.3525) loss 2.5218 (3.0523) grad_norm 2.5050 (1.6692) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][230/625] eta 0:02:20 lr 0.000873 wd 0.0500 time 0.3485 (0.3567) data time 0.0008 (0.0025) model time 0.3476 (0.3523) loss 2.8557 (3.0548) grad_norm 1.7051 (1.6716) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][240/625] eta 0:02:17 lr 0.000873 wd 0.0500 time 0.3484 (0.3565) data time 0.0007 (0.0024) model time 0.3477 (0.3522) loss 3.2808 (3.0642) grad_norm 1.1482 (1.6642) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][250/625] eta 0:02:13 lr 0.000873 wd 0.0500 time 0.3506 (0.3564) data time 0.0007 (0.0024) model time 0.3499 (0.3522) loss 3.5421 (3.0648) grad_norm 1.4036 (1.6554) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][260/625] eta 0:02:09 lr 0.000873 wd 0.0500 time 0.3422 (0.3561) data time 0.0007 (0.0023) model time 0.3415 (0.3520) loss 2.7804 (3.0646) grad_norm 1.4712 (1.6606) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][270/625] eta 0:02:06 lr 0.000872 wd 0.0500 time 0.3487 (0.3557) data time 0.0007 (0.0023) model time 0.3480 (0.3517) loss 3.1845 (3.0672) grad_norm 1.2911 (1.6550) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][280/625] eta 0:02:03 lr 0.000872 wd 0.0500 time 0.5709 (0.3576) data time 0.0006 (0.0022) model time 0.5702 (0.3542) loss 2.8070 (3.0681) grad_norm 2.0236 (1.6564) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][290/625] eta 0:01:59 lr 0.000872 wd 0.0500 time 0.3512 (0.3574) data time 0.0006 (0.0022) model time 0.3506 (0.3540) loss 3.8591 (3.0729) grad_norm 1.8143 (1.6519) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][300/625] eta 0:01:56 lr 0.000872 wd 0.0500 time 0.3471 (0.3571) data time 0.0007 (0.0021) model time 0.3464 (0.3537) loss 3.1149 (3.0590) grad_norm 2.0565 (1.6536) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][310/625] eta 0:01:52 lr 0.000872 wd 0.0500 time 0.3440 (0.3568) data time 0.0009 (0.0021) model time 0.3431 (0.3535) loss 3.3021 (3.0626) grad_norm 1.8029 (1.6572) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][320/625] eta 0:01:48 lr 0.000871 wd 0.0500 time 0.3484 (0.3565) data time 0.0006 (0.0020) model time 0.3477 (0.3532) loss 2.4423 (3.0625) grad_norm 1.5305 (1.6521) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][330/625] eta 0:01:45 lr 0.000871 wd 0.0500 time 0.3496 (0.3562) data time 0.0007 (0.0020) model time 0.3490 (0.3530) loss 3.8888 (3.0730) grad_norm 1.2359 (1.6460) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][340/625] eta 0:01:41 lr 0.000871 wd 0.0500 time 0.3480 (0.3560) data time 0.0007 (0.0020) model time 0.3473 (0.3528) loss 2.4369 (3.0783) grad_norm 1.6868 (1.6510) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:50:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][350/625] eta 0:01:37 lr 0.000871 wd 0.0500 time 0.3576 (0.3557) data time 0.0006 (0.0019) model time 0.3569 (0.3526) loss 3.3296 (3.0826) grad_norm 2.1386 (1.6471) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][360/625] eta 0:01:34 lr 0.000871 wd 0.0500 time 0.3393 (0.3554) data time 0.0009 (0.0019) model time 0.3384 (0.3523) loss 2.8403 (3.0851) grad_norm 1.2107 (1.6411) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][370/625] eta 0:01:30 lr 0.000871 wd 0.0500 time 0.3465 (0.3553) data time 0.0009 (0.0019) model time 0.3456 (0.3522) loss 3.4038 (3.0825) grad_norm 1.7150 (1.6446) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][380/625] eta 0:01:26 lr 0.000870 wd 0.0500 time 0.3435 (0.3551) data time 0.0009 (0.0019) model time 0.3427 (0.3520) loss 2.7722 (3.0793) grad_norm 1.9626 (1.6455) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][390/625] eta 0:01:23 lr 0.000870 wd 0.0500 time 0.3505 (0.3550) data time 0.0008 (0.0018) model time 0.3497 (0.3520) loss 3.5809 (3.0833) grad_norm 1.3212 (1.6439) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][400/625] eta 0:01:19 lr 0.000870 wd 0.0500 time 0.3494 (0.3552) data time 0.0006 (0.0018) model time 0.3488 (0.3523) loss 3.4289 (3.0822) grad_norm 1.8396 (1.6483) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][410/625] eta 0:01:16 lr 0.000870 wd 0.0500 time 0.3488 (0.3550) data time 0.0009 (0.0018) model time 0.3480 (0.3521) loss 2.4464 (3.0792) grad_norm 1.6808 (1.6468) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][420/625] eta 0:01:12 lr 0.000870 wd 0.0500 time 0.3438 (0.3548) data time 0.0010 (0.0018) model time 0.3428 (0.3520) loss 3.5703 (3.0796) grad_norm 2.2477 (1.6533) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][430/625] eta 0:01:09 lr 0.000870 wd 0.0500 time 0.3581 (0.3547) data time 0.0008 (0.0017) model time 0.3573 (0.3518) loss 2.5262 (3.0851) grad_norm 1.5794 (1.6483) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][440/625] eta 0:01:05 lr 0.000869 wd 0.0500 time 0.3489 (0.3545) data time 0.0008 (0.0017) model time 0.3481 (0.3517) loss 3.4523 (3.0850) grad_norm 1.4415 (1.6530) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][450/625] eta 0:01:02 lr 0.000869 wd 0.0500 time 0.3609 (0.3544) data time 0.0006 (0.0017) model time 0.3603 (0.3517) loss 4.0833 (3.0899) grad_norm 1.5834 (1.6539) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][460/625] eta 0:00:58 lr 0.000869 wd 0.0500 time 0.5449 (0.3551) data time 0.0008 (0.0017) model time 0.5441 (0.3525) loss 3.1036 (3.0930) grad_norm 1.1157 (1.6457) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][470/625] eta 0:00:55 lr 0.000869 wd 0.0500 time 0.3497 (0.3561) data time 0.0009 (0.0017) model time 0.3488 (0.3537) loss 3.8575 (3.0961) grad_norm 1.7994 (1.6439) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][480/625] eta 0:00:51 lr 0.000869 wd 0.0500 time 0.3496 (0.3559) data time 0.0008 (0.0016) model time 0.3487 (0.3535) loss 1.9720 (3.0923) grad_norm 1.2433 (1.6473) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][490/625] eta 0:00:48 lr 0.000868 wd 0.0500 time 0.3466 (0.3557) data time 0.0009 (0.0016) model time 0.3457 (0.3533) loss 3.0799 (3.0909) grad_norm 1.9684 (1.6456) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][500/625] eta 0:00:44 lr 0.000868 wd 0.0500 time 0.3465 (0.3556) data time 0.0007 (0.0016) model time 0.3458 (0.3532) loss 2.6460 (3.0883) grad_norm 1.4058 (1.6419) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][510/625] eta 0:00:40 lr 0.000868 wd 0.0500 time 0.3444 (0.3555) data time 0.0008 (0.0016) model time 0.3436 (0.3531) loss 3.1076 (3.0892) grad_norm 1.9984 (1.6489) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:51:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][520/625] eta 0:00:37 lr 0.000868 wd 0.0500 time 0.3458 (0.3553) data time 0.0007 (0.0016) model time 0.3452 (0.3530) loss 3.3053 (3.0944) grad_norm 1.3322 (1.6523) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][530/625] eta 0:00:33 lr 0.000868 wd 0.0500 time 0.3492 (0.3552) data time 0.0006 (0.0016) model time 0.3486 (0.3529) loss 3.9121 (3.0978) grad_norm 1.3939 (1.6475) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][540/625] eta 0:00:30 lr 0.000868 wd 0.0500 time 0.3483 (0.3551) data time 0.0008 (0.0016) model time 0.3475 (0.3527) loss 2.2509 (3.0925) grad_norm 1.3172 (1.6443) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][550/625] eta 0:00:26 lr 0.000867 wd 0.0500 time 0.3408 (0.3549) data time 0.0006 (0.0015) model time 0.3401 (0.3526) loss 3.0601 (3.0902) grad_norm 1.6199 (1.6443) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][560/625] eta 0:00:23 lr 0.000867 wd 0.0500 time 0.3433 (0.3548) data time 0.0007 (0.0015) model time 0.3426 (0.3525) loss 3.1588 (3.0945) grad_norm 1.7519 (1.6506) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][570/625] eta 0:00:19 lr 0.000867 wd 0.0500 time 0.3504 (0.3547) data time 0.0006 (0.0015) model time 0.3498 (0.3524) loss 3.5990 (3.0975) grad_norm 1.7286 (1.6525) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][580/625] eta 0:00:15 lr 0.000867 wd 0.0500 time 0.3453 (0.3552) data time 0.0007 (0.0015) model time 0.3446 (0.3530) loss 3.0902 (3.1005) grad_norm 2.1360 (1.6625) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][590/625] eta 0:00:12 lr 0.000867 wd 0.0500 time 0.3491 (0.3558) data time 0.0007 (0.0015) model time 0.3484 (0.3537) loss 3.4728 (3.0992) grad_norm 1.9376 (1.6634) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][600/625] eta 0:00:08 lr 0.000867 wd 0.0500 time 0.3492 (0.3557) data time 0.0007 (0.0015) model time 0.3485 (0.3536) loss 3.1842 (3.1019) grad_norm 1.7659 (1.6585) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][610/625] eta 0:00:05 lr 0.000866 wd 0.0500 time 0.3064 (0.3559) data time 0.0005 (0.0015) model time 0.3060 (0.3538) loss 3.1536 (3.1037) grad_norm 1.4209 (1.6622) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [172/300][620/625] eta 0:00:01 lr 0.000866 wd 0.0500 time 0.3403 (0.3556) data time 0.0004 (0.0015) model time 0.3399 (0.3536) loss 3.6137 (3.1070) grad_norm 1.4729 (1.6710) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 172 training takes 0:03:42 [2024-09-01 06:52:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:52:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:52:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.471 (0.471) Loss 0.5586 (0.5586) Acc@1 88.574 (88.574) Acc@5 98.096 (98.096) Mem 11224MB [2024-09-01 06:52:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.133) Loss 0.9370 (0.7081) Acc@1 76.904 (84.335) Acc@5 95.215 (97.110) Mem 11224MB [2024-09-01 06:52:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 1.0762 (0.8451) Acc@1 73.486 (80.992) Acc@5 93.701 (95.596) Mem 11224MB [2024-09-01 06:52:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.746 Acc@5 95.641 [2024-09-01 06:52:40 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.7% [2024-09-01 06:52:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 1.272 (1.272) Loss 0.5020 (0.5020) Acc@1 88.916 (88.916) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 06:52:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.209) Loss 0.8257 (0.6262) Acc@1 80.127 (85.929) Acc@5 96.045 (97.643) Mem 11224MB [2024-09-01 06:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.153) Loss 0.9404 (0.7445) Acc@1 77.051 (82.850) Acc@5 94.727 (96.496) Mem 11224MB [2024-09-01 06:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.584 Acc@5 96.501 [2024-09-01 06:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.6% [2024-09-01 06:52:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][0/625] eta 0:14:59 lr 0.000866 wd 0.0500 time 1.4389 (1.4389) data time 0.5943 (0.5943) model time 0.0000 (0.0000) loss 2.1539 (2.1539) grad_norm 1.4666 (1.4666) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][10/625] eta 0:04:35 lr 0.000866 wd 0.0500 time 0.3475 (0.4473) data time 0.0006 (0.0549) model time 0.0000 (0.0000) loss 2.0992 (2.9687) grad_norm 1.4187 (1.6017) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 06:52:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][20/625] eta 0:04:01 lr 0.000866 wd 0.0500 time 0.3488 (0.3991) data time 0.0008 (0.0292) model time 0.0000 (0.0000) loss 3.1872 (2.9964) grad_norm 1.2540 (inf) loss_scale 512.0000 (877.7143) mem 11224MB [2024-09-01 06:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][30/625] eta 0:03:48 lr 0.000866 wd 0.0500 time 0.3512 (0.3836) data time 0.0008 (0.0200) model time 0.0000 (0.0000) loss 3.0379 (3.0182) grad_norm 1.5341 (inf) loss_scale 512.0000 (759.7419) mem 11224MB [2024-09-01 06:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][40/625] eta 0:03:39 lr 0.000865 wd 0.0500 time 0.3507 (0.3749) data time 0.0007 (0.0153) model time 0.0000 (0.0000) loss 3.7808 (3.0884) grad_norm 2.0365 (inf) loss_scale 512.0000 (699.3171) mem 11224MB [2024-09-01 06:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][50/625] eta 0:03:32 lr 0.000865 wd 0.0500 time 0.3449 (0.3699) data time 0.0009 (0.0125) model time 0.0000 (0.0000) loss 2.4086 (3.1052) grad_norm 1.1336 (inf) loss_scale 512.0000 (662.5882) mem 11224MB [2024-09-01 06:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][60/625] eta 0:03:27 lr 0.000865 wd 0.0500 time 0.3439 (0.3665) data time 0.0009 (0.0106) model time 0.3430 (0.3480) loss 3.4058 (3.1547) grad_norm 2.0812 (inf) loss_scale 512.0000 (637.9016) mem 11224MB [2024-09-01 06:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][70/625] eta 0:03:21 lr 0.000865 wd 0.0500 time 0.3439 (0.3638) data time 0.0009 (0.0092) model time 0.3430 (0.3473) loss 3.1105 (3.1688) grad_norm 1.2868 (inf) loss_scale 512.0000 (620.1690) mem 11224MB [2024-09-01 06:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][80/625] eta 0:03:17 lr 0.000865 wd 0.0500 time 0.3466 (0.3617) data time 0.0009 (0.0082) model time 0.3457 (0.3469) loss 3.2751 (3.1297) grad_norm 2.0942 (inf) loss_scale 512.0000 (606.8148) mem 11224MB [2024-09-01 06:53:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][90/625] eta 0:03:12 lr 0.000865 wd 0.0500 time 0.3471 (0.3601) data time 0.0008 (0.0074) model time 0.3463 (0.3466) loss 2.2535 (3.1271) grad_norm 1.5903 (inf) loss_scale 512.0000 (596.3956) mem 11224MB [2024-09-01 06:53:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][100/625] eta 0:03:11 lr 0.000864 wd 0.0500 time 0.3486 (0.3640) data time 0.0006 (0.0067) model time 0.3480 (0.3571) loss 2.5998 (3.0997) grad_norm 1.7294 (inf) loss_scale 512.0000 (588.0396) mem 11224MB [2024-09-01 06:53:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][110/625] eta 0:03:06 lr 0.000864 wd 0.0500 time 0.3433 (0.3626) data time 0.0008 (0.0062) model time 0.3425 (0.3556) loss 3.0931 (3.1056) grad_norm 1.7148 (inf) loss_scale 512.0000 (581.1892) mem 11224MB [2024-09-01 06:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][120/625] eta 0:03:02 lr 0.000864 wd 0.0500 time 0.3502 (0.3614) data time 0.0008 (0.0058) model time 0.3494 (0.3544) loss 3.5075 (3.0930) grad_norm 1.7110 (inf) loss_scale 512.0000 (575.4711) mem 11224MB [2024-09-01 06:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][130/625] eta 0:02:58 lr 0.000864 wd 0.0500 time 0.3450 (0.3603) data time 0.0009 (0.0054) model time 0.3441 (0.3533) loss 3.1152 (3.0820) grad_norm 1.4167 (inf) loss_scale 512.0000 (570.6260) mem 11224MB [2024-09-01 06:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][140/625] eta 0:02:54 lr 0.000864 wd 0.0500 time 0.3560 (0.3594) data time 0.0007 (0.0051) model time 0.3553 (0.3526) loss 2.4377 (3.0795) grad_norm 1.3057 (inf) loss_scale 512.0000 (566.4681) mem 11224MB [2024-09-01 06:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][150/625] eta 0:02:50 lr 0.000863 wd 0.0500 time 0.3486 (0.3588) data time 0.0007 (0.0048) model time 0.3479 (0.3523) loss 2.8982 (3.0888) grad_norm 1.4181 (inf) loss_scale 512.0000 (562.8609) mem 11224MB [2024-09-01 06:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][160/625] eta 0:02:46 lr 0.000863 wd 0.0500 time 0.3459 (0.3581) data time 0.0009 (0.0045) model time 0.3450 (0.3518) loss 3.4165 (3.0922) grad_norm 2.1364 (inf) loss_scale 512.0000 (559.7019) mem 11224MB [2024-09-01 06:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][170/625] eta 0:02:42 lr 0.000863 wd 0.0500 time 0.3497 (0.3576) data time 0.0007 (0.0043) model time 0.3491 (0.3515) loss 2.0247 (3.0869) grad_norm 1.4210 (inf) loss_scale 512.0000 (556.9123) mem 11224MB [2024-09-01 06:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][180/625] eta 0:02:38 lr 0.000863 wd 0.0500 time 0.3513 (0.3571) data time 0.0008 (0.0041) model time 0.3505 (0.3512) loss 3.3362 (3.1065) grad_norm 1.8716 (inf) loss_scale 512.0000 (554.4309) mem 11224MB [2024-09-01 06:53:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][190/625] eta 0:02:35 lr 0.000863 wd 0.0500 time 0.3528 (0.3566) data time 0.0007 (0.0040) model time 0.3521 (0.3510) loss 2.8541 (3.1131) grad_norm 2.4491 (inf) loss_scale 512.0000 (552.2094) mem 11224MB [2024-09-01 06:53:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][200/625] eta 0:02:31 lr 0.000863 wd 0.0500 time 0.3587 (0.3563) data time 0.0006 (0.0038) model time 0.3581 (0.3509) loss 3.8184 (3.1228) grad_norm 2.1084 (inf) loss_scale 512.0000 (550.2090) mem 11224MB [2024-09-01 06:53:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][210/625] eta 0:02:27 lr 0.000862 wd 0.0500 time 0.3556 (0.3560) data time 0.0009 (0.0037) model time 0.3547 (0.3507) loss 2.7379 (3.1241) grad_norm 1.5999 (inf) loss_scale 512.0000 (548.3981) mem 11224MB [2024-09-01 06:54:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][220/625] eta 0:02:24 lr 0.000862 wd 0.0500 time 0.3474 (0.3557) data time 0.0008 (0.0035) model time 0.3466 (0.3505) loss 3.1724 (3.1255) grad_norm 1.2238 (inf) loss_scale 512.0000 (546.7511) mem 11224MB [2024-09-01 06:54:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][230/625] eta 0:02:20 lr 0.000862 wd 0.0500 time 0.3435 (0.3553) data time 0.0009 (0.0034) model time 0.3426 (0.3503) loss 3.1602 (3.1229) grad_norm 1.3524 (inf) loss_scale 512.0000 (545.2468) mem 11224MB [2024-09-01 06:54:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][240/625] eta 0:02:16 lr 0.000862 wd 0.0500 time 0.3455 (0.3550) data time 0.0007 (0.0033) model time 0.3448 (0.3501) loss 2.2228 (3.1218) grad_norm 1.8075 (inf) loss_scale 512.0000 (543.8672) mem 11224MB [2024-09-01 06:54:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][250/625] eta 0:02:13 lr 0.000862 wd 0.0500 time 0.3473 (0.3547) data time 0.0008 (0.0032) model time 0.3466 (0.3500) loss 3.6856 (3.1223) grad_norm 2.9410 (inf) loss_scale 512.0000 (542.5976) mem 11224MB [2024-09-01 06:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][260/625] eta 0:02:09 lr 0.000862 wd 0.0500 time 0.3467 (0.3544) data time 0.0009 (0.0031) model time 0.3458 (0.3498) loss 3.6974 (3.1218) grad_norm 1.9725 (inf) loss_scale 512.0000 (541.4253) mem 11224MB [2024-09-01 06:54:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][270/625] eta 0:02:05 lr 0.000861 wd 0.0500 time 0.3459 (0.3541) data time 0.0008 (0.0030) model time 0.3451 (0.3496) loss 2.6977 (3.1248) grad_norm 1.5554 (inf) loss_scale 512.0000 (540.3395) mem 11224MB [2024-09-01 06:54:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][280/625] eta 0:02:02 lr 0.000861 wd 0.0500 time 0.3492 (0.3540) data time 0.0009 (0.0030) model time 0.3483 (0.3496) loss 2.7607 (3.1217) grad_norm 1.1693 (inf) loss_scale 512.0000 (539.3310) mem 11224MB [2024-09-01 06:54:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][290/625] eta 0:01:58 lr 0.000861 wd 0.0500 time 0.3445 (0.3538) data time 0.0007 (0.0029) model time 0.3438 (0.3495) loss 2.7027 (3.1267) grad_norm 1.5062 (inf) loss_scale 512.0000 (538.3918) mem 11224MB [2024-09-01 06:54:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][300/625] eta 0:01:54 lr 0.000861 wd 0.0500 time 0.3523 (0.3537) data time 0.0009 (0.0028) model time 0.3514 (0.3495) loss 3.2549 (3.1280) grad_norm 1.3465 (inf) loss_scale 512.0000 (537.5150) mem 11224MB [2024-09-01 06:54:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][310/625] eta 0:01:51 lr 0.000861 wd 0.0500 time 0.3425 (0.3549) data time 0.0007 (0.0028) model time 0.3418 (0.3511) loss 3.7085 (3.1260) grad_norm 1.4924 (inf) loss_scale 512.0000 (536.6945) mem 11224MB [2024-09-01 06:54:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][320/625] eta 0:01:48 lr 0.000860 wd 0.0500 time 0.3501 (0.3546) data time 0.0008 (0.0027) model time 0.3493 (0.3509) loss 3.5175 (3.1243) grad_norm 3.1138 (inf) loss_scale 512.0000 (535.9252) mem 11224MB [2024-09-01 06:54:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][330/625] eta 0:01:44 lr 0.000860 wd 0.0500 time 0.3477 (0.3545) data time 0.0006 (0.0027) model time 0.3471 (0.3507) loss 2.0586 (3.1190) grad_norm 1.8707 (inf) loss_scale 512.0000 (535.2024) mem 11224MB [2024-09-01 06:54:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][340/625] eta 0:01:40 lr 0.000860 wd 0.0500 time 0.3522 (0.3543) data time 0.0007 (0.0027) model time 0.3516 (0.3506) loss 1.8971 (3.1159) grad_norm 1.8506 (inf) loss_scale 512.0000 (534.5220) mem 11224MB [2024-09-01 06:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][350/625] eta 0:01:37 lr 0.000860 wd 0.0500 time 0.3453 (0.3542) data time 0.0007 (0.0026) model time 0.3447 (0.3505) loss 3.9211 (3.1148) grad_norm 1.8086 (inf) loss_scale 512.0000 (533.8803) mem 11224MB [2024-09-01 06:54:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][360/625] eta 0:01:33 lr 0.000860 wd 0.0500 time 0.3518 (0.3540) data time 0.0009 (0.0026) model time 0.3509 (0.3504) loss 2.8821 (3.1160) grad_norm 1.5036 (inf) loss_scale 512.0000 (533.2742) mem 11224MB [2024-09-01 06:54:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][370/625] eta 0:01:30 lr 0.000860 wd 0.0500 time 0.3480 (0.3538) data time 0.0009 (0.0025) model time 0.3471 (0.3503) loss 3.3969 (3.1204) grad_norm 2.4008 (inf) loss_scale 512.0000 (532.7008) mem 11224MB [2024-09-01 06:54:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][380/625] eta 0:01:26 lr 0.000859 wd 0.0500 time 0.3454 (0.3537) data time 0.0006 (0.0025) model time 0.3448 (0.3503) loss 3.4956 (3.1252) grad_norm 1.9422 (inf) loss_scale 512.0000 (532.1575) mem 11224MB [2024-09-01 06:55:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][390/625] eta 0:01:23 lr 0.000859 wd 0.0500 time 0.3469 (0.3536) data time 0.0009 (0.0024) model time 0.3459 (0.3502) loss 2.8554 (3.1197) grad_norm 2.1365 (inf) loss_scale 512.0000 (531.6419) mem 11224MB [2024-09-01 06:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][400/625] eta 0:01:19 lr 0.000859 wd 0.0500 time 0.3479 (0.3534) data time 0.0007 (0.0024) model time 0.3472 (0.3501) loss 3.8326 (3.1186) grad_norm 2.1545 (inf) loss_scale 512.0000 (531.1521) mem 11224MB [2024-09-01 06:55:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][410/625] eta 0:01:15 lr 0.000859 wd 0.0500 time 0.3500 (0.3532) data time 0.0009 (0.0024) model time 0.3491 (0.3499) loss 3.3368 (3.1293) grad_norm 1.6273 (inf) loss_scale 512.0000 (530.6861) mem 11224MB [2024-09-01 06:55:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][420/625] eta 0:01:12 lr 0.000859 wd 0.0500 time 0.3428 (0.3531) data time 0.0008 (0.0023) model time 0.3420 (0.3498) loss 2.2769 (3.1266) grad_norm 1.4673 (inf) loss_scale 512.0000 (530.2423) mem 11224MB [2024-09-01 06:55:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][430/625] eta 0:01:08 lr 0.000859 wd 0.0500 time 0.3478 (0.3530) data time 0.0006 (0.0023) model time 0.3472 (0.3498) loss 3.2083 (3.1296) grad_norm 1.3342 (inf) loss_scale 512.0000 (529.8190) mem 11224MB [2024-09-01 06:55:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][440/625] eta 0:01:05 lr 0.000858 wd 0.0500 time 0.3543 (0.3529) data time 0.0006 (0.0023) model time 0.3536 (0.3498) loss 3.4827 (3.1246) grad_norm 1.0011 (inf) loss_scale 512.0000 (529.4150) mem 11224MB [2024-09-01 06:55:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][450/625] eta 0:01:01 lr 0.000858 wd 0.0500 time 0.3522 (0.3529) data time 0.0009 (0.0022) model time 0.3514 (0.3498) loss 2.7698 (3.1162) grad_norm 1.2619 (inf) loss_scale 512.0000 (529.0288) mem 11224MB [2024-09-01 06:55:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][460/625] eta 0:00:58 lr 0.000858 wd 0.0500 time 0.3486 (0.3527) data time 0.0006 (0.0022) model time 0.3480 (0.3497) loss 3.2578 (3.1173) grad_norm 1.4613 (inf) loss_scale 512.0000 (528.6594) mem 11224MB [2024-09-01 06:55:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][470/625] eta 0:00:54 lr 0.000858 wd 0.0500 time 0.3528 (0.3526) data time 0.0007 (0.0022) model time 0.3522 (0.3496) loss 2.8892 (3.1205) grad_norm 1.8410 (inf) loss_scale 512.0000 (528.3057) mem 11224MB [2024-09-01 06:55:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][480/625] eta 0:00:51 lr 0.000858 wd 0.0500 time 0.3504 (0.3526) data time 0.0008 (0.0021) model time 0.3496 (0.3496) loss 3.1977 (3.1215) grad_norm 1.8620 (inf) loss_scale 512.0000 (527.9667) mem 11224MB [2024-09-01 06:55:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][490/625] eta 0:00:47 lr 0.000858 wd 0.0500 time 0.5720 (0.3541) data time 0.0006 (0.0021) model time 0.5713 (0.3514) loss 2.5710 (3.1152) grad_norm 1.1907 (inf) loss_scale 512.0000 (527.6415) mem 11224MB [2024-09-01 06:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][500/625] eta 0:00:44 lr 0.000857 wd 0.0500 time 0.3497 (0.3540) data time 0.0007 (0.0021) model time 0.3491 (0.3513) loss 2.3611 (3.1144) grad_norm 1.3749 (inf) loss_scale 512.0000 (527.3293) mem 11224MB [2024-09-01 06:55:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][510/625] eta 0:00:40 lr 0.000857 wd 0.0500 time 0.3451 (0.3539) data time 0.0008 (0.0021) model time 0.3443 (0.3512) loss 2.5318 (3.1127) grad_norm 1.2818 (inf) loss_scale 512.0000 (527.0294) mem 11224MB [2024-09-01 06:55:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][520/625] eta 0:00:37 lr 0.000857 wd 0.0500 time 0.3507 (0.3537) data time 0.0007 (0.0020) model time 0.3500 (0.3511) loss 2.5997 (3.1142) grad_norm 2.0822 (inf) loss_scale 512.0000 (526.7409) mem 11224MB [2024-09-01 06:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][530/625] eta 0:00:33 lr 0.000857 wd 0.0500 time 0.3484 (0.3537) data time 0.0009 (0.0020) model time 0.3475 (0.3510) loss 3.5282 (3.1161) grad_norm 2.0174 (inf) loss_scale 512.0000 (526.4633) mem 11224MB [2024-09-01 06:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][540/625] eta 0:00:30 lr 0.000857 wd 0.0500 time 0.3468 (0.3535) data time 0.0006 (0.0020) model time 0.3462 (0.3509) loss 3.4457 (3.1153) grad_norm 1.3622 (inf) loss_scale 512.0000 (526.1959) mem 11224MB [2024-09-01 06:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][550/625] eta 0:00:26 lr 0.000856 wd 0.0500 time 0.3448 (0.3534) data time 0.0008 (0.0020) model time 0.3440 (0.3508) loss 3.3327 (3.1181) grad_norm 1.3819 (inf) loss_scale 512.0000 (525.9383) mem 11224MB [2024-09-01 06:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][560/625] eta 0:00:22 lr 0.000856 wd 0.0500 time 0.3482 (0.3533) data time 0.0008 (0.0019) model time 0.3474 (0.3508) loss 2.8454 (3.1136) grad_norm 1.3961 (inf) loss_scale 512.0000 (525.6898) mem 11224MB [2024-09-01 06:56:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][570/625] eta 0:00:19 lr 0.000856 wd 0.0500 time 0.3494 (0.3532) data time 0.0006 (0.0019) model time 0.3488 (0.3507) loss 3.1479 (3.1160) grad_norm 1.7290 (inf) loss_scale 512.0000 (525.4501) mem 11224MB [2024-09-01 06:56:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][580/625] eta 0:00:15 lr 0.000856 wd 0.0500 time 0.3466 (0.3531) data time 0.0008 (0.0019) model time 0.3458 (0.3506) loss 2.8734 (3.1160) grad_norm 1.9381 (inf) loss_scale 512.0000 (525.2186) mem 11224MB [2024-09-01 06:56:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][590/625] eta 0:00:12 lr 0.000856 wd 0.0500 time 0.3513 (0.3531) data time 0.0006 (0.0019) model time 0.3507 (0.3506) loss 2.3196 (3.1163) grad_norm 1.6316 (inf) loss_scale 512.0000 (524.9949) mem 11224MB [2024-09-01 06:56:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][600/625] eta 0:00:08 lr 0.000856 wd 0.0500 time 0.3491 (0.3530) data time 0.0006 (0.0019) model time 0.3484 (0.3505) loss 2.4286 (3.1165) grad_norm 1.5583 (inf) loss_scale 512.0000 (524.7787) mem 11224MB [2024-09-01 06:56:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][610/625] eta 0:00:05 lr 0.000855 wd 0.0500 time 0.3406 (0.3542) data time 0.0007 (0.0019) model time 0.3399 (0.3518) loss 1.7646 (3.1179) grad_norm 1.5931 (inf) loss_scale 512.0000 (524.5696) mem 11224MB [2024-09-01 06:56:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [173/300][620/625] eta 0:00:01 lr 0.000855 wd 0.0500 time 0.3394 (0.3539) data time 0.0006 (0.0019) model time 0.3388 (0.3516) loss 2.7875 (3.1161) grad_norm 1.5095 (inf) loss_scale 512.0000 (524.3671) mem 11224MB [2024-09-01 06:56:24 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 173 training takes 0:03:41 [2024-09-01 06:56:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 06:56:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 06:56:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.432 (0.432) Loss 0.5796 (0.5796) Acc@1 86.328 (86.328) Acc@5 97.998 (97.998) Mem 11224MB [2024-09-01 06:56:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.128) Loss 0.9336 (0.7006) Acc@1 76.660 (84.184) Acc@5 94.922 (96.964) Mem 11224MB [2024-09-01 06:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.111) Loss 1.0293 (0.8288) Acc@1 75.879 (80.997) Acc@5 93.701 (95.650) Mem 11224MB [2024-09-01 06:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.776 Acc@5 95.637 [2024-09-01 06:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.8% [2024-09-01 06:56:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.905 (0.905) Loss 0.5024 (0.5024) Acc@1 89.062 (89.062) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 06:56:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.172) Loss 0.8228 (0.6258) Acc@1 80.225 (85.991) Acc@5 96.045 (97.643) Mem 11224MB [2024-09-01 06:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.133) Loss 0.9399 (0.7437) Acc@1 77.197 (82.885) Acc@5 94.678 (96.489) Mem 11224MB [2024-09-01 06:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.612 Acc@5 96.501 [2024-09-01 06:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.6% [2024-09-01 06:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.61% [2024-09-01 06:56:31 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 06:56:32 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 06:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][0/625] eta 0:07:07 lr 0.000855 wd 0.0500 time 0.6839 (0.6839) data time 0.3700 (0.3700) model time 0.0000 (0.0000) loss 3.6095 (3.6095) grad_norm 1.3896 (1.3896) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][10/625] eta 0:03:52 lr 0.000855 wd 0.0500 time 0.3462 (0.3785) data time 0.0008 (0.0345) model time 0.0000 (0.0000) loss 3.0251 (2.9354) grad_norm 2.4232 (2.1289) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][20/625] eta 0:03:40 lr 0.000855 wd 0.0500 time 0.3478 (0.3640) data time 0.0007 (0.0184) model time 0.0000 (0.0000) loss 3.5243 (3.0564) grad_norm 1.4624 (2.0049) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:56:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][30/625] eta 0:03:33 lr 0.000855 wd 0.0500 time 0.3450 (0.3585) data time 0.0008 (0.0128) model time 0.0000 (0.0000) loss 2.5114 (3.0682) grad_norm 1.2695 (1.8656) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:56:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][40/625] eta 0:03:28 lr 0.000854 wd 0.0500 time 0.3527 (0.3558) data time 0.0006 (0.0099) model time 0.0000 (0.0000) loss 2.3441 (3.1238) grad_norm 1.3418 (1.7813) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:56:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][50/625] eta 0:03:23 lr 0.000854 wd 0.0500 time 0.3496 (0.3544) data time 0.0008 (0.0081) model time 0.0000 (0.0000) loss 2.8184 (3.0974) grad_norm 1.9440 (1.7929) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:56:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][60/625] eta 0:03:19 lr 0.000854 wd 0.0500 time 0.3452 (0.3535) data time 0.0006 (0.0069) model time 0.3446 (0.3481) loss 1.7067 (3.0822) grad_norm 3.0800 (1.8632) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:56:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][70/625] eta 0:03:16 lr 0.000854 wd 0.0500 time 0.3132 (0.3541) data time 0.0008 (0.0061) model time 0.3124 (0.3526) loss 2.5189 (3.0884) grad_norm 1.6913 (1.8524) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][80/625] eta 0:03:12 lr 0.000854 wd 0.0500 time 0.3516 (0.3536) data time 0.0006 (0.0054) model time 0.3509 (0.3514) loss 3.7816 (3.1073) grad_norm 1.4145 (1.8078) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][90/625] eta 0:03:08 lr 0.000854 wd 0.0500 time 0.3544 (0.3532) data time 0.0006 (0.0049) model time 0.3538 (0.3509) loss 3.2639 (3.1162) grad_norm 1.4605 (1.7761) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][100/625] eta 0:03:05 lr 0.000853 wd 0.0500 time 0.3423 (0.3527) data time 0.0008 (0.0045) model time 0.3415 (0.3502) loss 3.0333 (3.1318) grad_norm 2.4438 (1.7704) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][110/625] eta 0:03:01 lr 0.000853 wd 0.0500 time 0.3556 (0.3525) data time 0.0007 (0.0042) model time 0.3548 (0.3500) loss 3.5980 (3.1205) grad_norm 1.5713 (1.7814) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][120/625] eta 0:02:57 lr 0.000853 wd 0.0500 time 0.3499 (0.3522) data time 0.0006 (0.0039) model time 0.3492 (0.3498) loss 3.2995 (3.1058) grad_norm 1.3488 (1.7862) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][130/625] eta 0:02:54 lr 0.000853 wd 0.0500 time 0.3456 (0.3520) data time 0.0007 (0.0037) model time 0.3449 (0.3497) loss 2.7758 (3.0981) grad_norm 2.3192 (1.8216) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][140/625] eta 0:02:50 lr 0.000853 wd 0.0500 time 0.3435 (0.3518) data time 0.0007 (0.0035) model time 0.3429 (0.3496) loss 2.6793 (3.1021) grad_norm 1.2911 (1.8154) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][150/625] eta 0:02:46 lr 0.000852 wd 0.0500 time 0.3484 (0.3516) data time 0.0009 (0.0033) model time 0.3475 (0.3493) loss 2.5791 (3.0880) grad_norm 1.8203 (1.8034) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][160/625] eta 0:02:43 lr 0.000852 wd 0.0500 time 0.3480 (0.3513) data time 0.0006 (0.0032) model time 0.3474 (0.3490) loss 3.1937 (3.0904) grad_norm 1.4314 (1.7895) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][170/625] eta 0:02:41 lr 0.000852 wd 0.0500 time 0.3489 (0.3544) data time 0.0008 (0.0030) model time 0.3480 (0.3536) loss 1.9897 (3.0818) grad_norm 1.9232 (1.7885) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][180/625] eta 0:02:37 lr 0.000852 wd 0.0500 time 0.3445 (0.3550) data time 0.0006 (0.0029) model time 0.3438 (0.3544) loss 4.0287 (3.0954) grad_norm 1.7323 (1.7941) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][190/625] eta 0:02:34 lr 0.000852 wd 0.0500 time 0.3489 (0.3546) data time 0.0008 (0.0028) model time 0.3482 (0.3538) loss 3.5740 (3.0990) grad_norm 1.6343 (1.7916) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][200/625] eta 0:02:30 lr 0.000852 wd 0.0500 time 0.3524 (0.3543) data time 0.0006 (0.0027) model time 0.3517 (0.3534) loss 2.3783 (3.1014) grad_norm 2.1120 (1.7991) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][210/625] eta 0:02:26 lr 0.000851 wd 0.0500 time 0.3486 (0.3541) data time 0.0008 (0.0026) model time 0.3478 (0.3531) loss 3.6846 (3.1081) grad_norm 1.8452 (1.7933) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][220/625] eta 0:02:23 lr 0.000851 wd 0.0500 time 0.3535 (0.3539) data time 0.0008 (0.0025) model time 0.3527 (0.3529) loss 2.0012 (3.1050) grad_norm 2.0859 (1.7865) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][230/625] eta 0:02:19 lr 0.000851 wd 0.0500 time 0.3464 (0.3537) data time 0.0008 (0.0024) model time 0.3456 (0.3527) loss 3.2927 (3.1077) grad_norm 2.2417 (1.7780) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][240/625] eta 0:02:16 lr 0.000851 wd 0.0500 time 0.3508 (0.3535) data time 0.0007 (0.0024) model time 0.3502 (0.3524) loss 3.1327 (3.1196) grad_norm 1.0240 (1.7640) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][250/625] eta 0:02:12 lr 0.000851 wd 0.0500 time 0.3482 (0.3532) data time 0.0008 (0.0023) model time 0.3474 (0.3521) loss 2.3184 (3.1179) grad_norm 1.5982 (1.7498) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][260/625] eta 0:02:08 lr 0.000851 wd 0.0500 time 0.3472 (0.3532) data time 0.0008 (0.0023) model time 0.3464 (0.3520) loss 2.9247 (3.1213) grad_norm 1.3014 (1.7372) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][270/625] eta 0:02:05 lr 0.000850 wd 0.0500 time 0.3483 (0.3530) data time 0.0008 (0.0022) model time 0.3475 (0.3519) loss 3.2152 (3.1218) grad_norm 2.0827 (1.7305) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][280/625] eta 0:02:01 lr 0.000850 wd 0.0500 time 0.4996 (0.3534) data time 0.0009 (0.0022) model time 0.4987 (0.3523) loss 3.5696 (3.1246) grad_norm 1.1824 (1.7171) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][290/625] eta 0:01:58 lr 0.000850 wd 0.0500 time 0.3454 (0.3546) data time 0.0006 (0.0021) model time 0.3448 (0.3538) loss 3.6628 (3.1296) grad_norm 1.6123 (1.7155) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][300/625] eta 0:01:55 lr 0.000850 wd 0.0500 time 0.3472 (0.3551) data time 0.0007 (0.0021) model time 0.3465 (0.3544) loss 3.9339 (3.1329) grad_norm 2.3282 (1.7137) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][310/625] eta 0:01:51 lr 0.000850 wd 0.0500 time 0.3487 (0.3549) data time 0.0008 (0.0020) model time 0.3479 (0.3541) loss 2.9617 (3.1359) grad_norm 1.9791 (1.7157) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][320/625] eta 0:01:48 lr 0.000850 wd 0.0500 time 0.3469 (0.3546) data time 0.0006 (0.0020) model time 0.3463 (0.3538) loss 2.5831 (3.1290) grad_norm 1.5405 (1.7115) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][330/625] eta 0:01:44 lr 0.000849 wd 0.0500 time 0.3429 (0.3543) data time 0.0009 (0.0020) model time 0.3420 (0.3535) loss 2.7554 (3.1173) grad_norm 1.1109 (1.7041) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][340/625] eta 0:01:40 lr 0.000849 wd 0.0500 time 0.3457 (0.3541) data time 0.0008 (0.0019) model time 0.3449 (0.3532) loss 2.3426 (3.1121) grad_norm 1.1030 (1.6973) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][350/625] eta 0:01:37 lr 0.000849 wd 0.0500 time 0.3474 (0.3539) data time 0.0006 (0.0019) model time 0.3468 (0.3530) loss 3.8385 (3.1158) grad_norm 1.8625 (1.6929) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][360/625] eta 0:01:33 lr 0.000849 wd 0.0500 time 0.3428 (0.3537) data time 0.0007 (0.0019) model time 0.3421 (0.3528) loss 3.1536 (3.1207) grad_norm 1.1500 (1.6967) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][370/625] eta 0:01:30 lr 0.000849 wd 0.0500 time 0.3539 (0.3536) data time 0.0008 (0.0018) model time 0.3531 (0.3527) loss 2.0678 (3.1149) grad_norm 1.3776 (1.6974) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][380/625] eta 0:01:26 lr 0.000848 wd 0.0500 time 0.3488 (0.3535) data time 0.0009 (0.0018) model time 0.3479 (0.3525) loss 2.6881 (3.1145) grad_norm 1.1807 (1.6911) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][390/625] eta 0:01:23 lr 0.000848 wd 0.0500 time 0.3492 (0.3534) data time 0.0006 (0.0018) model time 0.3485 (0.3524) loss 3.5472 (3.1148) grad_norm 2.2195 (1.6937) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][400/625] eta 0:01:19 lr 0.000848 wd 0.0500 time 0.3503 (0.3533) data time 0.0009 (0.0018) model time 0.3494 (0.3523) loss 3.1786 (3.1150) grad_norm 1.6110 (1.6929) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:58:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][410/625] eta 0:01:15 lr 0.000848 wd 0.0500 time 0.3508 (0.3532) data time 0.0007 (0.0017) model time 0.3501 (0.3522) loss 3.5180 (3.1115) grad_norm 1.2297 (1.6861) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][420/625] eta 0:01:12 lr 0.000848 wd 0.0500 time 0.3461 (0.3530) data time 0.0006 (0.0017) model time 0.3455 (0.3520) loss 3.5925 (3.1083) grad_norm 1.5715 (1.6844) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][430/625] eta 0:01:08 lr 0.000848 wd 0.0500 time 0.3501 (0.3529) data time 0.0010 (0.0017) model time 0.3491 (0.3519) loss 3.3975 (3.1096) grad_norm 1.6047 (1.6809) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][440/625] eta 0:01:05 lr 0.000847 wd 0.0500 time 0.3533 (0.3528) data time 0.0008 (0.0017) model time 0.3524 (0.3518) loss 3.6006 (3.1078) grad_norm 1.2264 (1.6743) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][450/625] eta 0:01:01 lr 0.000847 wd 0.0500 time 0.3563 (0.3527) data time 0.0009 (0.0017) model time 0.3554 (0.3517) loss 2.9418 (3.1083) grad_norm 1.7265 (1.6722) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][460/625] eta 0:00:58 lr 0.000847 wd 0.0500 time 0.3486 (0.3526) data time 0.0006 (0.0016) model time 0.3480 (0.3516) loss 2.8716 (3.1017) grad_norm 1.6313 (1.6759) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][470/625] eta 0:00:54 lr 0.000847 wd 0.0500 time 0.3512 (0.3529) data time 0.0007 (0.0016) model time 0.3505 (0.3519) loss 3.0108 (3.1062) grad_norm 1.6431 (1.6830) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][480/625] eta 0:00:51 lr 0.000847 wd 0.0500 time 0.3406 (0.3528) data time 0.0006 (0.0016) model time 0.3400 (0.3517) loss 3.0960 (3.1053) grad_norm 2.1151 (1.6902) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][490/625] eta 0:00:47 lr 0.000847 wd 0.0500 time 0.3504 (0.3526) data time 0.0007 (0.0016) model time 0.3498 (0.3516) loss 3.4719 (3.1086) grad_norm 1.2389 (1.6900) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][500/625] eta 0:00:44 lr 0.000846 wd 0.0500 time 0.5654 (0.3538) data time 0.0009 (0.0016) model time 0.5646 (0.3529) loss 3.0375 (3.1116) grad_norm 2.1001 (1.6842) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][510/625] eta 0:00:40 lr 0.000846 wd 0.0500 time 0.3409 (0.3536) data time 0.0009 (0.0016) model time 0.3401 (0.3527) loss 2.9028 (3.1137) grad_norm 2.4426 (1.6836) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][520/625] eta 0:00:37 lr 0.000846 wd 0.0500 time 0.3463 (0.3534) data time 0.0008 (0.0015) model time 0.3455 (0.3525) loss 3.6944 (3.1176) grad_norm 1.8763 (1.6837) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][530/625] eta 0:00:33 lr 0.000846 wd 0.0500 time 0.3515 (0.3533) data time 0.0008 (0.0015) model time 0.3507 (0.3523) loss 2.6921 (3.1197) grad_norm 1.3458 (1.6797) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][540/625] eta 0:00:30 lr 0.000846 wd 0.0500 time 0.3442 (0.3532) data time 0.0010 (0.0015) model time 0.3432 (0.3522) loss 3.5315 (3.1174) grad_norm 2.2630 (1.6787) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][550/625] eta 0:00:26 lr 0.000845 wd 0.0500 time 0.3489 (0.3531) data time 0.0009 (0.0015) model time 0.3480 (0.3521) loss 2.6943 (3.1175) grad_norm 1.6375 (1.6790) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][560/625] eta 0:00:22 lr 0.000845 wd 0.0500 time 0.3506 (0.3530) data time 0.0008 (0.0015) model time 0.3498 (0.3520) loss 3.0968 (3.1193) grad_norm 1.7401 (1.6761) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][570/625] eta 0:00:19 lr 0.000845 wd 0.0500 time 0.3535 (0.3530) data time 0.0008 (0.0015) model time 0.3527 (0.3520) loss 2.5587 (3.1144) grad_norm 2.2259 (1.6742) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 06:59:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][580/625] eta 0:00:15 lr 0.000845 wd 0.0500 time 0.3462 (0.3529) data time 0.0009 (0.0015) model time 0.3452 (0.3519) loss 2.9971 (3.1142) grad_norm 1.1097 (1.6797) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][590/625] eta 0:00:12 lr 0.000845 wd 0.0500 time 0.3499 (0.3531) data time 0.0009 (0.0015) model time 0.3490 (0.3521) loss 2.9415 (3.1122) grad_norm 1.7321 (1.6803) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][600/625] eta 0:00:08 lr 0.000845 wd 0.0500 time 0.3462 (0.3530) data time 0.0006 (0.0015) model time 0.3456 (0.3520) loss 3.4637 (3.1128) grad_norm 1.5472 (1.6774) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][610/625] eta 0:00:05 lr 0.000844 wd 0.0500 time 0.3482 (0.3530) data time 0.0004 (0.0015) model time 0.3477 (0.3520) loss 3.4361 (3.1118) grad_norm 1.4323 (1.6754) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [174/300][620/625] eta 0:00:01 lr 0.000844 wd 0.0500 time 0.3465 (0.3529) data time 0.0004 (0.0014) model time 0.3461 (0.3519) loss 3.7135 (3.1120) grad_norm 1.3375 (1.6733) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:13 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 174 training takes 0:03:40 [2024-09-01 07:00:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:00:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.536 (0.536) Loss 0.5396 (0.5396) Acc@1 88.184 (88.184) Acc@5 98.242 (98.242) Mem 11224MB [2024-09-01 07:00:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 0.9282 (0.6838) Acc@1 77.148 (84.446) Acc@5 95.117 (97.084) Mem 11224MB [2024-09-01 07:00:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.097 (0.120) Loss 1.0410 (0.8142) Acc@1 74.658 (81.220) Acc@5 94.141 (95.671) Mem 11224MB [2024-09-01 07:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.972 Acc@5 95.689 [2024-09-01 07:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.0% [2024-09-01 07:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 80.97% [2024-09-01 07:00:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 07:00:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 07:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.451 (0.451) Loss 0.5024 (0.5024) Acc@1 89.014 (89.014) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 07:00:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.132) Loss 0.8232 (0.6250) Acc@1 80.371 (86.026) Acc@5 96.094 (97.665) Mem 11224MB [2024-09-01 07:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 0.9385 (0.7430) Acc@1 77.051 (82.926) Acc@5 94.678 (96.482) Mem 11224MB [2024-09-01 07:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.660 Acc@5 96.491 [2024-09-01 07:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.7% [2024-09-01 07:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.66% [2024-09-01 07:00:21 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:00:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][0/625] eta 0:07:52 lr 0.000844 wd 0.0500 time 0.7561 (0.7561) data time 0.4445 (0.4445) model time 0.0000 (0.0000) loss 3.3260 (3.3260) grad_norm 1.6904 (1.6904) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][10/625] eta 0:03:57 lr 0.000844 wd 0.0500 time 0.3513 (0.3864) data time 0.0007 (0.0412) model time 0.0000 (0.0000) loss 3.7718 (3.0738) grad_norm 1.6064 (1.8627) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][20/625] eta 0:03:42 lr 0.000844 wd 0.0500 time 0.3438 (0.3673) data time 0.0007 (0.0220) model time 0.0000 (0.0000) loss 2.3665 (3.0280) grad_norm 2.1523 (1.8695) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][30/625] eta 0:06:47 lr 0.000844 wd 0.0500 time 0.3511 (0.6844) data time 0.0008 (0.0152) model time 0.0000 (0.0000) loss 2.8079 (3.0440) grad_norm 2.3332 (1.8845) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][40/625] eta 0:06:10 lr 0.000843 wd 0.0500 time 0.3651 (0.6335) data time 0.0009 (0.0117) model time 0.0000 (0.0000) loss 3.2195 (3.0441) grad_norm 1.3464 (1.7970) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][50/625] eta 0:05:33 lr 0.000843 wd 0.0500 time 0.3521 (0.5794) data time 0.0008 (0.0096) model time 0.0000 (0.0000) loss 2.8650 (3.0201) grad_norm 1.2420 (1.7850) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][60/625] eta 0:05:08 lr 0.000843 wd 0.0500 time 0.3440 (0.5453) data time 0.0008 (0.0082) model time 0.3432 (0.3701) loss 3.6746 (3.0503) grad_norm 1.5124 (1.7450) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:00:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][70/625] eta 0:04:50 lr 0.000843 wd 0.0500 time 0.3392 (0.5234) data time 0.0007 (0.0071) model time 0.3385 (0.3797) loss 3.2665 (3.0480) grad_norm 1.1935 (1.7364) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][80/625] eta 0:04:34 lr 0.000843 wd 0.0500 time 0.3154 (0.5035) data time 0.0007 (0.0064) model time 0.3147 (0.3734) loss 3.1006 (3.0531) grad_norm 1.3976 (1.7159) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][90/625] eta 0:04:20 lr 0.000843 wd 0.0500 time 0.3412 (0.4867) data time 0.0007 (0.0058) model time 0.3405 (0.3675) loss 3.2818 (3.0498) grad_norm 1.2759 (1.7013) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][100/625] eta 0:04:08 lr 0.000842 wd 0.0500 time 0.3559 (0.4730) data time 0.0008 (0.0053) model time 0.3551 (0.3635) loss 3.1971 (3.0489) grad_norm 1.0722 (1.6837) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][110/625] eta 0:04:00 lr 0.000842 wd 0.0500 time 0.3514 (0.4668) data time 0.0006 (0.0049) model time 0.3508 (0.3701) loss 2.0656 (3.0618) grad_norm 1.6001 (1.7110) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][120/625] eta 0:03:50 lr 0.000842 wd 0.0500 time 0.3484 (0.4570) data time 0.0005 (0.0046) model time 0.3478 (0.3669) loss 2.5854 (3.0621) grad_norm 2.0976 (1.7008) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][130/625] eta 0:03:43 lr 0.000842 wd 0.0500 time 0.3505 (0.4524) data time 0.0006 (0.0072) model time 0.3499 (0.3656) loss 3.2781 (3.0716) grad_norm 1.0050 (1.7005) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][140/625] eta 0:03:35 lr 0.000842 wd 0.0500 time 0.3438 (0.4450) data time 0.0008 (0.0068) model time 0.3430 (0.3636) loss 3.3586 (3.0725) grad_norm 2.6589 (1.7807) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][150/625] eta 0:03:28 lr 0.000842 wd 0.0500 time 0.3425 (0.4390) data time 0.0008 (0.0064) model time 0.3417 (0.3626) loss 3.2895 (3.0625) grad_norm 2.1194 (1.7879) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][160/625] eta 0:03:21 lr 0.000841 wd 0.0500 time 0.3463 (0.4336) data time 0.0017 (0.0060) model time 0.3446 (0.3616) loss 3.3140 (3.0681) grad_norm 1.9309 (1.7807) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][170/625] eta 0:03:15 lr 0.000841 wd 0.0500 time 0.3500 (0.4287) data time 0.0008 (0.0057) model time 0.3492 (0.3605) loss 2.9820 (3.0619) grad_norm 1.2654 (1.7643) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][180/625] eta 0:03:08 lr 0.000841 wd 0.0500 time 0.3521 (0.4242) data time 0.0006 (0.0055) model time 0.3515 (0.3595) loss 3.4030 (3.0627) grad_norm 1.8360 (1.7631) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][190/625] eta 0:03:02 lr 0.000841 wd 0.0500 time 0.3527 (0.4204) data time 0.0009 (0.0052) model time 0.3518 (0.3588) loss 3.3097 (3.0585) grad_norm 1.9879 (1.7589) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][200/625] eta 0:02:57 lr 0.000841 wd 0.0500 time 0.3372 (0.4169) data time 0.0008 (0.0050) model time 0.3364 (0.3582) loss 3.2025 (3.0616) grad_norm 3.9363 (1.7615) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][210/625] eta 0:02:51 lr 0.000840 wd 0.0500 time 0.3525 (0.4137) data time 0.0009 (0.0048) model time 0.3516 (0.3575) loss 2.8373 (3.0681) grad_norm 1.7729 (1.7578) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:01:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][220/625] eta 0:02:49 lr 0.000840 wd 0.0500 time 2.0479 (0.4184) data time 0.0006 (0.0046) model time 2.0473 (0.3670) loss 3.9015 (3.0765) grad_norm 2.6161 (1.7583) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][230/625] eta 0:02:51 lr 0.000840 wd 0.0500 time 0.3520 (0.4337) data time 0.0006 (0.0045) model time 0.3514 (0.3894) loss 3.8303 (3.0865) grad_norm 1.4061 (1.7514) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][240/625] eta 0:02:47 lr 0.000840 wd 0.0500 time 0.3419 (0.4338) data time 0.0008 (0.0043) model time 0.3410 (0.3918) loss 3.2608 (3.0781) grad_norm 2.8883 (1.7640) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][250/625] eta 0:02:44 lr 0.000840 wd 0.0500 time 0.3492 (0.4394) data time 0.0007 (0.0042) model time 0.3485 (0.4008) loss 3.4894 (3.0862) grad_norm 2.0263 (1.7652) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][260/625] eta 0:02:39 lr 0.000840 wd 0.0500 time 0.3469 (0.4368) data time 0.0008 (0.0041) model time 0.3461 (0.3994) loss 2.4683 (3.0858) grad_norm 1.7551 (1.7582) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][270/625] eta 0:02:34 lr 0.000839 wd 0.0500 time 0.3487 (0.4339) data time 0.0010 (0.0039) model time 0.3478 (0.3976) loss 3.2318 (3.0940) grad_norm 2.1088 (1.7569) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][280/625] eta 0:02:28 lr 0.000839 wd 0.0500 time 0.3515 (0.4310) data time 0.0008 (0.0038) model time 0.3506 (0.3955) loss 2.1250 (3.0949) grad_norm 1.6230 (1.7464) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][290/625] eta 0:02:23 lr 0.000839 wd 0.0500 time 0.3426 (0.4281) data time 0.0009 (0.0037) model time 0.3418 (0.3934) loss 3.3646 (3.1030) grad_norm 1.6498 (1.7405) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][300/625] eta 0:02:18 lr 0.000839 wd 0.0500 time 0.3444 (0.4255) data time 0.0008 (0.0036) model time 0.3436 (0.3916) loss 2.4715 (3.0988) grad_norm 2.4844 (1.7542) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][310/625] eta 0:02:13 lr 0.000839 wd 0.0500 time 0.3478 (0.4230) data time 0.0009 (0.0035) model time 0.3470 (0.3899) loss 3.5977 (3.1068) grad_norm 1.7513 (1.7579) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][320/625] eta 0:02:08 lr 0.000839 wd 0.0500 time 0.3448 (0.4207) data time 0.0008 (0.0035) model time 0.3440 (0.3884) loss 2.9622 (3.1092) grad_norm 1.5584 (1.7532) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][330/625] eta 0:02:03 lr 0.000838 wd 0.0500 time 0.3381 (0.4185) data time 0.0008 (0.0034) model time 0.3373 (0.3870) loss 3.7033 (3.1135) grad_norm 1.4699 (1.7483) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][340/625] eta 0:01:58 lr 0.000838 wd 0.0500 time 0.3458 (0.4166) data time 0.0007 (0.0033) model time 0.3451 (0.3858) loss 2.1232 (3.1069) grad_norm 2.1101 (1.7654) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:02:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][350/625] eta 0:01:54 lr 0.000838 wd 0.0500 time 0.3447 (0.4147) data time 0.0008 (0.0032) model time 0.3439 (0.3845) loss 2.6746 (3.1063) grad_norm 2.5891 (1.7805) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][360/625] eta 0:01:56 lr 0.000838 wd 0.0500 time 10.1926 (0.4400) data time 0.0007 (0.0032) model time 10.1919 (0.4150) loss 3.8177 (3.1083) grad_norm 1.6350 (1.7805) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][370/625] eta 0:01:52 lr 0.000838 wd 0.0500 time 0.5717 (0.4416) data time 0.0009 (0.0037) model time 0.5708 (0.4169) loss 3.1593 (3.1139) grad_norm 1.3787 (1.7748) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][380/625] eta 0:01:47 lr 0.000838 wd 0.0500 time 0.3531 (0.4393) data time 0.0007 (0.0036) model time 0.3525 (0.4149) loss 3.3091 (3.1109) grad_norm 1.6454 (1.7663) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][390/625] eta 0:01:42 lr 0.000837 wd 0.0500 time 0.3557 (0.4374) data time 0.0006 (0.0035) model time 0.3551 (0.4135) loss 3.6125 (3.1157) grad_norm 1.5658 (1.7607) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][400/625] eta 0:01:37 lr 0.000837 wd 0.0500 time 0.3494 (0.4351) data time 0.0006 (0.0034) model time 0.3488 (0.4116) loss 2.2436 (3.1189) grad_norm 1.7328 (1.7540) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][410/625] eta 0:01:33 lr 0.000837 wd 0.0500 time 0.3472 (0.4330) data time 0.0006 (0.0034) model time 0.3466 (0.4098) loss 1.9462 (3.1150) grad_norm 1.2727 (1.7481) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][420/625] eta 0:01:28 lr 0.000837 wd 0.0500 time 0.3459 (0.4310) data time 0.0009 (0.0033) model time 0.3450 (0.4081) loss 2.4526 (3.1072) grad_norm 1.1115 (1.7426) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][430/625] eta 0:01:24 lr 0.000837 wd 0.0500 time 0.3521 (0.4352) data time 0.0007 (0.0033) model time 0.3513 (0.4134) loss 1.9459 (3.1038) grad_norm 2.1062 (1.7497) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][440/625] eta 0:01:20 lr 0.000836 wd 0.0500 time 0.3510 (0.4332) data time 0.0007 (0.0032) model time 0.3503 (0.4117) loss 4.0187 (3.1083) grad_norm 2.0201 (1.7509) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][450/625] eta 0:01:15 lr 0.000836 wd 0.0500 time 0.3456 (0.4314) data time 0.0007 (0.0032) model time 0.3448 (0.4102) loss 3.2621 (3.1160) grad_norm 1.5176 (1.7455) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][460/625] eta 0:01:10 lr 0.000836 wd 0.0500 time 0.3486 (0.4296) data time 0.0007 (0.0031) model time 0.3479 (0.4087) loss 3.2636 (3.1175) grad_norm 1.2289 (1.7399) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][470/625] eta 0:01:06 lr 0.000836 wd 0.0500 time 0.6142 (0.4309) data time 0.0007 (0.0031) model time 0.6135 (0.4106) loss 3.5054 (3.1163) grad_norm 1.7345 (1.7371) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][480/625] eta 0:01:02 lr 0.000836 wd 0.0500 time 0.3425 (0.4304) data time 0.0007 (0.0030) model time 0.3418 (0.4105) loss 3.7443 (3.1169) grad_norm 1.5683 (1.7333) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][490/625] eta 0:00:57 lr 0.000836 wd 0.0500 time 0.3465 (0.4287) data time 0.0006 (0.0030) model time 0.3459 (0.4091) loss 3.4600 (3.1167) grad_norm 1.5390 (1.7359) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:03:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][500/625] eta 0:00:54 lr 0.000835 wd 0.0500 time 0.3493 (0.4335) data time 0.0009 (0.0035) model time 0.3484 (0.4141) loss 2.5231 (3.1224) grad_norm 1.1084 (1.7386) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][510/625] eta 0:00:49 lr 0.000835 wd 0.0500 time 0.3538 (0.4319) data time 0.0006 (0.0034) model time 0.3532 (0.4128) loss 2.8200 (3.1183) grad_norm 1.8640 (1.7398) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][520/625] eta 0:00:45 lr 0.000835 wd 0.0500 time 0.3484 (0.4303) data time 0.0009 (0.0034) model time 0.3475 (0.4113) loss 2.9321 (3.1159) grad_norm 1.5772 (1.7358) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][530/625] eta 0:00:41 lr 0.000835 wd 0.0500 time 0.3761 (0.4335) data time 0.0007 (0.0059) model time 0.3753 (0.4125) loss 3.8433 (3.1201) grad_norm 1.2467 (1.7311) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][540/625] eta 0:00:36 lr 0.000835 wd 0.0500 time 0.3457 (0.4332) data time 0.0006 (0.0058) model time 0.3451 (0.4126) loss 2.5957 (3.1191) grad_norm 2.4314 (1.7339) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][550/625] eta 0:00:32 lr 0.000835 wd 0.0500 time 0.3478 (0.4324) data time 0.0008 (0.0057) model time 0.3470 (0.4120) loss 3.1445 (3.1216) grad_norm 1.4055 (1.7353) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][560/625] eta 0:00:28 lr 0.000834 wd 0.0500 time 0.3448 (0.4309) data time 0.0008 (0.0056) model time 0.3440 (0.4108) loss 3.1495 (3.1237) grad_norm 2.2817 (1.7325) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][570/625] eta 0:00:23 lr 0.000834 wd 0.0500 time 0.3469 (0.4294) data time 0.0006 (0.0055) model time 0.3463 (0.4095) loss 2.3909 (3.1182) grad_norm 1.1490 (1.7333) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][580/625] eta 0:00:19 lr 0.000834 wd 0.0500 time 0.3543 (0.4295) data time 0.0008 (0.0055) model time 0.3535 (0.4100) loss 3.0977 (3.1137) grad_norm 1.9724 (1.7339) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][590/625] eta 0:00:14 lr 0.000834 wd 0.0500 time 0.3437 (0.4285) data time 0.0006 (0.0054) model time 0.3430 (0.4092) loss 2.6704 (3.1115) grad_norm 2.2387 (1.7387) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][600/625] eta 0:00:10 lr 0.000834 wd 0.0500 time 0.3475 (0.4275) data time 0.0009 (0.0053) model time 0.3467 (0.4085) loss 3.1962 (3.1094) grad_norm 1.4264 (1.7385) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][610/625] eta 0:00:06 lr 0.000833 wd 0.0500 time 0.3415 (0.4262) data time 0.0005 (0.0052) model time 0.3409 (0.4074) loss 2.4524 (3.1081) grad_norm 1.4328 (1.7361) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [175/300][620/625] eta 0:00:02 lr 0.000833 wd 0.0500 time 0.3370 (0.4248) data time 0.0006 (0.0052) model time 0.3364 (0.4062) loss 2.2725 (3.1045) grad_norm 1.7218 (1.7323) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:04:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 175 training takes 0:04:25 [2024-09-01 07:04:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:04:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:04:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 10.495 (10.495) Loss 0.5503 (0.5503) Acc@1 88.477 (88.477) Acc@5 98.242 (98.242) Mem 11224MB [2024-09-01 07:05:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (1.140) Loss 0.9409 (0.6973) Acc@1 76.807 (84.468) Acc@5 95.117 (97.155) Mem 11224MB [2024-09-01 07:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.641) Loss 1.0039 (0.8303) Acc@1 75.098 (81.120) Acc@5 94.189 (95.712) Mem 11224MB [2024-09-01 07:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.898 Acc@5 95.683 [2024-09-01 07:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.9% [2024-09-01 07:05:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 6.594 (6.594) Loss 0.5024 (0.5024) Acc@1 89.111 (89.111) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 07:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.691) Loss 0.8237 (0.6243) Acc@1 80.469 (86.071) Acc@5 96.094 (97.661) Mem 11224MB [2024-09-01 07:05:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.406) Loss 0.9375 (0.7417) Acc@1 76.758 (82.982) Acc@5 94.775 (96.487) Mem 11224MB [2024-09-01 07:05:10 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.728 Acc@5 96.501 [2024-09-01 07:05:10 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.7% [2024-09-01 07:05:10 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.73% [2024-09-01 07:05:10 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:05:11 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][0/625] eta 0:12:44 lr 0.000833 wd 0.0500 time 1.2235 (1.2235) data time 0.8784 (0.8784) model time 0.0000 (0.0000) loss 3.6451 (3.6451) grad_norm 2.0675 (2.0675) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][10/625] eta 0:04:23 lr 0.000833 wd 0.0500 time 0.3563 (0.4288) data time 0.0006 (0.0806) model time 0.0000 (0.0000) loss 2.1086 (2.8730) grad_norm 2.6623 (1.9252) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][20/625] eta 0:03:56 lr 0.000833 wd 0.0500 time 0.3539 (0.3909) data time 0.0006 (0.0426) model time 0.0000 (0.0000) loss 3.3173 (3.0409) grad_norm 1.2705 (1.7006) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][30/625] eta 0:03:44 lr 0.000833 wd 0.0500 time 0.3422 (0.3767) data time 0.0009 (0.0294) model time 0.0000 (0.0000) loss 1.8721 (2.9486) grad_norm 1.6588 (1.6462) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][40/625] eta 0:03:36 lr 0.000833 wd 0.0500 time 0.3389 (0.3694) data time 0.0007 (0.0228) model time 0.0000 (0.0000) loss 3.7025 (3.0660) grad_norm 1.5387 (1.6239) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][50/625] eta 0:05:16 lr 0.000832 wd 0.0500 time 9.7619 (0.5496) data time 8.4331 (0.1839) model time 0.0000 (0.0000) loss 3.8058 (3.1030) grad_norm 1.4486 (1.7137) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][60/625] eta 0:04:54 lr 0.000832 wd 0.0500 time 0.3467 (0.5209) data time 0.0006 (0.1539) model time 0.3461 (0.3736) loss 2.3117 (3.0962) grad_norm 2.3055 (1.7099) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][70/625] eta 0:04:42 lr 0.000832 wd 0.0500 time 1.2629 (0.5092) data time 0.0007 (0.1323) model time 1.2622 (0.4053) loss 3.2729 (3.1087) grad_norm 1.4202 (1.6754) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][80/625] eta 0:04:26 lr 0.000832 wd 0.0500 time 0.3434 (0.4890) data time 0.0008 (0.1161) model time 0.3426 (0.3852) loss 3.5937 (3.1305) grad_norm 1.2605 (1.6734) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][90/625] eta 0:04:13 lr 0.000832 wd 0.0500 time 0.3520 (0.4735) data time 0.0008 (0.1034) model time 0.3512 (0.3757) loss 3.4134 (3.1116) grad_norm 1.4950 (1.7325) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:06:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][100/625] eta 0:04:34 lr 0.000831 wd 0.0500 time 0.3490 (0.5222) data time 0.0006 (0.1543) model time 0.3484 (0.3699) loss 2.5020 (3.1187) grad_norm 3.6518 (1.7313) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][110/625] eta 0:04:21 lr 0.000831 wd 0.0500 time 0.3547 (0.5077) data time 0.0008 (0.1405) model time 0.3538 (0.3684) loss 2.2691 (3.1027) grad_norm 2.1481 (1.7455) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:06:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][120/625] eta 0:04:16 lr 0.000831 wd 0.0500 time 0.3573 (0.5079) data time 0.0009 (0.1290) model time 0.3564 (0.3886) loss 2.4846 (3.1039) grad_norm 1.7639 (1.7364) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:06:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][130/625] eta 0:04:05 lr 0.000831 wd 0.0500 time 0.3462 (0.4960) data time 0.0007 (0.1192) model time 0.3456 (0.3838) loss 2.9534 (3.0718) grad_norm 2.2424 (1.7523) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:06:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][140/625] eta 0:03:55 lr 0.000831 wd 0.0500 time 0.3550 (0.4858) data time 0.0010 (0.1108) model time 0.3541 (0.3803) loss 3.3689 (3.0576) grad_norm 1.5528 (1.7472) loss_scale 1024.0000 (515.6312) mem 11224MB [2024-09-01 07:06:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][150/625] eta 0:03:47 lr 0.000831 wd 0.0500 time 0.3453 (0.4780) data time 0.0009 (0.1035) model time 0.3444 (0.3790) loss 2.1753 (3.0637) grad_norm 1.3019 (1.7392) loss_scale 1024.0000 (549.2980) mem 11224MB [2024-09-01 07:06:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][160/625] eta 0:03:40 lr 0.000830 wd 0.0500 time 0.3419 (0.4740) data time 0.0006 (0.0971) model time 0.3413 (0.3820) loss 3.2895 (3.0576) grad_norm 1.7311 (1.7353) loss_scale 1024.0000 (578.7826) mem 11224MB [2024-09-01 07:06:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][170/625] eta 0:03:46 lr 0.000830 wd 0.0500 time 0.3545 (0.4981) data time 0.0009 (0.0923) model time 0.3536 (0.4228) loss 3.4761 (3.0446) grad_norm 1.6155 (1.7256) loss_scale 1024.0000 (604.8187) mem 11224MB [2024-09-01 07:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][180/625] eta 0:03:38 lr 0.000830 wd 0.0500 time 0.3451 (0.4900) data time 0.0009 (0.0872) model time 0.3442 (0.4172) loss 2.6856 (3.0343) grad_norm 3.0840 (1.7485) loss_scale 1024.0000 (627.9779) mem 11224MB [2024-09-01 07:06:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][190/625] eta 0:03:29 lr 0.000830 wd 0.0500 time 0.3633 (0.4828) data time 0.0008 (0.0827) model time 0.3626 (0.4125) loss 3.9343 (3.0379) grad_norm 2.2733 (1.7547) loss_scale 1024.0000 (648.7120) mem 11224MB [2024-09-01 07:06:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][200/625] eta 0:03:22 lr 0.000830 wd 0.0500 time 0.3457 (0.4760) data time 0.0006 (0.0786) model time 0.3451 (0.4082) loss 3.6431 (3.0538) grad_norm 1.6020 (1.7566) loss_scale 1024.0000 (667.3831) mem 11224MB [2024-09-01 07:06:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][210/625] eta 0:03:15 lr 0.000830 wd 0.0500 time 0.3492 (0.4700) data time 0.0011 (0.0749) model time 0.3482 (0.4044) loss 3.3421 (3.0517) grad_norm 3.6323 (1.7847) loss_scale 1024.0000 (684.2844) mem 11224MB [2024-09-01 07:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][220/625] eta 0:03:12 lr 0.000829 wd 0.0500 time 0.3572 (0.4764) data time 0.0006 (0.0716) model time 0.3565 (0.4166) loss 2.9488 (3.0564) grad_norm 2.0476 (1.7967) loss_scale 1024.0000 (699.6561) mem 11224MB [2024-09-01 07:07:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][230/625] eta 0:03:06 lr 0.000829 wd 0.0500 time 0.3392 (0.4715) data time 0.0008 (0.0685) model time 0.3384 (0.4136) loss 2.6460 (3.0473) grad_norm 1.4779 (1.7904) loss_scale 1024.0000 (713.6970) mem 11224MB [2024-09-01 07:07:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][240/625] eta 0:03:01 lr 0.000829 wd 0.0500 time 0.6312 (0.4719) data time 0.0007 (0.0657) model time 0.6305 (0.4170) loss 3.5100 (3.0618) grad_norm 1.7152 (1.7840) loss_scale 1024.0000 (726.5726) mem 11224MB [2024-09-01 07:07:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][250/625] eta 0:02:56 lr 0.000829 wd 0.0500 time 0.3502 (0.4708) data time 0.0008 (0.0631) model time 0.3494 (0.4183) loss 3.1962 (3.0642) grad_norm 1.5476 (1.7748) loss_scale 1024.0000 (738.4223) mem 11224MB [2024-09-01 07:07:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][260/625] eta 0:02:50 lr 0.000829 wd 0.0500 time 0.3415 (0.4661) data time 0.0010 (0.0608) model time 0.3405 (0.4150) loss 3.5099 (3.0714) grad_norm 1.4530 (1.7667) loss_scale 1024.0000 (749.3640) mem 11224MB [2024-09-01 07:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][270/625] eta 0:02:43 lr 0.000829 wd 0.0500 time 0.3461 (0.4616) data time 0.0007 (0.0585) model time 0.3454 (0.4117) loss 3.5796 (3.0744) grad_norm 1.8942 (1.7765) loss_scale 1024.0000 (759.4982) mem 11224MB [2024-09-01 07:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][280/625] eta 0:02:38 lr 0.000828 wd 0.0500 time 0.3441 (0.4582) data time 0.0008 (0.0565) model time 0.3433 (0.4097) loss 3.4050 (3.0796) grad_norm 1.4984 (1.7818) loss_scale 1024.0000 (768.9110) mem 11224MB [2024-09-01 07:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][290/625] eta 0:02:32 lr 0.000828 wd 0.0500 time 0.3500 (0.4544) data time 0.0009 (0.0546) model time 0.3491 (0.4071) loss 2.1049 (3.0754) grad_norm 1.5067 (1.7703) loss_scale 1024.0000 (777.6770) mem 11224MB [2024-09-01 07:07:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][300/625] eta 0:02:26 lr 0.000828 wd 0.0500 time 0.3536 (0.4510) data time 0.0007 (0.0528) model time 0.3530 (0.4049) loss 3.4650 (3.0807) grad_norm 2.0462 (1.7603) loss_scale 1024.0000 (785.8605) mem 11224MB [2024-09-01 07:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][310/625] eta 0:02:21 lr 0.000828 wd 0.0500 time 0.3462 (0.4477) data time 0.0008 (0.0511) model time 0.3454 (0.4026) loss 2.6682 (3.0818) grad_norm 1.5679 (1.7535) loss_scale 1024.0000 (793.5177) mem 11224MB [2024-09-01 07:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][320/625] eta 0:02:15 lr 0.000828 wd 0.0500 time 0.3457 (0.4446) data time 0.0010 (0.0496) model time 0.3447 (0.4006) loss 2.4784 (3.0820) grad_norm 1.9936 (1.7421) loss_scale 1024.0000 (800.6978) mem 11224MB [2024-09-01 07:07:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][330/625] eta 0:02:10 lr 0.000827 wd 0.0500 time 0.3472 (0.4417) data time 0.0008 (0.0481) model time 0.3464 (0.3987) loss 2.9271 (3.0856) grad_norm 2.4226 (1.7389) loss_scale 1024.0000 (807.4441) mem 11224MB [2024-09-01 07:07:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][340/625] eta 0:02:07 lr 0.000827 wd 0.0500 time 0.4864 (0.4478) data time 0.0008 (0.0467) model time 0.4855 (0.4074) loss 3.5825 (3.0879) grad_norm 1.2614 (1.7357) loss_scale 1024.0000 (813.7947) mem 11224MB [2024-09-01 07:07:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][350/625] eta 0:02:06 lr 0.000827 wd 0.0500 time 0.3477 (0.4596) data time 0.0008 (0.0454) model time 0.3469 (0.4225) loss 3.5113 (3.0835) grad_norm 1.5986 (1.7317) loss_scale 1024.0000 (819.7835) mem 11224MB [2024-09-01 07:07:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][360/625] eta 0:02:00 lr 0.000827 wd 0.0500 time 0.3472 (0.4565) data time 0.0006 (0.0442) model time 0.3465 (0.4201) loss 2.5834 (3.0840) grad_norm 1.2236 (1.7241) loss_scale 1024.0000 (825.4404) mem 11224MB [2024-09-01 07:07:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][370/625] eta 0:01:55 lr 0.000827 wd 0.0500 time 0.3442 (0.4535) data time 0.0008 (0.0430) model time 0.3434 (0.4177) loss 1.8976 (3.0802) grad_norm 2.8714 (1.7326) loss_scale 1024.0000 (830.7925) mem 11224MB [2024-09-01 07:08:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][380/625] eta 0:01:54 lr 0.000827 wd 0.0500 time 0.3555 (0.4655) data time 0.0007 (0.0419) model time 0.3548 (0.4326) loss 3.4974 (3.0789) grad_norm 1.9542 (inf) loss_scale 512.0000 (831.8320) mem 11224MB [2024-09-01 07:08:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][390/625] eta 0:01:48 lr 0.000826 wd 0.0500 time 0.3506 (0.4625) data time 0.0009 (0.0408) model time 0.3497 (0.4301) loss 2.1527 (3.0787) grad_norm 1.5593 (inf) loss_scale 512.0000 (823.6522) mem 11224MB [2024-09-01 07:08:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][400/625] eta 0:01:43 lr 0.000826 wd 0.0500 time 0.3443 (0.4597) data time 0.0009 (0.0398) model time 0.3434 (0.4277) loss 3.1076 (3.0768) grad_norm 2.2538 (inf) loss_scale 512.0000 (815.8803) mem 11224MB [2024-09-01 07:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][410/625] eta 0:01:38 lr 0.000826 wd 0.0500 time 0.3483 (0.4569) data time 0.0008 (0.0389) model time 0.3474 (0.4254) loss 2.2915 (3.0840) grad_norm 1.5885 (inf) loss_scale 512.0000 (808.4866) mem 11224MB [2024-09-01 07:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][420/625] eta 0:01:33 lr 0.000826 wd 0.0500 time 0.3484 (0.4544) data time 0.0008 (0.0380) model time 0.3476 (0.4234) loss 2.1615 (3.0879) grad_norm 1.7343 (inf) loss_scale 512.0000 (801.4442) mem 11224MB [2024-09-01 07:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][430/625] eta 0:01:28 lr 0.000826 wd 0.0500 time 0.3475 (0.4520) data time 0.0007 (0.0371) model time 0.3468 (0.4214) loss 2.4348 (3.0892) grad_norm 1.8167 (inf) loss_scale 512.0000 (794.7285) mem 11224MB [2024-09-01 07:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][440/625] eta 0:01:23 lr 0.000826 wd 0.0500 time 0.3504 (0.4501) data time 0.0008 (0.0363) model time 0.3496 (0.4200) loss 3.5481 (3.0913) grad_norm 1.5206 (inf) loss_scale 512.0000 (788.3175) mem 11224MB [2024-09-01 07:08:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][450/625] eta 0:01:18 lr 0.000825 wd 0.0500 time 0.3612 (0.4478) data time 0.0006 (0.0355) model time 0.3605 (0.4183) loss 3.3812 (3.0917) grad_norm 1.8334 (inf) loss_scale 512.0000 (782.1907) mem 11224MB [2024-09-01 07:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][460/625] eta 0:01:13 lr 0.000825 wd 0.0500 time 0.5419 (0.4474) data time 0.0006 (0.0348) model time 0.5412 (0.4184) loss 2.9146 (3.0940) grad_norm 1.5871 (inf) loss_scale 512.0000 (776.3297) mem 11224MB [2024-09-01 07:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][470/625] eta 0:01:09 lr 0.000825 wd 0.0500 time 0.3570 (0.4458) data time 0.0008 (0.0341) model time 0.3561 (0.4173) loss 3.3512 (3.0901) grad_norm 1.5093 (inf) loss_scale 512.0000 (770.7176) mem 11224MB [2024-09-01 07:08:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][480/625] eta 0:01:04 lr 0.000825 wd 0.0500 time 0.3436 (0.4437) data time 0.0008 (0.0334) model time 0.3428 (0.4156) loss 3.4987 (3.0880) grad_norm 1.8741 (inf) loss_scale 512.0000 (765.3389) mem 11224MB [2024-09-01 07:08:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][490/625] eta 0:00:59 lr 0.000825 wd 0.0500 time 0.3428 (0.4417) data time 0.0007 (0.0327) model time 0.3421 (0.4140) loss 3.1716 (3.0808) grad_norm 1.5502 (inf) loss_scale 512.0000 (760.1792) mem 11224MB [2024-09-01 07:08:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][500/625] eta 0:00:56 lr 0.000824 wd 0.0500 time 0.3464 (0.4499) data time 0.0008 (0.0321) model time 0.3457 (0.4237) loss 3.3659 (3.0834) grad_norm 2.1882 (inf) loss_scale 512.0000 (755.2255) mem 11224MB [2024-09-01 07:09:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][510/625] eta 0:00:51 lr 0.000824 wd 0.0500 time 0.3405 (0.4480) data time 0.0005 (0.0315) model time 0.3400 (0.4222) loss 4.0967 (3.0830) grad_norm 1.8921 (inf) loss_scale 512.0000 (750.4658) mem 11224MB [2024-09-01 07:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][520/625] eta 0:00:46 lr 0.000824 wd 0.0500 time 0.3462 (0.4461) data time 0.0006 (0.0309) model time 0.3456 (0.4206) loss 2.3089 (3.0860) grad_norm 1.4291 (inf) loss_scale 512.0000 (745.8887) mem 11224MB [2024-09-01 07:09:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][530/625] eta 0:00:42 lr 0.000824 wd 0.0500 time 0.3582 (0.4485) data time 0.0010 (0.0303) model time 0.3572 (0.4237) loss 3.4324 (3.0881) grad_norm 1.6776 (inf) loss_scale 512.0000 (741.4840) mem 11224MB [2024-09-01 07:09:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][540/625] eta 0:00:37 lr 0.000824 wd 0.0500 time 0.3544 (0.4466) data time 0.0007 (0.0298) model time 0.3537 (0.4222) loss 2.2650 (3.0903) grad_norm 1.5257 (inf) loss_scale 512.0000 (737.2421) mem 11224MB [2024-09-01 07:09:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][550/625] eta 0:00:33 lr 0.000824 wd 0.0500 time 0.3502 (0.4449) data time 0.0009 (0.0292) model time 0.3493 (0.4207) loss 3.1042 (3.0904) grad_norm 1.3811 (inf) loss_scale 512.0000 (733.1543) mem 11224MB [2024-09-01 07:09:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][560/625] eta 0:00:28 lr 0.000823 wd 0.0500 time 0.3516 (0.4432) data time 0.0008 (0.0287) model time 0.3507 (0.4194) loss 2.6495 (3.0897) grad_norm 1.7270 (inf) loss_scale 512.0000 (729.2121) mem 11224MB [2024-09-01 07:09:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][570/625] eta 0:00:24 lr 0.000823 wd 0.0500 time 0.3399 (0.4416) data time 0.0006 (0.0283) model time 0.3392 (0.4180) loss 3.4217 (3.0903) grad_norm 1.3680 (inf) loss_scale 512.0000 (725.4081) mem 11224MB [2024-09-01 07:09:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][580/625] eta 0:00:19 lr 0.000823 wd 0.0500 time 0.3497 (0.4400) data time 0.0007 (0.0278) model time 0.3490 (0.4167) loss 2.0882 (3.0917) grad_norm 1.9762 (inf) loss_scale 512.0000 (721.7349) mem 11224MB [2024-09-01 07:09:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][590/625] eta 0:00:15 lr 0.000823 wd 0.0500 time 0.3521 (0.4385) data time 0.0006 (0.0273) model time 0.3515 (0.4154) loss 2.5952 (3.0943) grad_norm 1.2927 (inf) loss_scale 512.0000 (718.1861) mem 11224MB [2024-09-01 07:09:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][600/625] eta 0:00:10 lr 0.000823 wd 0.0500 time 0.3410 (0.4370) data time 0.0008 (0.0269) model time 0.3402 (0.4142) loss 2.4251 (3.0959) grad_norm 2.3568 (inf) loss_scale 512.0000 (714.7554) mem 11224MB [2024-09-01 07:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][610/625] eta 0:00:06 lr 0.000823 wd 0.0500 time 0.3432 (0.4355) data time 0.0005 (0.0265) model time 0.3427 (0.4130) loss 3.3659 (3.0964) grad_norm 1.9842 (inf) loss_scale 512.0000 (711.4370) mem 11224MB [2024-09-01 07:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [176/300][620/625] eta 0:00:02 lr 0.000822 wd 0.0500 time 0.3351 (0.4340) data time 0.0004 (0.0261) model time 0.3346 (0.4117) loss 1.6833 (3.0940) grad_norm 2.0257 (inf) loss_scale 512.0000 (708.2254) mem 11224MB [2024-09-01 07:09:42 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 176 training takes 0:04:31 [2024-09-01 07:09:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:09:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:09:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 4.116 (4.116) Loss 0.5591 (0.5591) Acc@1 88.281 (88.281) Acc@5 98.047 (98.047) Mem 11224MB [2024-09-01 07:09:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.468) Loss 0.9375 (0.6966) Acc@1 78.564 (84.561) Acc@5 94.775 (97.124) Mem 11224MB [2024-09-01 07:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.290) Loss 1.0400 (0.8336) Acc@1 74.854 (81.041) Acc@5 94.238 (95.717) Mem 11224MB [2024-09-01 07:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.820 Acc@5 95.717 [2024-09-01 07:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.8% [2024-09-01 07:09:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 3.936 (3.936) Loss 0.5020 (0.5020) Acc@1 89.111 (89.111) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 07:09:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.458) Loss 0.8223 (0.6230) Acc@1 80.518 (86.115) Acc@5 95.947 (97.647) Mem 11224MB [2024-09-01 07:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.284) Loss 0.9355 (0.7405) Acc@1 76.953 (83.026) Acc@5 94.873 (96.477) Mem 11224MB [2024-09-01 07:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.770 Acc@5 96.499 [2024-09-01 07:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.8% [2024-09-01 07:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.77% [2024-09-01 07:09:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:09:57 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:10:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][0/625] eta 2:03:41 lr 0.000822 wd 0.0500 time 11.8742 (11.8742) data time 9.1392 (9.1392) model time 0.0000 (0.0000) loss 3.4935 (3.4935) grad_norm 2.2230 (2.2230) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][10/625] eta 0:14:17 lr 0.000822 wd 0.0500 time 0.3432 (1.3941) data time 0.0008 (0.8316) model time 0.0000 (0.0000) loss 3.2474 (3.1974) grad_norm 1.2333 (1.6005) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][20/625] eta 0:09:01 lr 0.000822 wd 0.0500 time 0.3420 (0.8946) data time 0.0008 (0.4360) model time 0.0000 (0.0000) loss 3.0786 (3.1195) grad_norm 1.6694 (1.5841) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][30/625] eta 0:07:37 lr 0.000822 wd 0.0500 time 0.3443 (0.7685) data time 0.0006 (0.2956) model time 0.0000 (0.0000) loss 2.8211 (3.0588) grad_norm 1.5465 (1.5786) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][40/625] eta 0:06:32 lr 0.000822 wd 0.0500 time 0.3508 (0.6707) data time 0.0007 (0.2237) model time 0.0000 (0.0000) loss 3.9379 (3.1340) grad_norm 1.9686 (1.6253) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][50/625] eta 0:05:51 lr 0.000821 wd 0.0500 time 0.3498 (0.6113) data time 0.0009 (0.1800) model time 0.0000 (0.0000) loss 3.2732 (3.0733) grad_norm 1.2186 (1.5901) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][60/625] eta 0:05:21 lr 0.000821 wd 0.0500 time 0.3513 (0.5683) data time 0.0008 (0.1507) model time 0.3505 (0.3483) loss 3.2961 (3.0834) grad_norm 1.5583 (1.5630) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][70/625] eta 0:04:58 lr 0.000821 wd 0.0500 time 0.3563 (0.5375) data time 0.0008 (0.1296) model time 0.3555 (0.3487) loss 3.6447 (3.1024) grad_norm 1.8012 (1.5789) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][80/625] eta 0:04:40 lr 0.000821 wd 0.0500 time 0.3505 (0.5141) data time 0.0006 (0.1137) model time 0.3499 (0.3481) loss 3.1904 (3.1036) grad_norm 1.2323 (1.6071) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][90/625] eta 0:04:25 lr 0.000821 wd 0.0500 time 0.3471 (0.4958) data time 0.0010 (0.1013) model time 0.3461 (0.3477) loss 3.2570 (3.1345) grad_norm 1.5967 (1.6361) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][100/625] eta 0:04:12 lr 0.000821 wd 0.0500 time 0.3535 (0.4812) data time 0.0008 (0.0913) model time 0.3527 (0.3478) loss 2.8269 (3.1268) grad_norm 2.6126 (1.6505) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][110/625] eta 0:04:01 lr 0.000820 wd 0.0500 time 0.3488 (0.4693) data time 0.0006 (0.0832) model time 0.3482 (0.3478) loss 2.6311 (3.1278) grad_norm 1.3229 (1.6876) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][120/625] eta 0:03:52 lr 0.000820 wd 0.0500 time 0.3459 (0.4595) data time 0.0009 (0.0764) model time 0.3450 (0.3481) loss 2.9628 (3.1215) grad_norm 1.2307 (1.6889) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][130/625] eta 0:03:43 lr 0.000820 wd 0.0500 time 0.3456 (0.4516) data time 0.0009 (0.0706) model time 0.3447 (0.3490) loss 3.3656 (3.1142) grad_norm 1.7793 (1.6944) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:10:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][140/625] eta 0:03:35 lr 0.000820 wd 0.0500 time 0.3549 (0.4444) data time 0.0010 (0.0657) model time 0.3540 (0.3490) loss 3.6468 (3.1106) grad_norm 1.5171 (1.6949) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][150/625] eta 0:03:28 lr 0.000820 wd 0.0500 time 0.3540 (0.4381) data time 0.0007 (0.0614) model time 0.3533 (0.3489) loss 2.5101 (3.1086) grad_norm 1.6614 (1.6992) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][160/625] eta 0:03:21 lr 0.000820 wd 0.0500 time 0.3438 (0.4326) data time 0.0006 (0.0576) model time 0.3432 (0.3489) loss 3.6333 (3.1188) grad_norm 2.0150 (1.7030) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][170/625] eta 0:03:14 lr 0.000819 wd 0.0500 time 0.3477 (0.4278) data time 0.0007 (0.0543) model time 0.3470 (0.3489) loss 2.6309 (3.1159) grad_norm 2.1891 (1.7086) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][180/625] eta 0:03:08 lr 0.000819 wd 0.0500 time 0.3485 (0.4234) data time 0.0006 (0.0513) model time 0.3479 (0.3489) loss 3.1690 (3.1168) grad_norm 1.9371 (1.7122) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][190/625] eta 0:03:02 lr 0.000819 wd 0.0500 time 0.3477 (0.4197) data time 0.0006 (0.0487) model time 0.3470 (0.3490) loss 2.0822 (3.1027) grad_norm 1.8855 (1.7403) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][200/625] eta 0:02:56 lr 0.000819 wd 0.0500 time 0.3430 (0.4162) data time 0.0009 (0.0463) model time 0.3421 (0.3490) loss 3.4579 (3.1117) grad_norm 1.4457 (1.7800) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][210/625] eta 0:02:51 lr 0.000819 wd 0.0500 time 0.3408 (0.4130) data time 0.0007 (0.0442) model time 0.3401 (0.3489) loss 3.6001 (3.1086) grad_norm 1.3751 (1.7749) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][220/625] eta 0:02:46 lr 0.000818 wd 0.0500 time 0.3495 (0.4101) data time 0.0009 (0.0422) model time 0.3486 (0.3489) loss 2.9976 (3.1061) grad_norm 1.5829 (1.7690) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][230/625] eta 0:02:41 lr 0.000818 wd 0.0500 time 0.3613 (0.4082) data time 0.0008 (0.0404) model time 0.3605 (0.3498) loss 3.5828 (3.1092) grad_norm 1.7107 (1.7656) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][240/625] eta 0:02:37 lr 0.000818 wd 0.0500 time 0.3405 (0.4083) data time 0.0006 (0.0388) model time 0.3398 (0.3530) loss 3.5549 (3.1165) grad_norm 1.8382 (1.7659) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][250/625] eta 0:02:32 lr 0.000818 wd 0.0500 time 0.3518 (0.4068) data time 0.0008 (0.0373) model time 0.3510 (0.3538) loss 3.1876 (3.1223) grad_norm 1.6933 (1.7678) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][260/625] eta 0:02:27 lr 0.000818 wd 0.0500 time 0.3539 (0.4045) data time 0.0006 (0.0359) model time 0.3533 (0.3534) loss 2.8709 (3.1245) grad_norm 1.5163 (1.7614) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][270/625] eta 0:02:22 lr 0.000818 wd 0.0500 time 0.3537 (0.4025) data time 0.0008 (0.0346) model time 0.3529 (0.3533) loss 3.0091 (3.1301) grad_norm 1.2832 (1.7549) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][280/625] eta 0:02:18 lr 0.000817 wd 0.0500 time 0.3452 (0.4006) data time 0.0007 (0.0334) model time 0.3445 (0.3531) loss 3.2865 (3.1271) grad_norm 1.3596 (1.7457) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][290/625] eta 0:02:13 lr 0.000817 wd 0.0500 time 0.3490 (0.3989) data time 0.0007 (0.0323) model time 0.3483 (0.3529) loss 3.7954 (3.1325) grad_norm 1.4310 (1.7350) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:11:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][300/625] eta 0:02:09 lr 0.000817 wd 0.0500 time 0.3471 (0.3972) data time 0.0007 (0.0312) model time 0.3464 (0.3527) loss 3.2785 (3.1327) grad_norm 1.2507 (1.7283) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][310/625] eta 0:02:04 lr 0.000817 wd 0.0500 time 0.3435 (0.3957) data time 0.0008 (0.0302) model time 0.3427 (0.3525) loss 3.1248 (3.1348) grad_norm 1.5479 (1.7352) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][320/625] eta 0:02:00 lr 0.000817 wd 0.0500 time 0.3780 (0.3944) data time 0.0009 (0.0293) model time 0.3771 (0.3525) loss 2.5226 (3.1348) grad_norm 1.1751 (1.7371) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][330/625] eta 0:01:55 lr 0.000817 wd 0.0500 time 0.3505 (0.3930) data time 0.0009 (0.0285) model time 0.3495 (0.3524) loss 3.1634 (3.1330) grad_norm 1.9907 (1.7317) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][340/625] eta 0:01:51 lr 0.000816 wd 0.0500 time 0.3459 (0.3917) data time 0.0007 (0.0277) model time 0.3452 (0.3522) loss 3.0993 (3.1316) grad_norm 1.6152 (1.7279) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][350/625] eta 0:01:47 lr 0.000816 wd 0.0500 time 0.5470 (0.3915) data time 0.0009 (0.0269) model time 0.5461 (0.3533) loss 3.4322 (3.1269) grad_norm 1.6139 (1.7200) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][360/625] eta 0:01:43 lr 0.000816 wd 0.0500 time 0.3497 (0.3903) data time 0.0008 (0.0262) model time 0.3489 (0.3531) loss 3.2970 (3.1353) grad_norm 2.0413 (1.7199) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][370/625] eta 0:01:39 lr 0.000816 wd 0.0500 time 0.3479 (0.3892) data time 0.0010 (0.0255) model time 0.3469 (0.3530) loss 2.8098 (3.1325) grad_norm 2.1043 (1.7211) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][380/625] eta 0:01:35 lr 0.000816 wd 0.0500 time 0.3472 (0.3882) data time 0.0009 (0.0248) model time 0.3462 (0.3528) loss 3.3072 (3.1351) grad_norm 2.1299 (1.7275) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][390/625] eta 0:01:30 lr 0.000816 wd 0.0500 time 0.3553 (0.3872) data time 0.0009 (0.0242) model time 0.3545 (0.3527) loss 2.9619 (3.1328) grad_norm 1.7630 (1.7376) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][400/625] eta 0:01:26 lr 0.000815 wd 0.0500 time 0.3513 (0.3863) data time 0.0006 (0.0236) model time 0.3507 (0.3527) loss 3.3680 (3.1332) grad_norm 1.9098 (1.7389) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][410/625] eta 0:01:22 lr 0.000815 wd 0.0500 time 0.3476 (0.3855) data time 0.0009 (0.0231) model time 0.3468 (0.3526) loss 2.9555 (3.1354) grad_norm 1.3152 (1.7365) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][420/625] eta 0:01:19 lr 0.000815 wd 0.0500 time 0.3430 (0.3860) data time 0.0006 (0.0226) model time 0.3424 (0.3541) loss 2.8134 (3.1360) grad_norm 2.3710 (1.7373) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][430/625] eta 0:01:15 lr 0.000815 wd 0.0500 time 0.3533 (0.3857) data time 0.0008 (0.0221) model time 0.3525 (0.3546) loss 3.3764 (3.1408) grad_norm 1.9605 (1.7396) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][440/625] eta 0:01:11 lr 0.000815 wd 0.0500 time 0.3542 (0.3849) data time 0.0007 (0.0216) model time 0.3535 (0.3545) loss 3.2511 (3.1467) grad_norm 1.5201 (1.7364) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][450/625] eta 0:01:07 lr 0.000814 wd 0.0500 time 0.3428 (0.3841) data time 0.0008 (0.0211) model time 0.3420 (0.3543) loss 2.7065 (3.1494) grad_norm 1.6948 (1.7336) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][460/625] eta 0:01:03 lr 0.000814 wd 0.0500 time 0.3651 (0.3838) data time 0.0009 (0.0207) model time 0.3642 (0.3546) loss 3.3209 (3.1504) grad_norm 1.3544 (1.7253) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][470/625] eta 0:00:59 lr 0.000814 wd 0.0500 time 0.3577 (0.3831) data time 0.0007 (0.0203) model time 0.3569 (0.3545) loss 2.0334 (3.1431) grad_norm 1.1608 (1.7232) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][480/625] eta 0:00:55 lr 0.000814 wd 0.0500 time 0.3515 (0.3823) data time 0.0007 (0.0199) model time 0.3508 (0.3543) loss 2.9530 (3.1413) grad_norm 1.4719 (1.7177) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][490/625] eta 0:00:51 lr 0.000814 wd 0.0500 time 0.3520 (0.3816) data time 0.0008 (0.0195) model time 0.3511 (0.3542) loss 2.8970 (3.1437) grad_norm 1.9777 (1.7168) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][500/625] eta 0:00:47 lr 0.000814 wd 0.0500 time 0.3454 (0.3809) data time 0.0007 (0.0191) model time 0.3447 (0.3540) loss 2.2830 (3.1427) grad_norm 3.1006 (1.7222) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][510/625] eta 0:00:43 lr 0.000813 wd 0.0500 time 0.3450 (0.3803) data time 0.0008 (0.0187) model time 0.3442 (0.3538) loss 3.0592 (3.1406) grad_norm 1.7982 (1.7325) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][520/625] eta 0:00:39 lr 0.000813 wd 0.0500 time 0.3490 (0.3796) data time 0.0006 (0.0184) model time 0.3484 (0.3537) loss 2.0422 (3.1347) grad_norm 1.5097 (1.7353) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][530/625] eta 0:00:36 lr 0.000813 wd 0.0500 time 0.3546 (0.3791) data time 0.0008 (0.0181) model time 0.3538 (0.3535) loss 3.3550 (3.1321) grad_norm 1.5133 (1.7297) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][540/625] eta 0:00:32 lr 0.000813 wd 0.0500 time 0.3560 (0.3797) data time 0.0006 (0.0178) model time 0.3554 (0.3547) loss 3.2601 (3.1310) grad_norm 1.4169 (1.7235) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][550/625] eta 0:00:28 lr 0.000813 wd 0.0500 time 0.5350 (0.3798) data time 0.0008 (0.0174) model time 0.5343 (0.3554) loss 2.9660 (3.1300) grad_norm 1.9857 (1.7325) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][560/625] eta 0:00:24 lr 0.000813 wd 0.0500 time 0.3477 (0.3793) data time 0.0007 (0.0171) model time 0.3470 (0.3552) loss 2.6223 (3.1303) grad_norm 1.4090 (1.7357) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][570/625] eta 0:00:20 lr 0.000812 wd 0.0500 time 0.3472 (0.3791) data time 0.0008 (0.0169) model time 0.3464 (0.3554) loss 2.8674 (3.1271) grad_norm 1.0911 (1.7334) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][580/625] eta 0:00:17 lr 0.000812 wd 0.0500 time 0.3432 (0.3785) data time 0.0009 (0.0166) model time 0.3424 (0.3552) loss 3.6250 (3.1255) grad_norm 2.0329 (1.7357) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][590/625] eta 0:00:13 lr 0.000812 wd 0.0500 time 0.3413 (0.3780) data time 0.0006 (0.0163) model time 0.3406 (0.3551) loss 1.5291 (3.1236) grad_norm 2.1154 (1.7402) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][600/625] eta 0:00:09 lr 0.000812 wd 0.0500 time 0.3454 (0.3775) data time 0.0007 (0.0161) model time 0.3446 (0.3549) loss 3.3232 (3.1234) grad_norm 1.1030 (1.7380) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][610/625] eta 0:00:05 lr 0.000812 wd 0.0500 time 0.3411 (0.3770) data time 0.0004 (0.0158) model time 0.3407 (0.3548) loss 2.9415 (3.1231) grad_norm 1.6943 (1.7387) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [177/300][620/625] eta 0:00:01 lr 0.000812 wd 0.0500 time 0.3432 (0.3764) data time 0.0004 (0.0156) model time 0.3428 (0.3545) loss 3.3475 (3.1238) grad_norm 1.3631 (1.7383) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:13:52 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 177 training takes 0:03:55 [2024-09-01 07:13:52 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:13:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:13:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.459 (0.459) Loss 0.6055 (0.6055) Acc@1 87.207 (87.207) Acc@5 98.096 (98.096) Mem 11224MB [2024-09-01 07:13:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.9678 (0.7220) Acc@1 77.637 (84.566) Acc@5 94.727 (97.066) Mem 11224MB [2024-09-01 07:13:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 1.0928 (0.8499) Acc@1 73.535 (81.159) Acc@5 93.994 (95.687) Mem 11224MB [2024-09-01 07:13:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.886 Acc@5 95.679 [2024-09-01 07:13:56 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 80.9% [2024-09-01 07:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 1.114 (1.114) Loss 0.5010 (0.5010) Acc@1 89.258 (89.258) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 07:13:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.193) Loss 0.8218 (0.6218) Acc@1 80.469 (86.111) Acc@5 95.850 (97.634) Mem 11224MB [2024-09-01 07:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.144) Loss 0.9321 (0.7390) Acc@1 77.051 (83.036) Acc@5 94.971 (96.475) Mem 11224MB [2024-09-01 07:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.770 Acc@5 96.503 [2024-09-01 07:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.8% [2024-09-01 07:14:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][0/625] eta 0:20:05 lr 0.000811 wd 0.0500 time 1.9291 (1.9291) data time 1.0227 (1.0227) model time 0.0000 (0.0000) loss 3.2533 (3.2533) grad_norm 1.4328 (1.4328) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][10/625] eta 0:05:04 lr 0.000811 wd 0.0500 time 0.3464 (0.4945) data time 0.0006 (0.0938) model time 0.0000 (0.0000) loss 3.9772 (3.2127) grad_norm 1.0910 (1.3605) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][20/625] eta 0:04:16 lr 0.000811 wd 0.0500 time 0.3433 (0.4247) data time 0.0007 (0.0495) model time 0.0000 (0.0000) loss 2.8726 (3.1042) grad_norm 2.4538 (1.4332) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][30/625] eta 0:03:57 lr 0.000811 wd 0.0500 time 0.3472 (0.3993) data time 0.0006 (0.0338) model time 0.0000 (0.0000) loss 3.4899 (3.0990) grad_norm 1.1795 (1.3993) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][40/625] eta 0:03:46 lr 0.000811 wd 0.0500 time 0.3447 (0.3865) data time 0.0008 (0.0258) model time 0.0000 (0.0000) loss 2.3404 (2.9953) grad_norm 1.0395 (1.3758) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][50/625] eta 0:03:37 lr 0.000811 wd 0.0500 time 0.3555 (0.3790) data time 0.0009 (0.0209) model time 0.0000 (0.0000) loss 3.2550 (2.9949) grad_norm 2.4822 (1.6859) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][60/625] eta 0:03:31 lr 0.000810 wd 0.0500 time 0.3581 (0.3739) data time 0.0008 (0.0176) model time 0.3573 (0.3467) loss 3.2410 (3.0193) grad_norm 1.6977 (1.7044) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][70/625] eta 0:03:27 lr 0.000810 wd 0.0500 time 0.3516 (0.3746) data time 0.0008 (0.0152) model time 0.3509 (0.3624) loss 3.7184 (3.0055) grad_norm 1.3070 (1.7011) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][80/625] eta 0:03:26 lr 0.000810 wd 0.0500 time 0.3368 (0.3785) data time 0.0008 (0.0135) model time 0.3361 (0.3767) loss 3.1596 (3.0124) grad_norm 1.2010 (1.6761) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][90/625] eta 0:03:20 lr 0.000810 wd 0.0500 time 0.3488 (0.3754) data time 0.0010 (0.0121) model time 0.3479 (0.3700) loss 3.4477 (3.0062) grad_norm 1.1962 (1.6377) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][100/625] eta 0:03:15 lr 0.000810 wd 0.0500 time 0.3650 (0.3730) data time 0.0008 (0.0110) model time 0.3641 (0.3659) loss 2.7759 (3.0215) grad_norm 1.7466 (1.6243) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][110/625] eta 0:03:10 lr 0.000810 wd 0.0500 time 0.3445 (0.3707) data time 0.0006 (0.0101) model time 0.3438 (0.3629) loss 2.5237 (3.0186) grad_norm 2.3134 (1.6271) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][120/625] eta 0:03:07 lr 0.000809 wd 0.0500 time 0.3515 (0.3704) data time 0.0008 (0.0093) model time 0.3508 (0.3632) loss 2.3062 (3.0249) grad_norm 1.4266 (1.6176) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][130/625] eta 0:03:03 lr 0.000809 wd 0.0500 time 0.3549 (0.3702) data time 0.0007 (0.0087) model time 0.3542 (0.3637) loss 3.7630 (3.0538) grad_norm 2.1410 (1.6721) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][140/625] eta 0:02:59 lr 0.000809 wd 0.0500 time 0.3143 (0.3699) data time 0.0007 (0.0081) model time 0.3136 (0.3639) loss 3.5590 (3.0658) grad_norm 2.5746 (1.6883) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][150/625] eta 0:02:55 lr 0.000809 wd 0.0500 time 0.3463 (0.3687) data time 0.0006 (0.0076) model time 0.3457 (0.3627) loss 3.3132 (3.0611) grad_norm 1.7952 (1.7250) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:14:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][160/625] eta 0:02:50 lr 0.000809 wd 0.0500 time 0.3567 (0.3675) data time 0.0008 (0.0072) model time 0.3560 (0.3614) loss 2.5604 (3.0647) grad_norm 1.7231 (1.7284) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][170/625] eta 0:02:46 lr 0.000808 wd 0.0500 time 0.3436 (0.3664) data time 0.0009 (0.0068) model time 0.3427 (0.3601) loss 2.6331 (3.0868) grad_norm 1.9714 (1.7290) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][180/625] eta 0:02:42 lr 0.000808 wd 0.0500 time 0.3479 (0.3655) data time 0.0009 (0.0065) model time 0.3471 (0.3593) loss 2.9274 (3.0790) grad_norm 2.0626 (1.7231) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][190/625] eta 0:02:38 lr 0.000808 wd 0.0500 time 0.3484 (0.3645) data time 0.0006 (0.0062) model time 0.3478 (0.3584) loss 2.6866 (3.0804) grad_norm 1.6733 (1.7177) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][200/625] eta 0:02:34 lr 0.000808 wd 0.0500 time 0.3449 (0.3638) data time 0.0006 (0.0059) model time 0.3442 (0.3578) loss 2.4885 (3.0807) grad_norm 1.3606 (1.7163) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][210/625] eta 0:02:30 lr 0.000808 wd 0.0500 time 0.3463 (0.3631) data time 0.0009 (0.0057) model time 0.3454 (0.3572) loss 3.0710 (3.0749) grad_norm 5.6439 (1.7467) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][220/625] eta 0:02:26 lr 0.000808 wd 0.0500 time 0.3487 (0.3624) data time 0.0008 (0.0055) model time 0.3479 (0.3565) loss 3.2539 (3.0977) grad_norm 2.0392 (1.7825) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][230/625] eta 0:02:22 lr 0.000807 wd 0.0500 time 0.3449 (0.3618) data time 0.0008 (0.0053) model time 0.3441 (0.3562) loss 2.8248 (3.0902) grad_norm 3.5279 (1.7898) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][240/625] eta 0:02:19 lr 0.000807 wd 0.0500 time 0.3504 (0.3613) data time 0.0009 (0.0051) model time 0.3495 (0.3557) loss 3.4129 (3.0941) grad_norm 1.4559 (1.7784) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][250/625] eta 0:02:15 lr 0.000807 wd 0.0500 time 0.3498 (0.3608) data time 0.0008 (0.0049) model time 0.3490 (0.3553) loss 2.8922 (3.0924) grad_norm 1.3333 (1.7691) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][260/625] eta 0:02:11 lr 0.000807 wd 0.0500 time 0.3458 (0.3603) data time 0.0008 (0.0048) model time 0.3449 (0.3549) loss 3.9132 (3.0955) grad_norm 1.3598 (1.7601) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][270/625] eta 0:02:07 lr 0.000807 wd 0.0500 time 0.3492 (0.3599) data time 0.0008 (0.0046) model time 0.3484 (0.3547) loss 3.2981 (3.0947) grad_norm 1.3436 (1.7452) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][280/625] eta 0:02:04 lr 0.000807 wd 0.0500 time 0.3518 (0.3603) data time 0.0009 (0.0045) model time 0.3509 (0.3553) loss 2.4532 (3.0874) grad_norm 1.5928 (1.7547) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][290/625] eta 0:02:01 lr 0.000806 wd 0.0500 time 0.3575 (0.3614) data time 0.0008 (0.0043) model time 0.3567 (0.3568) loss 3.4560 (3.0917) grad_norm 1.8762 (1.7537) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][300/625] eta 0:01:57 lr 0.000806 wd 0.0500 time 0.3494 (0.3609) data time 0.0008 (0.0042) model time 0.3486 (0.3564) loss 2.6290 (3.0921) grad_norm 2.6486 (1.7666) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][310/625] eta 0:01:53 lr 0.000806 wd 0.0500 time 0.3476 (0.3610) data time 0.0009 (0.0041) model time 0.3467 (0.3566) loss 3.4400 (3.0863) grad_norm 2.1186 (1.7677) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][320/625] eta 0:01:49 lr 0.000806 wd 0.0500 time 0.3559 (0.3606) data time 0.0009 (0.0040) model time 0.3551 (0.3563) loss 3.0631 (3.0878) grad_norm 1.2476 (1.7567) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:15:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][330/625] eta 0:01:46 lr 0.000806 wd 0.0500 time 0.3560 (0.3613) data time 0.0006 (0.0039) model time 0.3554 (0.3572) loss 2.7091 (3.0872) grad_norm 1.3526 (1.7496) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][340/625] eta 0:01:42 lr 0.000806 wd 0.0500 time 0.3486 (0.3610) data time 0.0009 (0.0038) model time 0.3478 (0.3570) loss 3.3099 (3.0889) grad_norm 1.2548 (1.7596) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][350/625] eta 0:01:39 lr 0.000805 wd 0.0500 time 0.3472 (0.3610) data time 0.0009 (0.0038) model time 0.3463 (0.3571) loss 3.7328 (3.0887) grad_norm 2.2928 (1.7557) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][360/625] eta 0:01:35 lr 0.000805 wd 0.0500 time 0.3544 (0.3606) data time 0.0009 (0.0037) model time 0.3535 (0.3568) loss 2.6310 (3.0929) grad_norm 1.5043 (1.7536) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][370/625] eta 0:01:31 lr 0.000805 wd 0.0500 time 0.3512 (0.3603) data time 0.0008 (0.0036) model time 0.3503 (0.3565) loss 2.8985 (3.0911) grad_norm 1.3810 (1.7510) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][380/625] eta 0:01:28 lr 0.000805 wd 0.0500 time 0.3501 (0.3600) data time 0.0007 (0.0035) model time 0.3494 (0.3562) loss 3.6859 (3.0970) grad_norm 1.2756 (1.7464) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][390/625] eta 0:01:24 lr 0.000805 wd 0.0500 time 0.3381 (0.3597) data time 0.0010 (0.0035) model time 0.3371 (0.3559) loss 3.1221 (3.0998) grad_norm 1.8006 (1.7398) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][400/625] eta 0:01:20 lr 0.000804 wd 0.0500 time 0.3477 (0.3594) data time 0.0008 (0.0034) model time 0.3469 (0.3557) loss 3.3560 (3.0965) grad_norm 1.3723 (1.7391) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][410/625] eta 0:01:17 lr 0.000804 wd 0.0500 time 0.3486 (0.3591) data time 0.0007 (0.0033) model time 0.3480 (0.3554) loss 3.0022 (3.0913) grad_norm 2.4459 (1.7361) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][420/625] eta 0:01:13 lr 0.000804 wd 0.0500 time 0.3470 (0.3588) data time 0.0008 (0.0033) model time 0.3463 (0.3551) loss 3.4662 (3.0839) grad_norm 2.3425 (1.7403) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][430/625] eta 0:01:09 lr 0.000804 wd 0.0500 time 0.3502 (0.3589) data time 0.0007 (0.0032) model time 0.3495 (0.3554) loss 3.6635 (3.0853) grad_norm 1.4101 (1.7394) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][440/625] eta 0:01:06 lr 0.000804 wd 0.0500 time 0.5283 (0.3592) data time 0.0008 (0.0032) model time 0.5275 (0.3557) loss 2.4951 (3.0824) grad_norm 1.4743 (1.7317) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][450/625] eta 0:01:02 lr 0.000804 wd 0.0500 time 0.3466 (0.3589) data time 0.0008 (0.0031) model time 0.3459 (0.3555) loss 2.2679 (3.0786) grad_norm 2.0218 (1.7302) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][460/625] eta 0:00:59 lr 0.000803 wd 0.0500 time 0.3463 (0.3587) data time 0.0006 (0.0031) model time 0.3457 (0.3553) loss 2.8973 (3.0738) grad_norm 1.3871 (1.7275) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][470/625] eta 0:00:55 lr 0.000803 wd 0.0500 time 0.3525 (0.3598) data time 0.0007 (0.0030) model time 0.3518 (0.3566) loss 2.2084 (3.0727) grad_norm 2.0274 (1.7337) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][480/625] eta 0:00:52 lr 0.000803 wd 0.0500 time 0.3549 (0.3596) data time 0.0007 (0.0030) model time 0.3542 (0.3564) loss 3.1771 (3.0729) grad_norm 1.7266 (1.7428) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][490/625] eta 0:00:48 lr 0.000803 wd 0.0500 time 0.3520 (0.3594) data time 0.0009 (0.0029) model time 0.3511 (0.3562) loss 2.4500 (3.0729) grad_norm 3.5295 (1.7457) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:16:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][500/625] eta 0:00:44 lr 0.000803 wd 0.0500 time 0.3521 (0.3592) data time 0.0009 (0.0029) model time 0.3513 (0.3561) loss 3.6073 (3.0705) grad_norm 1.2210 (1.7402) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][510/625] eta 0:00:41 lr 0.000803 wd 0.0500 time 0.3480 (0.3590) data time 0.0007 (0.0028) model time 0.3473 (0.3560) loss 2.9259 (3.0668) grad_norm 1.2382 (1.7362) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][520/625] eta 0:00:37 lr 0.000802 wd 0.0500 time 0.3486 (0.3592) data time 0.0007 (0.0028) model time 0.3479 (0.3562) loss 3.8295 (3.0666) grad_norm 1.9714 (1.7317) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][530/625] eta 0:00:34 lr 0.000802 wd 0.0500 time 0.3464 (0.3590) data time 0.0007 (0.0028) model time 0.3457 (0.3560) loss 2.9200 (3.0626) grad_norm 1.6582 (1.7281) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][540/625] eta 0:00:30 lr 0.000802 wd 0.0500 time 0.3447 (0.3587) data time 0.0007 (0.0027) model time 0.3441 (0.3558) loss 2.7753 (3.0610) grad_norm 1.6083 (1.7294) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][550/625] eta 0:00:26 lr 0.000802 wd 0.0500 time 0.3497 (0.3585) data time 0.0006 (0.0027) model time 0.3490 (0.3556) loss 3.3171 (3.0609) grad_norm 1.1023 (1.7250) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][560/625] eta 0:00:23 lr 0.000802 wd 0.0500 time 0.3521 (0.3584) data time 0.0009 (0.0027) model time 0.3512 (0.3555) loss 2.7944 (3.0594) grad_norm 1.3023 (1.7177) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][570/625] eta 0:00:19 lr 0.000802 wd 0.0500 time 0.3642 (0.3582) data time 0.0008 (0.0026) model time 0.3634 (0.3554) loss 3.1112 (3.0620) grad_norm 1.4715 (1.7202) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][580/625] eta 0:00:16 lr 0.000801 wd 0.0500 time 0.5498 (0.3584) data time 0.0006 (0.0026) model time 0.5492 (0.3556) loss 3.5580 (3.0623) grad_norm 1.7167 (1.7193) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][590/625] eta 0:00:12 lr 0.000801 wd 0.0500 time 0.3462 (0.3590) data time 0.0009 (0.0026) model time 0.3453 (0.3562) loss 3.2447 (3.0606) grad_norm 1.6112 (1.7173) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][600/625] eta 0:00:08 lr 0.000801 wd 0.0500 time 0.3407 (0.3588) data time 0.0007 (0.0026) model time 0.3399 (0.3560) loss 3.1351 (3.0620) grad_norm 1.6691 (1.7147) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][610/625] eta 0:00:05 lr 0.000801 wd 0.0500 time 0.3433 (0.3587) data time 0.0004 (0.0025) model time 0.3429 (0.3560) loss 3.3909 (3.0643) grad_norm 1.5756 (1.7125) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [178/300][620/625] eta 0:00:01 lr 0.000801 wd 0.0500 time 0.3395 (0.3584) data time 0.0006 (0.0025) model time 0.3389 (0.3557) loss 3.3165 (3.0660) grad_norm 1.2506 (1.7095) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:43 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 178 training takes 0:03:43 [2024-09-01 07:17:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:17:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:17:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.445 (0.445) Loss 0.5449 (0.5449) Acc@1 88.623 (88.623) Acc@5 98.193 (98.193) Mem 11224MB [2024-09-01 07:17:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 0.9473 (0.6862) Acc@1 77.539 (84.584) Acc@5 94.482 (97.110) Mem 11224MB [2024-09-01 07:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 1.0000 (0.8145) Acc@1 76.416 (81.436) Acc@5 93.799 (95.729) Mem 11224MB [2024-09-01 07:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.058 Acc@5 95.725 [2024-09-01 07:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.1% [2024-09-01 07:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.06% [2024-09-01 07:17:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 07:17:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 07:17:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.476 (0.476) Loss 0.5015 (0.5015) Acc@1 89.307 (89.307) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 07:17:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.132) Loss 0.8198 (0.6210) Acc@1 80.420 (86.142) Acc@5 95.898 (97.665) Mem 11224MB [2024-09-01 07:17:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 0.9307 (0.7380) Acc@1 77.051 (83.059) Acc@5 94.922 (96.484) Mem 11224MB [2024-09-01 07:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.796 Acc@5 96.509 [2024-09-01 07:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.8% [2024-09-01 07:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.80% [2024-09-01 07:17:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:17:52 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:17:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][0/625] eta 0:08:12 lr 0.000801 wd 0.0500 time 0.7887 (0.7887) data time 0.4724 (0.4724) model time 0.0000 (0.0000) loss 3.3727 (3.3727) grad_norm 1.2170 (1.2170) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:17:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][10/625] eta 0:04:09 lr 0.000800 wd 0.0500 time 0.3413 (0.4059) data time 0.0007 (0.0437) model time 0.0000 (0.0000) loss 2.9816 (3.0606) grad_norm 2.3884 (1.7337) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][20/625] eta 0:03:54 lr 0.000800 wd 0.0500 time 0.3456 (0.3876) data time 0.0006 (0.0233) model time 0.0000 (0.0000) loss 1.7670 (3.1391) grad_norm 2.9587 (1.8371) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][30/625] eta 0:03:43 lr 0.000800 wd 0.0500 time 0.3499 (0.3754) data time 0.0006 (0.0160) model time 0.0000 (0.0000) loss 3.1485 (3.0634) grad_norm 2.8745 (1.8020) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][40/625] eta 0:03:35 lr 0.000800 wd 0.0500 time 0.3467 (0.3685) data time 0.0007 (0.0123) model time 0.0000 (0.0000) loss 2.6104 (3.0921) grad_norm 1.6768 (1.7673) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][50/625] eta 0:03:29 lr 0.000800 wd 0.0500 time 0.3538 (0.3648) data time 0.0010 (0.0101) model time 0.0000 (0.0000) loss 3.1917 (3.0508) grad_norm 1.1613 (1.7138) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][60/625] eta 0:03:24 lr 0.000800 wd 0.0500 time 0.3435 (0.3621) data time 0.0008 (0.0086) model time 0.3428 (0.3478) loss 2.2402 (3.0062) grad_norm 2.2052 (1.9975) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][70/625] eta 0:03:19 lr 0.000799 wd 0.0500 time 0.3500 (0.3601) data time 0.0006 (0.0075) model time 0.3494 (0.3472) loss 3.5134 (2.9987) grad_norm 1.4184 (1.9362) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][80/625] eta 0:03:15 lr 0.000799 wd 0.0500 time 0.3456 (0.3585) data time 0.0007 (0.0066) model time 0.3448 (0.3471) loss 2.3283 (2.9932) grad_norm 1.6150 (1.8898) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][90/625] eta 0:03:11 lr 0.000799 wd 0.0500 time 0.3467 (0.3574) data time 0.0008 (0.0060) model time 0.3459 (0.3472) loss 3.2199 (2.9821) grad_norm 1.2161 (1.8630) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][100/625] eta 0:03:10 lr 0.000799 wd 0.0500 time 0.3516 (0.3623) data time 0.0006 (0.0055) model time 0.3510 (0.3590) loss 4.1511 (3.0071) grad_norm 1.2432 (1.8285) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][110/625] eta 0:03:06 lr 0.000799 wd 0.0500 time 0.3460 (0.3612) data time 0.0006 (0.0051) model time 0.3453 (0.3573) loss 3.7674 (2.9850) grad_norm 1.4497 (1.8003) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][120/625] eta 0:03:01 lr 0.000798 wd 0.0500 time 0.3547 (0.3602) data time 0.0008 (0.0047) model time 0.3539 (0.3560) loss 3.4633 (2.9929) grad_norm 1.6520 (1.8102) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][130/625] eta 0:02:57 lr 0.000798 wd 0.0500 time 0.3442 (0.3592) data time 0.0006 (0.0044) model time 0.3436 (0.3548) loss 3.0937 (2.9836) grad_norm 2.3784 (1.8322) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][140/625] eta 0:02:53 lr 0.000798 wd 0.0500 time 0.3474 (0.3585) data time 0.0008 (0.0043) model time 0.3466 (0.3539) loss 3.2842 (2.9907) grad_norm 1.5340 (1.8482) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][150/625] eta 0:02:50 lr 0.000798 wd 0.0500 time 0.3483 (0.3590) data time 0.0008 (0.0040) model time 0.3475 (0.3551) loss 3.5586 (2.9821) grad_norm 2.0711 (1.8326) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][160/625] eta 0:02:46 lr 0.000798 wd 0.0500 time 0.3497 (0.3585) data time 0.0008 (0.0038) model time 0.3489 (0.3547) loss 3.6776 (2.9963) grad_norm 1.4947 (1.8336) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][170/625] eta 0:02:42 lr 0.000798 wd 0.0500 time 0.3469 (0.3579) data time 0.0008 (0.0037) model time 0.3461 (0.3541) loss 2.9213 (3.0008) grad_norm 1.2150 (1.8218) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:18:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][180/625] eta 0:02:39 lr 0.000797 wd 0.0500 time 0.3577 (0.3575) data time 0.0008 (0.0035) model time 0.3570 (0.3537) loss 3.7631 (2.9977) grad_norm 1.5063 (1.8092) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][190/625] eta 0:02:35 lr 0.000797 wd 0.0500 time 0.3493 (0.3572) data time 0.0010 (0.0034) model time 0.3484 (0.3535) loss 2.6708 (3.0058) grad_norm 1.2495 (1.7962) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][200/625] eta 0:02:31 lr 0.000797 wd 0.0500 time 0.3561 (0.3568) data time 0.0008 (0.0032) model time 0.3553 (0.3531) loss 2.2556 (3.0137) grad_norm 1.3785 (1.7856) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][210/625] eta 0:02:27 lr 0.000797 wd 0.0500 time 0.3492 (0.3565) data time 0.0008 (0.0033) model time 0.3484 (0.3527) loss 3.3596 (3.0299) grad_norm 2.2068 (1.7925) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][220/625] eta 0:02:24 lr 0.000797 wd 0.0500 time 0.3413 (0.3569) data time 0.0010 (0.0032) model time 0.3403 (0.3535) loss 3.1162 (3.0360) grad_norm 4.1943 (1.8019) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][230/625] eta 0:02:21 lr 0.000797 wd 0.0500 time 0.3462 (0.3575) data time 0.0009 (0.0031) model time 0.3453 (0.3543) loss 3.2795 (3.0287) grad_norm 1.4769 (1.8021) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][240/625] eta 0:02:17 lr 0.000796 wd 0.0500 time 0.3458 (0.3571) data time 0.0006 (0.0030) model time 0.3451 (0.3539) loss 2.9611 (3.0338) grad_norm 1.8608 (1.8047) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][250/625] eta 0:02:13 lr 0.000796 wd 0.0500 time 0.3438 (0.3568) data time 0.0007 (0.0029) model time 0.3431 (0.3537) loss 3.0009 (3.0338) grad_norm 1.3079 (1.8087) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][260/625] eta 0:02:10 lr 0.000796 wd 0.0500 time 0.3524 (0.3565) data time 0.0008 (0.0028) model time 0.3516 (0.3534) loss 2.9315 (3.0303) grad_norm 1.4632 (1.8029) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][270/625] eta 0:02:06 lr 0.000796 wd 0.0500 time 0.3459 (0.3562) data time 0.0009 (0.0027) model time 0.3450 (0.3532) loss 3.5661 (3.0369) grad_norm 1.5070 (1.8050) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][280/625] eta 0:02:02 lr 0.000796 wd 0.0500 time 0.3463 (0.3559) data time 0.0008 (0.0027) model time 0.3455 (0.3529) loss 3.1319 (3.0417) grad_norm 1.4323 (1.7945) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][290/625] eta 0:01:59 lr 0.000796 wd 0.0500 time 0.3515 (0.3557) data time 0.0008 (0.0026) model time 0.3507 (0.3527) loss 3.8505 (3.0468) grad_norm 1.4876 (1.7897) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][300/625] eta 0:01:55 lr 0.000795 wd 0.0500 time 0.3407 (0.3554) data time 0.0009 (0.0025) model time 0.3399 (0.3525) loss 3.4226 (3.0485) grad_norm 1.9690 (1.7819) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][310/625] eta 0:01:52 lr 0.000795 wd 0.0500 time 0.3399 (0.3572) data time 0.0008 (0.0025) model time 0.3391 (0.3547) loss 1.9151 (3.0510) grad_norm 1.3373 (1.7905) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][320/625] eta 0:01:48 lr 0.000795 wd 0.0500 time 0.3401 (0.3569) data time 0.0008 (0.0024) model time 0.3392 (0.3545) loss 3.4152 (3.0542) grad_norm 1.9170 (1.7838) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][330/625] eta 0:01:45 lr 0.000795 wd 0.0500 time 0.3509 (0.3567) data time 0.0008 (0.0024) model time 0.3501 (0.3543) loss 2.6751 (3.0532) grad_norm 1.2298 (1.7774) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][340/625] eta 0:01:41 lr 0.000795 wd 0.0500 time 0.3413 (0.3570) data time 0.0010 (0.0023) model time 0.3403 (0.3546) loss 3.2752 (3.0558) grad_norm 7.6250 (1.7874) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:19:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][350/625] eta 0:01:38 lr 0.000794 wd 0.0500 time 0.3488 (0.3568) data time 0.0007 (0.0023) model time 0.3481 (0.3545) loss 3.0392 (3.0595) grad_norm 1.8625 (1.8039) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][360/625] eta 0:01:34 lr 0.000794 wd 0.0500 time 0.3689 (0.3567) data time 0.0009 (0.0023) model time 0.3680 (0.3543) loss 3.1942 (3.0609) grad_norm 1.7321 (1.7977) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][370/625] eta 0:01:30 lr 0.000794 wd 0.0500 time 0.3425 (0.3564) data time 0.0008 (0.0022) model time 0.3417 (0.3540) loss 3.5159 (3.0651) grad_norm 1.7367 (1.8033) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][380/625] eta 0:01:27 lr 0.000794 wd 0.0500 time 0.3491 (0.3562) data time 0.0007 (0.0022) model time 0.3484 (0.3538) loss 3.3073 (3.0695) grad_norm 1.6873 (1.7999) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][390/625] eta 0:01:23 lr 0.000794 wd 0.0500 time 0.3512 (0.3560) data time 0.0008 (0.0022) model time 0.3504 (0.3537) loss 3.4712 (3.0707) grad_norm 2.0472 (1.8000) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][400/625] eta 0:01:20 lr 0.000794 wd 0.0500 time 0.3525 (0.3562) data time 0.0009 (0.0021) model time 0.3516 (0.3540) loss 3.0844 (3.0677) grad_norm 2.0493 (1.8013) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][410/625] eta 0:01:16 lr 0.000793 wd 0.0500 time 0.3502 (0.3565) data time 0.0009 (0.0021) model time 0.3493 (0.3544) loss 3.2842 (3.0730) grad_norm 1.5350 (1.7957) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][420/625] eta 0:01:13 lr 0.000793 wd 0.0500 time 0.3526 (0.3563) data time 0.0008 (0.0021) model time 0.3518 (0.3542) loss 2.6379 (3.0743) grad_norm 1.2107 (1.7852) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][430/625] eta 0:01:09 lr 0.000793 wd 0.0500 time 0.3506 (0.3561) data time 0.0008 (0.0020) model time 0.3498 (0.3539) loss 2.3116 (3.0727) grad_norm 1.8440 (1.7853) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][440/625] eta 0:01:05 lr 0.000793 wd 0.0500 time 0.3438 (0.3560) data time 0.0007 (0.0020) model time 0.3431 (0.3539) loss 2.4970 (3.0699) grad_norm 2.1657 (1.7861) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][450/625] eta 0:01:02 lr 0.000793 wd 0.0500 time 0.5386 (0.3562) data time 0.0006 (0.0020) model time 0.5380 (0.3542) loss 3.5435 (3.0729) grad_norm 1.7010 (1.7956) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][460/625] eta 0:00:58 lr 0.000793 wd 0.0500 time 0.3429 (0.3560) data time 0.0009 (0.0020) model time 0.3420 (0.3539) loss 3.3382 (3.0772) grad_norm 1.7519 (1.7931) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][470/625] eta 0:00:55 lr 0.000792 wd 0.0500 time 0.3448 (0.3558) data time 0.0007 (0.0019) model time 0.3441 (0.3537) loss 2.4844 (3.0738) grad_norm 1.4539 (1.7913) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][480/625] eta 0:00:51 lr 0.000792 wd 0.0500 time 0.3493 (0.3556) data time 0.0007 (0.0019) model time 0.3486 (0.3536) loss 3.2789 (3.0716) grad_norm 1.8986 (1.7898) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][490/625] eta 0:00:48 lr 0.000792 wd 0.0500 time 0.5676 (0.3564) data time 0.0006 (0.0019) model time 0.5670 (0.3544) loss 2.1099 (3.0712) grad_norm 1.3441 (1.7826) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][500/625] eta 0:00:44 lr 0.000792 wd 0.0500 time 0.3455 (0.3567) data time 0.0008 (0.0019) model time 0.3446 (0.3548) loss 3.1731 (3.0682) grad_norm 1.5198 (1.7764) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:20:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][510/625] eta 0:00:40 lr 0.000792 wd 0.0500 time 0.3473 (0.3565) data time 0.0008 (0.0019) model time 0.3465 (0.3546) loss 2.9194 (3.0738) grad_norm 2.6830 (1.7760) loss_scale 1024.0000 (520.0157) mem 11224MB [2024-09-01 07:20:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][520/625] eta 0:00:37 lr 0.000792 wd 0.0500 time 0.3432 (0.3567) data time 0.0008 (0.0018) model time 0.3424 (0.3549) loss 2.4149 (3.0733) grad_norm 1.8486 (1.7767) loss_scale 1024.0000 (529.6891) mem 11224MB [2024-09-01 07:21:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][530/625] eta 0:00:33 lr 0.000791 wd 0.0500 time 0.3792 (0.3569) data time 0.0008 (0.0018) model time 0.3784 (0.3552) loss 3.4756 (3.0750) grad_norm 2.0209 (1.7890) loss_scale 1024.0000 (538.9981) mem 11224MB [2024-09-01 07:21:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][540/625] eta 0:00:30 lr 0.000791 wd 0.0500 time 0.3498 (0.3568) data time 0.0007 (0.0018) model time 0.3491 (0.3551) loss 3.8562 (3.0774) grad_norm 2.0014 (1.7846) loss_scale 1024.0000 (547.9630) mem 11224MB [2024-09-01 07:21:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][550/625] eta 0:00:26 lr 0.000791 wd 0.0500 time 0.3520 (0.3567) data time 0.0008 (0.0018) model time 0.3512 (0.3549) loss 2.0185 (3.0731) grad_norm 1.9120 (1.7822) loss_scale 1024.0000 (556.6025) mem 11224MB [2024-09-01 07:21:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][560/625] eta 0:00:23 lr 0.000791 wd 0.0500 time 0.3511 (0.3566) data time 0.0009 (0.0018) model time 0.3502 (0.3548) loss 3.4056 (3.0776) grad_norm 2.1725 (1.7821) loss_scale 1024.0000 (564.9340) mem 11224MB [2024-09-01 07:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][570/625] eta 0:00:19 lr 0.000791 wd 0.0500 time 0.3543 (0.3564) data time 0.0006 (0.0018) model time 0.3537 (0.3547) loss 2.7340 (3.0796) grad_norm 1.3792 (1.7812) loss_scale 1024.0000 (572.9737) mem 11224MB [2024-09-01 07:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][580/625] eta 0:00:16 lr 0.000790 wd 0.0500 time 0.3586 (0.3563) data time 0.0006 (0.0017) model time 0.3579 (0.3546) loss 3.5343 (3.0814) grad_norm 1.6167 (1.7865) loss_scale 1024.0000 (580.7367) mem 11224MB [2024-09-01 07:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][590/625] eta 0:00:12 lr 0.000790 wd 0.0500 time 0.3534 (0.3562) data time 0.0008 (0.0017) model time 0.3525 (0.3544) loss 3.2045 (3.0838) grad_norm 1.3733 (1.7848) loss_scale 1024.0000 (588.2369) mem 11224MB [2024-09-01 07:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][600/625] eta 0:00:08 lr 0.000790 wd 0.0500 time 0.3412 (0.3560) data time 0.0007 (0.0017) model time 0.3405 (0.3543) loss 3.8576 (3.0881) grad_norm 3.4750 (1.7901) loss_scale 1024.0000 (595.4875) mem 11224MB [2024-09-01 07:21:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][610/625] eta 0:00:05 lr 0.000790 wd 0.0500 time 0.5306 (0.3568) data time 0.0007 (0.0017) model time 0.5299 (0.3552) loss 3.1444 (3.0901) grad_norm 1.9081 (1.7915) loss_scale 1024.0000 (602.5008) mem 11224MB [2024-09-01 07:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [179/300][620/625] eta 0:00:01 lr 0.000790 wd 0.0500 time 0.3366 (0.3566) data time 0.0004 (0.0017) model time 0.3362 (0.3549) loss 2.6454 (3.0915) grad_norm 1.4125 (1.7876) loss_scale 1024.0000 (609.2882) mem 11224MB [2024-09-01 07:21:34 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 179 training takes 0:03:42 [2024-09-01 07:21:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:21:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.456 (0.456) Loss 0.5596 (0.5596) Acc@1 88.379 (88.379) Acc@5 98.389 (98.389) Mem 11224MB [2024-09-01 07:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.132) Loss 0.9399 (0.7005) Acc@1 77.344 (84.508) Acc@5 94.971 (97.226) Mem 11224MB [2024-09-01 07:21:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.114) Loss 1.0645 (0.8325) Acc@1 74.219 (81.266) Acc@5 94.092 (95.824) Mem 11224MB [2024-09-01 07:21:38 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.984 Acc@5 95.833 [2024-09-01 07:21:38 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.0% [2024-09-01 07:21:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.775 (0.775) Loss 0.5020 (0.5020) Acc@1 89.307 (89.307) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 07:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.163) Loss 0.8154 (0.6202) Acc@1 80.664 (86.159) Acc@5 96.045 (97.692) Mem 11224MB [2024-09-01 07:21:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.129) Loss 0.9302 (0.7368) Acc@1 77.295 (83.087) Acc@5 94.971 (96.494) Mem 11224MB [2024-09-01 07:21:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.808 Acc@5 96.517 [2024-09-01 07:21:41 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.8% [2024-09-01 07:21:41 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.81% [2024-09-01 07:21:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:21:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:21:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][0/625] eta 0:07:25 lr 0.000790 wd 0.0500 time 0.7121 (0.7121) data time 0.4043 (0.4043) model time 0.0000 (0.0000) loss 3.2126 (3.2126) grad_norm 1.5562 (1.5562) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][10/625] eta 0:03:54 lr 0.000790 wd 0.0500 time 0.3547 (0.3820) data time 0.0007 (0.0375) model time 0.0000 (0.0000) loss 2.9770 (3.1963) grad_norm 2.4638 (1.6511) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:21:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][20/625] eta 0:03:41 lr 0.000789 wd 0.0500 time 0.3451 (0.3654) data time 0.0007 (0.0201) model time 0.0000 (0.0000) loss 3.7727 (3.1305) grad_norm 1.7540 (1.6771) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:21:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][30/625] eta 0:03:33 lr 0.000789 wd 0.0500 time 0.3471 (0.3595) data time 0.0006 (0.0139) model time 0.0000 (0.0000) loss 4.0030 (3.1343) grad_norm 1.7750 (1.8621) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:21:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][40/625] eta 0:03:28 lr 0.000789 wd 0.0500 time 0.3473 (0.3564) data time 0.0006 (0.0107) model time 0.0000 (0.0000) loss 2.4235 (3.0896) grad_norm 1.7784 (1.8206) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][50/625] eta 0:03:23 lr 0.000789 wd 0.0500 time 0.3498 (0.3546) data time 0.0009 (0.0088) model time 0.0000 (0.0000) loss 3.3208 (3.0543) grad_norm 2.4480 (1.8439) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][60/625] eta 0:03:20 lr 0.000789 wd 0.0500 time 0.3505 (0.3554) data time 0.0008 (0.0075) model time 0.3497 (0.3585) loss 2.7236 (3.0278) grad_norm 1.5038 (1.9045) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][70/625] eta 0:03:18 lr 0.000788 wd 0.0500 time 0.3488 (0.3573) data time 0.0006 (0.0065) model time 0.3481 (0.3632) loss 3.3331 (3.0139) grad_norm 1.1432 (1.8708) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][80/625] eta 0:03:14 lr 0.000788 wd 0.0500 time 0.3462 (0.3563) data time 0.0006 (0.0058) model time 0.3456 (0.3582) loss 3.5949 (3.0519) grad_norm 2.0129 (1.8626) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][90/625] eta 0:03:10 lr 0.000788 wd 0.0500 time 0.3480 (0.3554) data time 0.0006 (0.0053) model time 0.3474 (0.3555) loss 3.5004 (3.0941) grad_norm 1.9744 (1.8870) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][100/625] eta 0:03:06 lr 0.000788 wd 0.0500 time 0.3493 (0.3547) data time 0.0009 (0.0048) model time 0.3484 (0.3540) loss 2.9288 (3.0950) grad_norm 1.3206 (1.8578) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][110/625] eta 0:03:02 lr 0.000788 wd 0.0500 time 0.3513 (0.3542) data time 0.0007 (0.0045) model time 0.3507 (0.3530) loss 3.1706 (3.1011) grad_norm 2.0915 (1.8445) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][120/625] eta 0:02:58 lr 0.000788 wd 0.0500 time 0.3509 (0.3536) data time 0.0009 (0.0042) model time 0.3500 (0.3521) loss 3.0470 (3.1177) grad_norm 1.3292 (1.8562) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][130/625] eta 0:02:54 lr 0.000787 wd 0.0500 time 0.3503 (0.3532) data time 0.0009 (0.0039) model time 0.3495 (0.3514) loss 2.3341 (3.1075) grad_norm 1.9141 (1.8482) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][140/625] eta 0:02:51 lr 0.000787 wd 0.0500 time 0.3420 (0.3529) data time 0.0007 (0.0037) model time 0.3414 (0.3512) loss 3.1826 (3.0946) grad_norm 2.1437 (1.8401) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][150/625] eta 0:02:47 lr 0.000787 wd 0.0500 time 0.3458 (0.3525) data time 0.0007 (0.0035) model time 0.3451 (0.3506) loss 3.2340 (3.0798) grad_norm 1.7793 (1.8402) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][160/625] eta 0:02:43 lr 0.000787 wd 0.0500 time 0.3486 (0.3522) data time 0.0008 (0.0033) model time 0.3478 (0.3502) loss 3.3549 (3.0894) grad_norm 1.6558 (1.8222) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][170/625] eta 0:02:41 lr 0.000787 wd 0.0500 time 0.5406 (0.3550) data time 0.0006 (0.0032) model time 0.5400 (0.3543) loss 3.6103 (3.1058) grad_norm 1.8427 (1.8555) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][180/625] eta 0:02:38 lr 0.000787 wd 0.0500 time 0.3515 (0.3568) data time 0.0007 (0.0030) model time 0.3507 (0.3568) loss 2.7502 (3.1077) grad_norm 1.4725 (1.8417) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][190/625] eta 0:02:34 lr 0.000786 wd 0.0500 time 0.3486 (0.3562) data time 0.0008 (0.0029) model time 0.3477 (0.3560) loss 3.5091 (3.1024) grad_norm 1.8350 (1.8276) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][200/625] eta 0:02:31 lr 0.000786 wd 0.0500 time 0.3442 (0.3557) data time 0.0007 (0.0028) model time 0.3435 (0.3553) loss 3.9675 (3.1053) grad_norm 1.7973 (1.8187) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:22:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][210/625] eta 0:02:27 lr 0.000786 wd 0.0500 time 0.3426 (0.3554) data time 0.0007 (0.0027) model time 0.3419 (0.3548) loss 2.2633 (3.0966) grad_norm 2.1102 (1.8162) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][220/625] eta 0:02:23 lr 0.000786 wd 0.0500 time 0.3447 (0.3550) data time 0.0008 (0.0026) model time 0.3439 (0.3543) loss 3.2371 (3.1107) grad_norm 1.6075 (1.8238) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][230/625] eta 0:02:20 lr 0.000786 wd 0.0500 time 0.3470 (0.3547) data time 0.0007 (0.0026) model time 0.3463 (0.3539) loss 2.7891 (3.0952) grad_norm 1.7893 (1.8366) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][240/625] eta 0:02:16 lr 0.000786 wd 0.0500 time 0.3516 (0.3544) data time 0.0009 (0.0025) model time 0.3507 (0.3535) loss 2.8546 (3.0906) grad_norm 1.5240 (1.8320) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][250/625] eta 0:02:12 lr 0.000785 wd 0.0500 time 0.3449 (0.3541) data time 0.0006 (0.0024) model time 0.3443 (0.3532) loss 2.4281 (3.0986) grad_norm 1.3064 (1.8246) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][260/625] eta 0:02:09 lr 0.000785 wd 0.0500 time 0.3533 (0.3539) data time 0.0008 (0.0024) model time 0.3525 (0.3529) loss 2.6382 (3.0930) grad_norm 1.7833 (1.8277) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][270/625] eta 0:02:05 lr 0.000785 wd 0.0500 time 0.3460 (0.3543) data time 0.0009 (0.0023) model time 0.3451 (0.3534) loss 2.4963 (3.0914) grad_norm 1.2540 (1.8189) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][280/625] eta 0:02:02 lr 0.000785 wd 0.0500 time 0.3443 (0.3548) data time 0.0008 (0.0023) model time 0.3435 (0.3540) loss 3.0988 (3.1007) grad_norm 1.3763 (1.8133) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][290/625] eta 0:01:59 lr 0.000785 wd 0.0500 time 0.3482 (0.3558) data time 0.0006 (0.0022) model time 0.3475 (0.3553) loss 3.9930 (3.1033) grad_norm 1.3330 (1.8065) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][300/625] eta 0:01:55 lr 0.000785 wd 0.0500 time 0.3487 (0.3569) data time 0.0008 (0.0022) model time 0.3480 (0.3565) loss 2.6971 (3.1043) grad_norm 1.4531 (1.8001) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][310/625] eta 0:01:52 lr 0.000784 wd 0.0500 time 0.3510 (0.3567) data time 0.0007 (0.0021) model time 0.3503 (0.3563) loss 2.3839 (3.0964) grad_norm 1.5825 (1.7979) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][320/625] eta 0:01:48 lr 0.000784 wd 0.0500 time 0.3581 (0.3565) data time 0.0009 (0.0021) model time 0.3573 (0.3560) loss 3.2545 (3.0969) grad_norm 1.4400 (1.7876) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][330/625] eta 0:01:45 lr 0.000784 wd 0.0500 time 0.3491 (0.3563) data time 0.0009 (0.0020) model time 0.3482 (0.3558) loss 3.4119 (3.0956) grad_norm 1.6041 (1.7835) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][340/625] eta 0:01:41 lr 0.000784 wd 0.0500 time 0.3437 (0.3560) data time 0.0008 (0.0020) model time 0.3429 (0.3555) loss 2.9410 (3.0875) grad_norm 1.3505 (1.7807) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][350/625] eta 0:01:37 lr 0.000784 wd 0.0500 time 0.3490 (0.3559) data time 0.0007 (0.0020) model time 0.3483 (0.3553) loss 2.5050 (3.0899) grad_norm 1.1403 (1.7744) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][360/625] eta 0:01:34 lr 0.000783 wd 0.0500 time 0.3531 (0.3557) data time 0.0009 (0.0019) model time 0.3521 (0.3550) loss 3.4821 (3.0950) grad_norm 2.2591 (1.7788) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][370/625] eta 0:01:30 lr 0.000783 wd 0.0500 time 0.3481 (0.3555) data time 0.0006 (0.0019) model time 0.3475 (0.3548) loss 2.4564 (3.0854) grad_norm 1.4565 (1.7762) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:23:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][380/625] eta 0:01:27 lr 0.000783 wd 0.0500 time 0.3471 (0.3552) data time 0.0007 (0.0019) model time 0.3464 (0.3545) loss 3.0251 (3.0775) grad_norm 2.3776 (1.7779) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][390/625] eta 0:01:23 lr 0.000783 wd 0.0500 time 0.3352 (0.3551) data time 0.0008 (0.0019) model time 0.3344 (0.3543) loss 3.3408 (3.0768) grad_norm 1.9311 (1.7729) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][400/625] eta 0:01:19 lr 0.000783 wd 0.0500 time 0.3469 (0.3549) data time 0.0008 (0.0018) model time 0.3461 (0.3541) loss 2.5746 (3.0764) grad_norm 1.1620 (1.7720) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][410/625] eta 0:01:16 lr 0.000783 wd 0.0500 time 0.3517 (0.3547) data time 0.0009 (0.0018) model time 0.3509 (0.3539) loss 2.8977 (3.0784) grad_norm 3.9571 (1.7712) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][420/625] eta 0:01:12 lr 0.000782 wd 0.0500 time 0.3491 (0.3546) data time 0.0007 (0.0018) model time 0.3484 (0.3538) loss 2.7416 (3.0807) grad_norm 1.8662 (1.7723) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][430/625] eta 0:01:09 lr 0.000782 wd 0.0500 time 0.3415 (0.3545) data time 0.0007 (0.0018) model time 0.3408 (0.3536) loss 2.2561 (3.0802) grad_norm 1.7866 (1.7859) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][440/625] eta 0:01:05 lr 0.000782 wd 0.0500 time 0.3583 (0.3543) data time 0.0006 (0.0017) model time 0.3577 (0.3535) loss 3.0517 (3.0889) grad_norm 1.4878 (1.7824) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][450/625] eta 0:01:01 lr 0.000782 wd 0.0500 time 0.3478 (0.3541) data time 0.0007 (0.0017) model time 0.3471 (0.3533) loss 4.1165 (3.0949) grad_norm 3.1384 (1.7833) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][460/625] eta 0:00:58 lr 0.000782 wd 0.0500 time 0.3459 (0.3549) data time 0.0006 (0.0017) model time 0.3453 (0.3541) loss 2.0403 (3.0889) grad_norm 2.2412 (1.7883) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][470/625] eta 0:00:54 lr 0.000782 wd 0.0500 time 0.3512 (0.3547) data time 0.0006 (0.0017) model time 0.3506 (0.3539) loss 2.2194 (3.0790) grad_norm 1.4931 (1.7834) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][480/625] eta 0:00:51 lr 0.000781 wd 0.0500 time 0.3485 (0.3546) data time 0.0006 (0.0017) model time 0.3479 (0.3537) loss 3.6586 (3.0754) grad_norm 1.4863 (1.7806) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][490/625] eta 0:00:47 lr 0.000781 wd 0.0500 time 0.3498 (0.3544) data time 0.0009 (0.0016) model time 0.3489 (0.3536) loss 2.4766 (3.0712) grad_norm 1.9600 (1.7763) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][500/625] eta 0:00:44 lr 0.000781 wd 0.0500 time 0.5511 (0.3551) data time 0.0008 (0.0016) model time 0.5503 (0.3543) loss 2.7936 (3.0760) grad_norm 1.3383 (1.7766) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][510/625] eta 0:00:40 lr 0.000781 wd 0.0500 time 0.3462 (0.3557) data time 0.0009 (0.0016) model time 0.3453 (0.3551) loss 2.8922 (3.0825) grad_norm 1.4610 (1.7762) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][520/625] eta 0:00:37 lr 0.000781 wd 0.0500 time 0.3485 (0.3556) data time 0.0007 (0.0016) model time 0.3478 (0.3549) loss 2.9330 (3.0826) grad_norm 1.6607 (1.7732) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][530/625] eta 0:00:33 lr 0.000781 wd 0.0500 time 0.3462 (0.3554) data time 0.0008 (0.0016) model time 0.3454 (0.3547) loss 3.4884 (3.0828) grad_norm 1.3960 (1.7705) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][540/625] eta 0:00:30 lr 0.000780 wd 0.0500 time 0.3507 (0.3553) data time 0.0006 (0.0016) model time 0.3501 (0.3545) loss 2.6665 (3.0832) grad_norm 1.4015 (1.7660) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:24:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][550/625] eta 0:00:26 lr 0.000780 wd 0.0500 time 0.3464 (0.3551) data time 0.0007 (0.0016) model time 0.3457 (0.3543) loss 2.6289 (3.0838) grad_norm 1.3959 (1.7634) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][560/625] eta 0:00:23 lr 0.000780 wd 0.0500 time 0.3442 (0.3550) data time 0.0009 (0.0015) model time 0.3433 (0.3542) loss 3.1298 (3.0852) grad_norm 1.8921 (1.7592) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][570/625] eta 0:00:19 lr 0.000780 wd 0.0500 time 0.3420 (0.3548) data time 0.0007 (0.0015) model time 0.3413 (0.3540) loss 3.2538 (3.0843) grad_norm 1.7976 (1.7580) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][580/625] eta 0:00:15 lr 0.000780 wd 0.0500 time 0.3485 (0.3553) data time 0.0008 (0.0015) model time 0.3477 (0.3545) loss 3.1837 (3.0837) grad_norm 1.1814 (1.7548) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][590/625] eta 0:00:12 lr 0.000779 wd 0.0500 time 0.3460 (0.3551) data time 0.0006 (0.0015) model time 0.3454 (0.3544) loss 2.1348 (3.0783) grad_norm 1.0109 (1.7505) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][600/625] eta 0:00:08 lr 0.000779 wd 0.0500 time 0.3498 (0.3550) data time 0.0006 (0.0015) model time 0.3491 (0.3542) loss 2.3092 (3.0751) grad_norm 3.1515 (1.7539) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][610/625] eta 0:00:05 lr 0.000779 wd 0.0500 time 0.3446 (0.3549) data time 0.0004 (0.0015) model time 0.3442 (0.3541) loss 3.4999 (3.0747) grad_norm 2.0085 (1.7558) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [180/300][620/625] eta 0:00:01 lr 0.000779 wd 0.0500 time 0.3415 (0.3547) data time 0.0004 (0.0015) model time 0.3412 (0.3539) loss 3.6329 (3.0740) grad_norm 1.1888 (1.7499) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:24 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 180 training takes 0:03:41 [2024-09-01 07:25:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:25:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.452 (0.452) Loss 0.5854 (0.5854) Acc@1 87.158 (87.158) Acc@5 98.096 (98.096) Mem 11224MB [2024-09-01 07:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.130) Loss 0.9727 (0.6950) Acc@1 76.367 (84.819) Acc@5 94.238 (97.079) Mem 11224MB [2024-09-01 07:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 1.0645 (0.8286) Acc@1 75.391 (81.445) Acc@5 93.848 (95.710) Mem 11224MB [2024-09-01 07:25:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.236 Acc@5 95.729 [2024-09-01 07:25:28 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.2% [2024-09-01 07:25:28 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.24% [2024-09-01 07:25:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 07:25:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 07:25:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.447 (0.447) Loss 0.5029 (0.5029) Acc@1 89.404 (89.404) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 07:25:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.8140 (0.6192) Acc@1 80.859 (86.186) Acc@5 95.947 (97.692) Mem 11224MB [2024-09-01 07:25:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.111) Loss 0.9272 (0.7357) Acc@1 77.344 (83.154) Acc@5 95.166 (96.531) Mem 11224MB [2024-09-01 07:25:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.875 Acc@5 96.549 [2024-09-01 07:25:32 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 07:25:32 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.88% [2024-09-01 07:25:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:25:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:25:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][0/625] eta 0:07:32 lr 0.000779 wd 0.0500 time 0.7248 (0.7248) data time 0.4139 (0.4139) model time 0.0000 (0.0000) loss 3.1795 (3.1795) grad_norm 1.6895 (1.6895) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][10/625] eta 0:03:55 lr 0.000779 wd 0.0500 time 0.3582 (0.3837) data time 0.0008 (0.0384) model time 0.0000 (0.0000) loss 2.3880 (2.8530) grad_norm 1.3593 (1.7217) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][20/625] eta 0:03:41 lr 0.000779 wd 0.0500 time 0.3476 (0.3665) data time 0.0008 (0.0205) model time 0.0000 (0.0000) loss 3.3346 (2.9436) grad_norm 1.7026 (1.6838) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][30/625] eta 0:03:34 lr 0.000778 wd 0.0500 time 0.3512 (0.3602) data time 0.0006 (0.0142) model time 0.0000 (0.0000) loss 2.4994 (2.9833) grad_norm 4.7153 (1.9350) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][40/625] eta 0:03:28 lr 0.000778 wd 0.0500 time 0.3429 (0.3566) data time 0.0007 (0.0109) model time 0.0000 (0.0000) loss 3.2975 (2.9601) grad_norm 1.3071 (1.9380) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][50/625] eta 0:03:24 lr 0.000778 wd 0.0500 time 0.3421 (0.3552) data time 0.0006 (0.0089) model time 0.0000 (0.0000) loss 3.4288 (2.9751) grad_norm 1.9423 (1.8901) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][60/625] eta 0:03:21 lr 0.000778 wd 0.0500 time 0.3475 (0.3559) data time 0.0008 (0.0076) model time 0.3467 (0.3584) loss 2.7614 (3.0007) grad_norm 1.6154 (1.8645) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][70/625] eta 0:03:23 lr 0.000778 wd 0.0500 time 0.5475 (0.3660) data time 0.0007 (0.0066) model time 0.5468 (0.3927) loss 2.7915 (3.0105) grad_norm 1.5990 (1.8318) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][80/625] eta 0:03:18 lr 0.000778 wd 0.0500 time 0.3488 (0.3637) data time 0.0008 (0.0059) model time 0.3480 (0.3771) loss 2.3902 (2.9961) grad_norm 1.9397 (1.8239) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][90/625] eta 0:03:14 lr 0.000777 wd 0.0500 time 0.3426 (0.3636) data time 0.0008 (0.0054) model time 0.3418 (0.3735) loss 2.7839 (2.9592) grad_norm 1.2727 (1.7818) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][100/625] eta 0:03:11 lr 0.000777 wd 0.0500 time 0.3433 (0.3641) data time 0.0008 (0.0049) model time 0.3425 (0.3722) loss 3.1293 (2.9340) grad_norm 2.2700 (1.8043) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][110/625] eta 0:03:06 lr 0.000777 wd 0.0500 time 0.3499 (0.3628) data time 0.0009 (0.0045) model time 0.3490 (0.3683) loss 3.7449 (2.9577) grad_norm 1.4736 (1.7898) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][120/625] eta 0:03:02 lr 0.000777 wd 0.0500 time 0.3418 (0.3616) data time 0.0009 (0.0042) model time 0.3409 (0.3653) loss 3.1868 (2.9731) grad_norm 1.7853 (1.7644) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][130/625] eta 0:02:58 lr 0.000777 wd 0.0500 time 0.3520 (0.3606) data time 0.0006 (0.0040) model time 0.3514 (0.3632) loss 2.0160 (2.9680) grad_norm 1.6217 (1.7517) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][140/625] eta 0:02:54 lr 0.000776 wd 0.0500 time 0.3445 (0.3597) data time 0.0006 (0.0038) model time 0.3439 (0.3614) loss 2.8471 (2.9788) grad_norm 1.7890 (1.7598) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][150/625] eta 0:02:50 lr 0.000776 wd 0.0500 time 0.3469 (0.3589) data time 0.0008 (0.0036) model time 0.3461 (0.3600) loss 3.6795 (2.9860) grad_norm 1.4628 (1.7706) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][160/625] eta 0:02:46 lr 0.000776 wd 0.0500 time 0.3471 (0.3582) data time 0.0006 (0.0034) model time 0.3464 (0.3588) loss 2.9465 (3.0013) grad_norm 2.0685 (1.7748) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][170/625] eta 0:02:42 lr 0.000776 wd 0.0500 time 0.3443 (0.3578) data time 0.0008 (0.0032) model time 0.3434 (0.3580) loss 3.3379 (3.0060) grad_norm 1.4148 (1.7715) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][180/625] eta 0:02:38 lr 0.000776 wd 0.0500 time 0.3440 (0.3573) data time 0.0006 (0.0031) model time 0.3434 (0.3572) loss 2.6832 (3.0045) grad_norm 1.4342 (1.7884) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][190/625] eta 0:02:35 lr 0.000776 wd 0.0500 time 0.3429 (0.3568) data time 0.0007 (0.0030) model time 0.3422 (0.3566) loss 2.8187 (3.0155) grad_norm 2.2477 (1.7895) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][200/625] eta 0:02:31 lr 0.000775 wd 0.0500 time 0.3501 (0.3564) data time 0.0006 (0.0029) model time 0.3495 (0.3560) loss 3.2599 (3.0182) grad_norm 1.7688 (1.7843) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][210/625] eta 0:02:28 lr 0.000775 wd 0.0500 time 0.3459 (0.3566) data time 0.0008 (0.0028) model time 0.3451 (0.3563) loss 3.2936 (3.0269) grad_norm 1.1739 (1.7860) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][220/625] eta 0:02:24 lr 0.000775 wd 0.0500 time 0.3462 (0.3563) data time 0.0006 (0.0027) model time 0.3456 (0.3558) loss 3.5238 (3.0367) grad_norm 2.0786 (1.7771) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][230/625] eta 0:02:20 lr 0.000775 wd 0.0500 time 0.3398 (0.3559) data time 0.0006 (0.0026) model time 0.3392 (0.3553) loss 3.3765 (3.0348) grad_norm 1.6408 (1.7798) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:26:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][240/625] eta 0:02:16 lr 0.000775 wd 0.0500 time 0.3549 (0.3557) data time 0.0006 (0.0025) model time 0.3543 (0.3550) loss 4.1374 (3.0370) grad_norm 1.5398 (1.7656) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][250/625] eta 0:02:13 lr 0.000775 wd 0.0500 time 0.5121 (0.3568) data time 0.0008 (0.0025) model time 0.5113 (0.3564) loss 1.8888 (3.0288) grad_norm 1.7413 (1.7613) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][260/625] eta 0:02:10 lr 0.000774 wd 0.0500 time 0.3442 (0.3581) data time 0.0006 (0.0024) model time 0.3436 (0.3580) loss 3.1721 (3.0266) grad_norm 1.6001 (1.7596) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][270/625] eta 0:02:07 lr 0.000774 wd 0.0500 time 0.3512 (0.3578) data time 0.0006 (0.0023) model time 0.3505 (0.3576) loss 3.0280 (3.0231) grad_norm 2.6070 (1.7595) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][280/625] eta 0:02:03 lr 0.000774 wd 0.0500 time 0.3469 (0.3576) data time 0.0008 (0.0023) model time 0.3461 (0.3573) loss 3.5206 (3.0217) grad_norm 1.8557 (1.7637) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][290/625] eta 0:01:59 lr 0.000774 wd 0.0500 time 0.3523 (0.3573) data time 0.0006 (0.0022) model time 0.3517 (0.3569) loss 2.7840 (3.0230) grad_norm 1.7810 (1.7663) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][300/625] eta 0:01:56 lr 0.000774 wd 0.0500 time 0.3556 (0.3577) data time 0.0008 (0.0022) model time 0.3548 (0.3574) loss 2.9869 (3.0327) grad_norm 1.6724 (1.7739) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][310/625] eta 0:01:52 lr 0.000774 wd 0.0500 time 0.3471 (0.3579) data time 0.0006 (0.0021) model time 0.3465 (0.3576) loss 3.7079 (3.0341) grad_norm 1.2969 (1.7664) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][320/625] eta 0:01:49 lr 0.000773 wd 0.0500 time 0.3451 (0.3575) data time 0.0009 (0.0021) model time 0.3442 (0.3571) loss 3.5012 (3.0346) grad_norm 1.7011 (1.7609) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][330/625] eta 0:01:45 lr 0.000773 wd 0.0500 time 0.3469 (0.3572) data time 0.0007 (0.0021) model time 0.3461 (0.3568) loss 3.2642 (3.0306) grad_norm 1.4133 (1.7711) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][340/625] eta 0:01:41 lr 0.000773 wd 0.0500 time 0.3450 (0.3570) data time 0.0009 (0.0020) model time 0.3441 (0.3564) loss 3.0054 (3.0359) grad_norm 1.4619 (1.7829) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][350/625] eta 0:01:38 lr 0.000773 wd 0.0500 time 0.3491 (0.3568) data time 0.0009 (0.0020) model time 0.3483 (0.3562) loss 3.4821 (3.0385) grad_norm 1.1641 (1.7890) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][360/625] eta 0:01:34 lr 0.000773 wd 0.0500 time 0.3458 (0.3565) data time 0.0009 (0.0020) model time 0.3448 (0.3559) loss 2.3109 (3.0401) grad_norm 1.9139 (1.8038) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][370/625] eta 0:01:31 lr 0.000773 wd 0.0500 time 0.5505 (0.3573) data time 0.0006 (0.0019) model time 0.5499 (0.3569) loss 3.3730 (3.0401) grad_norm 1.5125 (1.7947) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][380/625] eta 0:01:27 lr 0.000772 wd 0.0500 time 0.3461 (0.3582) data time 0.0008 (0.0019) model time 0.3453 (0.3579) loss 3.4245 (3.0393) grad_norm 1.3532 (1.7933) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][390/625] eta 0:01:24 lr 0.000772 wd 0.0500 time 0.4900 (0.3583) data time 0.0006 (0.0019) model time 0.4894 (0.3579) loss 3.1283 (3.0394) grad_norm 1.3691 (1.7910) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:27:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][400/625] eta 0:01:20 lr 0.000772 wd 0.0500 time 0.3480 (0.3579) data time 0.0006 (0.0018) model time 0.3474 (0.3575) loss 2.5421 (3.0371) grad_norm 1.6075 (1.7895) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][410/625] eta 0:01:16 lr 0.000772 wd 0.0500 time 0.3476 (0.3577) data time 0.0008 (0.0018) model time 0.3468 (0.3572) loss 3.1626 (3.0367) grad_norm 2.4146 (1.7844) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][420/625] eta 0:01:13 lr 0.000772 wd 0.0500 time 0.3411 (0.3574) data time 0.0009 (0.0018) model time 0.3402 (0.3569) loss 3.4614 (3.0334) grad_norm 2.4868 (1.7908) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][430/625] eta 0:01:09 lr 0.000771 wd 0.0500 time 0.3429 (0.3572) data time 0.0008 (0.0018) model time 0.3421 (0.3566) loss 3.4657 (3.0387) grad_norm 1.9603 (1.7841) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][440/625] eta 0:01:06 lr 0.000771 wd 0.0500 time 0.3454 (0.3569) data time 0.0008 (0.0017) model time 0.3446 (0.3563) loss 3.4279 (3.0461) grad_norm 1.6053 (1.7862) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][450/625] eta 0:01:02 lr 0.000771 wd 0.0500 time 0.3445 (0.3567) data time 0.0008 (0.0017) model time 0.3436 (0.3561) loss 3.0112 (3.0467) grad_norm 1.9578 (1.7897) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][460/625] eta 0:00:58 lr 0.000771 wd 0.0500 time 0.3466 (0.3565) data time 0.0007 (0.0017) model time 0.3459 (0.3559) loss 3.6608 (3.0504) grad_norm 1.2291 (1.7917) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][470/625] eta 0:00:55 lr 0.000771 wd 0.0500 time 0.3459 (0.3563) data time 0.0006 (0.0017) model time 0.3453 (0.3556) loss 2.6016 (3.0538) grad_norm 1.1536 (1.7882) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][480/625] eta 0:00:51 lr 0.000771 wd 0.0500 time 0.3432 (0.3561) data time 0.0006 (0.0017) model time 0.3426 (0.3554) loss 2.8779 (3.0550) grad_norm 3.4470 (1.7997) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][490/625] eta 0:00:48 lr 0.000770 wd 0.0500 time 0.3489 (0.3565) data time 0.0008 (0.0017) model time 0.3481 (0.3559) loss 3.3934 (3.0583) grad_norm 1.6721 (1.8187) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][500/625] eta 0:00:44 lr 0.000770 wd 0.0500 time 0.3468 (0.3564) data time 0.0007 (0.0016) model time 0.3461 (0.3557) loss 2.8517 (3.0561) grad_norm 0.9877 (1.8116) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][510/625] eta 0:00:41 lr 0.000770 wd 0.0500 time 0.5732 (0.3566) data time 0.0008 (0.0016) model time 0.5724 (0.3560) loss 3.8235 (3.0623) grad_norm 1.3697 (1.8084) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][520/625] eta 0:00:37 lr 0.000770 wd 0.0500 time 0.3441 (0.3564) data time 0.0006 (0.0016) model time 0.3435 (0.3557) loss 3.2570 (3.0656) grad_norm 2.4241 (1.8145) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][530/625] eta 0:00:33 lr 0.000770 wd 0.0500 time 0.3444 (0.3562) data time 0.0008 (0.0016) model time 0.3436 (0.3555) loss 2.8378 (3.0715) grad_norm 1.2854 (1.8098) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][540/625] eta 0:00:30 lr 0.000770 wd 0.0500 time 0.3471 (0.3561) data time 0.0007 (0.0016) model time 0.3464 (0.3554) loss 4.0739 (3.0760) grad_norm 2.2426 (1.8040) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][550/625] eta 0:00:26 lr 0.000769 wd 0.0500 time 0.3470 (0.3559) data time 0.0009 (0.0016) model time 0.3462 (0.3552) loss 2.6668 (3.0713) grad_norm 1.7425 (1.8053) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][560/625] eta 0:00:23 lr 0.000769 wd 0.0500 time 0.3515 (0.3558) data time 0.0009 (0.0015) model time 0.3506 (0.3550) loss 3.5422 (3.0721) grad_norm 3.3324 (1.8096) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:28:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][570/625] eta 0:00:19 lr 0.000769 wd 0.0500 time 0.3454 (0.3556) data time 0.0006 (0.0015) model time 0.3448 (0.3549) loss 3.5422 (3.0727) grad_norm 1.7211 (1.8083) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][580/625] eta 0:00:16 lr 0.000769 wd 0.0500 time 0.3428 (0.3562) data time 0.0008 (0.0015) model time 0.3420 (0.3555) loss 3.5744 (3.0731) grad_norm 1.1758 (1.8020) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][590/625] eta 0:00:12 lr 0.000769 wd 0.0500 time 0.3487 (0.3568) data time 0.0008 (0.0015) model time 0.3479 (0.3561) loss 3.1913 (3.0792) grad_norm 2.0615 (1.8115) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][600/625] eta 0:00:08 lr 0.000769 wd 0.0500 time 0.3494 (0.3567) data time 0.0006 (0.0015) model time 0.3488 (0.3560) loss 3.4617 (3.0805) grad_norm 2.1027 (1.8207) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][610/625] eta 0:00:05 lr 0.000768 wd 0.0500 time 0.3461 (0.3571) data time 0.0006 (0.0015) model time 0.3455 (0.3564) loss 2.7173 (3.0819) grad_norm 1.7519 (1.8197) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [181/300][620/625] eta 0:00:01 lr 0.000768 wd 0.0500 time 0.3382 (0.3569) data time 0.0004 (0.0015) model time 0.3378 (0.3562) loss 3.0411 (3.0796) grad_norm 1.5434 (1.8143) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:16 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 181 training takes 0:03:42 [2024-09-01 07:29:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:29:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:29:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.462 (0.462) Loss 0.5688 (0.5688) Acc@1 88.086 (88.086) Acc@5 98.242 (98.242) Mem 11224MB [2024-09-01 07:29:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.131) Loss 0.9380 (0.6843) Acc@1 77.344 (84.686) Acc@5 95.312 (97.288) Mem 11224MB [2024-09-01 07:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.0078 (0.8160) Acc@1 76.807 (81.545) Acc@5 94.092 (95.780) Mem 11224MB [2024-09-01 07:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.240 Acc@5 95.797 [2024-09-01 07:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.2% [2024-09-01 07:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.24% [2024-09-01 07:29:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 07:29:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 07:29:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.447 (0.447) Loss 0.5034 (0.5034) Acc@1 89.502 (89.502) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 07:29:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.8120 (0.6189) Acc@1 80.664 (86.195) Acc@5 95.898 (97.678) Mem 11224MB [2024-09-01 07:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.9263 (0.7347) Acc@1 77.100 (83.164) Acc@5 95.068 (96.526) Mem 11224MB [2024-09-01 07:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.877 Acc@5 96.545 [2024-09-01 07:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 07:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.88% [2024-09-01 07:29:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:29:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:29:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][0/625] eta 0:07:38 lr 0.000768 wd 0.0500 time 0.7329 (0.7329) data time 0.4223 (0.4223) model time 0.0000 (0.0000) loss 3.5331 (3.5331) grad_norm 1.1541 (1.1541) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][10/625] eta 0:03:56 lr 0.000768 wd 0.0500 time 0.3487 (0.3840) data time 0.0008 (0.0391) model time 0.0000 (0.0000) loss 3.9302 (3.3907) grad_norm 1.6345 (1.6601) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][20/625] eta 0:03:45 lr 0.000768 wd 0.0500 time 0.3442 (0.3734) data time 0.0006 (0.0209) model time 0.0000 (0.0000) loss 2.4519 (3.2387) grad_norm 2.2633 (1.6981) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][30/625] eta 0:03:37 lr 0.000768 wd 0.0500 time 0.3522 (0.3652) data time 0.0006 (0.0144) model time 0.0000 (0.0000) loss 2.5195 (3.0936) grad_norm 1.7284 (1.6541) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][40/625] eta 0:03:31 lr 0.000767 wd 0.0500 time 0.3474 (0.3611) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 3.1407 (3.0945) grad_norm 1.2024 (1.6527) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][50/625] eta 0:03:25 lr 0.000767 wd 0.0500 time 0.3437 (0.3582) data time 0.0008 (0.0091) model time 0.0000 (0.0000) loss 2.5400 (3.0579) grad_norm 1.7858 (1.6612) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][60/625] eta 0:03:21 lr 0.000767 wd 0.0500 time 0.3438 (0.3564) data time 0.0009 (0.0077) model time 0.3429 (0.3460) loss 2.8054 (3.0818) grad_norm 2.4980 (1.6925) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][70/625] eta 0:03:17 lr 0.000767 wd 0.0500 time 0.3476 (0.3551) data time 0.0008 (0.0068) model time 0.3468 (0.3463) loss 2.0648 (3.0777) grad_norm 1.8698 (1.6934) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][80/625] eta 0:03:13 lr 0.000767 wd 0.0500 time 0.3464 (0.3544) data time 0.0007 (0.0060) model time 0.3457 (0.3470) loss 4.1218 (3.0803) grad_norm 1.5073 (1.6674) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:29:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][90/625] eta 0:03:09 lr 0.000767 wd 0.0500 time 0.3453 (0.3537) data time 0.0006 (0.0055) model time 0.3447 (0.3472) loss 3.4112 (3.0762) grad_norm 1.1554 (1.6500) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][100/625] eta 0:03:05 lr 0.000766 wd 0.0500 time 0.3453 (0.3532) data time 0.0008 (0.0050) model time 0.3445 (0.3472) loss 2.7837 (3.0647) grad_norm 1.4384 (1.6801) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][110/625] eta 0:03:01 lr 0.000766 wd 0.0500 time 0.3502 (0.3528) data time 0.0006 (0.0046) model time 0.3495 (0.3473) loss 2.5045 (3.0750) grad_norm 1.8488 (1.6924) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][120/625] eta 0:02:58 lr 0.000766 wd 0.0500 time 0.3477 (0.3525) data time 0.0009 (0.0043) model time 0.3468 (0.3475) loss 3.3304 (3.0699) grad_norm 1.4410 (1.6854) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][130/625] eta 0:02:54 lr 0.000766 wd 0.0500 time 0.3467 (0.3522) data time 0.0009 (0.0040) model time 0.3458 (0.3476) loss 3.3895 (3.0609) grad_norm 2.0921 (1.6881) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][140/625] eta 0:02:50 lr 0.000766 wd 0.0500 time 0.3491 (0.3520) data time 0.0007 (0.0038) model time 0.3484 (0.3476) loss 3.1787 (3.0630) grad_norm 1.3471 (1.6861) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][150/625] eta 0:02:47 lr 0.000766 wd 0.0500 time 0.3436 (0.3517) data time 0.0011 (0.0036) model time 0.3425 (0.3475) loss 2.7088 (3.0663) grad_norm 1.4481 (1.6811) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][160/625] eta 0:02:45 lr 0.000765 wd 0.0500 time 0.3386 (0.3566) data time 0.0008 (0.0035) model time 0.3378 (0.3550) loss 2.8813 (3.0624) grad_norm 1.4906 (1.6755) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][170/625] eta 0:02:43 lr 0.000765 wd 0.0500 time 0.3495 (0.3598) data time 0.0006 (0.0033) model time 0.3489 (0.3596) loss 1.7811 (3.0592) grad_norm 1.5680 (1.6724) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][180/625] eta 0:02:39 lr 0.000765 wd 0.0500 time 0.3526 (0.3592) data time 0.0006 (0.0032) model time 0.3520 (0.3587) loss 3.1833 (3.0646) grad_norm 2.0934 (1.6827) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][190/625] eta 0:02:35 lr 0.000765 wd 0.0500 time 0.3506 (0.3586) data time 0.0006 (0.0030) model time 0.3499 (0.3579) loss 3.5187 (3.0697) grad_norm 2.5922 (1.7359) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][200/625] eta 0:02:32 lr 0.000765 wd 0.0500 time 0.3471 (0.3588) data time 0.0008 (0.0029) model time 0.3463 (0.3582) loss 2.9119 (3.0716) grad_norm 1.9485 (1.7645) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][210/625] eta 0:02:28 lr 0.000764 wd 0.0500 time 0.3465 (0.3583) data time 0.0007 (0.0028) model time 0.3457 (0.3575) loss 3.1657 (3.0748) grad_norm 1.7233 (1.7744) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][220/625] eta 0:02:24 lr 0.000764 wd 0.0500 time 0.3477 (0.3579) data time 0.0008 (0.0027) model time 0.3470 (0.3570) loss 2.6981 (3.0830) grad_norm 1.4288 (1.7655) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][230/625] eta 0:02:21 lr 0.000764 wd 0.0500 time 0.3435 (0.3575) data time 0.0006 (0.0027) model time 0.3429 (0.3564) loss 2.2618 (3.0882) grad_norm 1.1875 (1.7554) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][240/625] eta 0:02:17 lr 0.000764 wd 0.0500 time 0.3539 (0.3571) data time 0.0008 (0.0026) model time 0.3531 (0.3559) loss 3.1620 (3.0891) grad_norm 1.6306 (1.7585) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][250/625] eta 0:02:13 lr 0.000764 wd 0.0500 time 0.3442 (0.3568) data time 0.0009 (0.0025) model time 0.3434 (0.3556) loss 3.5526 (3.0945) grad_norm 1.5454 (1.7519) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][260/625] eta 0:02:10 lr 0.000764 wd 0.0500 time 0.3502 (0.3565) data time 0.0009 (0.0025) model time 0.3493 (0.3552) loss 3.2799 (3.0922) grad_norm 1.4798 (1.7564) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][270/625] eta 0:02:06 lr 0.000763 wd 0.0500 time 0.3514 (0.3562) data time 0.0007 (0.0024) model time 0.3507 (0.3549) loss 2.9025 (3.0946) grad_norm 1.7237 (1.7551) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][280/625] eta 0:02:02 lr 0.000763 wd 0.0500 time 0.3390 (0.3560) data time 0.0008 (0.0023) model time 0.3383 (0.3547) loss 3.6951 (3.0988) grad_norm 1.7521 (1.7466) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][290/625] eta 0:01:59 lr 0.000763 wd 0.0500 time 0.3480 (0.3570) data time 0.0007 (0.0023) model time 0.3473 (0.3559) loss 3.7631 (3.1127) grad_norm 1.8505 (1.7421) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][300/625] eta 0:01:55 lr 0.000763 wd 0.0500 time 0.3478 (0.3567) data time 0.0009 (0.0022) model time 0.3469 (0.3555) loss 3.6596 (3.1132) grad_norm 1.9032 (1.7399) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][310/625] eta 0:01:52 lr 0.000763 wd 0.0500 time 0.3515 (0.3565) data time 0.0009 (0.0022) model time 0.3507 (0.3553) loss 3.0837 (3.1167) grad_norm 1.8692 (1.7346) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][320/625] eta 0:01:48 lr 0.000763 wd 0.0500 time 0.3556 (0.3568) data time 0.0006 (0.0022) model time 0.3549 (0.3557) loss 3.6474 (3.1173) grad_norm 2.0472 (1.7430) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][330/625] eta 0:01:45 lr 0.000762 wd 0.0500 time 0.3525 (0.3567) data time 0.0007 (0.0021) model time 0.3519 (0.3555) loss 2.3800 (3.1150) grad_norm 1.7713 (1.7500) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][340/625] eta 0:01:41 lr 0.000762 wd 0.0500 time 0.3462 (0.3573) data time 0.0008 (0.0021) model time 0.3454 (0.3562) loss 3.4836 (3.1199) grad_norm 5.7815 (1.7534) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][350/625] eta 0:01:38 lr 0.000762 wd 0.0500 time 0.3424 (0.3594) data time 0.0008 (0.0021) model time 0.3415 (0.3588) loss 3.1381 (3.1199) grad_norm 1.3617 (1.7638) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][360/625] eta 0:01:35 lr 0.000762 wd 0.0500 time 0.3489 (0.3591) data time 0.0006 (0.0020) model time 0.3482 (0.3584) loss 3.4601 (3.1203) grad_norm 1.9366 (1.7693) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][370/625] eta 0:01:31 lr 0.000762 wd 0.0500 time 0.3426 (0.3587) data time 0.0009 (0.0020) model time 0.3417 (0.3579) loss 3.1914 (3.1204) grad_norm 1.2264 (1.7854) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][380/625] eta 0:01:27 lr 0.000762 wd 0.0500 time 0.3470 (0.3585) data time 0.0008 (0.0020) model time 0.3462 (0.3577) loss 3.7183 (3.1199) grad_norm 1.9077 (1.8030) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][390/625] eta 0:01:24 lr 0.000761 wd 0.0500 time 0.3481 (0.3583) data time 0.0007 (0.0019) model time 0.3474 (0.3574) loss 2.9612 (3.1179) grad_norm 3.9416 (1.8096) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][400/625] eta 0:01:20 lr 0.000761 wd 0.0500 time 0.3715 (0.3582) data time 0.0009 (0.0019) model time 0.3706 (0.3573) loss 2.0943 (3.1136) grad_norm 1.3784 (1.8065) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][410/625] eta 0:01:16 lr 0.000761 wd 0.0500 time 0.3493 (0.3579) data time 0.0009 (0.0019) model time 0.3484 (0.3570) loss 3.6174 (3.1141) grad_norm 2.0441 (1.8073) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][420/625] eta 0:01:13 lr 0.000761 wd 0.0500 time 0.3462 (0.3578) data time 0.0009 (0.0019) model time 0.3453 (0.3569) loss 3.1368 (3.1159) grad_norm 1.5606 (1.7997) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][430/625] eta 0:01:09 lr 0.000761 wd 0.0500 time 0.3473 (0.3576) data time 0.0008 (0.0018) model time 0.3466 (0.3566) loss 2.6568 (3.1205) grad_norm 2.0828 (1.7957) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][440/625] eta 0:01:06 lr 0.000761 wd 0.0500 time 0.3523 (0.3574) data time 0.0007 (0.0018) model time 0.3516 (0.3564) loss 2.8437 (3.1207) grad_norm 2.5749 (1.8198) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][450/625] eta 0:01:02 lr 0.000760 wd 0.0500 time 0.3419 (0.3572) data time 0.0009 (0.0018) model time 0.3410 (0.3562) loss 3.4871 (3.1205) grad_norm 1.5770 (1.8116) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][460/625] eta 0:00:59 lr 0.000760 wd 0.0500 time 0.5493 (0.3580) data time 0.0008 (0.0018) model time 0.5485 (0.3570) loss 2.3202 (3.1158) grad_norm 1.7866 (1.8045) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][470/625] eta 0:00:55 lr 0.000760 wd 0.0500 time 0.3620 (0.3591) data time 0.0007 (0.0018) model time 0.3614 (0.3583) loss 3.6820 (3.1151) grad_norm 1.5224 (1.8031) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][480/625] eta 0:00:52 lr 0.000760 wd 0.0500 time 0.3557 (0.3589) data time 0.0009 (0.0018) model time 0.3547 (0.3581) loss 3.5163 (3.1156) grad_norm 2.1332 (1.7977) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][490/625] eta 0:00:48 lr 0.000760 wd 0.0500 time 0.3446 (0.3587) data time 0.0009 (0.0017) model time 0.3437 (0.3579) loss 1.8126 (3.1113) grad_norm 1.7281 (1.8164) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][500/625] eta 0:00:44 lr 0.000759 wd 0.0500 time 0.3505 (0.3592) data time 0.0006 (0.0017) model time 0.3498 (0.3585) loss 3.9494 (3.1071) grad_norm 2.4117 (1.8258) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][510/625] eta 0:00:41 lr 0.000759 wd 0.0500 time 0.3525 (0.3591) data time 0.0006 (0.0017) model time 0.3519 (0.3583) loss 1.8355 (3.0982) grad_norm 1.5797 (1.8249) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][520/625] eta 0:00:37 lr 0.000759 wd 0.0500 time 0.3456 (0.3589) data time 0.0006 (0.0017) model time 0.3450 (0.3581) loss 2.9265 (3.0920) grad_norm 2.5151 (1.8280) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][530/625] eta 0:00:34 lr 0.000759 wd 0.0500 time 0.3471 (0.3590) data time 0.0009 (0.0017) model time 0.3462 (0.3582) loss 2.6819 (3.0906) grad_norm 1.9136 (1.8264) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][540/625] eta 0:00:30 lr 0.000759 wd 0.0500 time 0.3471 (0.3588) data time 0.0007 (0.0017) model time 0.3464 (0.3580) loss 3.6146 (3.0914) grad_norm 1.7941 (1.8259) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][550/625] eta 0:00:26 lr 0.000759 wd 0.0500 time 0.3434 (0.3587) data time 0.0007 (0.0017) model time 0.3427 (0.3578) loss 3.4463 (3.0909) grad_norm 1.2521 (1.8265) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][560/625] eta 0:00:23 lr 0.000758 wd 0.0500 time 0.3566 (0.3585) data time 0.0009 (0.0016) model time 0.3557 (0.3576) loss 2.4903 (3.0907) grad_norm 1.2446 (1.8187) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][570/625] eta 0:00:19 lr 0.000758 wd 0.0500 time 0.3540 (0.3584) data time 0.0009 (0.0016) model time 0.3532 (0.3575) loss 2.4625 (3.0913) grad_norm 1.1688 (1.8108) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][580/625] eta 0:00:16 lr 0.000758 wd 0.0500 time 0.3493 (0.3582) data time 0.0008 (0.0016) model time 0.3485 (0.3574) loss 2.3504 (3.0874) grad_norm 1.2665 (1.8229) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][590/625] eta 0:00:12 lr 0.000758 wd 0.0500 time 0.3521 (0.3581) data time 0.0010 (0.0016) model time 0.3511 (0.3572) loss 2.7895 (3.0862) grad_norm 2.1084 (1.8300) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:32:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][600/625] eta 0:00:08 lr 0.000758 wd 0.0500 time 0.3480 (0.3579) data time 0.0010 (0.0016) model time 0.3470 (0.3570) loss 2.5214 (3.0826) grad_norm 1.8885 (1.8310) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:33:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][610/625] eta 0:00:05 lr 0.000758 wd 0.0500 time 0.3414 (0.3578) data time 0.0008 (0.0016) model time 0.3406 (0.3568) loss 3.4045 (3.0880) grad_norm 1.4492 (1.8295) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:33:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [182/300][620/625] eta 0:00:01 lr 0.000757 wd 0.0500 time 0.3439 (0.3575) data time 0.0004 (0.0016) model time 0.3436 (0.3565) loss 2.9727 (3.0881) grad_norm 1.5352 (1.8219) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:33:08 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 182 training takes 0:03:43 [2024-09-01 07:33:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:33:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:33:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 2.739 (2.739) Loss 0.5547 (0.5547) Acc@1 88.379 (88.379) Acc@5 98.145 (98.145) Mem 11224MB [2024-09-01 07:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.344) Loss 0.9180 (0.6871) Acc@1 77.051 (84.331) Acc@5 94.531 (97.093) Mem 11224MB [2024-09-01 07:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.224) Loss 0.9863 (0.8159) Acc@1 76.660 (81.287) Acc@5 93.994 (95.701) Mem 11224MB [2024-09-01 07:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 80.998 Acc@5 95.697 [2024-09-01 07:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.0% [2024-09-01 07:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 2.771 (2.771) Loss 0.5044 (0.5044) Acc@1 89.600 (89.600) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 07:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.344) Loss 0.8105 (0.6186) Acc@1 80.811 (86.235) Acc@5 96.094 (97.696) Mem 11224MB [2024-09-01 07:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.224) Loss 0.9243 (0.7343) Acc@1 77.246 (83.231) Acc@5 95.020 (96.540) Mem 11224MB [2024-09-01 07:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.937 Acc@5 96.551 [2024-09-01 07:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 07:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.94% [2024-09-01 07:33:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:33:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:33:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][0/625] eta 0:14:09 lr 0.000757 wd 0.0500 time 1.3590 (1.3590) data time 1.0296 (1.0296) model time 0.0000 (0.0000) loss 3.3155 (3.3155) grad_norm 1.7550 (1.7550) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:33:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][10/625] eta 0:04:30 lr 0.000757 wd 0.0500 time 0.3505 (0.4399) data time 0.0008 (0.0944) model time 0.0000 (0.0000) loss 2.9612 (2.9395) grad_norm 1.1444 (1.5707) loss_scale 2048.0000 (1768.7273) mem 11224MB [2024-09-01 07:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][20/625] eta 0:04:00 lr 0.000757 wd 0.0500 time 0.3487 (0.3975) data time 0.0006 (0.0498) model time 0.0000 (0.0000) loss 3.0010 (2.9249) grad_norm 1.6254 (1.5994) loss_scale 2048.0000 (1901.7143) mem 11224MB [2024-09-01 07:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][30/625] eta 0:04:00 lr 0.000757 wd 0.0500 time 0.3474 (0.4047) data time 0.0006 (0.0340) model time 0.0000 (0.0000) loss 3.2287 (3.0266) grad_norm 1.7658 (1.6231) loss_scale 2048.0000 (1948.9032) mem 11224MB [2024-09-01 07:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][40/625] eta 0:03:49 lr 0.000757 wd 0.0500 time 0.3519 (0.3915) data time 0.0007 (0.0259) model time 0.0000 (0.0000) loss 3.6227 (3.0360) grad_norm 1.6053 (1.6207) loss_scale 2048.0000 (1973.0732) mem 11224MB [2024-09-01 07:33:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][50/625] eta 0:03:42 lr 0.000756 wd 0.0500 time 0.3475 (0.3872) data time 0.0007 (0.0211) model time 0.0000 (0.0000) loss 3.6521 (3.0018) grad_norm 1.4219 (1.5883) loss_scale 2048.0000 (1987.7647) mem 11224MB [2024-09-01 07:33:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][60/625] eta 0:03:36 lr 0.000756 wd 0.0500 time 0.5325 (0.3841) data time 0.0008 (0.0177) model time 0.5317 (0.3671) loss 3.3691 (3.0416) grad_norm 1.9484 (1.6582) loss_scale 2048.0000 (1997.6393) mem 11224MB [2024-09-01 07:33:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][70/625] eta 0:03:31 lr 0.000756 wd 0.0500 time 0.3433 (0.3819) data time 0.0007 (0.0154) model time 0.3426 (0.3673) loss 3.2322 (3.0425) grad_norm 2.0592 (1.7597) loss_scale 2048.0000 (2004.7324) mem 11224MB [2024-09-01 07:33:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][80/625] eta 0:03:25 lr 0.000756 wd 0.0500 time 0.3474 (0.3779) data time 0.0009 (0.0136) model time 0.3465 (0.3611) loss 3.2548 (3.0291) grad_norm 1.7758 (1.7351) loss_scale 2048.0000 (2010.0741) mem 11224MB [2024-09-01 07:33:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][90/625] eta 0:03:20 lr 0.000756 wd 0.0500 time 0.3441 (0.3749) data time 0.0008 (0.0122) model time 0.3434 (0.3582) loss 3.2680 (3.0209) grad_norm 1.9763 (1.7591) loss_scale 2048.0000 (2014.2418) mem 11224MB [2024-09-01 07:33:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][100/625] eta 0:03:15 lr 0.000756 wd 0.0500 time 0.3454 (0.3720) data time 0.0008 (0.0111) model time 0.3446 (0.3556) loss 2.9228 (3.0385) grad_norm 1.8330 (1.7996) loss_scale 2048.0000 (2017.5842) mem 11224MB [2024-09-01 07:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][110/625] eta 0:03:10 lr 0.000755 wd 0.0500 time 0.3599 (0.3700) data time 0.0009 (0.0102) model time 0.3590 (0.3545) loss 3.0430 (3.0461) grad_norm 2.0462 (1.8262) loss_scale 2048.0000 (2020.3243) mem 11224MB [2024-09-01 07:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][120/625] eta 0:03:05 lr 0.000755 wd 0.0500 time 0.3479 (0.3681) data time 0.0009 (0.0094) model time 0.3470 (0.3533) loss 2.0655 (3.0299) grad_norm 1.5168 (1.8089) loss_scale 2048.0000 (2022.6116) mem 11224MB [2024-09-01 07:34:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][130/625] eta 0:03:01 lr 0.000755 wd 0.0500 time 0.3467 (0.3667) data time 0.0007 (0.0087) model time 0.3461 (0.3527) loss 3.5776 (3.0275) grad_norm 1.6862 (1.8127) loss_scale 2048.0000 (2024.5496) mem 11224MB [2024-09-01 07:34:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][140/625] eta 0:02:57 lr 0.000755 wd 0.0500 time 0.3548 (0.3655) data time 0.0009 (0.0082) model time 0.3538 (0.3523) loss 2.7962 (3.0379) grad_norm 1.6600 (1.8238) loss_scale 2048.0000 (2026.2128) mem 11224MB [2024-09-01 07:34:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][150/625] eta 0:02:53 lr 0.000755 wd 0.0500 time 0.3487 (0.3654) data time 0.0009 (0.0077) model time 0.3478 (0.3534) loss 3.2546 (3.0437) grad_norm 1.6503 (1.8163) loss_scale 2048.0000 (2027.6556) mem 11224MB [2024-09-01 07:34:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][160/625] eta 0:02:49 lr 0.000755 wd 0.0500 time 0.3398 (0.3643) data time 0.0006 (0.0073) model time 0.3391 (0.3529) loss 2.8168 (3.0348) grad_norm 1.3557 (1.8120) loss_scale 2048.0000 (2028.9193) mem 11224MB [2024-09-01 07:34:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][170/625] eta 0:02:45 lr 0.000754 wd 0.0500 time 0.3492 (0.3634) data time 0.0006 (0.0069) model time 0.3486 (0.3524) loss 3.5883 (3.0324) grad_norm 1.6013 (1.8363) loss_scale 2048.0000 (2030.0351) mem 11224MB [2024-09-01 07:34:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][180/625] eta 0:02:41 lr 0.000754 wd 0.0500 time 0.3536 (0.3626) data time 0.0008 (0.0066) model time 0.3528 (0.3520) loss 3.2484 (3.0386) grad_norm 1.2530 (1.8288) loss_scale 2048.0000 (2031.0276) mem 11224MB [2024-09-01 07:34:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][190/625] eta 0:02:37 lr 0.000754 wd 0.0500 time 0.3414 (0.3619) data time 0.0007 (0.0063) model time 0.3406 (0.3517) loss 3.0303 (3.0412) grad_norm 1.5205 (1.8133) loss_scale 2048.0000 (2031.9162) mem 11224MB [2024-09-01 07:34:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][200/625] eta 0:02:33 lr 0.000754 wd 0.0500 time 0.3662 (0.3614) data time 0.0007 (0.0060) model time 0.3655 (0.3517) loss 4.0188 (3.0501) grad_norm 2.1938 (1.8206) loss_scale 2048.0000 (2032.7164) mem 11224MB [2024-09-01 07:34:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][210/625] eta 0:02:29 lr 0.000754 wd 0.0500 time 0.3437 (0.3608) data time 0.0009 (0.0058) model time 0.3428 (0.3515) loss 2.6080 (3.0370) grad_norm 1.7121 (1.8227) loss_scale 2048.0000 (2033.4408) mem 11224MB [2024-09-01 07:34:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][220/625] eta 0:02:25 lr 0.000754 wd 0.0500 time 0.3505 (0.3603) data time 0.0006 (0.0056) model time 0.3499 (0.3513) loss 2.3137 (3.0385) grad_norm 1.7253 (1.8118) loss_scale 2048.0000 (2034.0995) mem 11224MB [2024-09-01 07:34:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][230/625] eta 0:02:22 lr 0.000753 wd 0.0500 time 0.3517 (0.3598) data time 0.0010 (0.0054) model time 0.3507 (0.3511) loss 2.0417 (3.0360) grad_norm 0.9455 (1.8097) loss_scale 2048.0000 (2034.7013) mem 11224MB [2024-09-01 07:34:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][240/625] eta 0:02:19 lr 0.000753 wd 0.0500 time 0.5624 (0.3628) data time 0.0006 (0.0052) model time 0.5618 (0.3554) loss 3.0952 (3.0380) grad_norm 1.8737 (1.8200) loss_scale 2048.0000 (2035.2531) mem 11224MB [2024-09-01 07:34:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][250/625] eta 0:02:16 lr 0.000753 wd 0.0500 time 0.3415 (0.3637) data time 0.0007 (0.0050) model time 0.3409 (0.3568) loss 3.5743 (3.0330) grad_norm 2.1442 (1.8144) loss_scale 2048.0000 (2035.7610) mem 11224MB [2024-09-01 07:34:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][260/625] eta 0:02:12 lr 0.000753 wd 0.0500 time 0.3455 (0.3632) data time 0.0008 (0.0049) model time 0.3447 (0.3565) loss 3.3631 (3.0249) grad_norm 2.9094 (1.8389) loss_scale 2048.0000 (2036.2299) mem 11224MB [2024-09-01 07:34:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][270/625] eta 0:02:08 lr 0.000753 wd 0.0500 time 0.3536 (0.3626) data time 0.0006 (0.0047) model time 0.3530 (0.3560) loss 3.6230 (3.0303) grad_norm 1.9431 (1.8558) loss_scale 2048.0000 (2036.6642) mem 11224MB [2024-09-01 07:35:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][280/625] eta 0:02:04 lr 0.000753 wd 0.0500 time 0.3468 (0.3622) data time 0.0006 (0.0046) model time 0.3462 (0.3557) loss 3.7585 (3.0310) grad_norm 1.6544 (1.8421) loss_scale 2048.0000 (2037.0676) mem 11224MB [2024-09-01 07:35:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][290/625] eta 0:02:01 lr 0.000752 wd 0.0500 time 0.3536 (0.3617) data time 0.0007 (0.0044) model time 0.3529 (0.3554) loss 3.6144 (3.0339) grad_norm 1.5111 (1.8435) loss_scale 2048.0000 (2037.4433) mem 11224MB [2024-09-01 07:35:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][300/625] eta 0:01:57 lr 0.000752 wd 0.0500 time 0.3459 (0.3613) data time 0.0008 (0.0043) model time 0.3451 (0.3551) loss 3.3223 (3.0298) grad_norm 2.3998 (1.8367) loss_scale 2048.0000 (2037.7940) mem 11224MB [2024-09-01 07:35:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][310/625] eta 0:01:53 lr 0.000752 wd 0.0500 time 0.3490 (0.3610) data time 0.0006 (0.0042) model time 0.3484 (0.3550) loss 2.8023 (3.0382) grad_norm 2.1885 (1.8654) loss_scale 2048.0000 (2038.1222) mem 11224MB [2024-09-01 07:35:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][320/625] eta 0:01:50 lr 0.000752 wd 0.0500 time 0.3430 (0.3607) data time 0.0007 (0.0041) model time 0.3424 (0.3548) loss 2.6813 (3.0364) grad_norm 1.2882 (1.8641) loss_scale 2048.0000 (2038.4299) mem 11224MB [2024-09-01 07:35:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][330/625] eta 0:01:46 lr 0.000752 wd 0.0500 time 0.3528 (0.3604) data time 0.0008 (0.0040) model time 0.3520 (0.3546) loss 3.3007 (3.0394) grad_norm 1.4729 (1.8665) loss_scale 2048.0000 (2038.7190) mem 11224MB [2024-09-01 07:35:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][340/625] eta 0:01:42 lr 0.000752 wd 0.0500 time 0.3393 (0.3600) data time 0.0009 (0.0039) model time 0.3385 (0.3543) loss 3.2026 (3.0443) grad_norm 1.7503 (1.8675) loss_scale 2048.0000 (2038.9912) mem 11224MB [2024-09-01 07:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][350/625] eta 0:01:39 lr 0.000751 wd 0.0500 time 0.5349 (0.3602) data time 0.0007 (0.0038) model time 0.5342 (0.3547) loss 2.6598 (3.0449) grad_norm 1.3664 (nan) loss_scale 1024.0000 (2018.8262) mem 11224MB [2024-09-01 07:35:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][360/625] eta 0:01:35 lr 0.000751 wd 0.0500 time 0.3476 (0.3604) data time 0.0008 (0.0037) model time 0.3467 (0.3551) loss 2.4201 (3.0413) grad_norm 3.7975 (nan) loss_scale 1024.0000 (1991.2687) mem 11224MB [2024-09-01 07:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][370/625] eta 0:01:32 lr 0.000751 wd 0.0500 time 0.3515 (0.3611) data time 0.0008 (0.0037) model time 0.3507 (0.3560) loss 3.3433 (3.0443) grad_norm 2.4874 (nan) loss_scale 1024.0000 (1965.1968) mem 11224MB [2024-09-01 07:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][380/625] eta 0:01:28 lr 0.000751 wd 0.0500 time 0.3472 (0.3608) data time 0.0009 (0.0036) model time 0.3464 (0.3558) loss 3.0673 (3.0448) grad_norm 1.2975 (nan) loss_scale 1024.0000 (1940.4934) mem 11224MB [2024-09-01 07:35:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][390/625] eta 0:01:24 lr 0.000751 wd 0.0500 time 0.3559 (0.3606) data time 0.0010 (0.0035) model time 0.3549 (0.3557) loss 3.4372 (3.0445) grad_norm 2.1384 (nan) loss_scale 1024.0000 (1917.0537) mem 11224MB [2024-09-01 07:35:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][400/625] eta 0:01:21 lr 0.000750 wd 0.0500 time 0.3422 (0.3602) data time 0.0009 (0.0035) model time 0.3413 (0.3554) loss 2.8256 (3.0436) grad_norm 1.5095 (nan) loss_scale 1024.0000 (1894.7830) mem 11224MB [2024-09-01 07:35:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][410/625] eta 0:01:17 lr 0.000750 wd 0.0500 time 0.3516 (0.3600) data time 0.0008 (0.0034) model time 0.3508 (0.3552) loss 3.1831 (3.0441) grad_norm 1.3931 (nan) loss_scale 1024.0000 (1873.5961) mem 11224MB [2024-09-01 07:35:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][420/625] eta 0:01:14 lr 0.000750 wd 0.0500 time 0.5555 (0.3615) data time 0.0009 (0.0033) model time 0.5546 (0.3571) loss 3.5111 (3.0475) grad_norm 1.8317 (nan) loss_scale 1024.0000 (1853.4157) mem 11224MB [2024-09-01 07:35:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][430/625] eta 0:01:10 lr 0.000750 wd 0.0500 time 0.3502 (0.3616) data time 0.0006 (0.0033) model time 0.3496 (0.3573) loss 2.1056 (3.0442) grad_norm 1.4628 (nan) loss_scale 1024.0000 (1834.1717) mem 11224MB [2024-09-01 07:35:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][440/625] eta 0:01:06 lr 0.000750 wd 0.0500 time 0.3471 (0.3613) data time 0.0008 (0.0032) model time 0.3463 (0.3571) loss 3.4645 (3.0482) grad_norm 2.0994 (nan) loss_scale 1024.0000 (1815.8005) mem 11224MB [2024-09-01 07:36:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][450/625] eta 0:01:03 lr 0.000750 wd 0.0500 time 0.3501 (0.3610) data time 0.0007 (0.0032) model time 0.3495 (0.3568) loss 2.3201 (3.0449) grad_norm 1.4621 (nan) loss_scale 1024.0000 (1798.2439) mem 11224MB [2024-09-01 07:36:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][460/625] eta 0:00:59 lr 0.000749 wd 0.0500 time 0.3419 (0.3607) data time 0.0010 (0.0031) model time 0.3410 (0.3565) loss 3.2222 (3.0468) grad_norm 1.5475 (nan) loss_scale 1024.0000 (1781.4490) mem 11224MB [2024-09-01 07:36:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][470/625] eta 0:00:55 lr 0.000749 wd 0.0500 time 0.3487 (0.3605) data time 0.0007 (0.0031) model time 0.3480 (0.3563) loss 3.2288 (3.0507) grad_norm 2.0708 (nan) loss_scale 1024.0000 (1765.3673) mem 11224MB [2024-09-01 07:36:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][480/625] eta 0:00:52 lr 0.000749 wd 0.0500 time 0.3447 (0.3602) data time 0.0008 (0.0030) model time 0.3438 (0.3561) loss 3.6681 (3.0508) grad_norm 1.4120 (nan) loss_scale 1024.0000 (1749.9543) mem 11224MB [2024-09-01 07:36:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][490/625] eta 0:00:48 lr 0.000749 wd 0.0500 time 0.3560 (0.3600) data time 0.0008 (0.0030) model time 0.3553 (0.3560) loss 2.2700 (3.0531) grad_norm 1.7692 (nan) loss_scale 1024.0000 (1735.1690) mem 11224MB [2024-09-01 07:36:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][500/625] eta 0:00:44 lr 0.000749 wd 0.0500 time 0.3495 (0.3598) data time 0.0009 (0.0029) model time 0.3486 (0.3558) loss 2.1769 (3.0470) grad_norm 1.6526 (nan) loss_scale 1024.0000 (1720.9741) mem 11224MB [2024-09-01 07:36:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][510/625] eta 0:00:41 lr 0.000749 wd 0.0500 time 0.3462 (0.3596) data time 0.0006 (0.0029) model time 0.3456 (0.3556) loss 1.9691 (3.0463) grad_norm 1.3466 (nan) loss_scale 512.0000 (1702.3249) mem 11224MB [2024-09-01 07:36:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][520/625] eta 0:00:37 lr 0.000748 wd 0.0500 time 0.3475 (0.3594) data time 0.0008 (0.0029) model time 0.3467 (0.3555) loss 3.6144 (3.0416) grad_norm 3.2718 (nan) loss_scale 512.0000 (1679.4779) mem 11224MB [2024-09-01 07:36:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][530/625] eta 0:00:34 lr 0.000748 wd 0.0500 time 0.3457 (0.3592) data time 0.0009 (0.0028) model time 0.3448 (0.3554) loss 3.2670 (3.0414) grad_norm 2.3094 (nan) loss_scale 512.0000 (1657.4915) mem 11224MB [2024-09-01 07:36:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][540/625] eta 0:00:30 lr 0.000748 wd 0.0500 time 0.3522 (0.3604) data time 0.0009 (0.0028) model time 0.3513 (0.3568) loss 2.0338 (3.0376) grad_norm 1.1533 (nan) loss_scale 512.0000 (1636.3179) mem 11224MB [2024-09-01 07:36:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][550/625] eta 0:00:27 lr 0.000748 wd 0.0500 time 0.3521 (0.3606) data time 0.0009 (0.0028) model time 0.3512 (0.3570) loss 3.5843 (3.0427) grad_norm 1.4168 (nan) loss_scale 512.0000 (1615.9129) mem 11224MB [2024-09-01 07:36:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][560/625] eta 0:00:23 lr 0.000748 wd 0.0500 time 0.3535 (0.3607) data time 0.0009 (0.0027) model time 0.3526 (0.3572) loss 3.1730 (3.0440) grad_norm 2.4445 (nan) loss_scale 512.0000 (1596.2353) mem 11224MB [2024-09-01 07:36:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][570/625] eta 0:00:19 lr 0.000748 wd 0.0500 time 0.3621 (0.3606) data time 0.0010 (0.0027) model time 0.3611 (0.3571) loss 2.1438 (3.0464) grad_norm 1.5129 (nan) loss_scale 512.0000 (1577.2469) mem 11224MB [2024-09-01 07:36:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][580/625] eta 0:00:16 lr 0.000747 wd 0.0500 time 0.3558 (0.3610) data time 0.0007 (0.0027) model time 0.3551 (0.3576) loss 2.9433 (3.0480) grad_norm 1.7861 (nan) loss_scale 512.0000 (1558.9122) mem 11224MB [2024-09-01 07:36:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][590/625] eta 0:00:12 lr 0.000747 wd 0.0500 time 0.3464 (0.3608) data time 0.0006 (0.0026) model time 0.3458 (0.3574) loss 3.2104 (3.0411) grad_norm 1.5329 (nan) loss_scale 512.0000 (1541.1980) mem 11224MB [2024-09-01 07:37:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][600/625] eta 0:00:09 lr 0.000747 wd 0.0500 time 1.5356 (0.3697) data time 0.0006 (0.0026) model time 1.5350 (0.3672) loss 3.2222 (3.0395) grad_norm 1.8831 (nan) loss_scale 512.0000 (1524.0732) mem 11224MB [2024-09-01 07:37:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][610/625] eta 0:00:05 lr 0.000747 wd 0.0500 time 0.3391 (0.3730) data time 0.0005 (0.0035) model time 0.3386 (0.3698) loss 3.6227 (3.0417) grad_norm 1.5029 (nan) loss_scale 512.0000 (1507.5090) mem 11224MB [2024-09-01 07:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [183/300][620/625] eta 0:00:01 lr 0.000747 wd 0.0500 time 0.3372 (0.3724) data time 0.0006 (0.0034) model time 0.3366 (0.3693) loss 3.3014 (3.0396) grad_norm 1.3741 (nan) loss_scale 512.0000 (1491.4783) mem 11224MB [2024-09-01 07:37:12 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 183 training takes 0:03:52 [2024-09-01 07:37:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:37:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:37:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.460 (0.460) Loss 0.5767 (0.5767) Acc@1 87.012 (87.012) Acc@5 98.193 (98.193) Mem 11224MB [2024-09-01 07:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.134) Loss 0.9233 (0.6915) Acc@1 78.467 (84.610) Acc@5 94.482 (97.164) Mem 11224MB [2024-09-01 07:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.114) Loss 1.0293 (0.8153) Acc@1 75.244 (81.473) Acc@5 93.750 (95.729) Mem 11224MB [2024-09-01 07:37:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.200 Acc@5 95.711 [2024-09-01 07:37:16 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.2% [2024-09-01 07:37:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 6.573 (6.573) Loss 0.5044 (0.5044) Acc@1 89.551 (89.551) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 07:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.807) Loss 0.8086 (0.6174) Acc@1 80.859 (86.275) Acc@5 96.240 (97.723) Mem 11224MB [2024-09-01 07:37:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.466) Loss 0.9229 (0.7330) Acc@1 77.197 (83.266) Acc@5 94.922 (96.561) Mem 11224MB [2024-09-01 07:37:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.967 Acc@5 96.567 [2024-09-01 07:37:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.0% [2024-09-01 07:37:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 82.97% [2024-09-01 07:37:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:37:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][0/625] eta 0:10:25 lr 0.000747 wd 0.0500 time 1.0004 (1.0004) data time 0.6909 (0.6909) model time 0.0000 (0.0000) loss 3.5006 (3.5006) grad_norm 1.6160 (1.6160) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:37:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][10/625] eta 0:04:18 lr 0.000746 wd 0.0500 time 0.3451 (0.4199) data time 0.0010 (0.0637) model time 0.0000 (0.0000) loss 3.3867 (3.1327) grad_norm 1.6365 (1.6228) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][20/625] eta 0:03:53 lr 0.000746 wd 0.0500 time 0.3510 (0.3861) data time 0.0006 (0.0338) model time 0.0000 (0.0000) loss 2.3679 (3.0375) grad_norm 1.6279 (1.7179) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][30/625] eta 0:03:42 lr 0.000746 wd 0.0500 time 0.3518 (0.3742) data time 0.0010 (0.0232) model time 0.0000 (0.0000) loss 1.8145 (2.9736) grad_norm 1.6833 (1.7070) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][40/625] eta 0:03:35 lr 0.000746 wd 0.0500 time 0.3466 (0.3681) data time 0.0009 (0.0177) model time 0.0000 (0.0000) loss 3.1609 (2.9245) grad_norm 1.5043 (1.8633) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:37:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][50/625] eta 0:03:29 lr 0.000746 wd 0.0500 time 0.3483 (0.3641) data time 0.0009 (0.0144) model time 0.0000 (0.0000) loss 3.1827 (2.9483) grad_norm 1.2554 (1.8701) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:37:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][60/625] eta 0:03:24 lr 0.000746 wd 0.0500 time 0.3535 (0.3619) data time 0.0008 (0.0122) model time 0.3527 (0.3498) loss 2.7077 (3.0024) grad_norm 1.3817 (1.8137) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][70/625] eta 0:03:25 lr 0.000745 wd 0.0500 time 0.5546 (0.3697) data time 0.0008 (0.0106) model time 0.5538 (0.3829) loss 3.4678 (3.0159) grad_norm 1.4760 (1.7580) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:37:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][80/625] eta 0:03:21 lr 0.000745 wd 0.0500 time 0.3551 (0.3698) data time 0.0009 (0.0094) model time 0.3542 (0.3787) loss 3.4700 (3.0228) grad_norm 1.3081 (1.7576) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][90/625] eta 0:03:16 lr 0.000745 wd 0.0500 time 0.3867 (0.3678) data time 0.0006 (0.0085) model time 0.3861 (0.3716) loss 2.1204 (3.0323) grad_norm 1.4158 (1.7706) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][100/625] eta 0:03:12 lr 0.000745 wd 0.0500 time 0.3398 (0.3660) data time 0.0009 (0.0077) model time 0.3390 (0.3670) loss 3.1926 (3.0365) grad_norm 1.1791 (1.7393) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][110/625] eta 0:03:07 lr 0.000745 wd 0.0500 time 0.3376 (0.3644) data time 0.0009 (0.0071) model time 0.3367 (0.3638) loss 2.2805 (3.0410) grad_norm 1.6778 (1.7209) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][120/625] eta 0:03:03 lr 0.000745 wd 0.0500 time 0.3510 (0.3631) data time 0.0007 (0.0066) model time 0.3502 (0.3614) loss 3.9985 (3.0492) grad_norm 1.3668 (1.7165) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][130/625] eta 0:02:59 lr 0.000744 wd 0.0500 time 0.3369 (0.3619) data time 0.0009 (0.0061) model time 0.3360 (0.3596) loss 3.1307 (3.0527) grad_norm 1.6671 (1.7939) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][140/625] eta 0:02:55 lr 0.000744 wd 0.0500 time 0.3387 (0.3622) data time 0.0008 (0.0058) model time 0.3379 (0.3602) loss 2.2385 (3.0591) grad_norm 1.4825 (1.7725) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][150/625] eta 0:02:51 lr 0.000744 wd 0.0500 time 0.3434 (0.3614) data time 0.0007 (0.0054) model time 0.3426 (0.3591) loss 2.6337 (3.0549) grad_norm 1.7893 (1.7533) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][160/625] eta 0:02:48 lr 0.000744 wd 0.0500 time 0.3465 (0.3623) data time 0.0008 (0.0051) model time 0.3457 (0.3606) loss 2.9359 (3.0519) grad_norm 3.1153 (1.7825) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][170/625] eta 0:02:44 lr 0.000744 wd 0.0500 time 0.3539 (0.3616) data time 0.0006 (0.0049) model time 0.3533 (0.3596) loss 3.8100 (3.0469) grad_norm 1.8286 (1.7816) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][180/625] eta 0:02:40 lr 0.000744 wd 0.0500 time 0.3480 (0.3611) data time 0.0006 (0.0047) model time 0.3474 (0.3591) loss 2.8496 (3.0466) grad_norm 2.1539 (1.7877) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][190/625] eta 0:02:36 lr 0.000743 wd 0.0500 time 0.3544 (0.3605) data time 0.0008 (0.0045) model time 0.3537 (0.3583) loss 3.8311 (3.0618) grad_norm 1.9981 (1.8085) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][200/625] eta 0:02:33 lr 0.000743 wd 0.0500 time 0.3648 (0.3601) data time 0.0008 (0.0043) model time 0.3640 (0.3578) loss 2.1640 (3.0593) grad_norm 2.3341 (1.8072) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][210/625] eta 0:02:29 lr 0.000743 wd 0.0500 time 0.3491 (0.3595) data time 0.0010 (0.0041) model time 0.3481 (0.3572) loss 3.4951 (3.0619) grad_norm 2.0298 (1.8125) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][220/625] eta 0:02:25 lr 0.000743 wd 0.0500 time 0.3459 (0.3600) data time 0.0010 (0.0040) model time 0.3449 (0.3579) loss 2.9561 (3.0532) grad_norm 2.7251 (1.8066) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][230/625] eta 0:02:22 lr 0.000743 wd 0.0500 time 0.3562 (0.3595) data time 0.0007 (0.0039) model time 0.3555 (0.3573) loss 1.8366 (3.0495) grad_norm 1.5597 (1.7988) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][240/625] eta 0:02:18 lr 0.000743 wd 0.0500 time 0.3603 (0.3591) data time 0.0009 (0.0038) model time 0.3595 (0.3569) loss 3.4250 (3.0613) grad_norm 1.6121 (1.7923) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][250/625] eta 0:02:14 lr 0.000742 wd 0.0500 time 0.3364 (0.3587) data time 0.0008 (0.0037) model time 0.3356 (0.3564) loss 2.4915 (3.0560) grad_norm 1.3895 (1.7817) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][260/625] eta 0:02:10 lr 0.000742 wd 0.0500 time 0.3507 (0.3582) data time 0.0008 (0.0036) model time 0.3499 (0.3559) loss 3.2735 (3.0574) grad_norm 4.2029 (1.8452) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][270/625] eta 0:02:07 lr 0.000742 wd 0.0500 time 0.3548 (0.3579) data time 0.0009 (0.0035) model time 0.3539 (0.3555) loss 3.2229 (3.0561) grad_norm 1.5607 (1.8569) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][280/625] eta 0:02:03 lr 0.000742 wd 0.0500 time 0.3469 (0.3589) data time 0.0010 (0.0034) model time 0.3459 (0.3568) loss 3.1373 (3.0595) grad_norm 2.3418 (1.8581) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][290/625] eta 0:02:00 lr 0.000742 wd 0.0500 time 0.3562 (0.3600) data time 0.0010 (0.0033) model time 0.3551 (0.3582) loss 3.1042 (3.0594) grad_norm 1.6540 (1.8564) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][300/625] eta 0:01:56 lr 0.000741 wd 0.0500 time 0.3457 (0.3595) data time 0.0009 (0.0032) model time 0.3448 (0.3577) loss 3.0934 (3.0576) grad_norm 1.6270 (1.8493) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][310/625] eta 0:01:53 lr 0.000741 wd 0.0500 time 0.3512 (0.3593) data time 0.0007 (0.0031) model time 0.3505 (0.3574) loss 3.5714 (3.0601) grad_norm 2.2129 (1.8571) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][320/625] eta 0:01:49 lr 0.000741 wd 0.0500 time 0.3639 (0.3596) data time 0.0008 (0.0031) model time 0.3631 (0.3578) loss 3.0122 (3.0612) grad_norm 1.4977 (1.8587) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][330/625] eta 0:01:46 lr 0.000741 wd 0.0500 time 0.3510 (0.3593) data time 0.0008 (0.0030) model time 0.3503 (0.3576) loss 2.6963 (3.0625) grad_norm 2.2052 (1.8619) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][340/625] eta 0:01:42 lr 0.000741 wd 0.0500 time 0.4591 (0.3598) data time 0.0008 (0.0029) model time 0.4583 (0.3581) loss 3.3149 (3.0606) grad_norm 1.2169 (1.8556) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][350/625] eta 0:01:38 lr 0.000741 wd 0.0500 time 0.3541 (0.3595) data time 0.0009 (0.0029) model time 0.3533 (0.3578) loss 3.0050 (3.0552) grad_norm 1.1614 (1.8436) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][360/625] eta 0:01:35 lr 0.000740 wd 0.0500 time 0.3467 (0.3592) data time 0.0006 (0.0028) model time 0.3461 (0.3575) loss 3.8853 (3.0621) grad_norm 1.6970 (1.8481) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][370/625] eta 0:01:31 lr 0.000740 wd 0.0500 time 0.3500 (0.3589) data time 0.0010 (0.0028) model time 0.3490 (0.3572) loss 3.1766 (3.0617) grad_norm 1.4399 (1.8458) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][380/625] eta 0:01:27 lr 0.000740 wd 0.0500 time 0.3442 (0.3586) data time 0.0007 (0.0027) model time 0.3435 (0.3569) loss 3.7801 (3.0573) grad_norm 2.6221 (1.8398) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][390/625] eta 0:01:24 lr 0.000740 wd 0.0500 time 0.3451 (0.3584) data time 0.0008 (0.0027) model time 0.3442 (0.3566) loss 3.0800 (3.0629) grad_norm 1.9018 (1.8378) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][400/625] eta 0:01:20 lr 0.000740 wd 0.0500 time 0.3465 (0.3586) data time 0.0007 (0.0026) model time 0.3457 (0.3568) loss 2.3858 (3.0658) grad_norm 1.8483 (1.8299) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][410/625] eta 0:01:17 lr 0.000740 wd 0.0500 time 0.3507 (0.3583) data time 0.0010 (0.0026) model time 0.3497 (0.3566) loss 4.0428 (3.0724) grad_norm 2.5140 (1.8381) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:39:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][420/625] eta 0:01:13 lr 0.000739 wd 0.0500 time 0.3635 (0.3581) data time 0.0007 (0.0025) model time 0.3628 (0.3564) loss 3.4157 (3.0668) grad_norm 1.4038 (1.8402) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][430/625] eta 0:01:09 lr 0.000739 wd 0.0500 time 0.3536 (0.3579) data time 0.0009 (0.0025) model time 0.3527 (0.3562) loss 2.4850 (3.0679) grad_norm 1.1527 (1.8324) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][440/625] eta 0:01:06 lr 0.000739 wd 0.0500 time 0.3609 (0.3583) data time 0.0006 (0.0025) model time 0.3603 (0.3566) loss 2.9882 (3.0686) grad_norm 2.1136 (1.8286) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][450/625] eta 0:01:02 lr 0.000739 wd 0.0500 time 0.3479 (0.3580) data time 0.0008 (0.0024) model time 0.3470 (0.3564) loss 2.2304 (3.0720) grad_norm 1.7953 (1.8226) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][460/625] eta 0:00:59 lr 0.000739 wd 0.0500 time 0.5111 (0.3586) data time 0.0006 (0.0024) model time 0.5105 (0.3570) loss 3.3442 (3.0757) grad_norm 1.3291 (1.8190) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][470/625] eta 0:00:55 lr 0.000739 wd 0.0500 time 0.3395 (0.3603) data time 0.0009 (0.0024) model time 0.3386 (0.3590) loss 3.2605 (3.0730) grad_norm 1.2325 (1.8159) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][480/625] eta 0:00:52 lr 0.000738 wd 0.0500 time 0.3467 (0.3601) data time 0.0006 (0.0023) model time 0.3461 (0.3587) loss 3.4366 (3.0728) grad_norm 1.6122 (1.8084) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][490/625] eta 0:00:48 lr 0.000738 wd 0.0500 time 0.3411 (0.3599) data time 0.0009 (0.0023) model time 0.3402 (0.3585) loss 2.4168 (3.0704) grad_norm 1.2546 (1.7999) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][500/625] eta 0:00:44 lr 0.000738 wd 0.0500 time 0.3435 (0.3596) data time 0.0007 (0.0023) model time 0.3428 (0.3582) loss 2.5022 (3.0627) grad_norm 1.7030 (1.7980) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][510/625] eta 0:00:41 lr 0.000738 wd 0.0500 time 0.3498 (0.3594) data time 0.0006 (0.0022) model time 0.3491 (0.3580) loss 2.9341 (3.0636) grad_norm 1.6584 (1.8002) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][520/625] eta 0:00:37 lr 0.000738 wd 0.0500 time 0.3488 (0.3595) data time 0.0007 (0.0022) model time 0.3481 (0.3581) loss 3.6533 (3.0610) grad_norm 1.3999 (1.7969) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][530/625] eta 0:00:34 lr 0.000738 wd 0.0500 time 0.3541 (0.3596) data time 0.0009 (0.0022) model time 0.3532 (0.3582) loss 3.2241 (3.0605) grad_norm 1.8649 (1.7972) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][540/625] eta 0:00:30 lr 0.000737 wd 0.0500 time 0.3394 (0.3595) data time 0.0009 (0.0022) model time 0.3385 (0.3581) loss 3.4750 (3.0610) grad_norm 2.4139 (1.7924) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][550/625] eta 0:00:26 lr 0.000737 wd 0.0500 time 0.3776 (0.3595) data time 0.0009 (0.0021) model time 0.3766 (0.3581) loss 2.7093 (3.0578) grad_norm 1.1855 (1.7939) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][560/625] eta 0:00:23 lr 0.000737 wd 0.0500 time 0.3335 (0.3593) data time 0.0010 (0.0021) model time 0.3325 (0.3579) loss 3.3740 (3.0544) grad_norm 2.1218 (1.7994) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][570/625] eta 0:00:19 lr 0.000737 wd 0.0500 time 0.3457 (0.3591) data time 0.0007 (0.0021) model time 0.3450 (0.3577) loss 3.9070 (3.0600) grad_norm 1.3002 (1.7967) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:40:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][580/625] eta 0:00:16 lr 0.000737 wd 0.0500 time 0.5411 (0.3592) data time 0.0008 (0.0021) model time 0.5402 (0.3579) loss 2.8333 (3.0554) grad_norm 1.3319 (1.7948) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][590/625] eta 0:00:12 lr 0.000737 wd 0.0500 time 0.3512 (0.3601) data time 0.0008 (0.0020) model time 0.3503 (0.3588) loss 3.4195 (3.0581) grad_norm 1.5814 (1.7924) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][600/625] eta 0:00:08 lr 0.000736 wd 0.0500 time 0.3525 (0.3599) data time 0.0008 (0.0020) model time 0.3517 (0.3587) loss 3.5296 (3.0567) grad_norm 1.8396 (1.7899) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][610/625] eta 0:00:05 lr 0.000736 wd 0.0500 time 0.3403 (0.3598) data time 0.0007 (0.0020) model time 0.3396 (0.3585) loss 3.1632 (3.0574) grad_norm 1.8240 (1.7946) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [184/300][620/625] eta 0:00:01 lr 0.000736 wd 0.0500 time 0.3362 (0.3594) data time 0.0004 (0.0020) model time 0.3359 (0.3581) loss 3.4408 (3.0555) grad_norm 1.8038 (1.7961) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:11 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 184 training takes 0:03:44 [2024-09-01 07:41:11 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:41:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:41:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.457 (0.457) Loss 0.5420 (0.5420) Acc@1 88.281 (88.281) Acc@5 98.193 (98.193) Mem 11224MB [2024-09-01 07:41:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.132) Loss 0.8936 (0.6653) Acc@1 78.271 (84.859) Acc@5 95.117 (97.195) Mem 11224MB [2024-09-01 07:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.113) Loss 1.0078 (0.7988) Acc@1 75.488 (81.545) Acc@5 93.652 (95.791) Mem 11224MB [2024-09-01 07:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.282 Acc@5 95.791 [2024-09-01 07:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.3% [2024-09-01 07:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.28% [2024-09-01 07:41:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 07:41:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 07:41:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.442 (0.442) Loss 0.5063 (0.5063) Acc@1 89.502 (89.502) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 07:41:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.131) Loss 0.8076 (0.6170) Acc@1 81.055 (86.350) Acc@5 96.338 (97.714) Mem 11224MB [2024-09-01 07:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.9224 (0.7322) Acc@1 77.051 (83.329) Acc@5 94.922 (96.556) Mem 11224MB [2024-09-01 07:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.021 Acc@5 96.565 [2024-09-01 07:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.0% [2024-09-01 07:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.02% [2024-09-01 07:41:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:41:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:41:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][0/625] eta 0:07:24 lr 0.000736 wd 0.0500 time 0.7113 (0.7113) data time 0.3656 (0.3656) model time 0.0000 (0.0000) loss 3.1083 (3.1083) grad_norm 1.2606 (1.2606) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][10/625] eta 0:04:04 lr 0.000736 wd 0.0500 time 0.3478 (0.3978) data time 0.0009 (0.0339) model time 0.0000 (0.0000) loss 2.0477 (3.0354) grad_norm 1.9063 (1.9231) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][20/625] eta 0:03:50 lr 0.000736 wd 0.0500 time 0.3488 (0.3808) data time 0.0009 (0.0182) model time 0.0000 (0.0000) loss 3.2198 (3.1406) grad_norm 2.5113 (1.9255) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][30/625] eta 0:03:47 lr 0.000735 wd 0.0500 time 0.3546 (0.3817) data time 0.0009 (0.0126) model time 0.0000 (0.0000) loss 2.4950 (3.1039) grad_norm 1.8524 (1.9731) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][40/625] eta 0:03:38 lr 0.000735 wd 0.0500 time 0.3656 (0.3737) data time 0.0008 (0.0097) model time 0.0000 (0.0000) loss 3.4240 (3.0513) grad_norm 1.5695 (1.8768) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][50/625] eta 0:03:31 lr 0.000735 wd 0.0500 time 0.3450 (0.3683) data time 0.0009 (0.0080) model time 0.0000 (0.0000) loss 3.3052 (3.0140) grad_norm 1.4850 (1.8114) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][60/625] eta 0:03:26 lr 0.000735 wd 0.0500 time 0.3430 (0.3648) data time 0.0006 (0.0068) model time 0.3424 (0.3463) loss 3.7017 (3.0294) grad_norm 2.7349 (1.8310) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][70/625] eta 0:03:21 lr 0.000735 wd 0.0500 time 0.3415 (0.3626) data time 0.0008 (0.0060) model time 0.3407 (0.3470) loss 3.3844 (3.0186) grad_norm 1.6596 (1.8368) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][80/625] eta 0:03:16 lr 0.000735 wd 0.0500 time 0.3525 (0.3613) data time 0.0009 (0.0054) model time 0.3516 (0.3486) loss 3.4468 (3.0339) grad_norm 1.4260 (1.8158) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][90/625] eta 0:03:12 lr 0.000734 wd 0.0500 time 0.3446 (0.3598) data time 0.0007 (0.0049) model time 0.3438 (0.3480) loss 3.6077 (3.0564) grad_norm 1.7043 (1.8120) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:41:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][100/625] eta 0:03:11 lr 0.000734 wd 0.0500 time 0.3479 (0.3651) data time 0.0008 (0.0045) model time 0.3470 (0.3609) loss 3.5054 (3.0563) grad_norm 1.9280 (1.8007) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][110/625] eta 0:03:07 lr 0.000734 wd 0.0500 time 0.3445 (0.3637) data time 0.0010 (0.0041) model time 0.3435 (0.3590) loss 3.3075 (3.0481) grad_norm 1.6712 (1.8393) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][120/625] eta 0:03:03 lr 0.000734 wd 0.0500 time 0.3471 (0.3624) data time 0.0007 (0.0039) model time 0.3464 (0.3573) loss 3.7042 (3.0335) grad_norm 2.4421 (1.8373) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][130/625] eta 0:02:58 lr 0.000734 wd 0.0500 time 0.3438 (0.3614) data time 0.0007 (0.0036) model time 0.3431 (0.3561) loss 2.8111 (3.0291) grad_norm 1.3553 (1.8638) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][140/625] eta 0:02:54 lr 0.000734 wd 0.0500 time 0.3465 (0.3605) data time 0.0009 (0.0034) model time 0.3456 (0.3552) loss 2.8298 (3.0350) grad_norm 1.7618 (1.8476) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][150/625] eta 0:02:50 lr 0.000733 wd 0.0500 time 0.3535 (0.3599) data time 0.0006 (0.0033) model time 0.3529 (0.3548) loss 3.8784 (3.0604) grad_norm 1.9436 (1.8369) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][160/625] eta 0:02:47 lr 0.000733 wd 0.0500 time 0.3485 (0.3593) data time 0.0008 (0.0031) model time 0.3478 (0.3543) loss 2.2445 (3.0424) grad_norm 1.1308 (1.8086) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][170/625] eta 0:02:43 lr 0.000733 wd 0.0500 time 0.3524 (0.3589) data time 0.0009 (0.0030) model time 0.3515 (0.3539) loss 3.1119 (3.0474) grad_norm 1.1392 (1.8005) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][180/625] eta 0:02:39 lr 0.000733 wd 0.0500 time 0.3538 (0.3584) data time 0.0006 (0.0029) model time 0.3532 (0.3537) loss 2.9685 (3.0297) grad_norm 1.4417 (1.7886) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][190/625] eta 0:02:36 lr 0.000733 wd 0.0500 time 0.3486 (0.3588) data time 0.0007 (0.0028) model time 0.3479 (0.3544) loss 3.7313 (3.0348) grad_norm 1.6911 (1.7857) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][200/625] eta 0:02:32 lr 0.000733 wd 0.0500 time 0.3560 (0.3584) data time 0.0008 (0.0027) model time 0.3552 (0.3542) loss 3.2006 (3.0376) grad_norm 1.9757 (1.7802) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][210/625] eta 0:02:28 lr 0.000732 wd 0.0500 time 0.3453 (0.3580) data time 0.0009 (0.0026) model time 0.3443 (0.3539) loss 3.1047 (3.0362) grad_norm 1.7417 (1.7825) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][220/625] eta 0:02:24 lr 0.000732 wd 0.0500 time 0.3646 (0.3578) data time 0.0007 (0.0025) model time 0.3639 (0.3538) loss 2.7713 (3.0266) grad_norm 2.4980 (1.7994) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][230/625] eta 0:02:21 lr 0.000732 wd 0.0500 time 0.3387 (0.3585) data time 0.0008 (0.0025) model time 0.3379 (0.3548) loss 2.7476 (3.0358) grad_norm 1.6910 (1.7978) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][240/625] eta 0:02:18 lr 0.000732 wd 0.0500 time 0.3502 (0.3597) data time 0.0007 (0.0024) model time 0.3495 (0.3565) loss 2.4366 (3.0287) grad_norm 1.5222 (1.7979) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][250/625] eta 0:02:14 lr 0.000732 wd 0.0500 time 0.3397 (0.3592) data time 0.0006 (0.0023) model time 0.3391 (0.3560) loss 3.0438 (3.0259) grad_norm 1.5402 (1.7934) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][260/625] eta 0:02:10 lr 0.000731 wd 0.0500 time 0.3490 (0.3588) data time 0.0009 (0.0023) model time 0.3482 (0.3556) loss 2.4139 (3.0273) grad_norm 1.3426 (1.7875) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:42:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][270/625] eta 0:02:07 lr 0.000731 wd 0.0500 time 0.3541 (0.3584) data time 0.0008 (0.0022) model time 0.3533 (0.3553) loss 3.3201 (3.0272) grad_norm 1.8207 (1.7834) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][280/625] eta 0:02:03 lr 0.000731 wd 0.0500 time 0.3456 (0.3580) data time 0.0008 (0.0022) model time 0.3449 (0.3548) loss 3.1190 (3.0219) grad_norm 1.8417 (1.7746) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][290/625] eta 0:01:59 lr 0.000731 wd 0.0500 time 0.3473 (0.3577) data time 0.0010 (0.0021) model time 0.3463 (0.3545) loss 3.3787 (3.0155) grad_norm 1.7063 (1.7797) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][300/625] eta 0:01:56 lr 0.000731 wd 0.0500 time 0.3614 (0.3576) data time 0.0009 (0.0021) model time 0.3605 (0.3545) loss 3.1342 (3.0090) grad_norm 2.3065 (1.7776) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][310/625] eta 0:01:53 lr 0.000731 wd 0.0500 time 0.3417 (0.3592) data time 0.0008 (0.0020) model time 0.3409 (0.3566) loss 2.8281 (3.0131) grad_norm 1.3688 (1.7845) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][320/625] eta 0:01:49 lr 0.000730 wd 0.0500 time 0.3483 (0.3590) data time 0.0009 (0.0020) model time 0.3473 (0.3563) loss 3.1706 (3.0126) grad_norm 1.6814 (1.7894) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][330/625] eta 0:01:45 lr 0.000730 wd 0.0500 time 0.3489 (0.3586) data time 0.0007 (0.0020) model time 0.3482 (0.3560) loss 3.5431 (3.0126) grad_norm 1.5560 (1.7863) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][340/625] eta 0:01:42 lr 0.000730 wd 0.0500 time 0.3432 (0.3585) data time 0.0008 (0.0019) model time 0.3424 (0.3558) loss 2.6870 (3.0150) grad_norm 1.0833 (1.7782) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][350/625] eta 0:01:38 lr 0.000730 wd 0.0500 time 0.3462 (0.3582) data time 0.0007 (0.0019) model time 0.3454 (0.3556) loss 3.8316 (3.0185) grad_norm 1.7063 (1.7837) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][360/625] eta 0:01:34 lr 0.000730 wd 0.0500 time 0.3522 (0.3580) data time 0.0009 (0.0019) model time 0.3513 (0.3555) loss 3.0242 (3.0161) grad_norm 1.5033 (1.7786) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][370/625] eta 0:01:31 lr 0.000730 wd 0.0500 time 0.3535 (0.3579) data time 0.0006 (0.0019) model time 0.3529 (0.3553) loss 4.0284 (3.0172) grad_norm 1.7097 (1.7754) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][380/625] eta 0:01:27 lr 0.000729 wd 0.0500 time 0.3493 (0.3576) data time 0.0006 (0.0018) model time 0.3487 (0.3551) loss 3.3356 (3.0212) grad_norm 1.0684 (1.7721) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][390/625] eta 0:01:23 lr 0.000729 wd 0.0500 time 0.3496 (0.3574) data time 0.0010 (0.0018) model time 0.3486 (0.3549) loss 3.6002 (3.0237) grad_norm 1.9530 (1.7728) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][400/625] eta 0:01:20 lr 0.000729 wd 0.0500 time 0.3458 (0.3572) data time 0.0007 (0.0018) model time 0.3451 (0.3547) loss 2.2797 (3.0218) grad_norm 1.9615 (1.7821) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][410/625] eta 0:01:16 lr 0.000729 wd 0.0500 time 0.3564 (0.3575) data time 0.0008 (0.0018) model time 0.3556 (0.3550) loss 3.5851 (3.0214) grad_norm 3.0530 (1.7880) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][420/625] eta 0:01:13 lr 0.000729 wd 0.0500 time 0.3541 (0.3586) data time 0.0006 (0.0017) model time 0.3535 (0.3564) loss 2.9921 (3.0208) grad_norm 2.2365 (1.7996) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][430/625] eta 0:01:09 lr 0.000729 wd 0.0500 time 0.3456 (0.3584) data time 0.0006 (0.0017) model time 0.3450 (0.3562) loss 2.6569 (3.0220) grad_norm 2.3300 (1.8048) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][440/625] eta 0:01:06 lr 0.000728 wd 0.0500 time 0.3472 (0.3582) data time 0.0007 (0.0017) model time 0.3465 (0.3560) loss 3.8129 (3.0245) grad_norm 2.3406 (1.8004) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][450/625] eta 0:01:02 lr 0.000728 wd 0.0500 time 0.3450 (0.3580) data time 0.0009 (0.0017) model time 0.3441 (0.3558) loss 3.2994 (3.0300) grad_norm 1.3092 (1.7904) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][460/625] eta 0:00:59 lr 0.000728 wd 0.0500 time 0.3457 (0.3578) data time 0.0007 (0.0017) model time 0.3450 (0.3556) loss 3.3861 (3.0311) grad_norm 1.7920 (1.7888) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][470/625] eta 0:00:55 lr 0.000728 wd 0.0500 time 0.3471 (0.3576) data time 0.0008 (0.0016) model time 0.3463 (0.3554) loss 2.7254 (3.0305) grad_norm 2.5097 (1.7918) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][480/625] eta 0:00:51 lr 0.000728 wd 0.0500 time 0.3445 (0.3576) data time 0.0008 (0.0016) model time 0.3437 (0.3554) loss 3.2461 (3.0283) grad_norm 1.2876 (1.7836) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][490/625] eta 0:00:48 lr 0.000728 wd 0.0500 time 0.5462 (0.3589) data time 0.0007 (0.0016) model time 0.5454 (0.3570) loss 3.1383 (3.0300) grad_norm 1.3311 (1.7804) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][500/625] eta 0:00:44 lr 0.000727 wd 0.0500 time 0.3512 (0.3591) data time 0.0008 (0.0016) model time 0.3504 (0.3572) loss 2.5095 (3.0229) grad_norm 2.0616 (1.7858) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][510/625] eta 0:00:41 lr 0.000727 wd 0.0500 time 0.3521 (0.3589) data time 0.0009 (0.0016) model time 0.3512 (0.3570) loss 2.2230 (3.0242) grad_norm 1.2063 (1.7902) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][520/625] eta 0:00:37 lr 0.000727 wd 0.0500 time 0.3448 (0.3591) data time 0.0008 (0.0016) model time 0.3439 (0.3572) loss 3.2752 (3.0272) grad_norm 1.5532 (1.7913) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][530/625] eta 0:00:34 lr 0.000727 wd 0.0500 time 0.3482 (0.3593) data time 0.0007 (0.0016) model time 0.3475 (0.3574) loss 2.1496 (3.0307) grad_norm 1.2641 (1.7881) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][540/625] eta 0:00:30 lr 0.000727 wd 0.0500 time 0.3449 (0.3601) data time 0.0010 (0.0015) model time 0.3440 (0.3583) loss 2.7743 (3.0258) grad_norm 1.6415 (1.7882) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][550/625] eta 0:00:26 lr 0.000727 wd 0.0500 time 0.3499 (0.3599) data time 0.0008 (0.0015) model time 0.3491 (0.3581) loss 3.3368 (3.0283) grad_norm 1.3050 (1.7854) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][560/625] eta 0:00:23 lr 0.000726 wd 0.0500 time 0.3471 (0.3597) data time 0.0009 (0.0015) model time 0.3462 (0.3580) loss 2.4563 (3.0277) grad_norm 1.6437 (1.7814) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][570/625] eta 0:00:19 lr 0.000726 wd 0.0500 time 0.3507 (0.3595) data time 0.0011 (0.0015) model time 0.3496 (0.3578) loss 2.3642 (3.0292) grad_norm 1.3211 (1.7778) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][580/625] eta 0:00:16 lr 0.000726 wd 0.0500 time 0.3506 (0.3593) data time 0.0007 (0.0015) model time 0.3498 (0.3576) loss 3.3397 (3.0304) grad_norm 1.7733 (1.7728) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][590/625] eta 0:00:12 lr 0.000726 wd 0.0500 time 0.3523 (0.3592) data time 0.0008 (0.0015) model time 0.3516 (0.3574) loss 3.4422 (3.0293) grad_norm 1.5974 (1.7712) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:44:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][600/625] eta 0:00:08 lr 0.000726 wd 0.0500 time 0.3554 (0.3591) data time 0.0009 (0.0015) model time 0.3544 (0.3573) loss 3.1992 (3.0305) grad_norm 1.7399 (1.7754) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][610/625] eta 0:00:05 lr 0.000726 wd 0.0500 time 0.5404 (0.3599) data time 0.0004 (0.0015) model time 0.5400 (0.3582) loss 2.1154 (3.0294) grad_norm 1.1833 (1.7757) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [185/300][620/625] eta 0:00:01 lr 0.000725 wd 0.0500 time 0.3417 (0.3596) data time 0.0004 (0.0015) model time 0.3412 (0.3580) loss 2.9692 (3.0293) grad_norm 1.3828 (1.7730) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:05 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 185 training takes 0:03:44 [2024-09-01 07:45:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:45:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:45:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.468 (0.468) Loss 0.5483 (0.5483) Acc@1 88.135 (88.135) Acc@5 98.242 (98.242) Mem 11224MB [2024-09-01 07:45:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.134) Loss 0.9062 (0.6744) Acc@1 78.467 (85.063) Acc@5 95.410 (97.288) Mem 11224MB [2024-09-01 07:45:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.115) Loss 0.9795 (0.8102) Acc@1 76.465 (81.687) Acc@5 94.189 (95.831) Mem 11224MB [2024-09-01 07:45:08 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.380 Acc@5 95.855 [2024-09-01 07:45:08 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.4% [2024-09-01 07:45:08 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.38% [2024-09-01 07:45:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 07:45:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 07:45:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.592 (0.592) Loss 0.5054 (0.5054) Acc@1 89.502 (89.502) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 07:45:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.144) Loss 0.8071 (0.6164) Acc@1 81.006 (86.346) Acc@5 96.240 (97.714) Mem 11224MB [2024-09-01 07:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.119) Loss 0.9199 (0.7310) Acc@1 77.148 (83.343) Acc@5 94.824 (96.540) Mem 11224MB [2024-09-01 07:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.057 Acc@5 96.545 [2024-09-01 07:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 07:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.06% [2024-09-01 07:45:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:45:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:45:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][0/625] eta 0:18:22 lr 0.000725 wd 0.0500 time 1.7643 (1.7643) data time 1.3022 (1.3022) model time 0.0000 (0.0000) loss 2.6413 (2.6413) grad_norm 1.4222 (1.4222) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][10/625] eta 0:04:52 lr 0.000725 wd 0.0500 time 0.3457 (0.4748) data time 0.0008 (0.1192) model time 0.0000 (0.0000) loss 2.6613 (3.0365) grad_norm 1.5713 (1.9369) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][20/625] eta 0:04:10 lr 0.000725 wd 0.0500 time 0.3459 (0.4144) data time 0.0007 (0.0629) model time 0.0000 (0.0000) loss 2.9337 (3.1019) grad_norm 1.6467 (1.9595) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][30/625] eta 0:03:54 lr 0.000725 wd 0.0500 time 0.3535 (0.3936) data time 0.0008 (0.0429) model time 0.0000 (0.0000) loss 3.6080 (3.1335) grad_norm 1.6343 (1.8831) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][40/625] eta 0:03:43 lr 0.000725 wd 0.0500 time 0.3518 (0.3826) data time 0.0009 (0.0326) model time 0.0000 (0.0000) loss 3.4692 (3.0550) grad_norm 1.4624 (1.8144) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][50/625] eta 0:03:36 lr 0.000724 wd 0.0500 time 0.3456 (0.3757) data time 0.0006 (0.0264) model time 0.0000 (0.0000) loss 3.4795 (3.0301) grad_norm 2.3420 (1.7919) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][60/625] eta 0:03:29 lr 0.000724 wd 0.0500 time 0.3543 (0.3715) data time 0.0006 (0.0222) model time 0.3537 (0.3487) loss 3.6414 (3.0393) grad_norm 2.5274 (1.8527) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][70/625] eta 0:03:29 lr 0.000724 wd 0.0500 time 0.3469 (0.3777) data time 0.0006 (0.0192) model time 0.3463 (0.3818) loss 3.6603 (3.0831) grad_norm 1.4558 (1.8185) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][80/625] eta 0:03:24 lr 0.000724 wd 0.0500 time 0.3493 (0.3747) data time 0.0007 (0.0169) model time 0.3486 (0.3720) loss 3.7959 (3.0858) grad_norm 1.6057 (1.7957) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][90/625] eta 0:03:18 lr 0.000724 wd 0.0500 time 0.3435 (0.3717) data time 0.0006 (0.0151) model time 0.3430 (0.3657) loss 2.9514 (3.1135) grad_norm 2.1624 (1.8738) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][100/625] eta 0:03:13 lr 0.000724 wd 0.0500 time 0.3431 (0.3694) data time 0.0006 (0.0137) model time 0.3425 (0.3622) loss 3.5457 (3.0980) grad_norm 1.8403 (1.8583) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][110/625] eta 0:03:09 lr 0.000723 wd 0.0500 time 0.3516 (0.3676) data time 0.0009 (0.0126) model time 0.3507 (0.3599) loss 3.2998 (3.0631) grad_norm 2.1146 (1.8346) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:45:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][120/625] eta 0:03:05 lr 0.000723 wd 0.0500 time 0.3427 (0.3664) data time 0.0009 (0.0116) model time 0.3418 (0.3588) loss 3.3747 (3.0529) grad_norm 2.0137 (1.8441) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][130/625] eta 0:03:01 lr 0.000723 wd 0.0500 time 0.3529 (0.3662) data time 0.0006 (0.0108) model time 0.3523 (0.3593) loss 3.7210 (3.0468) grad_norm 2.1494 (1.8417) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][140/625] eta 0:02:57 lr 0.000723 wd 0.0500 time 0.3526 (0.3653) data time 0.0008 (0.0101) model time 0.3518 (0.3585) loss 3.1343 (3.0406) grad_norm 2.0250 (1.9142) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][150/625] eta 0:02:53 lr 0.000723 wd 0.0500 time 0.3498 (0.3643) data time 0.0008 (0.0095) model time 0.3489 (0.3576) loss 2.7574 (3.0435) grad_norm 1.1741 (1.9201) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][160/625] eta 0:02:48 lr 0.000723 wd 0.0500 time 0.3471 (0.3633) data time 0.0010 (0.0089) model time 0.3461 (0.3567) loss 2.8700 (3.0595) grad_norm 1.8655 (1.9176) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][170/625] eta 0:02:46 lr 0.000722 wd 0.0500 time 0.3484 (0.3655) data time 0.0006 (0.0085) model time 0.3478 (0.3603) loss 3.3189 (3.0626) grad_norm 2.0030 (1.8991) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][180/625] eta 0:02:42 lr 0.000722 wd 0.0500 time 0.3481 (0.3646) data time 0.0006 (0.0080) model time 0.3474 (0.3594) loss 3.7243 (3.0557) grad_norm 2.5938 (1.8965) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][190/625] eta 0:02:38 lr 0.000722 wd 0.0500 time 0.3460 (0.3638) data time 0.0007 (0.0077) model time 0.3454 (0.3586) loss 3.7094 (3.0666) grad_norm 1.7736 (1.8803) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][200/625] eta 0:02:34 lr 0.000722 wd 0.0500 time 0.3467 (0.3630) data time 0.0006 (0.0073) model time 0.3461 (0.3578) loss 3.3563 (3.0707) grad_norm 2.5604 (1.8860) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][210/625] eta 0:02:30 lr 0.000722 wd 0.0500 time 0.3456 (0.3623) data time 0.0006 (0.0070) model time 0.3450 (0.3572) loss 2.6528 (3.0672) grad_norm 1.8998 (1.8962) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][220/625] eta 0:02:26 lr 0.000722 wd 0.0500 time 0.3507 (0.3618) data time 0.0006 (0.0067) model time 0.3501 (0.3568) loss 3.8442 (3.0652) grad_norm 2.2130 (1.8934) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][230/625] eta 0:02:22 lr 0.000721 wd 0.0500 time 0.3441 (0.3612) data time 0.0007 (0.0065) model time 0.3435 (0.3562) loss 3.2913 (3.0537) grad_norm 1.8067 (1.8876) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][240/625] eta 0:02:18 lr 0.000721 wd 0.0500 time 0.3612 (0.3606) data time 0.0006 (0.0063) model time 0.3606 (0.3557) loss 3.0271 (3.0628) grad_norm 1.6379 (1.8855) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][250/625] eta 0:02:15 lr 0.000721 wd 0.0500 time 0.3440 (0.3601) data time 0.0007 (0.0060) model time 0.3433 (0.3553) loss 2.3842 (3.0583) grad_norm 1.6535 (1.8711) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][260/625] eta 0:02:11 lr 0.000721 wd 0.0500 time 0.3497 (0.3597) data time 0.0010 (0.0058) model time 0.3487 (0.3550) loss 3.6702 (3.0554) grad_norm 9.9100 (1.9072) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][270/625] eta 0:02:07 lr 0.000721 wd 0.0500 time 0.3431 (0.3593) data time 0.0006 (0.0057) model time 0.3425 (0.3547) loss 2.2346 (3.0648) grad_norm 5.4194 (1.9456) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][280/625] eta 0:02:04 lr 0.000721 wd 0.0500 time 0.3524 (0.3609) data time 0.0007 (0.0055) model time 0.3517 (0.3567) loss 3.3985 (3.0678) grad_norm 1.2212 (1.9379) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][290/625] eta 0:02:01 lr 0.000720 wd 0.0500 time 0.3426 (0.3618) data time 0.0006 (0.0053) model time 0.3420 (0.3580) loss 3.6162 (3.0624) grad_norm 1.8309 (1.9310) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][300/625] eta 0:01:57 lr 0.000720 wd 0.0500 time 0.3480 (0.3614) data time 0.0008 (0.0052) model time 0.3472 (0.3577) loss 3.1926 (3.0631) grad_norm 2.8446 (1.9232) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][310/625] eta 0:01:53 lr 0.000720 wd 0.0500 time 0.3479 (0.3610) data time 0.0008 (0.0050) model time 0.3471 (0.3573) loss 3.0412 (3.0597) grad_norm 1.6267 (1.9237) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][320/625] eta 0:01:49 lr 0.000720 wd 0.0500 time 0.3593 (0.3607) data time 0.0008 (0.0049) model time 0.3584 (0.3569) loss 2.9302 (3.0634) grad_norm 2.5626 (1.9196) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][330/625] eta 0:01:46 lr 0.000720 wd 0.0500 time 0.3531 (0.3605) data time 0.0006 (0.0048) model time 0.3525 (0.3568) loss 2.8097 (3.0611) grad_norm 1.2537 (1.9138) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][340/625] eta 0:01:42 lr 0.000719 wd 0.0500 time 0.3499 (0.3607) data time 0.0009 (0.0047) model time 0.3491 (0.3572) loss 3.6803 (3.0507) grad_norm 1.3620 (1.9001) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][350/625] eta 0:01:39 lr 0.000719 wd 0.0500 time 0.3618 (0.3604) data time 0.0006 (0.0046) model time 0.3612 (0.3569) loss 2.6060 (3.0473) grad_norm 1.1705 (1.8833) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][360/625] eta 0:01:35 lr 0.000719 wd 0.0500 time 0.3513 (0.3601) data time 0.0007 (0.0045) model time 0.3505 (0.3566) loss 3.6615 (3.0484) grad_norm 1.9997 (1.8819) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][370/625] eta 0:01:31 lr 0.000719 wd 0.0500 time 0.3444 (0.3598) data time 0.0008 (0.0044) model time 0.3436 (0.3564) loss 2.3996 (3.0464) grad_norm 1.2558 (1.8843) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][380/625] eta 0:01:28 lr 0.000719 wd 0.0500 time 0.3512 (0.3595) data time 0.0006 (0.0043) model time 0.3505 (0.3562) loss 2.8636 (3.0453) grad_norm 1.2734 (1.8798) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][390/625] eta 0:01:24 lr 0.000719 wd 0.0500 time 0.3438 (0.3593) data time 0.0008 (0.0042) model time 0.3429 (0.3559) loss 3.1040 (3.0428) grad_norm 2.8165 (1.8863) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][400/625] eta 0:01:20 lr 0.000718 wd 0.0500 time 0.3467 (0.3590) data time 0.0009 (0.0041) model time 0.3458 (0.3557) loss 3.1702 (3.0481) grad_norm 1.6624 (1.8872) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][410/625] eta 0:01:17 lr 0.000718 wd 0.0500 time 0.3706 (0.3588) data time 0.0008 (0.0040) model time 0.3698 (0.3555) loss 3.1088 (3.0481) grad_norm 2.9118 (1.8907) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][420/625] eta 0:01:13 lr 0.000718 wd 0.0500 time 0.3539 (0.3586) data time 0.0009 (0.0039) model time 0.3529 (0.3554) loss 3.2984 (3.0517) grad_norm 1.6738 (1.8860) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][430/625] eta 0:01:09 lr 0.000718 wd 0.0500 time 0.3496 (0.3585) data time 0.0010 (0.0039) model time 0.3486 (0.3554) loss 3.0649 (3.0484) grad_norm 1.3589 (1.8766) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][440/625] eta 0:01:06 lr 0.000718 wd 0.0500 time 0.3509 (0.3584) data time 0.0008 (0.0038) model time 0.3501 (0.3552) loss 3.6434 (3.0545) grad_norm 1.4297 (1.8749) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][450/625] eta 0:01:02 lr 0.000718 wd 0.0500 time 0.3494 (0.3582) data time 0.0007 (0.0037) model time 0.3487 (0.3551) loss 3.3687 (3.0531) grad_norm 1.4973 (1.8700) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:47:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][460/625] eta 0:00:59 lr 0.000717 wd 0.0500 time 0.3539 (0.3584) data time 0.0008 (0.0037) model time 0.3531 (0.3554) loss 1.7958 (3.0498) grad_norm 1.1947 (1.8667) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][470/625] eta 0:00:55 lr 0.000717 wd 0.0500 time 0.3550 (0.3591) data time 0.0007 (0.0036) model time 0.3544 (0.3562) loss 2.6482 (3.0518) grad_norm 1.4471 (1.8586) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][480/625] eta 0:00:52 lr 0.000717 wd 0.0500 time 0.3443 (0.3589) data time 0.0006 (0.0036) model time 0.3437 (0.3561) loss 2.3778 (3.0552) grad_norm 1.7734 (1.8522) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][490/625] eta 0:00:48 lr 0.000717 wd 0.0500 time 0.3514 (0.3589) data time 0.0006 (0.0035) model time 0.3507 (0.3561) loss 3.4672 (3.0553) grad_norm 1.8272 (1.8477) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][500/625] eta 0:00:44 lr 0.000717 wd 0.0500 time 0.3446 (0.3597) data time 0.0007 (0.0035) model time 0.3440 (0.3570) loss 3.2293 (3.0487) grad_norm 1.2072 (1.8470) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][510/625] eta 0:00:41 lr 0.000717 wd 0.0500 time 0.3515 (0.3595) data time 0.0008 (0.0034) model time 0.3507 (0.3568) loss 3.4187 (3.0419) grad_norm 2.4967 (1.8459) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][520/625] eta 0:00:37 lr 0.000716 wd 0.0500 time 0.3542 (0.3596) data time 0.0007 (0.0034) model time 0.3535 (0.3570) loss 3.6485 (3.0462) grad_norm 8.4554 (1.8650) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][530/625] eta 0:00:34 lr 0.000716 wd 0.0500 time 0.3491 (0.3595) data time 0.0007 (0.0033) model time 0.3484 (0.3569) loss 3.0543 (3.0472) grad_norm 1.3585 (1.8656) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][540/625] eta 0:00:30 lr 0.000716 wd 0.0500 time 0.3487 (0.3594) data time 0.0007 (0.0033) model time 0.3480 (0.3568) loss 3.3797 (3.0463) grad_norm 1.7363 (1.8600) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][550/625] eta 0:00:26 lr 0.000716 wd 0.0500 time 0.3496 (0.3592) data time 0.0008 (0.0032) model time 0.3488 (0.3567) loss 1.9856 (3.0429) grad_norm 2.1442 (1.8570) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][560/625] eta 0:00:23 lr 0.000716 wd 0.0500 time 0.3561 (0.3591) data time 0.0009 (0.0032) model time 0.3552 (0.3565) loss 3.5223 (3.0474) grad_norm 1.1978 (1.8481) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][570/625] eta 0:00:19 lr 0.000716 wd 0.0500 time 0.3452 (0.3589) data time 0.0009 (0.0031) model time 0.3443 (0.3564) loss 2.6773 (3.0470) grad_norm 1.2878 (1.8439) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][580/625] eta 0:00:16 lr 0.000715 wd 0.0500 time 0.5447 (0.3595) data time 0.0009 (0.0031) model time 0.5438 (0.3571) loss 3.5479 (3.0495) grad_norm 1.6060 (1.8499) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][590/625] eta 0:00:12 lr 0.000715 wd 0.0500 time 0.3556 (0.3596) data time 0.0006 (0.0031) model time 0.3550 (0.3572) loss 3.9872 (3.0498) grad_norm 2.6035 (1.8716) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][600/625] eta 0:00:08 lr 0.000715 wd 0.0500 time 0.3460 (0.3594) data time 0.0007 (0.0030) model time 0.3453 (0.3570) loss 3.8144 (3.0513) grad_norm 1.4570 (1.8755) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][610/625] eta 0:00:05 lr 0.000715 wd 0.0500 time 0.3367 (0.3592) data time 0.0004 (0.0030) model time 0.3363 (0.3568) loss 3.5987 (3.0517) grad_norm 1.4224 (1.8734) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [186/300][620/625] eta 0:00:01 lr 0.000715 wd 0.0500 time 0.3424 (0.3589) data time 0.0006 (0.0030) model time 0.3418 (0.3566) loss 3.3454 (3.0525) grad_norm 1.7846 (1.8912) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:48:57 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 186 training takes 0:03:44 [2024-09-01 07:48:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:48:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:49:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 3.680 (3.680) Loss 0.5630 (0.5630) Acc@1 87.500 (87.500) Acc@5 98.145 (98.145) Mem 11224MB [2024-09-01 07:49:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.433) Loss 0.8936 (0.6877) Acc@1 78.369 (84.628) Acc@5 95.117 (97.164) Mem 11224MB [2024-09-01 07:49:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.270) Loss 1.0391 (0.8191) Acc@1 74.658 (81.376) Acc@5 93.896 (95.689) Mem 11224MB [2024-09-01 07:49:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.122 Acc@5 95.727 [2024-09-01 07:49:05 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.1% [2024-09-01 07:49:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 6.616 (6.616) Loss 0.5054 (0.5054) Acc@1 89.453 (89.453) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 07:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.693) Loss 0.8071 (0.6156) Acc@1 81.104 (86.377) Acc@5 96.240 (97.696) Mem 11224MB [2024-09-01 07:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.407) Loss 0.9180 (0.7302) Acc@1 77.148 (83.389) Acc@5 94.824 (96.519) Mem 11224MB [2024-09-01 07:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.099 Acc@5 96.529 [2024-09-01 07:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 07:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.10% [2024-09-01 07:49:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 07:49:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 07:49:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][0/625] eta 0:48:27 lr 0.000715 wd 0.0500 time 4.6522 (4.6522) data time 3.0326 (3.0326) model time 0.0000 (0.0000) loss 2.7757 (2.7757) grad_norm 3.5528 (3.5528) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 07:49:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][10/625] eta 0:07:35 lr 0.000714 wd 0.0500 time 0.3433 (0.7412) data time 0.0006 (0.2765) model time 0.0000 (0.0000) loss 3.5547 (3.1412) grad_norm 1.1964 (1.8498) loss_scale 1024.0000 (744.7273) mem 11224MB [2024-09-01 07:49:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][20/625] eta 0:05:35 lr 0.000714 wd 0.0500 time 0.3551 (0.5547) data time 0.0009 (0.1453) model time 0.0000 (0.0000) loss 2.4066 (3.1392) grad_norm 1.0703 (1.6930) loss_scale 1024.0000 (877.7143) mem 11224MB [2024-09-01 07:49:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][30/625] eta 0:04:52 lr 0.000714 wd 0.0500 time 0.3504 (0.4911) data time 0.0008 (0.0987) model time 0.0000 (0.0000) loss 3.0868 (3.0650) grad_norm 2.2786 (1.7908) loss_scale 1024.0000 (924.9032) mem 11224MB [2024-09-01 07:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][40/625] eta 0:04:27 lr 0.000714 wd 0.0500 time 0.3441 (0.4570) data time 0.0009 (0.0749) model time 0.0000 (0.0000) loss 2.6077 (3.0137) grad_norm 1.8037 (1.7646) loss_scale 1024.0000 (949.0732) mem 11224MB [2024-09-01 07:49:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][50/625] eta 0:04:10 lr 0.000714 wd 0.0500 time 0.3489 (0.4359) data time 0.0006 (0.0603) model time 0.0000 (0.0000) loss 3.1596 (3.0246) grad_norm 2.0781 (1.7407) loss_scale 1024.0000 (963.7647) mem 11224MB [2024-09-01 07:49:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][60/625] eta 0:03:59 lr 0.000714 wd 0.0500 time 0.5409 (0.4247) data time 0.0009 (0.0506) model time 0.5400 (0.3668) loss 3.3140 (3.0211) grad_norm 2.2386 (1.7538) loss_scale 1024.0000 (973.6393) mem 11224MB [2024-09-01 07:49:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][70/625] eta 0:03:53 lr 0.000713 wd 0.0500 time 0.3550 (0.4203) data time 0.0009 (0.0436) model time 0.3541 (0.3798) loss 2.6192 (2.9921) grad_norm 1.7338 (1.7723) loss_scale 1024.0000 (980.7324) mem 11224MB [2024-09-01 07:49:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][80/625] eta 0:03:44 lr 0.000713 wd 0.0500 time 0.3533 (0.4118) data time 0.0007 (0.0383) model time 0.3526 (0.3699) loss 2.4294 (2.9712) grad_norm 1.4160 (1.7322) loss_scale 1024.0000 (986.0741) mem 11224MB [2024-09-01 07:49:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][90/625] eta 0:03:36 lr 0.000713 wd 0.0500 time 0.3473 (0.4051) data time 0.0006 (0.0342) model time 0.3467 (0.3650) loss 3.6326 (2.9887) grad_norm 1.3176 (1.7222) loss_scale 1024.0000 (990.2418) mem 11224MB [2024-09-01 07:49:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][100/625] eta 0:03:32 lr 0.000713 wd 0.0500 time 0.3675 (0.4053) data time 0.0006 (0.0309) model time 0.3668 (0.3734) loss 3.6609 (3.0031) grad_norm 1.8741 (1.7129) loss_scale 1024.0000 (993.5842) mem 11224MB [2024-09-01 07:49:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][110/625] eta 0:03:26 lr 0.000713 wd 0.0500 time 0.3507 (0.4005) data time 0.0009 (0.0282) model time 0.3498 (0.3697) loss 3.3795 (3.0012) grad_norm 2.5531 (1.7136) loss_scale 1024.0000 (996.3243) mem 11224MB [2024-09-01 07:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][120/625] eta 0:03:20 lr 0.000713 wd 0.0500 time 0.3492 (0.3963) data time 0.0009 (0.0259) model time 0.3483 (0.3666) loss 3.0148 (3.0101) grad_norm 2.0339 (1.7401) loss_scale 1024.0000 (998.6116) mem 11224MB [2024-09-01 07:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][130/625] eta 0:03:14 lr 0.000712 wd 0.0500 time 0.3496 (0.3929) data time 0.0009 (0.0240) model time 0.3487 (0.3646) loss 3.2859 (3.0183) grad_norm 1.8273 (1.7438) loss_scale 1024.0000 (1000.5496) mem 11224MB [2024-09-01 07:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][140/625] eta 0:03:09 lr 0.000712 wd 0.0500 time 0.3455 (0.3899) data time 0.0009 (0.0224) model time 0.3446 (0.3629) loss 2.9129 (3.0116) grad_norm 1.5807 (1.7300) loss_scale 1024.0000 (1002.2128) mem 11224MB [2024-09-01 07:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][150/625] eta 0:03:03 lr 0.000712 wd 0.0500 time 0.3483 (0.3873) data time 0.0008 (0.0209) model time 0.3475 (0.3616) loss 3.1573 (2.9936) grad_norm 1.4408 (1.7329) loss_scale 1024.0000 (1003.6556) mem 11224MB [2024-09-01 07:50:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][160/625] eta 0:02:59 lr 0.000712 wd 0.0500 time 0.3486 (0.3850) data time 0.0006 (0.0197) model time 0.3480 (0.3606) loss 3.6395 (3.0051) grad_norm 2.4374 (1.7429) loss_scale 1024.0000 (1004.9193) mem 11224MB [2024-09-01 07:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][170/625] eta 0:02:54 lr 0.000712 wd 0.0500 time 0.3491 (0.3829) data time 0.0008 (0.0186) model time 0.3484 (0.3595) loss 2.7247 (3.0060) grad_norm 2.0882 (1.7457) loss_scale 1024.0000 (1006.0351) mem 11224MB [2024-09-01 07:50:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][180/625] eta 0:02:49 lr 0.000712 wd 0.0500 time 0.3459 (0.3810) data time 0.0007 (0.0176) model time 0.3453 (0.3587) loss 3.4549 (3.0138) grad_norm 1.4606 (1.7498) loss_scale 1024.0000 (1007.0276) mem 11224MB [2024-09-01 07:50:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][190/625] eta 0:02:45 lr 0.000711 wd 0.0500 time 0.3440 (0.3803) data time 0.0009 (0.0167) model time 0.3432 (0.3592) loss 3.2858 (3.0062) grad_norm 1.8533 (1.7419) loss_scale 1024.0000 (1007.9162) mem 11224MB [2024-09-01 07:50:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][200/625] eta 0:02:40 lr 0.000711 wd 0.0500 time 0.3473 (0.3788) data time 0.0006 (0.0159) model time 0.3467 (0.3585) loss 3.1386 (2.9996) grad_norm 1.7686 (1.7392) loss_scale 1024.0000 (1008.7164) mem 11224MB [2024-09-01 07:50:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][210/625] eta 0:02:36 lr 0.000711 wd 0.0500 time 0.3590 (0.3773) data time 0.0007 (0.0152) model time 0.3583 (0.3578) loss 1.9643 (3.0044) grad_norm 1.4612 (1.7346) loss_scale 1024.0000 (1009.4408) mem 11224MB [2024-09-01 07:50:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][220/625] eta 0:02:32 lr 0.000711 wd 0.0500 time 0.3733 (0.3763) data time 0.0006 (0.0146) model time 0.3728 (0.3575) loss 2.4004 (3.0039) grad_norm 1.8721 (1.7321) loss_scale 1024.0000 (1010.0995) mem 11224MB [2024-09-01 07:50:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][230/625] eta 0:02:28 lr 0.000711 wd 0.0500 time 0.3449 (0.3753) data time 0.0007 (0.0140) model time 0.3442 (0.3572) loss 3.3348 (3.0131) grad_norm 1.9866 (1.7396) loss_scale 1024.0000 (1010.7013) mem 11224MB [2024-09-01 07:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][240/625] eta 0:02:24 lr 0.000711 wd 0.0500 time 0.3489 (0.3742) data time 0.0010 (0.0134) model time 0.3479 (0.3568) loss 3.1860 (3.0099) grad_norm 1.5520 (1.7355) loss_scale 1024.0000 (1011.2531) mem 11224MB [2024-09-01 07:50:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][250/625] eta 0:02:21 lr 0.000710 wd 0.0500 time 0.3544 (0.3766) data time 0.0009 (0.0129) model time 0.3536 (0.3607) loss 3.6642 (3.0078) grad_norm 2.7695 (1.7771) loss_scale 1024.0000 (1011.7610) mem 11224MB [2024-09-01 07:50:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][260/625] eta 0:02:17 lr 0.000710 wd 0.0500 time 0.3491 (0.3755) data time 0.0006 (0.0125) model time 0.3485 (0.3600) loss 3.2013 (3.0116) grad_norm 2.6573 (1.7964) loss_scale 1024.0000 (1012.2299) mem 11224MB [2024-09-01 07:50:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][270/625] eta 0:02:12 lr 0.000710 wd 0.0500 time 0.3467 (0.3745) data time 0.0006 (0.0120) model time 0.3461 (0.3594) loss 2.8380 (3.0148) grad_norm 1.6958 (1.7888) loss_scale 1024.0000 (1012.6642) mem 11224MB [2024-09-01 07:50:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][280/625] eta 0:02:08 lr 0.000710 wd 0.0500 time 0.3477 (0.3736) data time 0.0008 (0.0116) model time 0.3470 (0.3590) loss 3.3608 (3.0201) grad_norm 1.4738 (1.7799) loss_scale 1024.0000 (1013.0676) mem 11224MB [2024-09-01 07:51:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][290/625] eta 0:02:04 lr 0.000710 wd 0.0500 time 0.3486 (0.3728) data time 0.0009 (0.0113) model time 0.3478 (0.3585) loss 3.5193 (3.0301) grad_norm 2.0168 (1.7814) loss_scale 1024.0000 (1013.4433) mem 11224MB [2024-09-01 07:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][300/625] eta 0:02:01 lr 0.000710 wd 0.0500 time 0.5535 (0.3726) data time 0.0007 (0.0109) model time 0.5528 (0.3589) loss 2.7171 (3.0332) grad_norm 1.6202 (1.7805) loss_scale 1024.0000 (1013.7940) mem 11224MB [2024-09-01 07:51:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][310/625] eta 0:01:57 lr 0.000709 wd 0.0500 time 0.3458 (0.3730) data time 0.0007 (0.0106) model time 0.3451 (0.3598) loss 2.3024 (3.0282) grad_norm 1.4733 (1.7747) loss_scale 1024.0000 (1014.1222) mem 11224MB [2024-09-01 07:51:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][320/625] eta 0:01:53 lr 0.000709 wd 0.0500 time 0.3483 (0.3722) data time 0.0006 (0.0103) model time 0.3476 (0.3593) loss 1.8975 (3.0264) grad_norm 1.5418 (1.7788) loss_scale 1024.0000 (1014.4299) mem 11224MB [2024-09-01 07:51:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][330/625] eta 0:01:49 lr 0.000709 wd 0.0500 time 0.3447 (0.3714) data time 0.0007 (0.0100) model time 0.3441 (0.3588) loss 3.8558 (3.0355) grad_norm 1.9414 (1.7718) loss_scale 1024.0000 (1014.7190) mem 11224MB [2024-09-01 07:51:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][340/625] eta 0:01:45 lr 0.000709 wd 0.0500 time 0.3470 (0.3707) data time 0.0007 (0.0097) model time 0.3462 (0.3584) loss 3.5732 (3.0351) grad_norm 1.8393 (1.7692) loss_scale 1024.0000 (1014.9912) mem 11224MB [2024-09-01 07:51:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][350/625] eta 0:01:41 lr 0.000709 wd 0.0500 time 0.3496 (0.3700) data time 0.0008 (0.0095) model time 0.3488 (0.3580) loss 3.1157 (3.0390) grad_norm 1.9076 (1.7792) loss_scale 1024.0000 (1015.2479) mem 11224MB [2024-09-01 07:51:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][360/625] eta 0:01:37 lr 0.000709 wd 0.0500 time 0.3455 (0.3694) data time 0.0009 (0.0093) model time 0.3446 (0.3576) loss 2.6004 (3.0343) grad_norm 1.8176 (1.7783) loss_scale 1024.0000 (1015.4903) mem 11224MB [2024-09-01 07:51:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][370/625] eta 0:01:34 lr 0.000708 wd 0.0500 time 0.3433 (0.3704) data time 0.0008 (0.0090) model time 0.3425 (0.3591) loss 3.5966 (3.0338) grad_norm 1.7900 (1.7773) loss_scale 1024.0000 (1015.7197) mem 11224MB [2024-09-01 07:51:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][380/625] eta 0:01:30 lr 0.000708 wd 0.0500 time 0.3488 (0.3699) data time 0.0008 (0.0088) model time 0.3480 (0.3588) loss 2.9657 (3.0246) grad_norm 1.4264 (1.7752) loss_scale 1024.0000 (1015.9370) mem 11224MB [2024-09-01 07:51:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][390/625] eta 0:01:26 lr 0.000708 wd 0.0500 time 0.3458 (0.3693) data time 0.0006 (0.0086) model time 0.3452 (0.3584) loss 2.7744 (3.0205) grad_norm 2.8526 (1.7763) loss_scale 1024.0000 (1016.1432) mem 11224MB [2024-09-01 07:51:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][400/625] eta 0:01:22 lr 0.000708 wd 0.0500 time 0.3484 (0.3687) data time 0.0006 (0.0084) model time 0.3477 (0.3581) loss 2.3649 (3.0264) grad_norm 1.3608 (1.7735) loss_scale 1024.0000 (1016.3392) mem 11224MB [2024-09-01 07:51:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][410/625] eta 0:01:19 lr 0.000708 wd 0.0500 time 0.3453 (0.3682) data time 0.0007 (0.0082) model time 0.3446 (0.3577) loss 3.2420 (3.0280) grad_norm 1.3532 (1.7812) loss_scale 1024.0000 (1016.5255) mem 11224MB [2024-09-01 07:51:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][420/625] eta 0:01:15 lr 0.000708 wd 0.0500 time 0.3585 (0.3677) data time 0.0007 (0.0081) model time 0.3577 (0.3574) loss 3.2693 (3.0219) grad_norm 2.0833 (1.7830) loss_scale 1024.0000 (1016.7031) mem 11224MB [2024-09-01 07:51:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][430/625] eta 0:01:11 lr 0.000707 wd 0.0500 time 0.3461 (0.3673) data time 0.0009 (0.0079) model time 0.3452 (0.3572) loss 3.5225 (3.0241) grad_norm 1.2341 (1.7821) loss_scale 1024.0000 (1016.8724) mem 11224MB [2024-09-01 07:51:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][440/625] eta 0:01:07 lr 0.000707 wd 0.0500 time 0.3485 (0.3668) data time 0.0008 (0.0077) model time 0.3477 (0.3570) loss 3.4558 (3.0230) grad_norm 1.3177 (1.7841) loss_scale 1024.0000 (1017.0340) mem 11224MB [2024-09-01 07:52:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][450/625] eta 0:01:04 lr 0.000707 wd 0.0500 time 0.3493 (0.3664) data time 0.0008 (0.0076) model time 0.3485 (0.3567) loss 2.7777 (3.0256) grad_norm 1.5890 (1.7803) loss_scale 1024.0000 (1017.1885) mem 11224MB [2024-09-01 07:52:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][460/625] eta 0:01:00 lr 0.000707 wd 0.0500 time 0.3468 (0.3660) data time 0.0006 (0.0074) model time 0.3462 (0.3565) loss 2.4771 (3.0271) grad_norm 1.3563 (1.7824) loss_scale 1024.0000 (1017.3362) mem 11224MB [2024-09-01 07:52:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][470/625] eta 0:00:56 lr 0.000707 wd 0.0500 time 0.3538 (0.3657) data time 0.0008 (0.0073) model time 0.3531 (0.3563) loss 3.4497 (3.0280) grad_norm 1.4311 (1.7806) loss_scale 1024.0000 (1017.4777) mem 11224MB [2024-09-01 07:52:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][480/625] eta 0:00:52 lr 0.000707 wd 0.0500 time 0.3499 (0.3653) data time 0.0009 (0.0072) model time 0.3490 (0.3561) loss 2.9902 (3.0239) grad_norm 1.5142 (1.7824) loss_scale 1024.0000 (1017.6133) mem 11224MB [2024-09-01 07:52:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][490/625] eta 0:00:49 lr 0.000706 wd 0.0500 time 0.3522 (0.3660) data time 0.0006 (0.0070) model time 0.3515 (0.3571) loss 3.2900 (3.0244) grad_norm 1.1918 (1.7878) loss_scale 1024.0000 (1017.7434) mem 11224MB [2024-09-01 07:52:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][500/625] eta 0:00:45 lr 0.000706 wd 0.0500 time 0.3486 (0.3657) data time 0.0006 (0.0069) model time 0.3480 (0.3569) loss 3.9101 (3.0268) grad_norm 1.7068 (1.7874) loss_scale 1024.0000 (1017.8683) mem 11224MB [2024-09-01 07:52:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][510/625] eta 0:00:42 lr 0.000706 wd 0.0500 time 0.3448 (0.3653) data time 0.0006 (0.0068) model time 0.3442 (0.3566) loss 2.0562 (3.0209) grad_norm 1.7457 (1.7892) loss_scale 1024.0000 (1017.9883) mem 11224MB [2024-09-01 07:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][520/625] eta 0:00:38 lr 0.000706 wd 0.0500 time 0.3457 (0.3649) data time 0.0006 (0.0067) model time 0.3451 (0.3564) loss 2.0843 (3.0178) grad_norm 1.8859 (1.7861) loss_scale 1024.0000 (1018.1036) mem 11224MB [2024-09-01 07:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][530/625] eta 0:00:34 lr 0.000706 wd 0.0500 time 0.3465 (0.3646) data time 0.0007 (0.0066) model time 0.3459 (0.3562) loss 3.7441 (3.0189) grad_norm 1.7910 (1.7830) loss_scale 1024.0000 (1018.2147) mem 11224MB [2024-09-01 07:52:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][540/625] eta 0:00:30 lr 0.000706 wd 0.0500 time 0.3483 (0.3644) data time 0.0006 (0.0065) model time 0.3476 (0.3561) loss 2.1411 (3.0130) grad_norm 1.6149 (1.7792) loss_scale 1024.0000 (1018.3216) mem 11224MB [2024-09-01 07:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][550/625] eta 0:00:27 lr 0.000705 wd 0.0500 time 0.3546 (0.3641) data time 0.0008 (0.0064) model time 0.3538 (0.3559) loss 3.1141 (3.0162) grad_norm 1.8569 (1.7800) loss_scale 1024.0000 (1018.4247) mem 11224MB [2024-09-01 07:52:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][560/625] eta 0:00:23 lr 0.000705 wd 0.0500 time 0.3525 (0.3639) data time 0.0006 (0.0063) model time 0.3519 (0.3558) loss 2.4338 (3.0144) grad_norm 1.4166 (1.7802) loss_scale 1024.0000 (1018.5241) mem 11224MB [2024-09-01 07:52:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][570/625] eta 0:00:19 lr 0.000705 wd 0.0500 time 0.3516 (0.3636) data time 0.0007 (0.0062) model time 0.3510 (0.3557) loss 2.2151 (3.0161) grad_norm 1.3064 (1.7805) loss_scale 1024.0000 (1018.6200) mem 11224MB [2024-09-01 07:52:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][580/625] eta 0:00:16 lr 0.000705 wd 0.0500 time 0.3460 (0.3644) data time 0.0008 (0.0061) model time 0.3452 (0.3567) loss 3.1099 (3.0133) grad_norm 1.2902 (1.7757) loss_scale 1024.0000 (1018.7126) mem 11224MB [2024-09-01 07:52:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][590/625] eta 0:00:12 lr 0.000705 wd 0.0500 time 0.3470 (0.3641) data time 0.0008 (0.0060) model time 0.3462 (0.3565) loss 3.3617 (3.0130) grad_norm 1.7425 (1.7799) loss_scale 1024.0000 (1018.8020) mem 11224MB [2024-09-01 07:52:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][600/625] eta 0:00:09 lr 0.000704 wd 0.0500 time 0.3505 (0.3639) data time 0.0009 (0.0059) model time 0.3496 (0.3563) loss 3.3080 (3.0122) grad_norm 3.1916 (1.7818) loss_scale 1024.0000 (1018.8885) mem 11224MB [2024-09-01 07:52:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][610/625] eta 0:00:05 lr 0.000704 wd 0.0500 time 0.3382 (0.3642) data time 0.0006 (0.0058) model time 0.3377 (0.3568) loss 2.6185 (3.0163) grad_norm 1.2148 (1.7849) loss_scale 1024.0000 (1018.9722) mem 11224MB [2024-09-01 07:53:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [187/300][620/625] eta 0:00:01 lr 0.000704 wd 0.0500 time 0.3367 (0.3638) data time 0.0004 (0.0057) model time 0.3363 (0.3565) loss 2.2079 (3.0176) grad_norm 1.3009 (1.7896) loss_scale 1024.0000 (1019.0531) mem 11224MB [2024-09-01 07:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 187 training takes 0:03:47 [2024-09-01 07:53:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:53:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:53:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.450 (0.450) Loss 0.5601 (0.5601) Acc@1 87.842 (87.842) Acc@5 98.242 (98.242) Mem 11224MB [2024-09-01 07:53:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.132) Loss 0.8940 (0.6827) Acc@1 78.906 (84.819) Acc@5 95.361 (97.306) Mem 11224MB [2024-09-01 07:53:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.113) Loss 1.0078 (0.8085) Acc@1 75.195 (81.592) Acc@5 94.482 (95.905) Mem 11224MB [2024-09-01 07:53:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.342 Acc@5 95.899 [2024-09-01 07:53:05 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.3% [2024-09-01 07:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.856 (0.856) Loss 0.5044 (0.5044) Acc@1 89.258 (89.258) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 07:53:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.170) Loss 0.8081 (0.6151) Acc@1 81.201 (86.373) Acc@5 96.045 (97.692) Mem 11224MB [2024-09-01 07:53:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.132) Loss 0.9155 (0.7292) Acc@1 77.295 (83.387) Acc@5 95.020 (96.508) Mem 11224MB [2024-09-01 07:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.081 Acc@5 96.523 [2024-09-01 07:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 07:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][0/625] eta 0:12:11 lr 0.000704 wd 0.0500 time 1.1698 (1.1698) data time 0.5244 (0.5244) model time 0.0000 (0.0000) loss 2.3783 (2.3783) grad_norm 1.2022 (1.2022) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][10/625] eta 0:04:21 lr 0.000704 wd 0.0500 time 0.3576 (0.4258) data time 0.0006 (0.0484) model time 0.0000 (0.0000) loss 3.5534 (3.0639) grad_norm 1.7231 (2.0359) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][20/625] eta 0:03:55 lr 0.000704 wd 0.0500 time 0.3510 (0.3888) data time 0.0008 (0.0258) model time 0.0000 (0.0000) loss 2.2514 (2.9193) grad_norm 1.6509 (1.9174) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][30/625] eta 0:03:43 lr 0.000704 wd 0.0500 time 0.3469 (0.3750) data time 0.0007 (0.0177) model time 0.0000 (0.0000) loss 3.2146 (2.9709) grad_norm 1.5946 (1.8251) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][40/625] eta 0:03:38 lr 0.000703 wd 0.0500 time 0.5370 (0.3729) data time 0.0007 (0.0136) model time 0.0000 (0.0000) loss 2.0565 (3.0096) grad_norm 1.6156 (1.8068) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][50/625] eta 0:03:31 lr 0.000703 wd 0.0500 time 0.3472 (0.3670) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 2.9495 (2.9579) grad_norm 1.2909 (1.8141) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][60/625] eta 0:03:25 lr 0.000703 wd 0.0500 time 0.3456 (0.3639) data time 0.0006 (0.0094) model time 0.3450 (0.3473) loss 2.4760 (2.9516) grad_norm 1.6644 (1.7694) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][70/625] eta 0:03:20 lr 0.000703 wd 0.0500 time 0.3441 (0.3617) data time 0.0009 (0.0082) model time 0.3432 (0.3473) loss 2.9689 (2.9426) grad_norm 1.8071 (1.7777) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][80/625] eta 0:03:16 lr 0.000703 wd 0.0500 time 0.3481 (0.3601) data time 0.0009 (0.0073) model time 0.3472 (0.3475) loss 2.4703 (2.9555) grad_norm 1.3419 (1.7628) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][90/625] eta 0:03:11 lr 0.000703 wd 0.0500 time 0.3587 (0.3587) data time 0.0009 (0.0066) model time 0.3579 (0.3473) loss 3.0576 (2.9872) grad_norm 1.5402 (1.7921) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][100/625] eta 0:03:07 lr 0.000702 wd 0.0500 time 0.3452 (0.3575) data time 0.0006 (0.0060) model time 0.3446 (0.3470) loss 3.6673 (2.9602) grad_norm 1.6638 (1.7967) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][110/625] eta 0:03:03 lr 0.000702 wd 0.0500 time 0.3469 (0.3566) data time 0.0007 (0.0055) model time 0.3462 (0.3470) loss 1.9311 (2.9583) grad_norm 1.4245 (1.7824) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][120/625] eta 0:02:59 lr 0.000702 wd 0.0500 time 0.3512 (0.3559) data time 0.0006 (0.0052) model time 0.3506 (0.3470) loss 3.2836 (2.9626) grad_norm 2.2426 (1.8067) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][130/625] eta 0:02:55 lr 0.000702 wd 0.0500 time 0.3431 (0.3552) data time 0.0008 (0.0048) model time 0.3423 (0.3469) loss 3.1487 (2.9676) grad_norm 1.6324 (1.8551) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][140/625] eta 0:02:51 lr 0.000702 wd 0.0500 time 0.3514 (0.3546) data time 0.0006 (0.0045) model time 0.3508 (0.3468) loss 3.5712 (2.9821) grad_norm 2.4952 (1.8547) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][150/625] eta 0:02:48 lr 0.000702 wd 0.0500 time 0.5252 (0.3553) data time 0.0007 (0.0043) model time 0.5245 (0.3485) loss 2.5228 (2.9816) grad_norm 2.6316 (1.8821) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][160/625] eta 0:02:47 lr 0.000701 wd 0.0500 time 0.3460 (0.3598) data time 0.0009 (0.0041) model time 0.3451 (0.3556) loss 3.4254 (2.9956) grad_norm 1.2125 (1.8628) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][170/625] eta 0:02:44 lr 0.000701 wd 0.0500 time 0.3466 (0.3619) data time 0.0009 (0.0039) model time 0.3457 (0.3590) loss 3.0672 (3.0243) grad_norm 1.5964 (1.8501) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][180/625] eta 0:02:40 lr 0.000701 wd 0.0500 time 0.3491 (0.3612) data time 0.0006 (0.0037) model time 0.3485 (0.3581) loss 2.0378 (3.0263) grad_norm 1.4030 (1.8370) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][190/625] eta 0:02:36 lr 0.000701 wd 0.0500 time 0.3443 (0.3605) data time 0.0008 (0.0036) model time 0.3435 (0.3573) loss 2.8414 (3.0179) grad_norm 1.4949 (1.8201) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][200/625] eta 0:02:32 lr 0.000701 wd 0.0500 time 0.3438 (0.3597) data time 0.0007 (0.0034) model time 0.3431 (0.3565) loss 2.9703 (3.0109) grad_norm 2.1879 (1.8291) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][210/625] eta 0:02:29 lr 0.000701 wd 0.0500 time 0.3463 (0.3592) data time 0.0008 (0.0033) model time 0.3455 (0.3559) loss 3.1636 (3.0139) grad_norm 2.0236 (1.8474) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][220/625] eta 0:02:25 lr 0.000700 wd 0.0500 time 0.3471 (0.3586) data time 0.0006 (0.0032) model time 0.3464 (0.3553) loss 2.7370 (3.0173) grad_norm 2.0897 (1.8486) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][230/625] eta 0:02:21 lr 0.000700 wd 0.0500 time 0.3464 (0.3586) data time 0.0006 (0.0031) model time 0.3458 (0.3554) loss 2.7126 (3.0139) grad_norm 2.1239 (1.8483) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][240/625] eta 0:02:17 lr 0.000700 wd 0.0500 time 0.3510 (0.3581) data time 0.0008 (0.0030) model time 0.3502 (0.3549) loss 3.2794 (3.0215) grad_norm 1.6084 (1.8406) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][250/625] eta 0:02:14 lr 0.000700 wd 0.0500 time 0.3438 (0.3576) data time 0.0008 (0.0029) model time 0.3431 (0.3544) loss 3.3208 (3.0181) grad_norm 2.5056 (1.8532) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][260/625] eta 0:02:10 lr 0.000700 wd 0.0500 time 0.3495 (0.3572) data time 0.0009 (0.0028) model time 0.3487 (0.3540) loss 3.2974 (3.0216) grad_norm 2.0180 (1.8445) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][270/625] eta 0:02:06 lr 0.000700 wd 0.0500 time 0.3529 (0.3568) data time 0.0006 (0.0028) model time 0.3523 (0.3537) loss 3.5073 (3.0248) grad_norm 1.4950 (1.8607) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][280/625] eta 0:02:02 lr 0.000699 wd 0.0500 time 0.3484 (0.3565) data time 0.0009 (0.0027) model time 0.3476 (0.3533) loss 3.0747 (3.0287) grad_norm 1.5664 (1.8539) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][290/625] eta 0:01:59 lr 0.000699 wd 0.0500 time 0.3414 (0.3575) data time 0.0007 (0.0026) model time 0.3407 (0.3546) loss 2.9492 (3.0333) grad_norm 1.5688 (1.8556) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:54:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][300/625] eta 0:01:56 lr 0.000699 wd 0.0500 time 0.3472 (0.3572) data time 0.0008 (0.0026) model time 0.3463 (0.3544) loss 3.2189 (3.0337) grad_norm 1.6145 (1.8666) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][310/625] eta 0:01:52 lr 0.000699 wd 0.0500 time 0.3506 (0.3569) data time 0.0006 (0.0025) model time 0.3500 (0.3541) loss 2.3799 (3.0369) grad_norm 1.2180 (1.8599) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][320/625] eta 0:01:48 lr 0.000699 wd 0.0500 time 0.3470 (0.3566) data time 0.0008 (0.0025) model time 0.3462 (0.3539) loss 3.3668 (3.0427) grad_norm 1.4573 (1.8501) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][330/625] eta 0:01:45 lr 0.000699 wd 0.0500 time 0.3470 (0.3564) data time 0.0008 (0.0024) model time 0.3461 (0.3536) loss 3.5426 (3.0535) grad_norm 1.4617 (1.8389) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][340/625] eta 0:01:42 lr 0.000698 wd 0.0500 time 0.3564 (0.3584) data time 0.0006 (0.0024) model time 0.3558 (0.3560) loss 3.4257 (3.0611) grad_norm 1.7271 (1.8636) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][350/625] eta 0:01:39 lr 0.000698 wd 0.0500 time 0.3464 (0.3602) data time 0.0008 (0.0023) model time 0.3455 (0.3583) loss 3.1711 (3.0657) grad_norm 2.3459 (1.8613) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][360/625] eta 0:01:35 lr 0.000698 wd 0.0500 time 0.3509 (0.3599) data time 0.0006 (0.0023) model time 0.3503 (0.3579) loss 3.1070 (3.0602) grad_norm 1.8708 (1.8598) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][370/625] eta 0:01:31 lr 0.000698 wd 0.0500 time 0.3590 (0.3596) data time 0.0007 (0.0022) model time 0.3584 (0.3576) loss 3.4813 (3.0627) grad_norm 1.9908 (1.8646) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][380/625] eta 0:01:28 lr 0.000698 wd 0.0500 time 0.3441 (0.3593) data time 0.0006 (0.0022) model time 0.3435 (0.3573) loss 3.1889 (3.0599) grad_norm 1.7217 (1.8647) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][390/625] eta 0:01:24 lr 0.000697 wd 0.0500 time 0.3449 (0.3590) data time 0.0008 (0.0022) model time 0.3441 (0.3570) loss 3.1236 (3.0588) grad_norm 1.3670 (1.8627) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][400/625] eta 0:01:20 lr 0.000697 wd 0.0500 time 0.3469 (0.3588) data time 0.0008 (0.0021) model time 0.3461 (0.3568) loss 3.2374 (3.0582) grad_norm 1.1751 (1.8619) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][410/625] eta 0:01:17 lr 0.000697 wd 0.0500 time 0.3461 (0.3585) data time 0.0007 (0.0021) model time 0.3454 (0.3565) loss 2.9984 (3.0552) grad_norm 1.4368 (1.8642) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][420/625] eta 0:01:13 lr 0.000697 wd 0.0500 time 0.3463 (0.3583) data time 0.0009 (0.0021) model time 0.3454 (0.3563) loss 3.1567 (3.0524) grad_norm 1.4459 (1.8613) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][430/625] eta 0:01:09 lr 0.000697 wd 0.0500 time 0.3545 (0.3581) data time 0.0008 (0.0020) model time 0.3537 (0.3561) loss 2.6257 (3.0482) grad_norm 1.0430 (1.8550) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][440/625] eta 0:01:06 lr 0.000697 wd 0.0500 time 0.3447 (0.3579) data time 0.0009 (0.0020) model time 0.3438 (0.3559) loss 2.4130 (3.0472) grad_norm 2.1165 (1.8492) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][450/625] eta 0:01:02 lr 0.000696 wd 0.0500 time 0.3465 (0.3577) data time 0.0007 (0.0020) model time 0.3458 (0.3557) loss 3.3422 (3.0466) grad_norm 1.6774 (1.8451) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][460/625] eta 0:00:59 lr 0.000696 wd 0.0500 time 0.3478 (0.3587) data time 0.0007 (0.0020) model time 0.3472 (0.3569) loss 1.8659 (3.0480) grad_norm 1.4760 (1.8408) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][470/625] eta 0:00:55 lr 0.000696 wd 0.0500 time 0.3489 (0.3594) data time 0.0008 (0.0019) model time 0.3481 (0.3576) loss 2.6981 (3.0468) grad_norm 1.8303 (1.8364) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][480/625] eta 0:00:52 lr 0.000696 wd 0.0500 time 0.3464 (0.3591) data time 0.0006 (0.0019) model time 0.3458 (0.3574) loss 3.6332 (3.0497) grad_norm 1.1090 (1.8318) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][490/625] eta 0:00:48 lr 0.000696 wd 0.0500 time 0.5442 (0.3593) data time 0.0008 (0.0019) model time 0.5434 (0.3576) loss 3.5556 (3.0528) grad_norm 1.3807 (1.8310) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][500/625] eta 0:00:44 lr 0.000696 wd 0.0500 time 0.3425 (0.3595) data time 0.0006 (0.0019) model time 0.3419 (0.3579) loss 4.0705 (3.0518) grad_norm 1.3833 (1.8470) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][510/625] eta 0:00:41 lr 0.000695 wd 0.0500 time 0.3457 (0.3593) data time 0.0008 (0.0018) model time 0.3450 (0.3577) loss 2.6663 (3.0433) grad_norm 2.1304 (1.8517) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][520/625] eta 0:00:37 lr 0.000695 wd 0.0500 time 0.3480 (0.3591) data time 0.0006 (0.0018) model time 0.3474 (0.3574) loss 3.9092 (3.0410) grad_norm 2.2004 (1.8567) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][530/625] eta 0:00:34 lr 0.000695 wd 0.0500 time 0.3369 (0.3589) data time 0.0008 (0.0018) model time 0.3361 (0.3572) loss 2.9306 (3.0411) grad_norm 2.7666 (1.8590) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][540/625] eta 0:00:30 lr 0.000695 wd 0.0500 time 0.3482 (0.3587) data time 0.0007 (0.0018) model time 0.3475 (0.3570) loss 3.2654 (3.0438) grad_norm 2.8559 (1.8583) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][550/625] eta 0:00:26 lr 0.000695 wd 0.0500 time 0.3523 (0.3585) data time 0.0006 (0.0018) model time 0.3517 (0.3568) loss 3.5701 (3.0452) grad_norm 1.3089 (1.8536) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][560/625] eta 0:00:23 lr 0.000695 wd 0.0500 time 0.3552 (0.3586) data time 0.0006 (0.0018) model time 0.3546 (0.3569) loss 3.5041 (3.0450) grad_norm 1.8443 (1.8493) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][570/625] eta 0:00:19 lr 0.000694 wd 0.0500 time 0.3458 (0.3584) data time 0.0009 (0.0017) model time 0.3449 (0.3567) loss 3.2634 (3.0508) grad_norm 1.7303 (1.8461) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][580/625] eta 0:00:16 lr 0.000694 wd 0.0500 time 0.3441 (0.3582) data time 0.0007 (0.0017) model time 0.3435 (0.3566) loss 3.6150 (3.0503) grad_norm 1.3914 (1.8446) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][590/625] eta 0:00:12 lr 0.000694 wd 0.0500 time 0.3533 (0.3581) data time 0.0010 (0.0017) model time 0.3523 (0.3564) loss 2.9131 (3.0502) grad_norm 1.6781 (1.8426) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][600/625] eta 0:00:08 lr 0.000694 wd 0.0500 time 0.3461 (0.3579) data time 0.0006 (0.0017) model time 0.3455 (0.3562) loss 3.5705 (3.0500) grad_norm 1.5515 (1.8469) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][610/625] eta 0:00:05 lr 0.000694 wd 0.0500 time 0.3402 (0.3577) data time 0.0004 (0.0017) model time 0.3398 (0.3560) loss 2.0190 (3.0409) grad_norm 3.5231 (1.8484) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [188/300][620/625] eta 0:00:01 lr 0.000694 wd 0.0500 time 0.3430 (0.3574) data time 0.0006 (0.0017) model time 0.3424 (0.3558) loss 3.3758 (3.0448) grad_norm 1.8650 (1.8470) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 188 training takes 0:03:43 [2024-09-01 07:56:52 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 07:56:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 07:56:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.437 (0.437) Loss 0.5483 (0.5483) Acc@1 88.281 (88.281) Acc@5 98.242 (98.242) Mem 11224MB [2024-09-01 07:56:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.131) Loss 0.9131 (0.6833) Acc@1 77.930 (84.703) Acc@5 95.215 (97.257) Mem 11224MB [2024-09-01 07:56:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.0312 (0.8092) Acc@1 75.879 (81.580) Acc@5 93.994 (95.845) Mem 11224MB [2024-09-01 07:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.304 Acc@5 95.861 [2024-09-01 07:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.3% [2024-09-01 07:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.898 (0.898) Loss 0.5034 (0.5034) Acc@1 89.404 (89.404) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 07:56:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.170) Loss 0.8066 (0.6144) Acc@1 81.201 (86.399) Acc@5 95.898 (97.683) Mem 11224MB [2024-09-01 07:56:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.133) Loss 0.9141 (0.7283) Acc@1 77.051 (83.380) Acc@5 95.020 (96.519) Mem 11224MB [2024-09-01 07:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.077 Acc@5 96.529 [2024-09-01 07:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 07:57:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][0/625] eta 0:12:26 lr 0.000694 wd 0.0500 time 1.1945 (1.1945) data time 0.6096 (0.6096) model time 0.0000 (0.0000) loss 3.0641 (3.0641) grad_norm 1.7962 (1.7962) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][10/625] eta 0:04:21 lr 0.000693 wd 0.0500 time 0.3482 (0.4253) data time 0.0009 (0.0562) model time 0.0000 (0.0000) loss 2.4588 (2.7161) grad_norm 1.6991 (1.5627) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][20/625] eta 0:03:54 lr 0.000693 wd 0.0500 time 0.3513 (0.3883) data time 0.0006 (0.0298) model time 0.0000 (0.0000) loss 2.7337 (2.8855) grad_norm 2.1934 (1.6198) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][30/625] eta 0:04:00 lr 0.000693 wd 0.0500 time 0.3449 (0.4045) data time 0.0008 (0.0205) model time 0.0000 (0.0000) loss 2.1691 (2.9021) grad_norm 2.2505 (1.8730) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][40/625] eta 0:03:48 lr 0.000693 wd 0.0500 time 0.3432 (0.3910) data time 0.0009 (0.0157) model time 0.0000 (0.0000) loss 2.2290 (2.9215) grad_norm 1.4557 (1.9024) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][50/625] eta 0:03:39 lr 0.000693 wd 0.0500 time 0.3452 (0.3826) data time 0.0007 (0.0128) model time 0.0000 (0.0000) loss 3.2705 (2.9438) grad_norm 1.3417 (1.8913) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][60/625] eta 0:03:36 lr 0.000693 wd 0.0500 time 0.5403 (0.3825) data time 0.0008 (0.0108) model time 0.5395 (0.3813) loss 3.1054 (2.9481) grad_norm 1.6286 (1.8576) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][70/625] eta 0:03:29 lr 0.000692 wd 0.0500 time 0.3460 (0.3778) data time 0.0009 (0.0094) model time 0.3451 (0.3647) loss 3.1561 (2.8966) grad_norm 1.6976 (1.8457) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][80/625] eta 0:03:23 lr 0.000692 wd 0.0500 time 0.3489 (0.3742) data time 0.0008 (0.0084) model time 0.3481 (0.3591) loss 3.6953 (2.9088) grad_norm 2.3999 (1.8866) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][90/625] eta 0:03:19 lr 0.000692 wd 0.0500 time 0.3131 (0.3726) data time 0.0007 (0.0075) model time 0.3124 (0.3591) loss 2.8571 (2.9196) grad_norm 1.2948 (1.9059) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][100/625] eta 0:03:14 lr 0.000692 wd 0.0500 time 0.3529 (0.3704) data time 0.0006 (0.0069) model time 0.3523 (0.3571) loss 3.6582 (2.9461) grad_norm 1.5217 (1.8599) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][110/625] eta 0:03:09 lr 0.000692 wd 0.0500 time 0.3483 (0.3683) data time 0.0007 (0.0063) model time 0.3476 (0.3554) loss 2.5517 (2.9513) grad_norm 1.4176 (1.8635) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][120/625] eta 0:03:05 lr 0.000692 wd 0.0500 time 0.3486 (0.3667) data time 0.0008 (0.0059) model time 0.3477 (0.3543) loss 3.6395 (2.9398) grad_norm 1.5688 (1.8338) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][130/625] eta 0:03:00 lr 0.000691 wd 0.0500 time 0.3536 (0.3653) data time 0.0007 (0.0055) model time 0.3530 (0.3534) loss 3.6879 (2.9611) grad_norm 2.3461 (1.8460) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][140/625] eta 0:02:56 lr 0.000691 wd 0.0500 time 0.3454 (0.3643) data time 0.0007 (0.0051) model time 0.3447 (0.3531) loss 3.2658 (2.9771) grad_norm 1.7005 (1.8457) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][150/625] eta 0:02:52 lr 0.000691 wd 0.0500 time 0.3551 (0.3632) data time 0.0008 (0.0049) model time 0.3543 (0.3525) loss 2.7516 (2.9708) grad_norm 1.6476 (1.8477) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 07:57:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][160/625] eta 0:02:48 lr 0.000691 wd 0.0500 time 0.3505 (0.3622) data time 0.0007 (0.0046) model time 0.3498 (0.3520) loss 2.5354 (2.9631) grad_norm 1.3862 (inf) loss_scale 512.0000 (1001.7391) mem 11224MB [2024-09-01 07:58:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][170/625] eta 0:02:44 lr 0.000691 wd 0.0500 time 0.3469 (0.3613) data time 0.0009 (0.0044) model time 0.3460 (0.3515) loss 2.2878 (2.9689) grad_norm 1.6719 (inf) loss_scale 512.0000 (973.0994) mem 11224MB [2024-09-01 07:58:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][180/625] eta 0:02:40 lr 0.000690 wd 0.0500 time 0.3503 (0.3606) data time 0.0006 (0.0042) model time 0.3497 (0.3511) loss 2.3322 (2.9613) grad_norm 1.9485 (inf) loss_scale 512.0000 (947.6243) mem 11224MB [2024-09-01 07:58:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][190/625] eta 0:02:36 lr 0.000690 wd 0.0500 time 0.3460 (0.3599) data time 0.0007 (0.0040) model time 0.3454 (0.3509) loss 3.1429 (2.9745) grad_norm 1.8254 (inf) loss_scale 512.0000 (924.8168) mem 11224MB [2024-09-01 07:58:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][200/625] eta 0:02:32 lr 0.000690 wd 0.0500 time 0.3509 (0.3594) data time 0.0006 (0.0039) model time 0.3502 (0.3507) loss 3.5674 (2.9658) grad_norm 3.4487 (inf) loss_scale 512.0000 (904.2786) mem 11224MB [2024-09-01 07:58:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][210/625] eta 0:02:29 lr 0.000690 wd 0.0500 time 0.3117 (0.3596) data time 0.0009 (0.0037) model time 0.3108 (0.3514) loss 2.9366 (2.9743) grad_norm 1.9966 (inf) loss_scale 512.0000 (885.6872) mem 11224MB [2024-09-01 07:58:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][220/625] eta 0:02:25 lr 0.000690 wd 0.0500 time 0.3518 (0.3591) data time 0.0006 (0.0036) model time 0.3512 (0.3512) loss 3.1006 (2.9857) grad_norm 2.6103 (inf) loss_scale 512.0000 (868.7783) mem 11224MB [2024-09-01 07:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][230/625] eta 0:02:21 lr 0.000690 wd 0.0500 time 0.3520 (0.3586) data time 0.0009 (0.0035) model time 0.3511 (0.3510) loss 3.1709 (2.9975) grad_norm 2.5188 (inf) loss_scale 512.0000 (853.3333) mem 11224MB [2024-09-01 07:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][240/625] eta 0:02:19 lr 0.000689 wd 0.0500 time 0.3449 (0.3613) data time 0.0007 (0.0034) model time 0.3442 (0.3547) loss 2.8498 (2.9963) grad_norm 2.0646 (inf) loss_scale 512.0000 (839.1701) mem 11224MB [2024-09-01 07:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][250/625] eta 0:02:15 lr 0.000689 wd 0.0500 time 0.3424 (0.3620) data time 0.0008 (0.0033) model time 0.3416 (0.3559) loss 3.2748 (2.9981) grad_norm 1.6124 (inf) loss_scale 512.0000 (826.1355) mem 11224MB [2024-09-01 07:58:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][260/625] eta 0:02:11 lr 0.000689 wd 0.0500 time 0.3469 (0.3615) data time 0.0010 (0.0032) model time 0.3459 (0.3555) loss 2.6187 (3.0044) grad_norm 1.9195 (inf) loss_scale 512.0000 (814.0996) mem 11224MB [2024-09-01 07:58:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][270/625] eta 0:02:08 lr 0.000689 wd 0.0500 time 0.3460 (0.3609) data time 0.0007 (0.0031) model time 0.3453 (0.3551) loss 3.7143 (3.0051) grad_norm 2.0810 (inf) loss_scale 512.0000 (802.9520) mem 11224MB [2024-09-01 07:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][280/625] eta 0:02:04 lr 0.000689 wd 0.0500 time 0.3448 (0.3605) data time 0.0006 (0.0030) model time 0.3442 (0.3547) loss 3.7724 (3.0018) grad_norm 1.4738 (inf) loss_scale 512.0000 (792.5979) mem 11224MB [2024-09-01 07:58:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][290/625] eta 0:02:00 lr 0.000689 wd 0.0500 time 0.3555 (0.3600) data time 0.0009 (0.0029) model time 0.3547 (0.3544) loss 3.0243 (2.9959) grad_norm 1.8384 (inf) loss_scale 512.0000 (782.9553) mem 11224MB [2024-09-01 07:58:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][300/625] eta 0:01:56 lr 0.000688 wd 0.0500 time 0.3480 (0.3596) data time 0.0010 (0.0029) model time 0.3470 (0.3540) loss 2.2876 (2.9943) grad_norm 4.1075 (inf) loss_scale 512.0000 (773.9535) mem 11224MB [2024-09-01 07:58:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][310/625] eta 0:01:53 lr 0.000688 wd 0.0500 time 0.3470 (0.3591) data time 0.0008 (0.0028) model time 0.3462 (0.3537) loss 2.6884 (2.9955) grad_norm 2.3457 (inf) loss_scale 512.0000 (765.5305) mem 11224MB [2024-09-01 07:58:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][320/625] eta 0:01:49 lr 0.000688 wd 0.0500 time 0.3485 (0.3588) data time 0.0008 (0.0027) model time 0.3477 (0.3534) loss 3.1197 (2.9976) grad_norm 1.8412 (inf) loss_scale 512.0000 (757.6324) mem 11224MB [2024-09-01 07:58:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][330/625] eta 0:01:45 lr 0.000688 wd 0.0500 time 0.3527 (0.3585) data time 0.0008 (0.0027) model time 0.3519 (0.3532) loss 3.2571 (3.0020) grad_norm 1.3155 (inf) loss_scale 512.0000 (750.2115) mem 11224MB [2024-09-01 07:59:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][340/625] eta 0:01:42 lr 0.000688 wd 0.0500 time 0.3500 (0.3582) data time 0.0006 (0.0026) model time 0.3494 (0.3530) loss 2.1117 (3.0094) grad_norm 2.0307 (inf) loss_scale 512.0000 (743.2258) mem 11224MB [2024-09-01 07:59:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][350/625] eta 0:01:38 lr 0.000688 wd 0.0500 time 0.3451 (0.3579) data time 0.0010 (0.0026) model time 0.3441 (0.3528) loss 2.7680 (3.0099) grad_norm 2.4253 (inf) loss_scale 512.0000 (736.6382) mem 11224MB [2024-09-01 07:59:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][360/625] eta 0:01:34 lr 0.000687 wd 0.0500 time 0.3477 (0.3581) data time 0.0009 (0.0025) model time 0.3468 (0.3532) loss 3.0125 (3.0060) grad_norm 1.9195 (inf) loss_scale 512.0000 (730.4155) mem 11224MB [2024-09-01 07:59:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][370/625] eta 0:01:31 lr 0.000687 wd 0.0500 time 0.3484 (0.3583) data time 0.0006 (0.0025) model time 0.3478 (0.3536) loss 4.0025 (3.0145) grad_norm 1.4936 (inf) loss_scale 512.0000 (724.5283) mem 11224MB [2024-09-01 07:59:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][380/625] eta 0:01:27 lr 0.000687 wd 0.0500 time 0.3440 (0.3581) data time 0.0008 (0.0025) model time 0.3431 (0.3535) loss 2.7408 (3.0115) grad_norm 2.1437 (inf) loss_scale 512.0000 (718.9501) mem 11224MB [2024-09-01 07:59:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][390/625] eta 0:01:24 lr 0.000687 wd 0.0500 time 0.3507 (0.3578) data time 0.0008 (0.0024) model time 0.3499 (0.3533) loss 3.0621 (3.0104) grad_norm 2.2545 (inf) loss_scale 512.0000 (713.6573) mem 11224MB [2024-09-01 07:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][400/625] eta 0:01:20 lr 0.000687 wd 0.0500 time 0.3547 (0.3577) data time 0.0009 (0.0024) model time 0.3538 (0.3532) loss 3.2349 (3.0039) grad_norm 2.1928 (inf) loss_scale 512.0000 (708.6284) mem 11224MB [2024-09-01 07:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][410/625] eta 0:01:16 lr 0.000687 wd 0.0500 time 0.3450 (0.3574) data time 0.0009 (0.0023) model time 0.3441 (0.3530) loss 3.0173 (3.0034) grad_norm 2.4423 (inf) loss_scale 512.0000 (703.8443) mem 11224MB [2024-09-01 07:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][420/625] eta 0:01:13 lr 0.000686 wd 0.0500 time 0.3442 (0.3590) data time 0.0007 (0.0023) model time 0.3435 (0.3549) loss 2.1308 (3.0076) grad_norm 2.0316 (inf) loss_scale 512.0000 (699.2874) mem 11224MB [2024-09-01 07:59:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][430/625] eta 0:01:10 lr 0.000686 wd 0.0500 time 0.3488 (0.3593) data time 0.0006 (0.0023) model time 0.3482 (0.3553) loss 2.6265 (3.0086) grad_norm 1.7604 (inf) loss_scale 512.0000 (694.9420) mem 11224MB [2024-09-01 07:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][440/625] eta 0:01:06 lr 0.000686 wd 0.0500 time 0.3442 (0.3590) data time 0.0008 (0.0022) model time 0.3435 (0.3551) loss 3.2965 (3.0055) grad_norm 2.9532 (inf) loss_scale 512.0000 (690.7937) mem 11224MB [2024-09-01 07:59:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][450/625] eta 0:01:02 lr 0.000686 wd 0.0500 time 0.3486 (0.3588) data time 0.0009 (0.0022) model time 0.3477 (0.3549) loss 3.5103 (3.0078) grad_norm 1.4863 (inf) loss_scale 512.0000 (686.8293) mem 11224MB [2024-09-01 07:59:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][460/625] eta 0:00:59 lr 0.000686 wd 0.0500 time 0.3505 (0.3586) data time 0.0007 (0.0022) model time 0.3499 (0.3548) loss 3.2438 (3.0077) grad_norm 1.8616 (inf) loss_scale 512.0000 (683.0369) mem 11224MB [2024-09-01 07:59:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][470/625] eta 0:00:55 lr 0.000686 wd 0.0500 time 0.3560 (0.3584) data time 0.0007 (0.0021) model time 0.3553 (0.3546) loss 2.1678 (3.0059) grad_norm 1.2723 (inf) loss_scale 512.0000 (679.4055) mem 11224MB [2024-09-01 07:59:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][480/625] eta 0:00:51 lr 0.000685 wd 0.0500 time 0.3480 (0.3582) data time 0.0006 (0.0021) model time 0.3474 (0.3544) loss 3.6916 (3.0052) grad_norm 2.0559 (inf) loss_scale 512.0000 (675.9252) mem 11224MB [2024-09-01 07:59:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][490/625] eta 0:00:48 lr 0.000685 wd 0.0500 time 0.3427 (0.3579) data time 0.0008 (0.0021) model time 0.3418 (0.3542) loss 2.7420 (3.0056) grad_norm 1.5529 (inf) loss_scale 512.0000 (672.5866) mem 11224MB [2024-09-01 07:59:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][500/625] eta 0:00:44 lr 0.000685 wd 0.0500 time 0.3482 (0.3578) data time 0.0007 (0.0021) model time 0.3475 (0.3541) loss 2.8014 (3.0047) grad_norm 2.1347 (inf) loss_scale 512.0000 (669.3812) mem 11224MB [2024-09-01 08:00:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][510/625] eta 0:00:41 lr 0.000685 wd 0.0500 time 0.3532 (0.3576) data time 0.0009 (0.0020) model time 0.3523 (0.3540) loss 2.8958 (3.0075) grad_norm 1.2016 (inf) loss_scale 512.0000 (666.3014) mem 11224MB [2024-09-01 08:00:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][520/625] eta 0:00:37 lr 0.000685 wd 0.0500 time 0.3482 (0.3575) data time 0.0009 (0.0020) model time 0.3473 (0.3539) loss 3.2152 (3.0112) grad_norm 1.8735 (inf) loss_scale 512.0000 (663.3397) mem 11224MB [2024-09-01 08:00:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][530/625] eta 0:00:33 lr 0.000685 wd 0.0500 time 0.3457 (0.3573) data time 0.0007 (0.0020) model time 0.3450 (0.3538) loss 2.5474 (3.0103) grad_norm 1.3585 (inf) loss_scale 512.0000 (660.4896) mem 11224MB [2024-09-01 08:00:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][540/625] eta 0:00:30 lr 0.000684 wd 0.0500 time 0.5571 (0.3583) data time 0.0006 (0.0020) model time 0.5565 (0.3549) loss 2.7477 (3.0109) grad_norm 1.6285 (inf) loss_scale 512.0000 (657.7449) mem 11224MB [2024-09-01 08:00:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][550/625] eta 0:00:26 lr 0.000684 wd 0.0500 time 0.3461 (0.3581) data time 0.0007 (0.0020) model time 0.3454 (0.3548) loss 3.6529 (3.0179) grad_norm 1.5234 (inf) loss_scale 512.0000 (655.0998) mem 11224MB [2024-09-01 08:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][560/625] eta 0:00:23 lr 0.000684 wd 0.0500 time 0.3544 (0.3579) data time 0.0008 (0.0019) model time 0.3536 (0.3546) loss 3.3522 (3.0160) grad_norm 1.8920 (inf) loss_scale 512.0000 (652.5490) mem 11224MB [2024-09-01 08:00:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][570/625] eta 0:00:19 lr 0.000684 wd 0.0500 time 0.3480 (0.3580) data time 0.0009 (0.0019) model time 0.3471 (0.3548) loss 3.0623 (3.0172) grad_norm 2.0991 (inf) loss_scale 512.0000 (650.0876) mem 11224MB [2024-09-01 08:00:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][580/625] eta 0:00:16 lr 0.000684 wd 0.0500 time 0.3494 (0.3582) data time 0.0006 (0.0019) model time 0.3488 (0.3550) loss 2.5219 (3.0102) grad_norm 1.6754 (inf) loss_scale 512.0000 (647.7108) mem 11224MB [2024-09-01 08:00:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][590/625] eta 0:00:12 lr 0.000684 wd 0.0500 time 0.3473 (0.3580) data time 0.0009 (0.0019) model time 0.3464 (0.3548) loss 2.0601 (3.0092) grad_norm 1.9933 (inf) loss_scale 512.0000 (645.4146) mem 11224MB [2024-09-01 08:00:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][600/625] eta 0:00:08 lr 0.000683 wd 0.0500 time 0.3488 (0.3579) data time 0.0009 (0.0019) model time 0.3479 (0.3547) loss 3.6659 (3.0119) grad_norm 2.4424 (inf) loss_scale 512.0000 (643.1947) mem 11224MB [2024-09-01 08:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][610/625] eta 0:00:05 lr 0.000683 wd 0.0500 time 0.3459 (0.3580) data time 0.0004 (0.0019) model time 0.3455 (0.3549) loss 4.1560 (3.0131) grad_norm 2.0114 (inf) loss_scale 512.0000 (641.0475) mem 11224MB [2024-09-01 08:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [189/300][620/625] eta 0:00:01 lr 0.000683 wd 0.0500 time 0.3441 (0.3577) data time 0.0006 (0.0018) model time 0.3436 (0.3546) loss 3.3064 (3.0162) grad_norm 1.6415 (inf) loss_scale 512.0000 (638.9694) mem 11224MB [2024-09-01 08:00:42 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 189 training takes 0:03:43 [2024-09-01 08:00:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:00:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:00:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.452 (0.452) Loss 0.5220 (0.5220) Acc@1 88.818 (88.818) Acc@5 98.291 (98.291) Mem 11224MB [2024-09-01 08:00:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.9229 (0.6721) Acc@1 78.516 (84.854) Acc@5 94.727 (97.283) Mem 11224MB [2024-09-01 08:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.9678 (0.7998) Acc@1 76.758 (81.745) Acc@5 94.824 (95.936) Mem 11224MB [2024-09-01 08:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.428 Acc@5 95.915 [2024-09-01 08:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.4% [2024-09-01 08:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.43% [2024-09-01 08:00:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:00:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:00:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.438 (0.438) Loss 0.5029 (0.5029) Acc@1 89.111 (89.111) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 08:00:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.8066 (0.6135) Acc@1 81.201 (86.364) Acc@5 95.898 (97.678) Mem 11224MB [2024-09-01 08:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.9121 (0.7272) Acc@1 77.246 (83.380) Acc@5 95.117 (96.515) Mem 11224MB [2024-09-01 08:00:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.083 Acc@5 96.535 [2024-09-01 08:00:50 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 08:00:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][0/625] eta 0:12:18 lr 0.000683 wd 0.0500 time 1.1818 (1.1818) data time 0.5154 (0.5154) model time 0.0000 (0.0000) loss 3.1256 (3.1256) grad_norm 1.5739 (1.5739) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:00:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][10/625] eta 0:04:21 lr 0.000683 wd 0.0500 time 0.3503 (0.4253) data time 0.0008 (0.0476) model time 0.0000 (0.0000) loss 3.3692 (3.0114) grad_norm 1.7547 (2.0837) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:00:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][20/625] eta 0:03:54 lr 0.000683 wd 0.0500 time 0.3474 (0.3882) data time 0.0007 (0.0253) model time 0.0000 (0.0000) loss 2.4777 (3.0653) grad_norm 1.6026 (1.8442) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][30/625] eta 0:03:42 lr 0.000683 wd 0.0500 time 0.3465 (0.3748) data time 0.0007 (0.0175) model time 0.0000 (0.0000) loss 3.5775 (3.1060) grad_norm 2.5274 (1.8887) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][40/625] eta 0:03:35 lr 0.000682 wd 0.0500 time 0.3447 (0.3679) data time 0.0009 (0.0134) model time 0.0000 (0.0000) loss 3.3669 (3.1408) grad_norm 2.1631 (1.8788) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][50/625] eta 0:03:29 lr 0.000682 wd 0.0500 time 0.3461 (0.3642) data time 0.0007 (0.0109) model time 0.0000 (0.0000) loss 3.2074 (3.1643) grad_norm 1.2134 (1.8199) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][60/625] eta 0:03:24 lr 0.000682 wd 0.0500 time 0.3479 (0.3614) data time 0.0007 (0.0093) model time 0.3472 (0.3459) loss 1.9959 (3.1180) grad_norm 1.5554 (1.8125) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][70/625] eta 0:03:24 lr 0.000682 wd 0.0500 time 0.5684 (0.3677) data time 0.0006 (0.0081) model time 0.5679 (0.3758) loss 2.7057 (3.0936) grad_norm 1.4397 (1.7987) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][80/625] eta 0:03:19 lr 0.000682 wd 0.0500 time 0.3403 (0.3652) data time 0.0007 (0.0072) model time 0.3397 (0.3661) loss 2.0320 (3.0615) grad_norm 1.4967 (1.8632) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][90/625] eta 0:03:14 lr 0.000682 wd 0.0500 time 0.3488 (0.3633) data time 0.0008 (0.0065) model time 0.3479 (0.3613) loss 2.7937 (3.0459) grad_norm 1.7618 (1.8697) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][100/625] eta 0:03:10 lr 0.000681 wd 0.0500 time 0.3487 (0.3630) data time 0.0008 (0.0059) model time 0.3479 (0.3610) loss 3.1474 (3.0209) grad_norm 1.5906 (1.8439) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][110/625] eta 0:03:06 lr 0.000681 wd 0.0500 time 0.3561 (0.3620) data time 0.0009 (0.0055) model time 0.3553 (0.3592) loss 2.4934 (3.0105) grad_norm 1.7231 (1.8292) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][120/625] eta 0:03:02 lr 0.000681 wd 0.0500 time 0.3462 (0.3608) data time 0.0008 (0.0051) model time 0.3454 (0.3576) loss 3.0136 (3.0070) grad_norm 2.9070 (1.8451) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][130/625] eta 0:02:58 lr 0.000681 wd 0.0500 time 0.3479 (0.3599) data time 0.0006 (0.0048) model time 0.3473 (0.3563) loss 3.1200 (3.0105) grad_norm 3.5281 (1.8652) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][140/625] eta 0:02:54 lr 0.000681 wd 0.0500 time 0.3471 (0.3591) data time 0.0007 (0.0045) model time 0.3464 (0.3553) loss 1.9889 (2.9958) grad_norm 1.9602 (1.8602) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][150/625] eta 0:02:51 lr 0.000681 wd 0.0500 time 0.5330 (0.3606) data time 0.0006 (0.0042) model time 0.5323 (0.3579) loss 3.1016 (2.9896) grad_norm 1.6197 (1.8388) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][160/625] eta 0:02:47 lr 0.000680 wd 0.0500 time 0.3491 (0.3597) data time 0.0009 (0.0040) model time 0.3483 (0.3568) loss 3.2447 (3.0026) grad_norm 2.1346 (1.8218) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][170/625] eta 0:02:43 lr 0.000680 wd 0.0500 time 0.3452 (0.3590) data time 0.0008 (0.0038) model time 0.3444 (0.3560) loss 2.3454 (3.0066) grad_norm 7.3048 (1.8446) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][180/625] eta 0:02:39 lr 0.000680 wd 0.0500 time 0.3474 (0.3583) data time 0.0009 (0.0037) model time 0.3465 (0.3552) loss 3.9143 (3.0153) grad_norm 1.9221 (1.8545) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:01:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][190/625] eta 0:02:35 lr 0.000680 wd 0.0500 time 0.3480 (0.3578) data time 0.0009 (0.0035) model time 0.3471 (0.3546) loss 2.7762 (3.0018) grad_norm 1.5894 (1.8510) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][200/625] eta 0:02:31 lr 0.000680 wd 0.0500 time 0.3517 (0.3573) data time 0.0008 (0.0034) model time 0.3509 (0.3540) loss 2.8338 (3.0019) grad_norm 2.3039 (1.8488) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][210/625] eta 0:02:28 lr 0.000680 wd 0.0500 time 0.3514 (0.3569) data time 0.0007 (0.0033) model time 0.3507 (0.3537) loss 3.6024 (2.9925) grad_norm 1.5094 (1.8580) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][220/625] eta 0:02:24 lr 0.000679 wd 0.0500 time 0.3484 (0.3572) data time 0.0007 (0.0032) model time 0.3477 (0.3543) loss 3.7574 (2.9976) grad_norm 1.4966 (1.8510) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][230/625] eta 0:02:20 lr 0.000679 wd 0.0500 time 0.3507 (0.3569) data time 0.0008 (0.0031) model time 0.3499 (0.3540) loss 2.6379 (3.0099) grad_norm 1.7127 (1.8402) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][240/625] eta 0:02:17 lr 0.000679 wd 0.0500 time 0.3500 (0.3566) data time 0.0008 (0.0030) model time 0.3491 (0.3537) loss 3.5677 (3.0257) grad_norm 1.5238 (1.8633) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][250/625] eta 0:02:13 lr 0.000679 wd 0.0500 time 0.3447 (0.3563) data time 0.0009 (0.0029) model time 0.3439 (0.3534) loss 3.4872 (3.0302) grad_norm 1.7864 (1.8709) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][260/625] eta 0:02:09 lr 0.000679 wd 0.0500 time 0.3650 (0.3560) data time 0.0007 (0.0028) model time 0.3643 (0.3532) loss 3.5947 (3.0320) grad_norm 1.3101 (1.8635) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][270/625] eta 0:02:06 lr 0.000678 wd 0.0500 time 0.3476 (0.3557) data time 0.0006 (0.0027) model time 0.3470 (0.3528) loss 3.5048 (3.0323) grad_norm 1.6920 (1.8641) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][280/625] eta 0:02:03 lr 0.000678 wd 0.0500 time 0.3460 (0.3568) data time 0.0009 (0.0027) model time 0.3450 (0.3543) loss 3.1773 (3.0361) grad_norm 2.1194 (1.8530) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][290/625] eta 0:01:59 lr 0.000678 wd 0.0500 time 0.3488 (0.3566) data time 0.0006 (0.0026) model time 0.3482 (0.3541) loss 3.5722 (3.0335) grad_norm 2.0541 (1.8519) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][300/625] eta 0:01:55 lr 0.000678 wd 0.0500 time 0.3520 (0.3563) data time 0.0006 (0.0026) model time 0.3513 (0.3539) loss 3.3813 (3.0391) grad_norm 2.7068 (1.8517) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][310/625] eta 0:01:52 lr 0.000678 wd 0.0500 time 0.3481 (0.3561) data time 0.0008 (0.0025) model time 0.3473 (0.3536) loss 3.5006 (3.0366) grad_norm 2.0440 (1.8515) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][320/625] eta 0:01:48 lr 0.000678 wd 0.0500 time 0.3457 (0.3559) data time 0.0007 (0.0024) model time 0.3450 (0.3535) loss 2.5354 (3.0378) grad_norm 2.0582 (1.8510) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][330/625] eta 0:01:45 lr 0.000677 wd 0.0500 time 0.3472 (0.3561) data time 0.0009 (0.0024) model time 0.3463 (0.3538) loss 3.3310 (3.0290) grad_norm 1.6536 (1.8537) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][340/625] eta 0:01:41 lr 0.000677 wd 0.0500 time 0.3451 (0.3565) data time 0.0007 (0.0024) model time 0.3445 (0.3543) loss 3.6114 (3.0353) grad_norm 1.7963 (1.8573) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][350/625] eta 0:01:37 lr 0.000677 wd 0.0500 time 0.3493 (0.3563) data time 0.0006 (0.0023) model time 0.3487 (0.3541) loss 3.2745 (3.0335) grad_norm 1.3225 (1.8564) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][360/625] eta 0:01:34 lr 0.000677 wd 0.0500 time 0.3478 (0.3561) data time 0.0008 (0.0023) model time 0.3469 (0.3539) loss 3.3478 (3.0409) grad_norm 1.3996 (1.8522) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][370/625] eta 0:01:30 lr 0.000677 wd 0.0500 time 0.3466 (0.3559) data time 0.0009 (0.0022) model time 0.3457 (0.3537) loss 3.2652 (3.0388) grad_norm 2.1043 (1.8478) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][380/625] eta 0:01:27 lr 0.000677 wd 0.0500 time 0.3477 (0.3557) data time 0.0007 (0.0022) model time 0.3470 (0.3535) loss 3.7137 (3.0410) grad_norm 1.4586 (1.8414) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][390/625] eta 0:01:23 lr 0.000676 wd 0.0500 time 0.3534 (0.3555) data time 0.0009 (0.0022) model time 0.3525 (0.3534) loss 2.9894 (3.0474) grad_norm 2.3780 (1.8560) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][400/625] eta 0:01:19 lr 0.000676 wd 0.0500 time 0.3500 (0.3553) data time 0.0006 (0.0021) model time 0.3494 (0.3532) loss 2.5718 (3.0500) grad_norm 1.3514 (1.8634) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][410/625] eta 0:01:16 lr 0.000676 wd 0.0500 time 0.3433 (0.3552) data time 0.0010 (0.0021) model time 0.3423 (0.3530) loss 3.2055 (3.0482) grad_norm 1.7591 (1.8616) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][420/625] eta 0:01:12 lr 0.000676 wd 0.0500 time 0.3462 (0.3550) data time 0.0007 (0.0021) model time 0.3455 (0.3529) loss 3.2439 (3.0482) grad_norm 1.4645 (1.8569) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][430/625] eta 0:01:09 lr 0.000676 wd 0.0500 time 0.3480 (0.3552) data time 0.0007 (0.0020) model time 0.3473 (0.3532) loss 3.6180 (3.0510) grad_norm 1.3280 (1.8457) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][440/625] eta 0:01:05 lr 0.000676 wd 0.0500 time 0.3533 (0.3551) data time 0.0007 (0.0020) model time 0.3526 (0.3530) loss 3.3774 (3.0493) grad_norm 2.0295 (1.8430) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][450/625] eta 0:01:02 lr 0.000675 wd 0.0500 time 0.3526 (0.3553) data time 0.0009 (0.0020) model time 0.3517 (0.3534) loss 3.4671 (3.0513) grad_norm 13.6898 (1.8706) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][460/625] eta 0:00:58 lr 0.000675 wd 0.0500 time 0.3492 (0.3555) data time 0.0006 (0.0020) model time 0.3486 (0.3536) loss 2.8635 (3.0509) grad_norm 1.9307 (1.8716) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][470/625] eta 0:00:55 lr 0.000675 wd 0.0500 time 0.3540 (0.3566) data time 0.0008 (0.0019) model time 0.3532 (0.3548) loss 2.7648 (3.0429) grad_norm 1.4859 (1.8697) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][480/625] eta 0:00:51 lr 0.000675 wd 0.0500 time 0.3410 (0.3564) data time 0.0006 (0.0019) model time 0.3404 (0.3547) loss 2.4412 (3.0432) grad_norm 1.8708 (1.8651) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][490/625] eta 0:00:48 lr 0.000675 wd 0.0500 time 0.3494 (0.3563) data time 0.0008 (0.0019) model time 0.3486 (0.3545) loss 3.1316 (3.0403) grad_norm 1.5917 (1.8686) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][500/625] eta 0:00:44 lr 0.000675 wd 0.0500 time 0.3460 (0.3561) data time 0.0007 (0.0019) model time 0.3453 (0.3544) loss 3.6668 (3.0401) grad_norm 1.6063 (1.8673) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][510/625] eta 0:00:40 lr 0.000674 wd 0.0500 time 0.3487 (0.3560) data time 0.0006 (0.0018) model time 0.3481 (0.3542) loss 3.4321 (3.0387) grad_norm 2.2208 (1.8654) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][520/625] eta 0:00:37 lr 0.000674 wd 0.0500 time 0.3447 (0.3558) data time 0.0009 (0.0018) model time 0.3438 (0.3541) loss 3.1687 (3.0418) grad_norm 1.9688 (1.8666) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:03:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][530/625] eta 0:00:33 lr 0.000674 wd 0.0500 time 0.3472 (0.3557) data time 0.0009 (0.0018) model time 0.3463 (0.3539) loss 3.0124 (3.0411) grad_norm 2.1165 (1.8645) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][540/625] eta 0:00:30 lr 0.000674 wd 0.0500 time 0.3458 (0.3555) data time 0.0009 (0.0018) model time 0.3449 (0.3537) loss 3.2220 (3.0411) grad_norm 1.8663 (1.8607) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][550/625] eta 0:00:26 lr 0.000674 wd 0.0500 time 0.3499 (0.3554) data time 0.0010 (0.0018) model time 0.3489 (0.3536) loss 3.5961 (3.0420) grad_norm 2.3484 (1.8633) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][560/625] eta 0:00:23 lr 0.000674 wd 0.0500 time 0.3457 (0.3553) data time 0.0010 (0.0018) model time 0.3447 (0.3536) loss 3.3412 (3.0419) grad_norm 1.7010 (1.8758) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][570/625] eta 0:00:19 lr 0.000673 wd 0.0500 time 0.3507 (0.3551) data time 0.0009 (0.0017) model time 0.3498 (0.3534) loss 3.5337 (3.0430) grad_norm 1.3269 (1.8707) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][580/625] eta 0:00:15 lr 0.000673 wd 0.0500 time 0.3481 (0.3550) data time 0.0008 (0.0017) model time 0.3472 (0.3533) loss 3.4031 (3.0446) grad_norm 1.5591 (1.8698) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][590/625] eta 0:00:12 lr 0.000673 wd 0.0500 time 0.3546 (0.3556) data time 0.0007 (0.0017) model time 0.3540 (0.3540) loss 2.9991 (3.0441) grad_norm 1.4452 (1.8704) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][600/625] eta 0:00:08 lr 0.000673 wd 0.0500 time 0.3472 (0.3555) data time 0.0008 (0.0017) model time 0.3464 (0.3539) loss 3.4154 (3.0417) grad_norm 2.0931 (1.8872) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][610/625] eta 0:00:05 lr 0.000673 wd 0.0500 time 0.3505 (0.3557) data time 0.0006 (0.0017) model time 0.3499 (0.3541) loss 3.4104 (3.0406) grad_norm 1.7657 (1.8832) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [190/300][620/625] eta 0:00:01 lr 0.000673 wd 0.0500 time 0.3400 (0.3555) data time 0.0004 (0.0017) model time 0.3396 (0.3539) loss 1.9898 (3.0386) grad_norm 1.6476 (1.8915) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:32 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 190 training takes 0:03:42 [2024-09-01 08:04:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:04:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:04:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.429 (0.429) Loss 0.5605 (0.5605) Acc@1 87.793 (87.793) Acc@5 97.998 (97.998) Mem 11224MB [2024-09-01 08:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.9258 (0.6902) Acc@1 79.199 (84.899) Acc@5 94.971 (97.306) Mem 11224MB [2024-09-01 08:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 0.9692 (0.8196) Acc@1 77.930 (81.752) Acc@5 94.189 (95.836) Mem 11224MB [2024-09-01 08:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.430 Acc@5 95.853 [2024-09-01 08:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.4% [2024-09-01 08:04:35 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.43% [2024-09-01 08:04:35 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:04:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:04:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.429 (0.429) Loss 0.5029 (0.5029) Acc@1 89.160 (89.160) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 08:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.129) Loss 0.8062 (0.6131) Acc@1 81.152 (86.390) Acc@5 95.996 (97.705) Mem 11224MB [2024-09-01 08:04:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 0.9097 (0.7265) Acc@1 77.393 (83.410) Acc@5 95.117 (96.547) Mem 11224MB [2024-09-01 08:04:39 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.107 Acc@5 96.571 [2024-09-01 08:04:39 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 08:04:39 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.11% [2024-09-01 08:04:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:04:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:04:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][0/625] eta 0:07:27 lr 0.000673 wd 0.0500 time 0.7161 (0.7161) data time 0.4109 (0.4109) model time 0.0000 (0.0000) loss 3.0039 (3.0039) grad_norm 1.4779 (1.4779) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][10/625] eta 0:03:54 lr 0.000672 wd 0.0500 time 0.3492 (0.3818) data time 0.0006 (0.0381) model time 0.0000 (0.0000) loss 3.6624 (3.0142) grad_norm 2.0478 (1.6642) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][20/625] eta 0:03:47 lr 0.000672 wd 0.0500 time 0.5425 (0.3759) data time 0.0007 (0.0204) model time 0.0000 (0.0000) loss 3.5053 (2.8898) grad_norm 1.7313 (1.6190) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][30/625] eta 0:03:43 lr 0.000672 wd 0.0500 time 0.3435 (0.3763) data time 0.0009 (0.0141) model time 0.0000 (0.0000) loss 2.4234 (2.9089) grad_norm 1.2158 (1.5881) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][40/625] eta 0:03:35 lr 0.000672 wd 0.0500 time 0.3399 (0.3689) data time 0.0007 (0.0108) model time 0.0000 (0.0000) loss 1.6718 (2.9372) grad_norm 1.7050 (1.5945) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][50/625] eta 0:03:29 lr 0.000672 wd 0.0500 time 0.3492 (0.3649) data time 0.0007 (0.0089) model time 0.0000 (0.0000) loss 3.3508 (2.9372) grad_norm 4.0802 (1.6760) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][60/625] eta 0:03:24 lr 0.000672 wd 0.0500 time 0.3530 (0.3621) data time 0.0006 (0.0076) model time 0.3524 (0.3469) loss 2.2837 (2.9615) grad_norm 2.4441 (1.7038) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][70/625] eta 0:03:19 lr 0.000671 wd 0.0500 time 0.3488 (0.3599) data time 0.0008 (0.0066) model time 0.3480 (0.3464) loss 3.5055 (2.9650) grad_norm 1.6248 (1.7062) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][80/625] eta 0:03:15 lr 0.000671 wd 0.0500 time 0.3468 (0.3584) data time 0.0009 (0.0059) model time 0.3459 (0.3464) loss 3.3195 (3.0057) grad_norm 1.7268 (1.7266) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][90/625] eta 0:03:11 lr 0.000671 wd 0.0500 time 0.3074 (0.3586) data time 0.0008 (0.0054) model time 0.3065 (0.3497) loss 3.1568 (3.0360) grad_norm 2.5945 (1.7734) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][100/625] eta 0:03:09 lr 0.000671 wd 0.0500 time 0.3453 (0.3614) data time 0.0006 (0.0049) model time 0.3447 (0.3569) loss 2.1168 (3.0075) grad_norm 1.7212 (1.8187) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][110/625] eta 0:03:05 lr 0.000671 wd 0.0500 time 0.3465 (0.3601) data time 0.0007 (0.0045) model time 0.3458 (0.3551) loss 2.8835 (3.0310) grad_norm 2.0564 (1.8139) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][120/625] eta 0:03:01 lr 0.000671 wd 0.0500 time 0.3499 (0.3591) data time 0.0009 (0.0042) model time 0.3490 (0.3541) loss 2.7248 (3.0358) grad_norm 2.5179 (1.8123) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][130/625] eta 0:02:57 lr 0.000670 wd 0.0500 time 0.3594 (0.3584) data time 0.0007 (0.0040) model time 0.3587 (0.3534) loss 3.1718 (3.0395) grad_norm 2.8519 (1.8271) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][140/625] eta 0:02:53 lr 0.000670 wd 0.0500 time 0.3458 (0.3579) data time 0.0007 (0.0038) model time 0.3451 (0.3530) loss 3.3862 (3.0370) grad_norm 1.9280 (1.8281) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][150/625] eta 0:02:49 lr 0.000670 wd 0.0500 time 0.3504 (0.3572) data time 0.0008 (0.0036) model time 0.3497 (0.3524) loss 3.4289 (3.0452) grad_norm 1.7334 (1.8480) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][160/625] eta 0:02:45 lr 0.000670 wd 0.0500 time 0.3526 (0.3566) data time 0.0009 (0.0034) model time 0.3517 (0.3519) loss 3.1045 (3.0420) grad_norm 1.5565 (1.8497) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][170/625] eta 0:02:42 lr 0.000670 wd 0.0500 time 0.3419 (0.3562) data time 0.0007 (0.0033) model time 0.3412 (0.3516) loss 3.5163 (3.0380) grad_norm 1.2185 (1.8458) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][180/625] eta 0:02:38 lr 0.000670 wd 0.0500 time 0.3491 (0.3559) data time 0.0006 (0.0031) model time 0.3484 (0.3515) loss 3.0586 (3.0371) grad_norm 1.3777 (1.8455) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][190/625] eta 0:02:34 lr 0.000669 wd 0.0500 time 0.3436 (0.3555) data time 0.0008 (0.0030) model time 0.3428 (0.3512) loss 2.9806 (3.0360) grad_norm 1.4798 (1.8440) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][200/625] eta 0:02:30 lr 0.000669 wd 0.0500 time 0.3419 (0.3550) data time 0.0007 (0.0029) model time 0.3412 (0.3508) loss 2.1753 (3.0321) grad_norm 1.9665 (1.8430) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][210/625] eta 0:02:27 lr 0.000669 wd 0.0500 time 0.3516 (0.3546) data time 0.0006 (0.0028) model time 0.3510 (0.3505) loss 3.1496 (3.0361) grad_norm 2.4447 (1.8436) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:05:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][220/625] eta 0:02:23 lr 0.000669 wd 0.0500 time 0.3433 (0.3544) data time 0.0006 (0.0027) model time 0.3427 (0.3503) loss 3.6578 (3.0366) grad_norm 1.6217 (1.8288) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][230/625] eta 0:02:20 lr 0.000669 wd 0.0500 time 0.3426 (0.3549) data time 0.0008 (0.0026) model time 0.3418 (0.3512) loss 3.3604 (3.0464) grad_norm 1.5922 (1.8343) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][240/625] eta 0:02:17 lr 0.000669 wd 0.0500 time 0.3459 (0.3562) data time 0.0008 (0.0025) model time 0.3451 (0.3531) loss 3.2125 (3.0519) grad_norm 1.5489 (1.8434) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][250/625] eta 0:02:13 lr 0.000668 wd 0.0500 time 0.3476 (0.3560) data time 0.0009 (0.0025) model time 0.3468 (0.3528) loss 3.1923 (3.0458) grad_norm 3.4247 (1.8559) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][260/625] eta 0:02:09 lr 0.000668 wd 0.0500 time 0.3493 (0.3557) data time 0.0008 (0.0024) model time 0.3486 (0.3526) loss 3.2684 (3.0468) grad_norm 1.7841 (1.8580) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][270/625] eta 0:02:06 lr 0.000668 wd 0.0500 time 0.3430 (0.3554) data time 0.0008 (0.0024) model time 0.3422 (0.3524) loss 3.3116 (3.0518) grad_norm 2.0029 (1.8570) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][280/625] eta 0:02:02 lr 0.000668 wd 0.0500 time 0.3412 (0.3556) data time 0.0008 (0.0023) model time 0.3404 (0.3527) loss 1.9857 (3.0500) grad_norm 1.5918 (1.8536) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][290/625] eta 0:01:59 lr 0.000668 wd 0.0500 time 0.3476 (0.3555) data time 0.0006 (0.0023) model time 0.3470 (0.3527) loss 3.6016 (3.0600) grad_norm 1.7450 (1.8529) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][300/625] eta 0:01:55 lr 0.000668 wd 0.0500 time 0.3509 (0.3553) data time 0.0006 (0.0022) model time 0.3503 (0.3526) loss 2.9244 (3.0586) grad_norm 1.6110 (1.8672) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][310/625] eta 0:01:52 lr 0.000667 wd 0.0500 time 0.3472 (0.3558) data time 0.0007 (0.0022) model time 0.3465 (0.3532) loss 1.9519 (3.0474) grad_norm 1.6324 (1.8764) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][320/625] eta 0:01:48 lr 0.000667 wd 0.0500 time 0.3435 (0.3556) data time 0.0007 (0.0021) model time 0.3428 (0.3530) loss 2.2750 (3.0449) grad_norm 1.6503 (1.8737) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][330/625] eta 0:01:44 lr 0.000667 wd 0.0500 time 0.3489 (0.3553) data time 0.0006 (0.0021) model time 0.3482 (0.3528) loss 3.1377 (3.0396) grad_norm 1.3390 (1.8677) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][340/625] eta 0:01:41 lr 0.000667 wd 0.0500 time 0.3446 (0.3551) data time 0.0009 (0.0021) model time 0.3437 (0.3525) loss 3.5304 (3.0375) grad_norm 1.9629 (1.8651) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][350/625] eta 0:01:37 lr 0.000667 wd 0.0500 time 0.3506 (0.3549) data time 0.0009 (0.0020) model time 0.3497 (0.3523) loss 2.4500 (3.0393) grad_norm 1.4053 (1.8540) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][360/625] eta 0:01:34 lr 0.000667 wd 0.0500 time 0.3502 (0.3547) data time 0.0006 (0.0020) model time 0.3496 (0.3522) loss 1.9418 (3.0354) grad_norm 1.9358 (1.8503) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][370/625] eta 0:01:30 lr 0.000666 wd 0.0500 time 0.3484 (0.3545) data time 0.0006 (0.0020) model time 0.3478 (0.3520) loss 2.2603 (3.0343) grad_norm 1.6102 (1.8510) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][380/625] eta 0:01:26 lr 0.000666 wd 0.0500 time 0.3491 (0.3544) data time 0.0008 (0.0019) model time 0.3483 (0.3519) loss 3.1996 (3.0282) grad_norm 1.6690 (1.8474) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][390/625] eta 0:01:23 lr 0.000666 wd 0.0500 time 0.3537 (0.3543) data time 0.0008 (0.0019) model time 0.3529 (0.3518) loss 3.2629 (3.0288) grad_norm 1.6827 (1.8481) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][400/625] eta 0:01:19 lr 0.000666 wd 0.0500 time 0.3523 (0.3545) data time 0.0006 (0.0019) model time 0.3516 (0.3521) loss 3.0500 (3.0248) grad_norm 1.3166 (1.8527) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][410/625] eta 0:01:16 lr 0.000666 wd 0.0500 time 0.3615 (0.3544) data time 0.0007 (0.0019) model time 0.3608 (0.3521) loss 3.1071 (3.0271) grad_norm 2.7602 (1.8522) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][420/625] eta 0:01:12 lr 0.000666 wd 0.0500 time 0.3508 (0.3556) data time 0.0009 (0.0018) model time 0.3498 (0.3535) loss 3.1275 (3.0247) grad_norm 13.6354 (1.8758) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][430/625] eta 0:01:09 lr 0.000665 wd 0.0500 time 0.3473 (0.3554) data time 0.0008 (0.0018) model time 0.3465 (0.3533) loss 3.3521 (3.0245) grad_norm 1.6296 (1.8782) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][440/625] eta 0:01:05 lr 0.000665 wd 0.0500 time 0.3533 (0.3553) data time 0.0007 (0.0018) model time 0.3526 (0.3532) loss 1.8919 (3.0270) grad_norm 1.7176 (1.8721) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][450/625] eta 0:01:02 lr 0.000665 wd 0.0500 time 0.3540 (0.3552) data time 0.0007 (0.0018) model time 0.3533 (0.3531) loss 3.6992 (3.0240) grad_norm 1.7034 (1.8694) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][460/625] eta 0:00:58 lr 0.000665 wd 0.0500 time 0.3489 (0.3550) data time 0.0009 (0.0017) model time 0.3480 (0.3529) loss 2.5567 (3.0232) grad_norm 1.5915 (1.8711) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][470/625] eta 0:00:55 lr 0.000665 wd 0.0500 time 0.3412 (0.3549) data time 0.0008 (0.0017) model time 0.3405 (0.3528) loss 3.0811 (3.0239) grad_norm 1.3830 (1.8792) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][480/625] eta 0:00:51 lr 0.000665 wd 0.0500 time 0.3465 (0.3548) data time 0.0008 (0.0017) model time 0.3457 (0.3528) loss 2.5297 (3.0281) grad_norm 1.5198 (1.8749) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][490/625] eta 0:00:48 lr 0.000664 wd 0.0500 time 0.3585 (0.3556) data time 0.0007 (0.0017) model time 0.3577 (0.3536) loss 2.9654 (3.0330) grad_norm 1.1632 (1.8693) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][500/625] eta 0:00:44 lr 0.000664 wd 0.0500 time 0.3466 (0.3555) data time 0.0007 (0.0017) model time 0.3459 (0.3535) loss 3.3592 (3.0279) grad_norm 1.9143 (1.8666) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][510/625] eta 0:00:40 lr 0.000664 wd 0.0500 time 0.3471 (0.3553) data time 0.0006 (0.0017) model time 0.3465 (0.3534) loss 2.4945 (3.0289) grad_norm 3.1332 (1.8687) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][520/625] eta 0:00:37 lr 0.000664 wd 0.0500 time 0.3499 (0.3552) data time 0.0008 (0.0016) model time 0.3491 (0.3533) loss 3.4302 (3.0303) grad_norm 1.4000 (1.8737) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][530/625] eta 0:00:33 lr 0.000664 wd 0.0500 time 0.3462 (0.3550) data time 0.0009 (0.0016) model time 0.3453 (0.3531) loss 2.8223 (3.0329) grad_norm 1.4953 (1.8657) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][540/625] eta 0:00:30 lr 0.000664 wd 0.0500 time 0.3431 (0.3559) data time 0.0008 (0.0016) model time 0.3423 (0.3542) loss 3.2229 (3.0287) grad_norm 1.5466 (1.8616) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:07:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][550/625] eta 0:00:26 lr 0.000663 wd 0.0500 time 0.3488 (0.3558) data time 0.0008 (0.0016) model time 0.3480 (0.3540) loss 2.0380 (3.0218) grad_norm 2.1199 (1.8575) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][560/625] eta 0:00:23 lr 0.000663 wd 0.0500 time 0.3530 (0.3557) data time 0.0008 (0.0016) model time 0.3522 (0.3539) loss 3.1008 (3.0229) grad_norm 1.4083 (1.8528) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][570/625] eta 0:00:19 lr 0.000663 wd 0.0500 time 0.3446 (0.3555) data time 0.0007 (0.0016) model time 0.3439 (0.3537) loss 2.7453 (3.0238) grad_norm 1.9280 (1.8542) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][580/625] eta 0:00:15 lr 0.000663 wd 0.0500 time 0.3454 (0.3554) data time 0.0009 (0.0016) model time 0.3446 (0.3536) loss 3.0226 (3.0246) grad_norm 1.8503 (1.8495) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][590/625] eta 0:00:12 lr 0.000663 wd 0.0500 time 0.3509 (0.3553) data time 0.0006 (0.0015) model time 0.3503 (0.3535) loss 3.0422 (3.0278) grad_norm 1.2292 (1.8506) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][600/625] eta 0:00:08 lr 0.000663 wd 0.0500 time 0.3444 (0.3552) data time 0.0008 (0.0015) model time 0.3436 (0.3534) loss 2.4639 (3.0221) grad_norm 1.5457 (1.8454) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][610/625] eta 0:00:05 lr 0.000662 wd 0.0500 time 0.3375 (0.3560) data time 0.0004 (0.0015) model time 0.3371 (0.3543) loss 2.5636 (3.0226) grad_norm 1.5136 (1.8449) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [191/300][620/625] eta 0:00:01 lr 0.000662 wd 0.0500 time 0.3384 (0.3557) data time 0.0006 (0.0015) model time 0.3378 (0.3541) loss 3.1493 (3.0257) grad_norm 2.8294 (1.8539) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 191 training takes 0:03:42 [2024-09-01 08:08:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:08:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:08:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.445 (0.445) Loss 0.5488 (0.5488) Acc@1 88.037 (88.037) Acc@5 98.438 (98.438) Mem 11224MB [2024-09-01 08:08:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.130) Loss 0.9102 (0.6682) Acc@1 78.174 (84.917) Acc@5 94.971 (97.328) Mem 11224MB [2024-09-01 08:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 0.9355 (0.7931) Acc@1 77.979 (81.806) Acc@5 94.824 (95.940) Mem 11224MB [2024-09-01 08:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.546 Acc@5 95.965 [2024-09-01 08:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.5% [2024-09-01 08:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.55% [2024-09-01 08:08:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:08:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:08:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.441 (0.441) Loss 0.5020 (0.5020) Acc@1 89.111 (89.111) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 08:08:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.8081 (0.6126) Acc@1 81.250 (86.444) Acc@5 96.045 (97.732) Mem 11224MB [2024-09-01 08:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.111) Loss 0.9058 (0.7258) Acc@1 77.344 (83.468) Acc@5 95.117 (96.554) Mem 11224MB [2024-09-01 08:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.165 Acc@5 96.573 [2024-09-01 08:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.2% [2024-09-01 08:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.17% [2024-09-01 08:08:30 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:08:31 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:08:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][0/625] eta 0:07:08 lr 0.000662 wd 0.0500 time 0.6855 (0.6855) data time 0.3671 (0.3671) model time 0.0000 (0.0000) loss 3.3976 (3.3976) grad_norm 1.7887 (1.7887) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][10/625] eta 0:03:52 lr 0.000662 wd 0.0500 time 0.3510 (0.3780) data time 0.0006 (0.0343) model time 0.0000 (0.0000) loss 3.2979 (2.9518) grad_norm 1.6246 (2.0606) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][20/625] eta 0:03:39 lr 0.000662 wd 0.0500 time 0.3517 (0.3634) data time 0.0009 (0.0184) model time 0.0000 (0.0000) loss 3.2184 (3.0297) grad_norm 1.4255 (2.0112) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][30/625] eta 0:03:33 lr 0.000662 wd 0.0500 time 0.3453 (0.3585) data time 0.0009 (0.0127) model time 0.0000 (0.0000) loss 3.2977 (3.0668) grad_norm 1.8854 (1.9948) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][40/625] eta 0:03:28 lr 0.000661 wd 0.0500 time 0.3489 (0.3556) data time 0.0006 (0.0098) model time 0.0000 (0.0000) loss 3.1769 (3.0766) grad_norm 1.8149 (1.9577) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][50/625] eta 0:03:23 lr 0.000661 wd 0.0500 time 0.3454 (0.3547) data time 0.0006 (0.0081) model time 0.0000 (0.0000) loss 3.3611 (3.0987) grad_norm 1.2222 (1.9640) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][60/625] eta 0:03:19 lr 0.000661 wd 0.0500 time 0.3471 (0.3537) data time 0.0009 (0.0069) model time 0.3462 (0.3476) loss 3.3011 (3.0557) grad_norm 1.7312 (1.9600) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:08:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][70/625] eta 0:03:19 lr 0.000661 wd 0.0500 time 0.3490 (0.3591) data time 0.0009 (0.0060) model time 0.3481 (0.3693) loss 3.1915 (3.0508) grad_norm 1.3139 (1.9298) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][80/625] eta 0:03:14 lr 0.000661 wd 0.0500 time 0.3478 (0.3577) data time 0.0008 (0.0054) model time 0.3470 (0.3619) loss 3.3121 (3.0617) grad_norm 1.6294 (1.9401) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][90/625] eta 0:03:10 lr 0.000661 wd 0.0500 time 0.3440 (0.3569) data time 0.0008 (0.0049) model time 0.3432 (0.3588) loss 3.3293 (3.0643) grad_norm 4.3326 (2.1060) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][100/625] eta 0:03:07 lr 0.000660 wd 0.0500 time 0.3513 (0.3577) data time 0.0007 (0.0045) model time 0.3506 (0.3600) loss 4.0164 (3.0656) grad_norm 2.4110 (2.1506) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][110/625] eta 0:03:03 lr 0.000660 wd 0.0500 time 0.3472 (0.3570) data time 0.0009 (0.0042) model time 0.3463 (0.3580) loss 2.9699 (3.0735) grad_norm 6.1349 (2.1436) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][120/625] eta 0:03:00 lr 0.000660 wd 0.0500 time 0.3421 (0.3566) data time 0.0006 (0.0039) model time 0.3415 (0.3570) loss 2.9767 (3.0607) grad_norm 1.7674 (2.1119) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][130/625] eta 0:02:56 lr 0.000660 wd 0.0500 time 0.3465 (0.3558) data time 0.0011 (0.0037) model time 0.3454 (0.3556) loss 3.6447 (3.0639) grad_norm 1.6100 (2.0789) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][140/625] eta 0:02:52 lr 0.000660 wd 0.0500 time 0.3501 (0.3552) data time 0.0006 (0.0035) model time 0.3495 (0.3545) loss 3.5398 (3.0752) grad_norm 1.6019 (2.0377) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][150/625] eta 0:02:48 lr 0.000660 wd 0.0500 time 0.3482 (0.3548) data time 0.0007 (0.0033) model time 0.3476 (0.3540) loss 3.5840 (3.0773) grad_norm 1.4755 (2.0083) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][160/625] eta 0:02:44 lr 0.000659 wd 0.0500 time 0.3456 (0.3544) data time 0.0008 (0.0031) model time 0.3447 (0.3534) loss 2.0521 (3.0737) grad_norm 2.0499 (2.0045) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][170/625] eta 0:02:42 lr 0.000659 wd 0.0500 time 0.5279 (0.3562) data time 0.0008 (0.0030) model time 0.5271 (0.3560) loss 3.3411 (3.0862) grad_norm 1.3448 (1.9851) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][180/625] eta 0:02:38 lr 0.000659 wd 0.0500 time 0.3415 (0.3557) data time 0.0007 (0.0029) model time 0.3408 (0.3552) loss 3.4355 (3.0948) grad_norm 1.8351 (2.0046) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][190/625] eta 0:02:34 lr 0.000659 wd 0.0500 time 0.3464 (0.3552) data time 0.0006 (0.0028) model time 0.3458 (0.3546) loss 3.1701 (3.0962) grad_norm 1.7925 (2.0030) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][200/625] eta 0:02:30 lr 0.000659 wd 0.0500 time 0.3485 (0.3549) data time 0.0007 (0.0027) model time 0.3478 (0.3541) loss 3.3526 (3.0860) grad_norm 1.7742 (1.9802) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][210/625] eta 0:02:27 lr 0.000659 wd 0.0500 time 0.3421 (0.3545) data time 0.0009 (0.0026) model time 0.3412 (0.3536) loss 3.1061 (3.0763) grad_norm 2.3884 (1.9624) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][220/625] eta 0:02:23 lr 0.000658 wd 0.0500 time 0.3561 (0.3543) data time 0.0008 (0.0025) model time 0.3554 (0.3533) loss 3.1499 (3.0819) grad_norm 1.4986 (1.9493) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][230/625] eta 0:02:19 lr 0.000658 wd 0.0500 time 0.3479 (0.3540) data time 0.0009 (0.0024) model time 0.3470 (0.3530) loss 2.8756 (3.0832) grad_norm 1.8604 (1.9453) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][240/625] eta 0:02:16 lr 0.000658 wd 0.0500 time 0.3476 (0.3537) data time 0.0006 (0.0024) model time 0.3470 (0.3526) loss 3.5936 (3.0782) grad_norm 1.6696 (1.9352) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][250/625] eta 0:02:12 lr 0.000658 wd 0.0500 time 0.3515 (0.3535) data time 0.0008 (0.0023) model time 0.3507 (0.3523) loss 2.8530 (3.0794) grad_norm 1.7027 (1.9366) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][260/625] eta 0:02:08 lr 0.000658 wd 0.0500 time 0.3541 (0.3533) data time 0.0006 (0.0023) model time 0.3535 (0.3521) loss 3.5302 (3.0798) grad_norm 2.7478 (1.9441) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][270/625] eta 0:02:05 lr 0.000658 wd 0.0500 time 0.3464 (0.3531) data time 0.0010 (0.0022) model time 0.3454 (0.3518) loss 3.1719 (3.0790) grad_norm 1.8120 (1.9411) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][280/625] eta 0:02:02 lr 0.000657 wd 0.0500 time 0.5811 (0.3557) data time 0.0009 (0.0022) model time 0.5802 (0.3551) loss 3.2572 (3.0754) grad_norm 2.0014 (1.9315) loss_scale 1024.0000 (515.6441) mem 11224MB [2024-09-01 08:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][290/625] eta 0:01:59 lr 0.000657 wd 0.0500 time 0.5574 (0.3568) data time 0.0007 (0.0021) model time 0.5567 (0.3563) loss 3.7039 (3.0725) grad_norm 1.9276 (1.9406) loss_scale 1024.0000 (533.1134) mem 11224MB [2024-09-01 08:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][300/625] eta 0:01:55 lr 0.000657 wd 0.0500 time 0.3401 (0.3565) data time 0.0009 (0.0021) model time 0.3392 (0.3560) loss 3.2652 (3.0708) grad_norm 1.4643 (1.9357) loss_scale 1024.0000 (549.4219) mem 11224MB [2024-09-01 08:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][310/625] eta 0:01:52 lr 0.000657 wd 0.0500 time 0.3443 (0.3562) data time 0.0007 (0.0020) model time 0.3437 (0.3556) loss 3.5044 (3.0666) grad_norm 1.2023 (1.9198) loss_scale 1024.0000 (564.6817) mem 11224MB [2024-09-01 08:10:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][320/625] eta 0:01:48 lr 0.000657 wd 0.0500 time 0.3441 (0.3560) data time 0.0008 (0.0020) model time 0.3433 (0.3554) loss 3.6138 (3.0684) grad_norm 2.2469 (1.9130) loss_scale 1024.0000 (578.9907) mem 11224MB [2024-09-01 08:10:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][330/625] eta 0:01:44 lr 0.000657 wd 0.0500 time 0.3486 (0.3557) data time 0.0009 (0.0020) model time 0.3477 (0.3551) loss 3.1628 (3.0666) grad_norm 2.7612 (1.9262) loss_scale 1024.0000 (592.4350) mem 11224MB [2024-09-01 08:10:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][340/625] eta 0:01:41 lr 0.000656 wd 0.0500 time 0.3497 (0.3555) data time 0.0009 (0.0019) model time 0.3488 (0.3548) loss 2.4760 (3.0652) grad_norm 1.6838 (1.9233) loss_scale 1024.0000 (605.0909) mem 11224MB [2024-09-01 08:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][350/625] eta 0:01:37 lr 0.000656 wd 0.0500 time 0.3510 (0.3552) data time 0.0008 (0.0019) model time 0.3502 (0.3545) loss 3.4423 (3.0661) grad_norm 2.0876 (1.9185) loss_scale 1024.0000 (617.0256) mem 11224MB [2024-09-01 08:10:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][360/625] eta 0:01:34 lr 0.000656 wd 0.0500 time 0.3467 (0.3550) data time 0.0006 (0.0019) model time 0.3460 (0.3542) loss 2.4656 (3.0632) grad_norm 2.1339 (1.9394) loss_scale 1024.0000 (628.2992) mem 11224MB [2024-09-01 08:10:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][370/625] eta 0:01:30 lr 0.000656 wd 0.0500 time 0.3455 (0.3548) data time 0.0009 (0.0018) model time 0.3447 (0.3539) loss 3.1058 (3.0529) grad_norm 2.7153 (1.9390) loss_scale 1024.0000 (638.9650) mem 11224MB [2024-09-01 08:10:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][380/625] eta 0:01:26 lr 0.000656 wd 0.0500 time 0.3419 (0.3546) data time 0.0010 (0.0018) model time 0.3410 (0.3537) loss 3.1972 (3.0573) grad_norm 1.4737 (1.9283) loss_scale 1024.0000 (649.0709) mem 11224MB [2024-09-01 08:10:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][390/625] eta 0:01:23 lr 0.000656 wd 0.0500 time 0.3490 (0.3544) data time 0.0006 (0.0018) model time 0.3483 (0.3535) loss 1.9934 (3.0578) grad_norm 1.7524 (1.9274) loss_scale 1024.0000 (658.6598) mem 11224MB [2024-09-01 08:10:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][400/625] eta 0:01:19 lr 0.000655 wd 0.0500 time 0.3109 (0.3547) data time 0.0007 (0.0018) model time 0.3102 (0.3539) loss 2.4604 (3.0556) grad_norm 1.3384 (1.9156) loss_scale 1024.0000 (667.7706) mem 11224MB [2024-09-01 08:10:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][410/625] eta 0:01:16 lr 0.000655 wd 0.0500 time 0.3456 (0.3546) data time 0.0009 (0.0017) model time 0.3447 (0.3537) loss 3.3199 (3.0600) grad_norm 1.7326 (1.9142) loss_scale 1024.0000 (676.4380) mem 11224MB [2024-09-01 08:11:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][420/625] eta 0:01:12 lr 0.000655 wd 0.0500 time 0.3419 (0.3544) data time 0.0008 (0.0017) model time 0.3411 (0.3535) loss 2.5874 (3.0558) grad_norm 1.3382 (1.9080) loss_scale 1024.0000 (684.6936) mem 11224MB [2024-09-01 08:11:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][430/625] eta 0:01:09 lr 0.000655 wd 0.0500 time 0.3486 (0.3543) data time 0.0006 (0.0017) model time 0.3479 (0.3534) loss 2.3535 (3.0511) grad_norm 1.8102 (1.9097) loss_scale 1024.0000 (692.5661) mem 11224MB [2024-09-01 08:11:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][440/625] eta 0:01:05 lr 0.000655 wd 0.0500 time 0.3492 (0.3541) data time 0.0006 (0.0017) model time 0.3486 (0.3532) loss 3.5995 (3.0510) grad_norm 2.2975 (1.9089) loss_scale 1024.0000 (700.0816) mem 11224MB [2024-09-01 08:11:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][450/625] eta 0:01:01 lr 0.000655 wd 0.0500 time 0.3452 (0.3540) data time 0.0009 (0.0017) model time 0.3443 (0.3530) loss 3.2781 (3.0508) grad_norm 1.7882 (1.9094) loss_scale 1024.0000 (707.2639) mem 11224MB [2024-09-01 08:11:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][460/625] eta 0:00:58 lr 0.000654 wd 0.0500 time 0.3541 (0.3543) data time 0.0007 (0.0016) model time 0.3534 (0.3534) loss 2.8893 (3.0534) grad_norm 2.0954 (1.9086) loss_scale 1024.0000 (714.1345) mem 11224MB [2024-09-01 08:11:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][470/625] eta 0:00:55 lr 0.000654 wd 0.0500 time 0.3491 (0.3549) data time 0.0008 (0.0016) model time 0.3483 (0.3540) loss 3.4984 (3.0536) grad_norm 1.3100 (1.8994) loss_scale 1024.0000 (720.7134) mem 11224MB [2024-09-01 08:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][480/625] eta 0:00:51 lr 0.000654 wd 0.0500 time 0.3475 (0.3547) data time 0.0006 (0.0016) model time 0.3469 (0.3539) loss 2.7591 (3.0521) grad_norm 1.8850 (1.8997) loss_scale 1024.0000 (727.0187) mem 11224MB [2024-09-01 08:11:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][490/625] eta 0:00:47 lr 0.000654 wd 0.0500 time 0.3527 (0.3546) data time 0.0009 (0.0016) model time 0.3518 (0.3538) loss 3.4310 (3.0541) grad_norm 1.5862 (1.8933) loss_scale 1024.0000 (733.0672) mem 11224MB [2024-09-01 08:11:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][500/625] eta 0:00:44 lr 0.000654 wd 0.0500 time 0.3467 (0.3554) data time 0.0007 (0.0016) model time 0.3460 (0.3547) loss 3.4918 (3.0499) grad_norm 1.7839 (1.8996) loss_scale 1024.0000 (738.8743) mem 11224MB [2024-09-01 08:11:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][510/625] eta 0:00:40 lr 0.000654 wd 0.0500 time 0.3505 (0.3553) data time 0.0008 (0.0016) model time 0.3497 (0.3546) loss 3.2138 (3.0454) grad_norm 7.3858 (1.9138) loss_scale 1024.0000 (744.4540) mem 11224MB [2024-09-01 08:11:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][520/625] eta 0:00:37 lr 0.000654 wd 0.0500 time 0.3554 (0.3552) data time 0.0009 (0.0015) model time 0.3545 (0.3544) loss 3.4718 (3.0479) grad_norm 1.7676 (1.9153) loss_scale 1024.0000 (749.8196) mem 11224MB [2024-09-01 08:11:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][530/625] eta 0:00:33 lr 0.000653 wd 0.0500 time 0.3429 (0.3551) data time 0.0006 (0.0015) model time 0.3423 (0.3543) loss 3.5189 (3.0504) grad_norm 1.6417 (1.9163) loss_scale 1024.0000 (754.9831) mem 11224MB [2024-09-01 08:11:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][540/625] eta 0:00:30 lr 0.000653 wd 0.0500 time 0.3663 (0.3550) data time 0.0007 (0.0015) model time 0.3656 (0.3542) loss 2.6832 (3.0479) grad_norm 1.5421 (1.9123) loss_scale 1024.0000 (759.9556) mem 11224MB [2024-09-01 08:11:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][550/625] eta 0:00:26 lr 0.000653 wd 0.0500 time 0.3439 (0.3549) data time 0.0006 (0.0015) model time 0.3432 (0.3541) loss 3.3999 (3.0460) grad_norm 1.4819 (1.9087) loss_scale 1024.0000 (764.7477) mem 11224MB [2024-09-01 08:11:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][560/625] eta 0:00:23 lr 0.000653 wd 0.0500 time 0.3513 (0.3548) data time 0.0008 (0.0015) model time 0.3505 (0.3540) loss 3.3629 (3.0461) grad_norm 1.6701 (1.9071) loss_scale 1024.0000 (769.3690) mem 11224MB [2024-09-01 08:11:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][570/625] eta 0:00:19 lr 0.000653 wd 0.0500 time 0.3526 (0.3548) data time 0.0008 (0.0015) model time 0.3518 (0.3539) loss 3.3668 (3.0483) grad_norm 1.7695 (1.9089) loss_scale 1024.0000 (773.8284) mem 11224MB [2024-09-01 08:11:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][580/625] eta 0:00:15 lr 0.000653 wd 0.0500 time 0.3480 (0.3550) data time 0.0007 (0.0015) model time 0.3474 (0.3541) loss 2.5800 (3.0494) grad_norm 1.5309 (1.9100) loss_scale 1024.0000 (778.1343) mem 11224MB [2024-09-01 08:12:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][590/625] eta 0:00:12 lr 0.000652 wd 0.0500 time 0.3491 (0.3552) data time 0.0006 (0.0015) model time 0.3484 (0.3544) loss 2.8950 (3.0427) grad_norm 1.7516 (1.9154) loss_scale 1024.0000 (782.2944) mem 11224MB [2024-09-01 08:12:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][600/625] eta 0:00:08 lr 0.000652 wd 0.0500 time 0.3543 (0.3551) data time 0.0006 (0.0015) model time 0.3537 (0.3543) loss 1.9451 (3.0436) grad_norm 1.0632 (1.9128) loss_scale 1024.0000 (786.3161) mem 11224MB [2024-09-01 08:12:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][610/625] eta 0:00:05 lr 0.000652 wd 0.0500 time 0.3373 (0.3553) data time 0.0006 (0.0014) model time 0.3366 (0.3545) loss 3.2096 (3.0463) grad_norm 1.9354 (1.9128) loss_scale 1024.0000 (790.2062) mem 11224MB [2024-09-01 08:12:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [192/300][620/625] eta 0:00:01 lr 0.000652 wd 0.0500 time 0.3413 (0.3550) data time 0.0006 (0.0014) model time 0.3407 (0.3542) loss 2.3656 (3.0427) grad_norm 3.5456 (1.9161) loss_scale 1024.0000 (793.9710) mem 11224MB [2024-09-01 08:12:13 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 192 training takes 0:03:41 [2024-09-01 08:12:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:12:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:12:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.460 (0.460) Loss 0.5591 (0.5591) Acc@1 88.672 (88.672) Acc@5 98.242 (98.242) Mem 11224MB [2024-09-01 08:12:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.133) Loss 0.9243 (0.6859) Acc@1 78.418 (85.041) Acc@5 94.824 (97.110) Mem 11224MB [2024-09-01 08:12:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.113) Loss 0.9634 (0.8072) Acc@1 76.611 (81.908) Acc@5 94.775 (95.815) Mem 11224MB [2024-09-01 08:12:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.638 Acc@5 95.845 [2024-09-01 08:12:17 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.6% [2024-09-01 08:12:17 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.64% [2024-09-01 08:12:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:12:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.439 (0.439) Loss 0.5015 (0.5015) Acc@1 89.160 (89.160) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 08:12:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.8066 (0.6118) Acc@1 81.299 (86.390) Acc@5 95.947 (97.701) Mem 11224MB [2024-09-01 08:12:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.111) Loss 0.9028 (0.7248) Acc@1 77.539 (83.459) Acc@5 95.312 (96.563) Mem 11224MB [2024-09-01 08:12:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.159 Acc@5 96.585 [2024-09-01 08:12:20 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.2% [2024-09-01 08:12:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][0/625] eta 0:12:20 lr 0.000652 wd 0.0500 time 1.1852 (1.1852) data time 0.6724 (0.6724) model time 0.0000 (0.0000) loss 3.5160 (3.5160) grad_norm 2.6133 (2.6133) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][10/625] eta 0:04:20 lr 0.000652 wd 0.0500 time 0.3424 (0.4243) data time 0.0008 (0.0619) model time 0.0000 (0.0000) loss 1.9892 (2.9305) grad_norm 2.1545 (2.0331) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][20/625] eta 0:03:54 lr 0.000651 wd 0.0500 time 0.3485 (0.3878) data time 0.0007 (0.0328) model time 0.0000 (0.0000) loss 3.4627 (2.9339) grad_norm 1.2910 (1.8247) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][30/625] eta 0:03:43 lr 0.000651 wd 0.0500 time 0.3505 (0.3753) data time 0.0009 (0.0225) model time 0.0000 (0.0000) loss 2.1149 (2.9098) grad_norm 1.5043 (1.8015) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][40/625] eta 0:03:35 lr 0.000651 wd 0.0500 time 0.3521 (0.3686) data time 0.0006 (0.0172) model time 0.0000 (0.0000) loss 2.2582 (2.9285) grad_norm 2.3965 (1.7699) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][50/625] eta 0:03:29 lr 0.000651 wd 0.0500 time 0.3498 (0.3644) data time 0.0009 (0.0140) model time 0.0000 (0.0000) loss 2.8789 (2.9396) grad_norm 1.4888 (1.7495) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][60/625] eta 0:03:24 lr 0.000651 wd 0.0500 time 0.3498 (0.3617) data time 0.0007 (0.0118) model time 0.3491 (0.3470) loss 2.1171 (2.9630) grad_norm 1.2818 (1.7207) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][70/625] eta 0:03:24 lr 0.000651 wd 0.0500 time 0.3489 (0.3684) data time 0.0007 (0.0103) model time 0.3483 (0.3778) loss 2.6132 (2.9451) grad_norm 1.8202 (1.7383) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][80/625] eta 0:03:20 lr 0.000650 wd 0.0500 time 0.3501 (0.3678) data time 0.0010 (0.0091) model time 0.3492 (0.3725) loss 3.3753 (2.9344) grad_norm 1.7143 (1.7112) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][90/625] eta 0:03:15 lr 0.000650 wd 0.0500 time 0.3453 (0.3657) data time 0.0008 (0.0082) model time 0.3446 (0.3664) loss 3.3652 (2.9743) grad_norm 2.3772 (1.7462) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:12:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][100/625] eta 0:03:12 lr 0.000650 wd 0.0500 time 0.3470 (0.3670) data time 0.0006 (0.0075) model time 0.3464 (0.3688) loss 3.5454 (2.9930) grad_norm 1.8989 (1.7417) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][110/625] eta 0:03:08 lr 0.000650 wd 0.0500 time 0.3555 (0.3652) data time 0.0008 (0.0069) model time 0.3547 (0.3650) loss 3.0324 (2.9798) grad_norm 1.3515 (1.7393) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][120/625] eta 0:03:03 lr 0.000650 wd 0.0500 time 0.3474 (0.3638) data time 0.0009 (0.0064) model time 0.3465 (0.3625) loss 3.1676 (2.9695) grad_norm 2.7496 (1.7688) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][130/625] eta 0:02:59 lr 0.000650 wd 0.0500 time 0.3450 (0.3626) data time 0.0006 (0.0060) model time 0.3445 (0.3606) loss 2.7918 (2.9792) grad_norm 2.3019 (1.7821) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][140/625] eta 0:02:55 lr 0.000649 wd 0.0500 time 0.3474 (0.3615) data time 0.0009 (0.0056) model time 0.3465 (0.3591) loss 3.4637 (2.9963) grad_norm 1.4204 (1.8102) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][150/625] eta 0:02:51 lr 0.000649 wd 0.0500 time 0.3530 (0.3607) data time 0.0007 (0.0053) model time 0.3523 (0.3580) loss 3.3954 (2.9989) grad_norm 2.1684 (1.8336) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][160/625] eta 0:02:47 lr 0.000649 wd 0.0500 time 0.3437 (0.3599) data time 0.0007 (0.0050) model time 0.3430 (0.3569) loss 2.9853 (2.9951) grad_norm 1.4560 (1.8496) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][170/625] eta 0:02:43 lr 0.000649 wd 0.0500 time 0.3483 (0.3592) data time 0.0008 (0.0048) model time 0.3475 (0.3561) loss 2.5187 (3.0031) grad_norm 1.6942 (1.8407) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][180/625] eta 0:02:39 lr 0.000649 wd 0.0500 time 0.3484 (0.3586) data time 0.0008 (0.0045) model time 0.3476 (0.3554) loss 3.0862 (3.0021) grad_norm 1.2931 (1.8235) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][190/625] eta 0:02:35 lr 0.000649 wd 0.0500 time 0.3455 (0.3580) data time 0.0009 (0.0043) model time 0.3447 (0.3549) loss 2.5593 (3.0080) grad_norm 1.3094 (1.8099) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][200/625] eta 0:02:31 lr 0.000648 wd 0.0500 time 0.3506 (0.3576) data time 0.0009 (0.0042) model time 0.3496 (0.3544) loss 3.2299 (3.0014) grad_norm 1.6248 (1.8031) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][210/625] eta 0:02:28 lr 0.000648 wd 0.0500 time 0.3469 (0.3571) data time 0.0009 (0.0040) model time 0.3460 (0.3539) loss 3.0194 (3.0066) grad_norm 1.7661 (1.8097) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][220/625] eta 0:02:24 lr 0.000648 wd 0.0500 time 0.3434 (0.3567) data time 0.0008 (0.0039) model time 0.3426 (0.3535) loss 3.1438 (3.0113) grad_norm 2.0654 (1.8136) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][230/625] eta 0:02:20 lr 0.000648 wd 0.0500 time 0.3478 (0.3563) data time 0.0007 (0.0037) model time 0.3471 (0.3532) loss 2.0588 (3.0115) grad_norm 2.1328 (1.8113) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][240/625] eta 0:02:17 lr 0.000648 wd 0.0500 time 0.3500 (0.3559) data time 0.0007 (0.0036) model time 0.3493 (0.3528) loss 2.3031 (3.0060) grad_norm 1.9576 (1.8193) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][250/625] eta 0:02:14 lr 0.000648 wd 0.0500 time 0.5634 (0.3594) data time 0.0007 (0.0035) model time 0.5627 (0.3573) loss 1.7306 (2.9916) grad_norm 1.6564 (1.8218) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][260/625] eta 0:02:11 lr 0.000647 wd 0.0500 time 0.3452 (0.3590) data time 0.0008 (0.0034) model time 0.3444 (0.3568) loss 3.4172 (2.9842) grad_norm 1.5488 (1.8116) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:13:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][270/625] eta 0:02:07 lr 0.000647 wd 0.0500 time 0.3516 (0.3586) data time 0.0007 (0.0033) model time 0.3510 (0.3563) loss 3.2833 (2.9846) grad_norm 1.4623 (1.7970) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][280/625] eta 0:02:03 lr 0.000647 wd 0.0500 time 0.3463 (0.3582) data time 0.0009 (0.0032) model time 0.3454 (0.3560) loss 2.9092 (2.9883) grad_norm 1.4494 (1.7878) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][290/625] eta 0:02:00 lr 0.000647 wd 0.0500 time 0.3493 (0.3584) data time 0.0009 (0.0031) model time 0.3484 (0.3563) loss 2.9392 (2.9910) grad_norm 1.6112 (1.7997) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][300/625] eta 0:01:56 lr 0.000647 wd 0.0500 time 0.3448 (0.3581) data time 0.0008 (0.0031) model time 0.3440 (0.3560) loss 3.0627 (2.9988) grad_norm 1.6877 (1.8054) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][310/625] eta 0:01:53 lr 0.000647 wd 0.0500 time 0.3603 (0.3592) data time 0.0009 (0.0030) model time 0.3594 (0.3573) loss 3.0299 (3.0055) grad_norm 1.8621 (1.8260) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][320/625] eta 0:01:49 lr 0.000646 wd 0.0500 time 0.3478 (0.3589) data time 0.0007 (0.0029) model time 0.3471 (0.3570) loss 2.3878 (3.0016) grad_norm 2.0104 (1.8400) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][330/625] eta 0:01:45 lr 0.000646 wd 0.0500 time 0.3636 (0.3586) data time 0.0008 (0.0029) model time 0.3627 (0.3567) loss 2.7259 (2.9984) grad_norm 1.1906 (1.8366) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][340/625] eta 0:01:42 lr 0.000646 wd 0.0500 time 0.3497 (0.3584) data time 0.0006 (0.0028) model time 0.3491 (0.3565) loss 3.9567 (3.0020) grad_norm 1.6427 (1.8284) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][350/625] eta 0:01:38 lr 0.000646 wd 0.0500 time 0.3487 (0.3581) data time 0.0006 (0.0028) model time 0.3480 (0.3562) loss 1.9847 (2.9975) grad_norm 1.7683 (1.8250) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][360/625] eta 0:01:34 lr 0.000646 wd 0.0500 time 0.3478 (0.3581) data time 0.0006 (0.0027) model time 0.3471 (0.3562) loss 2.9819 (2.9958) grad_norm 2.7954 (1.8310) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][370/625] eta 0:01:31 lr 0.000646 wd 0.0500 time 0.3393 (0.3599) data time 0.0007 (0.0026) model time 0.3387 (0.3583) loss 1.8919 (2.9921) grad_norm 1.5666 (1.8344) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][380/625] eta 0:01:28 lr 0.000645 wd 0.0500 time 0.3501 (0.3596) data time 0.0008 (0.0026) model time 0.3493 (0.3580) loss 3.2466 (2.9912) grad_norm 2.1830 (1.8461) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][390/625] eta 0:01:24 lr 0.000645 wd 0.0500 time 0.3569 (0.3593) data time 0.0008 (0.0026) model time 0.3561 (0.3577) loss 3.3338 (2.9944) grad_norm 1.5746 (1.8420) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][400/625] eta 0:01:20 lr 0.000645 wd 0.0500 time 0.3474 (0.3591) data time 0.0007 (0.0025) model time 0.3467 (0.3575) loss 1.9705 (2.9899) grad_norm 1.2197 (1.8345) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][410/625] eta 0:01:17 lr 0.000645 wd 0.0500 time 0.3428 (0.3588) data time 0.0010 (0.0025) model time 0.3419 (0.3571) loss 3.2206 (2.9905) grad_norm 1.6822 (1.8290) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][420/625] eta 0:01:13 lr 0.000645 wd 0.0500 time 0.3496 (0.3585) data time 0.0006 (0.0024) model time 0.3490 (0.3569) loss 2.4219 (2.9910) grad_norm 1.0547 (1.8267) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][430/625] eta 0:01:09 lr 0.000645 wd 0.0500 time 0.3456 (0.3583) data time 0.0006 (0.0024) model time 0.3450 (0.3566) loss 3.8644 (2.9925) grad_norm 1.7296 (1.8245) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:14:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][440/625] eta 0:01:06 lr 0.000644 wd 0.0500 time 0.3524 (0.3581) data time 0.0008 (0.0024) model time 0.3516 (0.3564) loss 2.6659 (2.9873) grad_norm 3.4911 (1.8215) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][450/625] eta 0:01:02 lr 0.000644 wd 0.0500 time 0.3430 (0.3579) data time 0.0008 (0.0023) model time 0.3422 (0.3562) loss 3.4884 (2.9834) grad_norm 1.4504 (1.8359) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][460/625] eta 0:00:59 lr 0.000644 wd 0.0500 time 0.3480 (0.3577) data time 0.0008 (0.0023) model time 0.3472 (0.3560) loss 3.2672 (2.9810) grad_norm 1.6447 (1.8413) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][470/625] eta 0:00:55 lr 0.000644 wd 0.0500 time 0.3135 (0.3578) data time 0.0009 (0.0023) model time 0.3126 (0.3562) loss 2.4600 (2.9833) grad_norm 1.1900 (1.8324) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][480/625] eta 0:00:51 lr 0.000644 wd 0.0500 time 0.3529 (0.3576) data time 0.0006 (0.0022) model time 0.3522 (0.3560) loss 2.3214 (2.9800) grad_norm 2.2530 (1.8373) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][490/625] eta 0:00:48 lr 0.000644 wd 0.0500 time 0.3442 (0.3586) data time 0.0006 (0.0022) model time 0.3436 (0.3571) loss 3.0250 (2.9792) grad_norm 1.9012 (1.8422) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][500/625] eta 0:00:44 lr 0.000643 wd 0.0500 time 0.3433 (0.3584) data time 0.0009 (0.0022) model time 0.3424 (0.3568) loss 2.3211 (2.9745) grad_norm 1.4538 (1.8496) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][510/625] eta 0:00:41 lr 0.000643 wd 0.0500 time 0.3569 (0.3582) data time 0.0007 (0.0022) model time 0.3562 (0.3566) loss 3.8680 (2.9802) grad_norm 1.5994 (1.8425) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][520/625] eta 0:00:37 lr 0.000643 wd 0.0500 time 0.3517 (0.3580) data time 0.0009 (0.0021) model time 0.3508 (0.3565) loss 3.1574 (2.9826) grad_norm 1.4960 (1.8454) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][530/625] eta 0:00:33 lr 0.000643 wd 0.0500 time 0.3441 (0.3579) data time 0.0008 (0.0021) model time 0.3433 (0.3563) loss 2.9897 (2.9849) grad_norm 1.1013 (1.8410) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][540/625] eta 0:00:30 lr 0.000643 wd 0.0500 time 0.3472 (0.3577) data time 0.0009 (0.0021) model time 0.3463 (0.3561) loss 3.3146 (2.9777) grad_norm 1.6638 (1.8377) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][550/625] eta 0:00:26 lr 0.000643 wd 0.0500 time 0.3530 (0.3575) data time 0.0007 (0.0021) model time 0.3523 (0.3559) loss 1.9148 (2.9750) grad_norm 1.6348 (1.8341) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][560/625] eta 0:00:23 lr 0.000643 wd 0.0500 time 0.3513 (0.3573) data time 0.0006 (0.0021) model time 0.3507 (0.3558) loss 3.4454 (2.9704) grad_norm 1.5039 (1.8342) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][570/625] eta 0:00:19 lr 0.000642 wd 0.0500 time 0.3467 (0.3572) data time 0.0009 (0.0020) model time 0.3458 (0.3556) loss 3.2078 (2.9727) grad_norm 1.9439 (1.8467) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][580/625] eta 0:00:16 lr 0.000642 wd 0.0500 time 0.5499 (0.3584) data time 0.0008 (0.0020) model time 0.5491 (0.3570) loss 2.9177 (2.9748) grad_norm 1.7482 (1.8464) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][590/625] eta 0:00:12 lr 0.000642 wd 0.0500 time 0.3107 (0.3586) data time 0.0009 (0.0020) model time 0.3098 (0.3572) loss 3.0701 (2.9776) grad_norm 1.7264 (1.8450) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][600/625] eta 0:00:08 lr 0.000642 wd 0.0500 time 0.3468 (0.3584) data time 0.0009 (0.0020) model time 0.3459 (0.3570) loss 2.8908 (2.9761) grad_norm 2.1160 (1.8476) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][610/625] eta 0:00:05 lr 0.000642 wd 0.0500 time 0.3404 (0.3591) data time 0.0006 (0.0020) model time 0.3398 (0.3578) loss 2.7087 (2.9759) grad_norm 1.5238 (1.8445) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [193/300][620/625] eta 0:00:01 lr 0.000642 wd 0.0500 time 0.3452 (0.3589) data time 0.0005 (0.0019) model time 0.3447 (0.3575) loss 3.9049 (2.9789) grad_norm 1.5424 (1.8422) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:05 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 193 training takes 0:03:44 [2024-09-01 08:16:05 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:16:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:16:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.460 (0.460) Loss 0.5430 (0.5430) Acc@1 87.988 (87.988) Acc@5 98.145 (98.145) Mem 11224MB [2024-09-01 08:16:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.131) Loss 0.8872 (0.6756) Acc@1 79.297 (84.992) Acc@5 95.312 (97.168) Mem 11224MB [2024-09-01 08:16:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 0.9829 (0.7964) Acc@1 75.977 (81.929) Acc@5 94.238 (95.868) Mem 11224MB [2024-09-01 08:16:08 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.586 Acc@5 95.889 [2024-09-01 08:16:08 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.6% [2024-09-01 08:16:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.817 (0.817) Loss 0.5015 (0.5015) Acc@1 89.062 (89.062) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 08:16:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.167) Loss 0.8062 (0.6115) Acc@1 81.396 (86.395) Acc@5 96.094 (97.705) Mem 11224MB [2024-09-01 08:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.131) Loss 0.8999 (0.7241) Acc@1 77.588 (83.440) Acc@5 95.264 (96.561) Mem 11224MB [2024-09-01 08:16:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.145 Acc@5 96.581 [2024-09-01 08:16:12 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 08:16:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][0/625] eta 0:12:35 lr 0.000641 wd 0.0500 time 1.2085 (1.2085) data time 0.4224 (0.4224) model time 0.0000 (0.0000) loss 2.9862 (2.9862) grad_norm 1.6648 (1.6648) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][10/625] eta 0:04:21 lr 0.000641 wd 0.0500 time 0.3430 (0.4254) data time 0.0008 (0.0391) model time 0.0000 (0.0000) loss 3.0638 (3.0766) grad_norm 1.3916 (1.7781) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][20/625] eta 0:03:54 lr 0.000641 wd 0.0500 time 0.3457 (0.3880) data time 0.0008 (0.0209) model time 0.0000 (0.0000) loss 3.0089 (2.9589) grad_norm 1.4336 (1.8149) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][30/625] eta 0:03:42 lr 0.000641 wd 0.0500 time 0.3478 (0.3747) data time 0.0006 (0.0144) model time 0.0000 (0.0000) loss 2.9487 (3.0535) grad_norm 1.8418 (1.8730) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][40/625] eta 0:03:35 lr 0.000641 wd 0.0500 time 0.3555 (0.3681) data time 0.0006 (0.0111) model time 0.0000 (0.0000) loss 3.5555 (3.0208) grad_norm 1.7069 (1.8992) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][50/625] eta 0:03:29 lr 0.000641 wd 0.0500 time 0.3486 (0.3643) data time 0.0008 (0.0091) model time 0.0000 (0.0000) loss 2.9600 (2.9976) grad_norm 2.7779 (1.9380) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][60/625] eta 0:03:24 lr 0.000640 wd 0.0500 time 0.3473 (0.3619) data time 0.0008 (0.0077) model time 0.3465 (0.3486) loss 2.8689 (2.9879) grad_norm 1.5804 (1.9410) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][70/625] eta 0:03:19 lr 0.000640 wd 0.0500 time 0.3515 (0.3598) data time 0.0009 (0.0068) model time 0.3507 (0.3474) loss 3.2909 (2.9756) grad_norm 1.4778 (1.9902) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][80/625] eta 0:03:15 lr 0.000640 wd 0.0500 time 0.3483 (0.3583) data time 0.0008 (0.0060) model time 0.3476 (0.3471) loss 3.2897 (3.0102) grad_norm 3.2198 (2.0129) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][90/625] eta 0:03:11 lr 0.000640 wd 0.0500 time 0.3494 (0.3573) data time 0.0006 (0.0054) model time 0.3487 (0.3475) loss 2.3170 (2.9915) grad_norm 1.9319 (2.0023) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][100/625] eta 0:03:07 lr 0.000640 wd 0.0500 time 0.3510 (0.3563) data time 0.0007 (0.0050) model time 0.3503 (0.3473) loss 2.9689 (2.9750) grad_norm 1.7279 (1.9805) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][110/625] eta 0:03:03 lr 0.000640 wd 0.0500 time 0.3486 (0.3554) data time 0.0008 (0.0046) model time 0.3478 (0.3470) loss 3.0683 (2.9604) grad_norm 2.9259 (2.0828) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][120/625] eta 0:02:59 lr 0.000639 wd 0.0500 time 0.3478 (0.3549) data time 0.0006 (0.0043) model time 0.3472 (0.3471) loss 3.9664 (2.9753) grad_norm 1.8561 (2.0917) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][130/625] eta 0:02:55 lr 0.000639 wd 0.0500 time 0.3479 (0.3543) data time 0.0008 (0.0040) model time 0.3471 (0.3471) loss 3.5034 (2.9838) grad_norm 1.5674 (2.0527) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:17:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][140/625] eta 0:02:51 lr 0.000639 wd 0.0500 time 0.3499 (0.3540) data time 0.0007 (0.0038) model time 0.3493 (0.3473) loss 3.6624 (2.9912) grad_norm 1.9877 (2.0485) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][150/625] eta 0:02:48 lr 0.000639 wd 0.0500 time 0.3461 (0.3547) data time 0.0006 (0.0036) model time 0.3455 (0.3490) loss 2.1670 (3.0101) grad_norm 2.0950 (inf) loss_scale 512.0000 (1000.2649) mem 11224MB [2024-09-01 08:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][160/625] eta 0:02:46 lr 0.000639 wd 0.0500 time 0.5582 (0.3582) data time 0.0006 (0.0034) model time 0.5576 (0.3546) loss 2.5746 (3.0019) grad_norm 1.5582 (inf) loss_scale 512.0000 (969.9379) mem 11224MB [2024-09-01 08:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][170/625] eta 0:02:43 lr 0.000639 wd 0.0500 time 0.4694 (0.3604) data time 0.0008 (0.0033) model time 0.4686 (0.3579) loss 2.8688 (2.9959) grad_norm 1.4826 (inf) loss_scale 512.0000 (943.1579) mem 11224MB [2024-09-01 08:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][180/625] eta 0:02:40 lr 0.000638 wd 0.0500 time 0.3479 (0.3598) data time 0.0006 (0.0032) model time 0.3473 (0.3571) loss 3.2883 (2.9972) grad_norm 2.0854 (inf) loss_scale 512.0000 (919.3370) mem 11224MB [2024-09-01 08:17:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][190/625] eta 0:02:36 lr 0.000638 wd 0.0500 time 0.3480 (0.3592) data time 0.0006 (0.0030) model time 0.3473 (0.3565) loss 3.5256 (3.0157) grad_norm 2.0973 (inf) loss_scale 512.0000 (898.0105) mem 11224MB [2024-09-01 08:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][200/625] eta 0:02:32 lr 0.000638 wd 0.0500 time 0.3447 (0.3586) data time 0.0007 (0.0029) model time 0.3440 (0.3559) loss 3.1835 (3.0104) grad_norm 1.9814 (inf) loss_scale 512.0000 (878.8060) mem 11224MB [2024-09-01 08:17:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][210/625] eta 0:02:28 lr 0.000638 wd 0.0500 time 0.3462 (0.3582) data time 0.0007 (0.0028) model time 0.3456 (0.3554) loss 3.0914 (3.0119) grad_norm 1.6543 (inf) loss_scale 512.0000 (861.4218) mem 11224MB [2024-09-01 08:17:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][220/625] eta 0:02:24 lr 0.000638 wd 0.0500 time 0.3466 (0.3577) data time 0.0007 (0.0027) model time 0.3459 (0.3549) loss 3.3368 (3.0191) grad_norm 1.2359 (inf) loss_scale 512.0000 (845.6109) mem 11224MB [2024-09-01 08:17:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][230/625] eta 0:02:21 lr 0.000638 wd 0.0500 time 0.3533 (0.3574) data time 0.0009 (0.0026) model time 0.3525 (0.3546) loss 3.1382 (3.0122) grad_norm 1.3495 (inf) loss_scale 512.0000 (831.1688) mem 11224MB [2024-09-01 08:17:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][240/625] eta 0:02:17 lr 0.000637 wd 0.0500 time 0.3599 (0.3571) data time 0.0009 (0.0026) model time 0.3590 (0.3543) loss 3.0189 (3.0170) grad_norm 2.0188 (inf) loss_scale 512.0000 (817.9253) mem 11224MB [2024-09-01 08:17:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][250/625] eta 0:02:13 lr 0.000637 wd 0.0500 time 0.3480 (0.3566) data time 0.0009 (0.0025) model time 0.3471 (0.3538) loss 3.1213 (3.0237) grad_norm 2.6983 (inf) loss_scale 512.0000 (805.7371) mem 11224MB [2024-09-01 08:17:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][260/625] eta 0:02:10 lr 0.000637 wd 0.0500 time 0.3462 (0.3563) data time 0.0008 (0.0024) model time 0.3455 (0.3535) loss 3.5254 (3.0192) grad_norm 2.2408 (inf) loss_scale 512.0000 (794.4828) mem 11224MB [2024-09-01 08:17:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][270/625] eta 0:02:06 lr 0.000637 wd 0.0500 time 0.3491 (0.3560) data time 0.0009 (0.0024) model time 0.3482 (0.3532) loss 3.2213 (3.0195) grad_norm 2.3278 (inf) loss_scale 512.0000 (784.0590) mem 11224MB [2024-09-01 08:17:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][280/625] eta 0:02:02 lr 0.000637 wd 0.0500 time 0.3552 (0.3558) data time 0.0009 (0.0023) model time 0.3543 (0.3530) loss 2.2987 (3.0108) grad_norm 1.3987 (inf) loss_scale 512.0000 (774.3772) mem 11224MB [2024-09-01 08:17:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][290/625] eta 0:01:59 lr 0.000637 wd 0.0500 time 0.5349 (0.3573) data time 0.0006 (0.0023) model time 0.5343 (0.3550) loss 3.9496 (3.0129) grad_norm 1.6624 (inf) loss_scale 512.0000 (765.3608) mem 11224MB [2024-09-01 08:17:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][300/625] eta 0:01:56 lr 0.000636 wd 0.0500 time 0.3458 (0.3570) data time 0.0008 (0.0022) model time 0.3450 (0.3547) loss 3.2406 (3.0183) grad_norm 1.7840 (inf) loss_scale 512.0000 (756.9435) mem 11224MB [2024-09-01 08:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][310/625] eta 0:01:52 lr 0.000636 wd 0.0500 time 0.3495 (0.3567) data time 0.0007 (0.0022) model time 0.3488 (0.3544) loss 2.9558 (3.0184) grad_norm 1.7340 (inf) loss_scale 512.0000 (749.0675) mem 11224MB [2024-09-01 08:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][320/625] eta 0:01:48 lr 0.000636 wd 0.0500 time 0.3495 (0.3565) data time 0.0008 (0.0021) model time 0.3487 (0.3542) loss 3.3372 (3.0198) grad_norm 3.1226 (inf) loss_scale 512.0000 (741.6822) mem 11224MB [2024-09-01 08:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][330/625] eta 0:01:45 lr 0.000636 wd 0.0500 time 0.3705 (0.3568) data time 0.0008 (0.0021) model time 0.3697 (0.3546) loss 2.9742 (3.0250) grad_norm 1.0698 (inf) loss_scale 512.0000 (734.7432) mem 11224MB [2024-09-01 08:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][340/625] eta 0:01:42 lr 0.000636 wd 0.0500 time 0.5624 (0.3590) data time 0.0008 (0.0021) model time 0.5615 (0.3572) loss 2.8186 (3.0204) grad_norm 1.3487 (inf) loss_scale 512.0000 (728.2111) mem 11224MB [2024-09-01 08:18:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][350/625] eta 0:01:38 lr 0.000636 wd 0.0500 time 0.3498 (0.3593) data time 0.0006 (0.0020) model time 0.3492 (0.3576) loss 2.4378 (3.0130) grad_norm 1.9238 (inf) loss_scale 512.0000 (722.0513) mem 11224MB [2024-09-01 08:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][360/625] eta 0:01:35 lr 0.000635 wd 0.0500 time 0.3473 (0.3590) data time 0.0007 (0.0020) model time 0.3466 (0.3573) loss 3.6889 (3.0136) grad_norm 1.3902 (inf) loss_scale 512.0000 (716.2327) mem 11224MB [2024-09-01 08:18:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][370/625] eta 0:01:31 lr 0.000635 wd 0.0500 time 0.3471 (0.3587) data time 0.0007 (0.0020) model time 0.3464 (0.3570) loss 2.8204 (3.0103) grad_norm 1.5318 (inf) loss_scale 512.0000 (710.7278) mem 11224MB [2024-09-01 08:18:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][380/625] eta 0:01:27 lr 0.000635 wd 0.0500 time 0.3542 (0.3584) data time 0.0007 (0.0019) model time 0.3536 (0.3566) loss 2.7431 (3.0098) grad_norm 1.5711 (inf) loss_scale 512.0000 (705.5118) mem 11224MB [2024-09-01 08:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][390/625] eta 0:01:24 lr 0.000635 wd 0.0500 time 0.3458 (0.3582) data time 0.0006 (0.0019) model time 0.3451 (0.3564) loss 2.4771 (3.0141) grad_norm 1.8315 (inf) loss_scale 512.0000 (700.5627) mem 11224MB [2024-09-01 08:18:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][400/625] eta 0:01:20 lr 0.000635 wd 0.0500 time 0.3520 (0.3579) data time 0.0009 (0.0019) model time 0.3511 (0.3561) loss 3.5339 (3.0136) grad_norm 1.5744 (inf) loss_scale 512.0000 (695.8603) mem 11224MB [2024-09-01 08:18:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][410/625] eta 0:01:16 lr 0.000635 wd 0.0500 time 0.3499 (0.3577) data time 0.0007 (0.0019) model time 0.3492 (0.3559) loss 2.0919 (3.0064) grad_norm 2.0228 (inf) loss_scale 512.0000 (691.3869) mem 11224MB [2024-09-01 08:18:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][420/625] eta 0:01:13 lr 0.000635 wd 0.0500 time 0.3500 (0.3575) data time 0.0008 (0.0018) model time 0.3492 (0.3557) loss 2.6839 (3.0077) grad_norm 1.4210 (inf) loss_scale 512.0000 (687.1259) mem 11224MB [2024-09-01 08:18:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][430/625] eta 0:01:09 lr 0.000634 wd 0.0500 time 0.3474 (0.3572) data time 0.0008 (0.0018) model time 0.3466 (0.3554) loss 3.4775 (3.0064) grad_norm 1.5737 (inf) loss_scale 512.0000 (683.0626) mem 11224MB [2024-09-01 08:18:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][440/625] eta 0:01:06 lr 0.000634 wd 0.0500 time 0.3465 (0.3570) data time 0.0007 (0.0018) model time 0.3458 (0.3552) loss 2.7946 (3.0001) grad_norm 6.7937 (inf) loss_scale 512.0000 (679.1837) mem 11224MB [2024-09-01 08:18:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][450/625] eta 0:01:02 lr 0.000634 wd 0.0500 time 0.3439 (0.3571) data time 0.0007 (0.0018) model time 0.3432 (0.3554) loss 3.9261 (3.0041) grad_norm 2.5121 (inf) loss_scale 512.0000 (675.4767) mem 11224MB [2024-09-01 08:18:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][460/625] eta 0:00:59 lr 0.000634 wd 0.0500 time 0.3418 (0.3585) data time 0.0008 (0.0017) model time 0.3410 (0.3570) loss 3.3923 (3.0018) grad_norm 1.4542 (inf) loss_scale 512.0000 (671.9306) mem 11224MB [2024-09-01 08:19:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][470/625] eta 0:00:55 lr 0.000634 wd 0.0500 time 0.3497 (0.3588) data time 0.0008 (0.0017) model time 0.3489 (0.3573) loss 2.4438 (3.0008) grad_norm 1.2496 (inf) loss_scale 512.0000 (668.5350) mem 11224MB [2024-09-01 08:19:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][480/625] eta 0:00:51 lr 0.000634 wd 0.0500 time 0.3454 (0.3585) data time 0.0008 (0.0017) model time 0.3446 (0.3570) loss 3.6060 (3.0006) grad_norm 1.4996 (inf) loss_scale 512.0000 (665.2807) mem 11224MB [2024-09-01 08:19:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][490/625] eta 0:00:48 lr 0.000633 wd 0.0500 time 0.3453 (0.3583) data time 0.0006 (0.0017) model time 0.3446 (0.3567) loss 3.7166 (3.0034) grad_norm 2.3343 (inf) loss_scale 512.0000 (662.1589) mem 11224MB [2024-09-01 08:19:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][500/625] eta 0:00:44 lr 0.000633 wd 0.0500 time 0.3439 (0.3592) data time 0.0007 (0.0017) model time 0.3432 (0.3578) loss 3.6477 (3.0076) grad_norm 1.4440 (inf) loss_scale 512.0000 (659.1617) mem 11224MB [2024-09-01 08:19:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][510/625] eta 0:00:41 lr 0.000633 wd 0.0500 time 0.3441 (0.3589) data time 0.0009 (0.0017) model time 0.3432 (0.3575) loss 3.1313 (3.0024) grad_norm 3.1532 (inf) loss_scale 512.0000 (656.2818) mem 11224MB [2024-09-01 08:19:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][520/625] eta 0:00:37 lr 0.000633 wd 0.0500 time 0.3440 (0.3587) data time 0.0011 (0.0016) model time 0.3429 (0.3573) loss 2.3967 (2.9988) grad_norm 1.7660 (inf) loss_scale 512.0000 (653.5125) mem 11224MB [2024-09-01 08:19:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][530/625] eta 0:00:34 lr 0.000633 wd 0.0500 time 0.3457 (0.3585) data time 0.0008 (0.0016) model time 0.3448 (0.3571) loss 2.1223 (2.9946) grad_norm 1.8531 (inf) loss_scale 512.0000 (650.8475) mem 11224MB [2024-09-01 08:19:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][540/625] eta 0:00:30 lr 0.000633 wd 0.0500 time 0.3459 (0.3583) data time 0.0007 (0.0016) model time 0.3452 (0.3568) loss 2.9919 (3.0003) grad_norm 2.4068 (inf) loss_scale 512.0000 (648.2810) mem 11224MB [2024-09-01 08:19:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][550/625] eta 0:00:26 lr 0.000632 wd 0.0500 time 0.3445 (0.3581) data time 0.0008 (0.0016) model time 0.3437 (0.3566) loss 3.1140 (3.0011) grad_norm 2.2339 (inf) loss_scale 512.0000 (645.8076) mem 11224MB [2024-09-01 08:19:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][560/625] eta 0:00:23 lr 0.000632 wd 0.0500 time 0.3482 (0.3579) data time 0.0009 (0.0016) model time 0.3473 (0.3564) loss 3.1072 (2.9984) grad_norm 2.1057 (inf) loss_scale 512.0000 (643.4225) mem 11224MB [2024-09-01 08:19:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][570/625] eta 0:00:19 lr 0.000632 wd 0.0500 time 0.3438 (0.3577) data time 0.0008 (0.0016) model time 0.3430 (0.3562) loss 3.0804 (3.0012) grad_norm 1.5274 (inf) loss_scale 512.0000 (641.1208) mem 11224MB [2024-09-01 08:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][580/625] eta 0:00:16 lr 0.000632 wd 0.0500 time 0.3404 (0.3575) data time 0.0008 (0.0015) model time 0.3396 (0.3560) loss 3.4988 (3.0084) grad_norm 1.8914 (inf) loss_scale 512.0000 (638.8985) mem 11224MB [2024-09-01 08:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][590/625] eta 0:00:12 lr 0.000632 wd 0.0500 time 0.3538 (0.3573) data time 0.0009 (0.0015) model time 0.3529 (0.3559) loss 3.1674 (3.0132) grad_norm 1.3998 (inf) loss_scale 512.0000 (636.7513) mem 11224MB [2024-09-01 08:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][600/625] eta 0:00:08 lr 0.000632 wd 0.0500 time 0.3453 (0.3572) data time 0.0008 (0.0015) model time 0.3445 (0.3557) loss 2.9433 (3.0185) grad_norm 1.3154 (inf) loss_scale 512.0000 (634.6755) mem 11224MB [2024-09-01 08:19:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][610/625] eta 0:00:05 lr 0.000631 wd 0.0500 time 0.3427 (0.3570) data time 0.0005 (0.0015) model time 0.3421 (0.3555) loss 2.4228 (3.0158) grad_norm 1.3554 (inf) loss_scale 512.0000 (632.6678) mem 11224MB [2024-09-01 08:19:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [194/300][620/625] eta 0:00:01 lr 0.000631 wd 0.0500 time 0.3422 (0.3568) data time 0.0005 (0.0015) model time 0.3417 (0.3553) loss 2.7994 (3.0127) grad_norm 2.2283 (inf) loss_scale 512.0000 (630.7246) mem 11224MB [2024-09-01 08:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 194 training takes 0:03:42 [2024-09-01 08:19:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:19:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:19:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.455 (0.455) Loss 0.5625 (0.5625) Acc@1 88.184 (88.184) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 08:19:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.132) Loss 0.9380 (0.6870) Acc@1 77.930 (84.961) Acc@5 94.873 (97.235) Mem 11224MB [2024-09-01 08:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.113) Loss 0.9839 (0.8144) Acc@1 75.732 (81.748) Acc@5 94.336 (95.924) Mem 11224MB [2024-09-01 08:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.500 Acc@5 95.979 [2024-09-01 08:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.5% [2024-09-01 08:19:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.762 (0.762) Loss 0.5005 (0.5005) Acc@1 89.062 (89.062) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 08:20:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.168) Loss 0.8047 (0.6109) Acc@1 81.494 (86.452) Acc@5 95.898 (97.692) Mem 11224MB [2024-09-01 08:20:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.132) Loss 0.8975 (0.7231) Acc@1 77.637 (83.505) Acc@5 95.312 (96.561) Mem 11224MB [2024-09-01 08:20:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.197 Acc@5 96.581 [2024-09-01 08:20:01 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.2% [2024-09-01 08:20:01 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.20% [2024-09-01 08:20:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:20:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][0/625] eta 0:07:18 lr 0.000631 wd 0.0500 time 0.7009 (0.7009) data time 0.3954 (0.3954) model time 0.0000 (0.0000) loss 2.9469 (2.9469) grad_norm 2.4620 (2.4620) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][10/625] eta 0:03:53 lr 0.000631 wd 0.0500 time 0.3446 (0.3794) data time 0.0007 (0.0367) model time 0.0000 (0.0000) loss 3.2348 (3.1147) grad_norm 1.8306 (1.6315) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][20/625] eta 0:03:40 lr 0.000631 wd 0.0500 time 0.3397 (0.3648) data time 0.0007 (0.0196) model time 0.0000 (0.0000) loss 3.5616 (3.0837) grad_norm 1.3478 (1.6384) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][30/625] eta 0:03:52 lr 0.000631 wd 0.0500 time 0.3430 (0.3904) data time 0.0008 (0.0136) model time 0.0000 (0.0000) loss 3.2489 (3.0949) grad_norm 1.6051 (1.7287) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][40/625] eta 0:03:42 lr 0.000630 wd 0.0500 time 0.3473 (0.3800) data time 0.0008 (0.0105) model time 0.0000 (0.0000) loss 2.8652 (3.0948) grad_norm 2.0214 (1.9713) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][50/625] eta 0:03:35 lr 0.000630 wd 0.0500 time 0.3477 (0.3740) data time 0.0008 (0.0086) model time 0.0000 (0.0000) loss 3.0860 (3.0492) grad_norm 2.0129 (1.9712) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][60/625] eta 0:03:30 lr 0.000630 wd 0.0500 time 0.3502 (0.3722) data time 0.0007 (0.0073) model time 0.3495 (0.3624) loss 1.7372 (3.0534) grad_norm 1.9091 (1.9648) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][70/625] eta 0:03:27 lr 0.000630 wd 0.0500 time 0.3492 (0.3740) data time 0.0008 (0.0064) model time 0.3484 (0.3731) loss 3.3870 (3.0684) grad_norm 2.6705 (1.9290) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][80/625] eta 0:03:22 lr 0.000630 wd 0.0500 time 0.3513 (0.3708) data time 0.0006 (0.0057) model time 0.3506 (0.3646) loss 3.6118 (3.0847) grad_norm 1.4490 (1.9064) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][90/625] eta 0:03:17 lr 0.000630 wd 0.0500 time 0.3491 (0.3682) data time 0.0009 (0.0052) model time 0.3482 (0.3601) loss 3.0682 (3.0774) grad_norm 1.2444 (1.9114) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][100/625] eta 0:03:12 lr 0.000630 wd 0.0500 time 0.3461 (0.3662) data time 0.0006 (0.0047) model time 0.3455 (0.3575) loss 2.8192 (3.0577) grad_norm 1.5428 (1.8791) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][110/625] eta 0:03:07 lr 0.000629 wd 0.0500 time 0.3498 (0.3647) data time 0.0008 (0.0044) model time 0.3490 (0.3560) loss 3.0279 (3.0652) grad_norm 1.6475 (1.8765) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][120/625] eta 0:03:03 lr 0.000629 wd 0.0500 time 0.3482 (0.3631) data time 0.0006 (0.0041) model time 0.3476 (0.3544) loss 3.0110 (3.0463) grad_norm 1.4280 (1.8511) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][130/625] eta 0:02:59 lr 0.000629 wd 0.0500 time 0.3501 (0.3621) data time 0.0007 (0.0038) model time 0.3494 (0.3536) loss 3.4591 (3.0587) grad_norm 2.0364 (1.8336) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][140/625] eta 0:02:55 lr 0.000629 wd 0.0500 time 0.3447 (0.3610) data time 0.0007 (0.0036) model time 0.3441 (0.3529) loss 2.2903 (3.0659) grad_norm 1.8586 (1.8242) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:20:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][150/625] eta 0:02:51 lr 0.000629 wd 0.0500 time 0.3481 (0.3601) data time 0.0008 (0.0035) model time 0.3472 (0.3521) loss 3.0847 (3.0657) grad_norm 2.0992 (1.8273) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][160/625] eta 0:02:47 lr 0.000629 wd 0.0500 time 0.3528 (0.3593) data time 0.0008 (0.0033) model time 0.3520 (0.3516) loss 2.4908 (3.0711) grad_norm 1.9010 (1.8138) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][170/625] eta 0:02:43 lr 0.000628 wd 0.0500 time 0.3465 (0.3587) data time 0.0006 (0.0032) model time 0.3459 (0.3513) loss 2.4741 (3.0599) grad_norm 2.3955 (1.8080) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][180/625] eta 0:02:39 lr 0.000628 wd 0.0500 time 0.3433 (0.3591) data time 0.0006 (0.0030) model time 0.3427 (0.3524) loss 1.8115 (3.0566) grad_norm 1.7959 (1.8130) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][190/625] eta 0:02:35 lr 0.000628 wd 0.0500 time 0.3450 (0.3585) data time 0.0007 (0.0029) model time 0.3443 (0.3520) loss 2.4825 (3.0509) grad_norm 1.6009 (1.8209) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][200/625] eta 0:02:32 lr 0.000628 wd 0.0500 time 0.3453 (0.3580) data time 0.0008 (0.0028) model time 0.3445 (0.3517) loss 3.5344 (3.0642) grad_norm 1.3680 (1.8147) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][210/625] eta 0:02:28 lr 0.000628 wd 0.0500 time 0.3509 (0.3575) data time 0.0009 (0.0027) model time 0.3500 (0.3514) loss 2.7165 (3.0752) grad_norm 3.1086 (1.8315) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][220/625] eta 0:02:24 lr 0.000628 wd 0.0500 time 0.3500 (0.3571) data time 0.0006 (0.0026) model time 0.3494 (0.3512) loss 3.6937 (3.0806) grad_norm 1.5557 (1.8357) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][230/625] eta 0:02:21 lr 0.000627 wd 0.0500 time 0.5337 (0.3575) data time 0.0008 (0.0025) model time 0.5329 (0.3519) loss 2.8640 (3.0766) grad_norm 2.0426 (1.8293) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][240/625] eta 0:02:18 lr 0.000627 wd 0.0500 time 0.4826 (0.3602) data time 0.0006 (0.0025) model time 0.4820 (0.3557) loss 3.2092 (3.0671) grad_norm 1.5011 (1.8455) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][250/625] eta 0:02:15 lr 0.000627 wd 0.0500 time 0.3445 (0.3608) data time 0.0008 (0.0024) model time 0.3437 (0.3566) loss 2.9934 (3.0696) grad_norm 1.2870 (1.8514) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][260/625] eta 0:02:11 lr 0.000627 wd 0.0500 time 0.3399 (0.3604) data time 0.0008 (0.0024) model time 0.3391 (0.3562) loss 2.4264 (3.0521) grad_norm 4.9852 (1.8848) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][270/625] eta 0:02:07 lr 0.000627 wd 0.0500 time 0.3515 (0.3599) data time 0.0008 (0.0023) model time 0.3507 (0.3558) loss 3.4914 (3.0543) grad_norm 2.3321 (1.8981) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][280/625] eta 0:02:04 lr 0.000627 wd 0.0500 time 0.3425 (0.3595) data time 0.0009 (0.0022) model time 0.3416 (0.3555) loss 2.3855 (3.0455) grad_norm 1.5348 (1.8981) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][290/625] eta 0:02:00 lr 0.000626 wd 0.0500 time 0.3440 (0.3590) data time 0.0009 (0.0022) model time 0.3432 (0.3550) loss 3.1539 (3.0371) grad_norm 2.5911 (1.9260) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][300/625] eta 0:01:56 lr 0.000626 wd 0.0500 time 0.3481 (0.3587) data time 0.0007 (0.0022) model time 0.3474 (0.3547) loss 3.2573 (3.0390) grad_norm 1.4422 (1.9325) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][310/625] eta 0:01:52 lr 0.000626 wd 0.0500 time 0.3526 (0.3584) data time 0.0008 (0.0021) model time 0.3518 (0.3545) loss 2.5952 (3.0343) grad_norm 1.5950 (1.9270) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:21:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][320/625] eta 0:01:49 lr 0.000626 wd 0.0500 time 0.3461 (0.3581) data time 0.0009 (0.0021) model time 0.3452 (0.3542) loss 2.4850 (3.0397) grad_norm 2.0002 (1.9329) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][330/625] eta 0:01:45 lr 0.000626 wd 0.0500 time 0.3595 (0.3578) data time 0.0006 (0.0020) model time 0.3589 (0.3540) loss 3.6627 (3.0458) grad_norm 1.4195 (1.9400) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][340/625] eta 0:01:41 lr 0.000626 wd 0.0500 time 0.3518 (0.3575) data time 0.0006 (0.0020) model time 0.3512 (0.3537) loss 2.7614 (3.0443) grad_norm 1.4164 (1.9382) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][350/625] eta 0:01:38 lr 0.000625 wd 0.0500 time 0.3527 (0.3572) data time 0.0008 (0.0020) model time 0.3519 (0.3535) loss 3.2337 (3.0483) grad_norm 1.6904 (1.9386) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][360/625] eta 0:01:34 lr 0.000625 wd 0.0500 time 0.5378 (0.3580) data time 0.0008 (0.0019) model time 0.5369 (0.3546) loss 2.9407 (3.0463) grad_norm 1.8538 (1.9324) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][370/625] eta 0:01:31 lr 0.000625 wd 0.0500 time 0.3451 (0.3588) data time 0.0006 (0.0019) model time 0.3445 (0.3555) loss 3.4941 (3.0412) grad_norm 1.7930 (1.9355) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][380/625] eta 0:01:27 lr 0.000625 wd 0.0500 time 0.3551 (0.3585) data time 0.0008 (0.0019) model time 0.3543 (0.3553) loss 3.2549 (3.0318) grad_norm 1.8836 (1.9267) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][390/625] eta 0:01:24 lr 0.000625 wd 0.0500 time 0.3499 (0.3583) data time 0.0006 (0.0018) model time 0.3493 (0.3551) loss 2.5896 (3.0260) grad_norm 1.3886 (1.9218) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][400/625] eta 0:01:20 lr 0.000625 wd 0.0500 time 0.3488 (0.3581) data time 0.0009 (0.0018) model time 0.3479 (0.3549) loss 3.4029 (3.0198) grad_norm 2.4881 (1.9253) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][410/625] eta 0:01:16 lr 0.000624 wd 0.0500 time 0.3456 (0.3578) data time 0.0006 (0.0018) model time 0.3450 (0.3547) loss 3.6312 (3.0220) grad_norm 1.4728 (1.9252) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][420/625] eta 0:01:13 lr 0.000624 wd 0.0500 time 0.3499 (0.3599) data time 0.0008 (0.0018) model time 0.3491 (0.3572) loss 3.0902 (3.0216) grad_norm 1.5502 (1.9181) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][430/625] eta 0:01:10 lr 0.000624 wd 0.0500 time 0.3492 (0.3602) data time 0.0008 (0.0017) model time 0.3484 (0.3575) loss 2.9011 (3.0218) grad_norm 1.7002 (1.9140) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][440/625] eta 0:01:06 lr 0.000624 wd 0.0500 time 0.3496 (0.3599) data time 0.0006 (0.0017) model time 0.3490 (0.3572) loss 3.4854 (3.0186) grad_norm 1.5719 (1.9070) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][450/625] eta 0:01:02 lr 0.000624 wd 0.0500 time 0.3526 (0.3597) data time 0.0006 (0.0017) model time 0.3520 (0.3571) loss 1.9512 (3.0207) grad_norm 1.9743 (1.9012) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][460/625] eta 0:00:59 lr 0.000624 wd 0.0500 time 0.3534 (0.3595) data time 0.0008 (0.0017) model time 0.3526 (0.3569) loss 2.7634 (3.0242) grad_norm 1.4438 (1.9037) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][470/625] eta 0:00:55 lr 0.000623 wd 0.0500 time 0.3448 (0.3593) data time 0.0008 (0.0017) model time 0.3440 (0.3567) loss 3.2431 (3.0189) grad_norm 1.8861 (1.8981) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][480/625] eta 0:00:52 lr 0.000623 wd 0.0500 time 0.3527 (0.3594) data time 0.0008 (0.0016) model time 0.3520 (0.3568) loss 2.5900 (3.0201) grad_norm 1.4744 (1.8929) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:22:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][490/625] eta 0:00:48 lr 0.000623 wd 0.0500 time 0.3435 (0.3591) data time 0.0007 (0.0016) model time 0.3429 (0.3565) loss 3.6817 (3.0188) grad_norm 1.7901 (1.8908) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][500/625] eta 0:00:44 lr 0.000623 wd 0.0500 time 0.3568 (0.3589) data time 0.0006 (0.0016) model time 0.3562 (0.3563) loss 3.4482 (3.0247) grad_norm 1.5342 (1.8886) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][510/625] eta 0:00:41 lr 0.000623 wd 0.0500 time 0.3447 (0.3586) data time 0.0008 (0.0016) model time 0.3438 (0.3561) loss 3.0964 (3.0229) grad_norm 1.6715 (1.8956) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][520/625] eta 0:00:37 lr 0.000623 wd 0.0500 time 0.3418 (0.3584) data time 0.0006 (0.0016) model time 0.3412 (0.3559) loss 2.7302 (3.0178) grad_norm 1.8262 (1.9102) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][530/625] eta 0:00:34 lr 0.000622 wd 0.0500 time 0.3475 (0.3582) data time 0.0008 (0.0016) model time 0.3468 (0.3557) loss 1.9549 (3.0182) grad_norm 1.5561 (1.9083) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][540/625] eta 0:00:30 lr 0.000622 wd 0.0500 time 0.3495 (0.3595) data time 0.0008 (0.0016) model time 0.3487 (0.3571) loss 3.2668 (3.0223) grad_norm 1.7686 (1.8997) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][550/625] eta 0:00:26 lr 0.000622 wd 0.0500 time 0.3496 (0.3597) data time 0.0008 (0.0015) model time 0.3487 (0.3574) loss 2.6784 (3.0212) grad_norm 1.2326 (1.8944) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][560/625] eta 0:00:23 lr 0.000622 wd 0.0500 time 0.3455 (0.3595) data time 0.0006 (0.0015) model time 0.3448 (0.3572) loss 3.7301 (3.0209) grad_norm 1.4294 (1.8886) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][570/625] eta 0:00:19 lr 0.000622 wd 0.0500 time 0.3405 (0.3595) data time 0.0007 (0.0015) model time 0.3399 (0.3573) loss 2.3215 (3.0124) grad_norm 1.3608 (1.8846) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][580/625] eta 0:00:16 lr 0.000622 wd 0.0500 time 0.3513 (0.3600) data time 0.0007 (0.0015) model time 0.3506 (0.3578) loss 2.9238 (3.0133) grad_norm 3.1560 (1.8820) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][590/625] eta 0:00:12 lr 0.000621 wd 0.0500 time 0.3512 (0.3598) data time 0.0006 (0.0015) model time 0.3506 (0.3576) loss 2.6347 (3.0114) grad_norm 2.3266 (1.8779) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][600/625] eta 0:00:08 lr 0.000621 wd 0.0500 time 0.3474 (0.3596) data time 0.0006 (0.0015) model time 0.3467 (0.3574) loss 3.4050 (3.0120) grad_norm 1.4542 (1.8765) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][610/625] eta 0:00:05 lr 0.000621 wd 0.0500 time 0.3414 (0.3594) data time 0.0006 (0.0015) model time 0.3408 (0.3572) loss 3.0853 (3.0136) grad_norm 2.3341 (1.8836) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [195/300][620/625] eta 0:00:01 lr 0.000621 wd 0.0500 time 0.3540 (0.3591) data time 0.0006 (0.0015) model time 0.3534 (0.3570) loss 2.6707 (3.0099) grad_norm 2.2368 (1.9010) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 195 training takes 0:03:44 [2024-09-01 08:23:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:23:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:23:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 0.5718 (0.5718) Acc@1 88.232 (88.232) Acc@5 98.193 (98.193) Mem 11224MB [2024-09-01 08:23:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.145) Loss 0.8970 (0.6843) Acc@1 79.443 (85.107) Acc@5 95.312 (97.385) Mem 11224MB [2024-09-01 08:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.120) Loss 0.9951 (0.8103) Acc@1 77.246 (81.968) Acc@5 94.189 (95.964) Mem 11224MB [2024-09-01 08:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.598 Acc@5 95.951 [2024-09-01 08:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.6% [2024-09-01 08:23:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.785 (0.785) Loss 0.5000 (0.5000) Acc@1 89.014 (89.014) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 08:23:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.166) Loss 0.8062 (0.6108) Acc@1 81.299 (86.417) Acc@5 95.947 (97.696) Mem 11224MB [2024-09-01 08:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.130) Loss 0.8950 (0.7227) Acc@1 77.686 (83.498) Acc@5 95.459 (96.554) Mem 11224MB [2024-09-01 08:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.187 Acc@5 96.581 [2024-09-01 08:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.2% [2024-09-01 08:23:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][0/625] eta 0:12:14 lr 0.000621 wd 0.0500 time 1.1747 (1.1747) data time 0.5850 (0.5850) model time 0.0000 (0.0000) loss 3.1111 (3.1111) grad_norm 1.8194 (1.8194) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:23:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][10/625] eta 0:04:20 lr 0.000621 wd 0.0500 time 0.3452 (0.4229) data time 0.0010 (0.0539) model time 0.0000 (0.0000) loss 2.5735 (3.0741) grad_norm 2.0930 (1.6934) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][20/625] eta 0:03:54 lr 0.000621 wd 0.0500 time 0.3432 (0.3874) data time 0.0007 (0.0287) model time 0.0000 (0.0000) loss 3.2375 (3.0737) grad_norm 1.8463 (1.7566) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][30/625] eta 0:03:43 lr 0.000620 wd 0.0500 time 0.3480 (0.3751) data time 0.0009 (0.0197) model time 0.0000 (0.0000) loss 2.2720 (3.0518) grad_norm 1.3786 (1.8005) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][40/625] eta 0:03:37 lr 0.000620 wd 0.0500 time 0.3470 (0.3723) data time 0.0007 (0.0151) model time 0.0000 (0.0000) loss 3.0925 (3.0171) grad_norm 2.6994 (1.8478) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][50/625] eta 0:03:31 lr 0.000620 wd 0.0500 time 0.3464 (0.3677) data time 0.0008 (0.0123) model time 0.0000 (0.0000) loss 3.5363 (3.0741) grad_norm 2.1828 (1.9183) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][60/625] eta 0:03:25 lr 0.000620 wd 0.0500 time 0.3559 (0.3645) data time 0.0006 (0.0104) model time 0.3553 (0.3470) loss 3.7070 (3.0403) grad_norm 1.7690 (1.9550) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][70/625] eta 0:03:26 lr 0.000620 wd 0.0500 time 0.5276 (0.3718) data time 0.0006 (0.0091) model time 0.5270 (0.3814) loss 3.7295 (3.0604) grad_norm 1.8713 (1.9738) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][80/625] eta 0:03:22 lr 0.000620 wd 0.0500 time 0.3460 (0.3715) data time 0.0008 (0.0081) model time 0.3452 (0.3772) loss 3.0009 (3.1051) grad_norm 2.1068 (1.9613) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][90/625] eta 0:03:17 lr 0.000619 wd 0.0500 time 0.3443 (0.3687) data time 0.0006 (0.0073) model time 0.3437 (0.3690) loss 2.1939 (3.0821) grad_norm 1.3699 (1.9230) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][100/625] eta 0:03:12 lr 0.000619 wd 0.0500 time 0.3463 (0.3664) data time 0.0008 (0.0066) model time 0.3455 (0.3643) loss 2.8434 (3.0516) grad_norm 1.5111 (1.8994) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][110/625] eta 0:03:07 lr 0.000619 wd 0.0500 time 0.3509 (0.3647) data time 0.0009 (0.0061) model time 0.3500 (0.3613) loss 2.5544 (3.0360) grad_norm 1.9306 (1.8795) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][120/625] eta 0:03:03 lr 0.000619 wd 0.0500 time 0.3481 (0.3632) data time 0.0008 (0.0057) model time 0.3473 (0.3590) loss 2.4865 (3.0134) grad_norm 3.7515 (1.9223) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][130/625] eta 0:02:59 lr 0.000619 wd 0.0500 time 0.3458 (0.3620) data time 0.0007 (0.0053) model time 0.3451 (0.3576) loss 3.6051 (3.0014) grad_norm 1.9891 (1.9341) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][140/625] eta 0:02:55 lr 0.000619 wd 0.0500 time 0.3470 (0.3610) data time 0.0007 (0.0050) model time 0.3464 (0.3563) loss 3.3403 (2.9991) grad_norm 2.4572 (1.9490) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][150/625] eta 0:02:51 lr 0.000618 wd 0.0500 time 0.3500 (0.3615) data time 0.0009 (0.0047) model time 0.3491 (0.3574) loss 3.2600 (3.0038) grad_norm 2.3177 (1.9638) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][160/625] eta 0:02:48 lr 0.000618 wd 0.0500 time 0.3534 (0.3630) data time 0.0008 (0.0045) model time 0.3525 (0.3599) loss 3.0991 (2.9989) grad_norm 2.6791 (1.9519) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][170/625] eta 0:02:44 lr 0.000618 wd 0.0500 time 0.3439 (0.3621) data time 0.0006 (0.0042) model time 0.3433 (0.3589) loss 2.3116 (3.0056) grad_norm 2.1090 (1.9364) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:24:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][180/625] eta 0:02:40 lr 0.000618 wd 0.0500 time 0.3542 (0.3614) data time 0.0006 (0.0041) model time 0.3536 (0.3580) loss 2.0900 (3.0036) grad_norm 3.4012 (1.9339) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][190/625] eta 0:02:36 lr 0.000618 wd 0.0500 time 0.3426 (0.3607) data time 0.0008 (0.0039) model time 0.3419 (0.3573) loss 2.0808 (2.9990) grad_norm 1.7686 (1.9294) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][200/625] eta 0:02:33 lr 0.000618 wd 0.0500 time 0.3540 (0.3601) data time 0.0009 (0.0037) model time 0.3531 (0.3567) loss 2.9793 (2.9962) grad_norm 2.2671 (1.9162) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][210/625] eta 0:02:29 lr 0.000617 wd 0.0500 time 0.3445 (0.3595) data time 0.0006 (0.0036) model time 0.3439 (0.3561) loss 3.4194 (2.9923) grad_norm 2.7457 (1.9300) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][220/625] eta 0:02:25 lr 0.000617 wd 0.0500 time 0.3523 (0.3595) data time 0.0006 (0.0035) model time 0.3517 (0.3562) loss 2.8465 (2.9933) grad_norm 1.8321 (1.9257) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][230/625] eta 0:02:21 lr 0.000617 wd 0.0500 time 0.3480 (0.3591) data time 0.0009 (0.0034) model time 0.3472 (0.3558) loss 3.3635 (2.9991) grad_norm 1.4247 (1.9145) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][240/625] eta 0:02:18 lr 0.000617 wd 0.0500 time 0.3531 (0.3587) data time 0.0009 (0.0033) model time 0.3523 (0.3555) loss 2.6829 (2.9935) grad_norm 1.9116 (1.8996) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][250/625] eta 0:02:14 lr 0.000617 wd 0.0500 time 0.3489 (0.3583) data time 0.0007 (0.0032) model time 0.3481 (0.3551) loss 1.6747 (2.9872) grad_norm 2.1859 (1.8971) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][260/625] eta 0:02:10 lr 0.000617 wd 0.0500 time 0.3490 (0.3580) data time 0.0009 (0.0031) model time 0.3481 (0.3548) loss 3.1189 (2.9941) grad_norm 2.5959 (1.9159) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][270/625] eta 0:02:06 lr 0.000616 wd 0.0500 time 0.3497 (0.3576) data time 0.0008 (0.0030) model time 0.3489 (0.3544) loss 2.2910 (2.9910) grad_norm 1.4952 (1.9190) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][280/625] eta 0:02:03 lr 0.000616 wd 0.0500 time 0.5769 (0.3594) data time 0.0009 (0.0029) model time 0.5761 (0.3567) loss 3.2648 (2.9932) grad_norm 1.3046 (1.9123) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][290/625] eta 0:02:00 lr 0.000616 wd 0.0500 time 0.3500 (0.3590) data time 0.0006 (0.0028) model time 0.3494 (0.3563) loss 3.5615 (2.9982) grad_norm 2.3199 (1.9137) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][300/625] eta 0:01:56 lr 0.000616 wd 0.0500 time 0.3462 (0.3586) data time 0.0009 (0.0028) model time 0.3453 (0.3559) loss 2.8616 (2.9993) grad_norm 1.6886 (1.9182) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][310/625] eta 0:01:52 lr 0.000616 wd 0.0500 time 0.3564 (0.3583) data time 0.0010 (0.0027) model time 0.3555 (0.3556) loss 2.4030 (3.0026) grad_norm 1.8157 (1.9161) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][320/625] eta 0:01:49 lr 0.000616 wd 0.0500 time 0.3470 (0.3581) data time 0.0008 (0.0027) model time 0.3462 (0.3554) loss 3.0550 (2.9996) grad_norm 1.7701 (1.9029) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][330/625] eta 0:01:45 lr 0.000616 wd 0.0500 time 0.3443 (0.3578) data time 0.0008 (0.0026) model time 0.3434 (0.3551) loss 3.0463 (3.0064) grad_norm 2.3856 (1.9089) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:25:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][340/625] eta 0:01:42 lr 0.000615 wd 0.0500 time 0.3445 (0.3595) data time 0.0008 (0.0025) model time 0.3436 (0.3572) loss 3.0525 (3.0091) grad_norm 1.9599 (1.9021) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][350/625] eta 0:01:38 lr 0.000615 wd 0.0500 time 0.3467 (0.3591) data time 0.0009 (0.0025) model time 0.3458 (0.3568) loss 2.4855 (3.0026) grad_norm 1.3115 (1.9141) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][360/625] eta 0:01:35 lr 0.000615 wd 0.0500 time 0.3428 (0.3589) data time 0.0007 (0.0024) model time 0.3420 (0.3566) loss 3.8777 (2.9970) grad_norm 1.7809 (1.9318) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][370/625] eta 0:01:31 lr 0.000615 wd 0.0500 time 0.3456 (0.3586) data time 0.0007 (0.0024) model time 0.3448 (0.3563) loss 2.9023 (2.9906) grad_norm 2.4590 (1.9340) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][380/625] eta 0:01:27 lr 0.000615 wd 0.0500 time 0.3493 (0.3583) data time 0.0007 (0.0024) model time 0.3486 (0.3561) loss 3.5280 (3.0025) grad_norm 2.0601 (1.9345) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][390/625] eta 0:01:24 lr 0.000615 wd 0.0500 time 0.3455 (0.3581) data time 0.0008 (0.0023) model time 0.3447 (0.3558) loss 3.4010 (3.0023) grad_norm 1.4057 (1.9279) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][400/625] eta 0:01:20 lr 0.000614 wd 0.0500 time 0.3515 (0.3579) data time 0.0009 (0.0023) model time 0.3506 (0.3556) loss 3.0915 (3.0035) grad_norm 1.5523 (1.9199) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][410/625] eta 0:01:16 lr 0.000614 wd 0.0500 time 0.3591 (0.3577) data time 0.0006 (0.0023) model time 0.3585 (0.3554) loss 1.8792 (3.0002) grad_norm 1.3325 (1.9198) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][420/625] eta 0:01:13 lr 0.000614 wd 0.0500 time 0.3471 (0.3575) data time 0.0009 (0.0022) model time 0.3462 (0.3552) loss 2.4962 (3.0018) grad_norm 1.9236 (1.9207) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][430/625] eta 0:01:09 lr 0.000614 wd 0.0500 time 0.3448 (0.3573) data time 0.0007 (0.0022) model time 0.3442 (0.3550) loss 3.3634 (3.0040) grad_norm 2.1993 (1.9144) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][440/625] eta 0:01:06 lr 0.000614 wd 0.0500 time 0.3509 (0.3571) data time 0.0009 (0.0022) model time 0.3500 (0.3549) loss 3.1648 (2.9987) grad_norm 1.2401 (1.9031) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][450/625] eta 0:01:02 lr 0.000614 wd 0.0500 time 0.5189 (0.3573) data time 0.0006 (0.0021) model time 0.5183 (0.3551) loss 2.2800 (2.9936) grad_norm 1.5427 (1.9056) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][460/625] eta 0:00:59 lr 0.000613 wd 0.0500 time 0.4926 (0.3586) data time 0.0009 (0.0021) model time 0.4917 (0.3566) loss 3.1823 (2.9970) grad_norm 3.0299 (1.9499) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][470/625] eta 0:00:55 lr 0.000613 wd 0.0500 time 0.3477 (0.3595) data time 0.0007 (0.0021) model time 0.3470 (0.3576) loss 2.4418 (2.9860) grad_norm 2.2434 (1.9534) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][480/625] eta 0:00:52 lr 0.000613 wd 0.0500 time 0.3481 (0.3593) data time 0.0006 (0.0021) model time 0.3475 (0.3575) loss 3.2665 (2.9863) grad_norm 3.1043 (1.9526) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][490/625] eta 0:00:48 lr 0.000613 wd 0.0500 time 0.3542 (0.3591) data time 0.0009 (0.0020) model time 0.3534 (0.3573) loss 3.1596 (2.9805) grad_norm 1.9683 (1.9492) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][500/625] eta 0:00:44 lr 0.000613 wd 0.0500 time 0.3476 (0.3589) data time 0.0009 (0.0020) model time 0.3467 (0.3571) loss 3.4081 (2.9816) grad_norm 1.7608 (1.9541) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:26:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][510/625] eta 0:00:41 lr 0.000613 wd 0.0500 time 0.3527 (0.3587) data time 0.0010 (0.0020) model time 0.3516 (0.3569) loss 2.6574 (2.9783) grad_norm 1.2310 (1.9508) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][520/625] eta 0:00:37 lr 0.000612 wd 0.0500 time 0.3532 (0.3585) data time 0.0006 (0.0020) model time 0.3526 (0.3567) loss 3.4990 (2.9799) grad_norm 1.7032 (1.9458) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][530/625] eta 0:00:34 lr 0.000612 wd 0.0500 time 0.3421 (0.3584) data time 0.0008 (0.0019) model time 0.3413 (0.3565) loss 3.0022 (2.9813) grad_norm 1.8687 (1.9760) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][540/625] eta 0:00:30 lr 0.000612 wd 0.0500 time 0.3442 (0.3582) data time 0.0007 (0.0019) model time 0.3435 (0.3563) loss 3.2509 (2.9852) grad_norm 1.9490 (1.9764) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][550/625] eta 0:00:26 lr 0.000612 wd 0.0500 time 0.3491 (0.3583) data time 0.0009 (0.0019) model time 0.3483 (0.3565) loss 3.3905 (2.9871) grad_norm 1.2746 (1.9737) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][560/625] eta 0:00:23 lr 0.000612 wd 0.0500 time 0.3493 (0.3581) data time 0.0006 (0.0019) model time 0.3487 (0.3563) loss 2.0259 (2.9844) grad_norm 1.7625 (1.9657) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][570/625] eta 0:00:19 lr 0.000612 wd 0.0500 time 0.3445 (0.3579) data time 0.0006 (0.0019) model time 0.3438 (0.3561) loss 2.6602 (2.9835) grad_norm 1.5880 (1.9608) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][580/625] eta 0:00:16 lr 0.000611 wd 0.0500 time 0.3432 (0.3583) data time 0.0007 (0.0018) model time 0.3426 (0.3566) loss 3.4819 (2.9861) grad_norm 5.0620 (1.9738) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][590/625] eta 0:00:12 lr 0.000611 wd 0.0500 time 0.3474 (0.3589) data time 0.0009 (0.0018) model time 0.3465 (0.3572) loss 3.0933 (2.9873) grad_norm 1.3946 (1.9754) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][600/625] eta 0:00:08 lr 0.000611 wd 0.0500 time 0.3457 (0.3587) data time 0.0008 (0.0018) model time 0.3449 (0.3570) loss 3.5101 (2.9894) grad_norm 1.5600 (1.9707) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][610/625] eta 0:00:05 lr 0.000611 wd 0.0500 time 0.3415 (0.3585) data time 0.0004 (0.0018) model time 0.3411 (0.3568) loss 2.5576 (2.9866) grad_norm 2.2019 (1.9691) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [196/300][620/625] eta 0:00:01 lr 0.000611 wd 0.0500 time 0.3416 (0.3582) data time 0.0006 (0.0018) model time 0.3410 (0.3565) loss 2.8861 (2.9863) grad_norm 2.2326 (1.9694) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 196 training takes 0:03:43 [2024-09-01 08:27:38 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:27:39 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:27:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.447 (0.447) Loss 0.5615 (0.5615) Acc@1 87.549 (87.549) Acc@5 98.389 (98.389) Mem 11224MB [2024-09-01 08:27:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.130) Loss 0.8765 (0.6739) Acc@1 79.004 (85.134) Acc@5 95.410 (97.341) Mem 11224MB [2024-09-01 08:27:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 1.0088 (0.8035) Acc@1 75.293 (81.822) Acc@5 93.994 (95.933) Mem 11224MB [2024-09-01 08:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.522 Acc@5 95.929 [2024-09-01 08:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.5% [2024-09-01 08:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.807 (0.807) Loss 0.4990 (0.4990) Acc@1 89.160 (89.160) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 08:27:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.169) Loss 0.8008 (0.6095) Acc@1 81.152 (86.399) Acc@5 95.996 (97.692) Mem 11224MB [2024-09-01 08:27:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.132) Loss 0.8945 (0.7216) Acc@1 77.490 (83.489) Acc@5 95.508 (96.566) Mem 11224MB [2024-09-01 08:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.177 Acc@5 96.593 [2024-09-01 08:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.2% [2024-09-01 08:27:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][0/625] eta 0:11:57 lr 0.000611 wd 0.0500 time 1.1481 (1.1481) data time 0.4892 (0.4892) model time 0.0000 (0.0000) loss 2.9965 (2.9965) grad_norm 3.0168 (3.0168) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][10/625] eta 0:04:18 lr 0.000611 wd 0.0500 time 0.3449 (0.4196) data time 0.0006 (0.0452) model time 0.0000 (0.0000) loss 2.6642 (3.0642) grad_norm 1.8992 (2.0199) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][20/625] eta 0:03:56 lr 0.000610 wd 0.0500 time 0.4534 (0.3909) data time 0.0008 (0.0240) model time 0.0000 (0.0000) loss 3.0784 (3.1790) grad_norm 1.5557 (1.8771) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:27:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][30/625] eta 0:03:50 lr 0.000610 wd 0.0500 time 0.3452 (0.3866) data time 0.0008 (0.0165) model time 0.0000 (0.0000) loss 2.9954 (3.1146) grad_norm 2.6702 (1.8491) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][40/625] eta 0:03:40 lr 0.000610 wd 0.0500 time 0.3449 (0.3769) data time 0.0009 (0.0127) model time 0.0000 (0.0000) loss 3.3168 (3.1194) grad_norm 1.9408 (1.8768) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][50/625] eta 0:03:33 lr 0.000610 wd 0.0500 time 0.3453 (0.3713) data time 0.0007 (0.0104) model time 0.0000 (0.0000) loss 3.2103 (3.1336) grad_norm 1.8488 (1.9078) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][60/625] eta 0:03:27 lr 0.000610 wd 0.0500 time 0.3484 (0.3674) data time 0.0007 (0.0088) model time 0.3477 (0.3467) loss 3.2264 (3.1141) grad_norm 1.8793 (1.8914) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][70/625] eta 0:03:22 lr 0.000610 wd 0.0500 time 0.3489 (0.3646) data time 0.0006 (0.0077) model time 0.3483 (0.3468) loss 3.3383 (3.1204) grad_norm 1.2820 (1.8818) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][80/625] eta 0:03:17 lr 0.000609 wd 0.0500 time 0.3467 (0.3626) data time 0.0008 (0.0069) model time 0.3459 (0.3470) loss 2.3764 (3.1053) grad_norm 1.5030 (1.8699) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][90/625] eta 0:03:13 lr 0.000609 wd 0.0500 time 0.3582 (0.3624) data time 0.0006 (0.0062) model time 0.3576 (0.3503) loss 2.5032 (3.0749) grad_norm 1.8055 (1.8550) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][100/625] eta 0:03:12 lr 0.000609 wd 0.0500 time 0.3469 (0.3665) data time 0.0008 (0.0057) model time 0.3460 (0.3607) loss 2.9716 (3.0564) grad_norm 1.4014 (1.8221) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][110/625] eta 0:03:07 lr 0.000609 wd 0.0500 time 0.3465 (0.3648) data time 0.0008 (0.0052) model time 0.3457 (0.3584) loss 3.1174 (3.0671) grad_norm 2.2081 (1.8285) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][120/625] eta 0:03:03 lr 0.000609 wd 0.0500 time 0.3474 (0.3634) data time 0.0006 (0.0049) model time 0.3468 (0.3567) loss 3.1855 (3.0802) grad_norm 2.1906 (1.8365) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][130/625] eta 0:02:59 lr 0.000609 wd 0.0500 time 0.3498 (0.3621) data time 0.0006 (0.0046) model time 0.3492 (0.3554) loss 4.3079 (3.0692) grad_norm 3.4667 (1.8524) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][140/625] eta 0:02:55 lr 0.000608 wd 0.0500 time 0.3465 (0.3611) data time 0.0007 (0.0043) model time 0.3458 (0.3545) loss 2.6572 (3.0683) grad_norm 1.1706 (1.8377) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][150/625] eta 0:02:51 lr 0.000608 wd 0.0500 time 0.3454 (0.3602) data time 0.0010 (0.0041) model time 0.3444 (0.3537) loss 3.4685 (3.0571) grad_norm 2.0700 (1.8276) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][160/625] eta 0:02:47 lr 0.000608 wd 0.0500 time 0.3454 (0.3594) data time 0.0007 (0.0039) model time 0.3447 (0.3530) loss 3.8243 (3.0626) grad_norm 1.6399 (1.8289) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][170/625] eta 0:02:43 lr 0.000608 wd 0.0500 time 0.3469 (0.3586) data time 0.0007 (0.0037) model time 0.3463 (0.3524) loss 2.3058 (3.0563) grad_norm 1.3821 (1.8190) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][180/625] eta 0:02:39 lr 0.000608 wd 0.0500 time 0.3498 (0.3582) data time 0.0007 (0.0035) model time 0.3491 (0.3522) loss 2.7798 (3.0576) grad_norm 1.4286 (1.8198) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][190/625] eta 0:02:35 lr 0.000608 wd 0.0500 time 0.3457 (0.3576) data time 0.0008 (0.0034) model time 0.3450 (0.3518) loss 2.4942 (3.0493) grad_norm 1.7666 (1.8106) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:28:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][200/625] eta 0:02:31 lr 0.000607 wd 0.0500 time 0.3421 (0.3571) data time 0.0006 (0.0033) model time 0.3415 (0.3515) loss 3.2401 (3.0382) grad_norm 1.4202 (1.8155) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:29:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][210/625] eta 0:02:28 lr 0.000607 wd 0.0500 time 0.3486 (0.3574) data time 0.0009 (0.0031) model time 0.3477 (0.3522) loss 2.7595 (3.0430) grad_norm 2.2169 (1.8162) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][220/625] eta 0:02:24 lr 0.000607 wd 0.0500 time 0.3461 (0.3570) data time 0.0007 (0.0030) model time 0.3454 (0.3519) loss 3.2244 (3.0388) grad_norm 1.6905 (1.8153) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][230/625] eta 0:02:21 lr 0.000607 wd 0.0500 time 0.5355 (0.3575) data time 0.0009 (0.0029) model time 0.5346 (0.3527) loss 3.0089 (3.0234) grad_norm 1.5394 (1.8075) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][240/625] eta 0:02:17 lr 0.000607 wd 0.0500 time 0.3453 (0.3578) data time 0.0006 (0.0029) model time 0.3447 (0.3534) loss 2.0667 (3.0171) grad_norm 1.3921 (1.8003) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:29:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][250/625] eta 0:02:14 lr 0.000607 wd 0.0500 time 0.3505 (0.3574) data time 0.0006 (0.0028) model time 0.3499 (0.3530) loss 1.8328 (3.0190) grad_norm 1.9737 (1.8180) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:29:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][260/625] eta 0:02:10 lr 0.000606 wd 0.0500 time 0.3485 (0.3571) data time 0.0007 (0.0027) model time 0.3478 (0.3528) loss 1.9549 (3.0149) grad_norm 1.3854 (1.8204) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:29:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][270/625] eta 0:02:06 lr 0.000606 wd 0.0500 time 0.3485 (0.3568) data time 0.0008 (0.0026) model time 0.3477 (0.3526) loss 3.2408 (3.0162) grad_norm 2.2081 (1.8119) loss_scale 1024.0000 (515.7786) mem 11224MB [2024-09-01 08:29:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][280/625] eta 0:02:02 lr 0.000606 wd 0.0500 time 0.3428 (0.3564) data time 0.0006 (0.0026) model time 0.3422 (0.3523) loss 1.9414 (3.0207) grad_norm 1.9476 (1.8060) loss_scale 1024.0000 (533.8648) mem 11224MB [2024-09-01 08:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][290/625] eta 0:01:59 lr 0.000606 wd 0.0500 time 0.3470 (0.3562) data time 0.0009 (0.0025) model time 0.3461 (0.3521) loss 3.5286 (3.0102) grad_norm 1.8804 (1.8012) loss_scale 1024.0000 (550.7079) mem 11224MB [2024-09-01 08:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][300/625] eta 0:01:55 lr 0.000606 wd 0.0500 time 0.3511 (0.3559) data time 0.0009 (0.0024) model time 0.3502 (0.3519) loss 2.7357 (3.0095) grad_norm 1.7498 (1.7994) loss_scale 1024.0000 (566.4319) mem 11224MB [2024-09-01 08:29:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][310/625] eta 0:01:52 lr 0.000606 wd 0.0500 time 0.3522 (0.3571) data time 0.0006 (0.0024) model time 0.3516 (0.3535) loss 2.9041 (3.0159) grad_norm 2.5184 (1.8041) loss_scale 1024.0000 (581.1447) mem 11224MB [2024-09-01 08:29:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][320/625] eta 0:01:48 lr 0.000606 wd 0.0500 time 0.3432 (0.3568) data time 0.0006 (0.0023) model time 0.3426 (0.3532) loss 1.8521 (3.0077) grad_norm 1.5898 (1.8024) loss_scale 1024.0000 (594.9408) mem 11224MB [2024-09-01 08:29:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][330/625] eta 0:01:45 lr 0.000605 wd 0.0500 time 0.3481 (0.3565) data time 0.0006 (0.0023) model time 0.3475 (0.3530) loss 3.8921 (3.0116) grad_norm 2.0611 (1.8078) loss_scale 1024.0000 (607.9033) mem 11224MB [2024-09-01 08:29:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][340/625] eta 0:01:41 lr 0.000605 wd 0.0500 time 0.3474 (0.3563) data time 0.0008 (0.0023) model time 0.3466 (0.3528) loss 3.1516 (3.0134) grad_norm 3.0952 (1.8297) loss_scale 1024.0000 (620.1056) mem 11224MB [2024-09-01 08:29:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][350/625] eta 0:01:37 lr 0.000605 wd 0.0500 time 0.3462 (0.3560) data time 0.0006 (0.0022) model time 0.3456 (0.3526) loss 2.3378 (3.0150) grad_norm 1.8233 (1.8372) loss_scale 1024.0000 (631.6125) mem 11224MB [2024-09-01 08:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][360/625] eta 0:01:34 lr 0.000605 wd 0.0500 time 0.3415 (0.3558) data time 0.0007 (0.0022) model time 0.3409 (0.3524) loss 3.2052 (3.0127) grad_norm 1.7290 (1.8344) loss_scale 1024.0000 (642.4820) mem 11224MB [2024-09-01 08:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][370/625] eta 0:01:30 lr 0.000605 wd 0.0500 time 0.3495 (0.3556) data time 0.0005 (0.0021) model time 0.3490 (0.3522) loss 3.2147 (3.0108) grad_norm 1.8188 (1.8310) loss_scale 1024.0000 (652.7655) mem 11224MB [2024-09-01 08:30:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][380/625] eta 0:01:27 lr 0.000605 wd 0.0500 time 0.3502 (0.3554) data time 0.0008 (0.0021) model time 0.3493 (0.3521) loss 2.9977 (3.0152) grad_norm 1.5157 (1.8228) loss_scale 1024.0000 (662.5092) mem 11224MB [2024-09-01 08:30:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][390/625] eta 0:01:23 lr 0.000604 wd 0.0500 time 0.3439 (0.3551) data time 0.0008 (0.0021) model time 0.3430 (0.3518) loss 2.9887 (3.0147) grad_norm 2.0683 (1.8222) loss_scale 1024.0000 (671.7545) mem 11224MB [2024-09-01 08:30:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][400/625] eta 0:01:19 lr 0.000604 wd 0.0500 time 0.3554 (0.3549) data time 0.0009 (0.0020) model time 0.3546 (0.3517) loss 3.5427 (3.0149) grad_norm 1.9154 (1.8193) loss_scale 1024.0000 (680.5387) mem 11224MB [2024-09-01 08:30:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][410/625] eta 0:01:16 lr 0.000604 wd 0.0500 time 0.3438 (0.3547) data time 0.0007 (0.0020) model time 0.3431 (0.3516) loss 3.3187 (3.0194) grad_norm 1.3485 (1.8208) loss_scale 1024.0000 (688.8954) mem 11224MB [2024-09-01 08:30:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][420/625] eta 0:01:13 lr 0.000604 wd 0.0500 time 0.3482 (0.3563) data time 0.0009 (0.0020) model time 0.3474 (0.3534) loss 3.1627 (3.0179) grad_norm 1.6909 (1.8263) loss_scale 1024.0000 (696.8551) mem 11224MB [2024-09-01 08:30:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][430/625] eta 0:01:09 lr 0.000604 wd 0.0500 time 0.3510 (0.3561) data time 0.0008 (0.0020) model time 0.3502 (0.3533) loss 3.2900 (3.0191) grad_norm 2.8227 (1.8290) loss_scale 1024.0000 (704.4455) mem 11224MB [2024-09-01 08:30:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][440/625] eta 0:01:05 lr 0.000604 wd 0.0500 time 0.3460 (0.3560) data time 0.0007 (0.0019) model time 0.3453 (0.3531) loss 2.0624 (3.0148) grad_norm 2.0890 (1.8512) loss_scale 1024.0000 (711.6916) mem 11224MB [2024-09-01 08:30:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][450/625] eta 0:01:02 lr 0.000603 wd 0.0500 time 0.3387 (0.3557) data time 0.0007 (0.0019) model time 0.3381 (0.3529) loss 3.4300 (3.0229) grad_norm 1.2939 (1.8497) loss_scale 1024.0000 (718.6164) mem 11224MB [2024-09-01 08:30:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][460/625] eta 0:00:58 lr 0.000603 wd 0.0500 time 0.3427 (0.3555) data time 0.0009 (0.0019) model time 0.3419 (0.3527) loss 3.2201 (3.0215) grad_norm 1.3979 (1.8475) loss_scale 1024.0000 (725.2408) mem 11224MB [2024-09-01 08:30:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][470/625] eta 0:00:55 lr 0.000603 wd 0.0500 time 0.3516 (0.3554) data time 0.0008 (0.0019) model time 0.3507 (0.3527) loss 2.2514 (3.0163) grad_norm 2.5984 (1.8510) loss_scale 1024.0000 (731.5839) mem 11224MB [2024-09-01 08:30:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][480/625] eta 0:00:51 lr 0.000603 wd 0.0500 time 0.3452 (0.3553) data time 0.0007 (0.0018) model time 0.3445 (0.3526) loss 3.0928 (3.0129) grad_norm 1.7151 (1.8486) loss_scale 1024.0000 (737.6632) mem 11224MB [2024-09-01 08:30:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][490/625] eta 0:00:48 lr 0.000603 wd 0.0500 time 0.5631 (0.3567) data time 0.0009 (0.0018) model time 0.5621 (0.3542) loss 2.7009 (3.0133) grad_norm 1.3525 (1.8465) loss_scale 1024.0000 (743.4949) mem 11224MB [2024-09-01 08:30:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][500/625] eta 0:00:44 lr 0.000603 wd 0.0500 time 0.3475 (0.3566) data time 0.0007 (0.0018) model time 0.3469 (0.3541) loss 1.9592 (3.0137) grad_norm 1.5190 (1.8493) loss_scale 1024.0000 (749.0938) mem 11224MB [2024-09-01 08:30:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][510/625] eta 0:00:40 lr 0.000602 wd 0.0500 time 0.3505 (0.3564) data time 0.0006 (0.0018) model time 0.3499 (0.3539) loss 3.3016 (3.0133) grad_norm 1.9814 (1.8506) loss_scale 1024.0000 (754.4736) mem 11224MB [2024-09-01 08:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][520/625] eta 0:00:37 lr 0.000602 wd 0.0500 time 0.3479 (0.3562) data time 0.0009 (0.0018) model time 0.3471 (0.3537) loss 2.3666 (3.0125) grad_norm 1.6099 (1.8499) loss_scale 1024.0000 (759.6468) mem 11224MB [2024-09-01 08:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][530/625] eta 0:00:33 lr 0.000602 wd 0.0500 time 0.3412 (0.3560) data time 0.0008 (0.0017) model time 0.3403 (0.3536) loss 3.0222 (3.0140) grad_norm 2.2995 (1.8531) loss_scale 1024.0000 (764.6252) mem 11224MB [2024-09-01 08:30:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][540/625] eta 0:00:30 lr 0.000602 wd 0.0500 time 0.3455 (0.3569) data time 0.0006 (0.0017) model time 0.3448 (0.3546) loss 3.0233 (3.0189) grad_norm 1.3056 (1.8535) loss_scale 1024.0000 (769.4196) mem 11224MB [2024-09-01 08:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][550/625] eta 0:00:26 lr 0.000602 wd 0.0500 time 0.3484 (0.3568) data time 0.0008 (0.0017) model time 0.3476 (0.3545) loss 3.1907 (3.0170) grad_norm 2.0220 (1.8516) loss_scale 1024.0000 (774.0399) mem 11224MB [2024-09-01 08:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][560/625] eta 0:00:23 lr 0.000602 wd 0.0500 time 0.3520 (0.3566) data time 0.0006 (0.0017) model time 0.3514 (0.3543) loss 1.7772 (3.0145) grad_norm 1.7563 (1.8473) loss_scale 1024.0000 (778.4955) mem 11224MB [2024-09-01 08:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][570/625] eta 0:00:19 lr 0.000601 wd 0.0500 time 0.3500 (0.3565) data time 0.0009 (0.0017) model time 0.3491 (0.3542) loss 2.9223 (3.0114) grad_norm 1.5791 (1.8451) loss_scale 1024.0000 (782.7951) mem 11224MB [2024-09-01 08:31:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][580/625] eta 0:00:16 lr 0.000601 wd 0.0500 time 0.3466 (0.3563) data time 0.0008 (0.0017) model time 0.3458 (0.3541) loss 3.0755 (3.0131) grad_norm 1.4533 (1.8440) loss_scale 1024.0000 (786.9466) mem 11224MB [2024-09-01 08:31:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][590/625] eta 0:00:12 lr 0.000601 wd 0.0500 time 0.3437 (0.3561) data time 0.0006 (0.0017) model time 0.3431 (0.3539) loss 1.8525 (3.0147) grad_norm 1.7372 (1.8434) loss_scale 1024.0000 (790.9577) mem 11224MB [2024-09-01 08:31:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][600/625] eta 0:00:08 lr 0.000601 wd 0.0500 time 0.3531 (0.3563) data time 0.0006 (0.0016) model time 0.3524 (0.3540) loss 3.6655 (3.0154) grad_norm 4.0229 (1.8516) loss_scale 1024.0000 (794.8353) mem 11224MB [2024-09-01 08:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][610/625] eta 0:00:05 lr 0.000601 wd 0.0500 time 0.3410 (0.3574) data time 0.0004 (0.0016) model time 0.3406 (0.3553) loss 2.5351 (3.0191) grad_norm 3.2668 (1.8544) loss_scale 1024.0000 (798.5859) mem 11224MB [2024-09-01 08:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [197/300][620/625] eta 0:00:01 lr 0.000601 wd 0.0500 time 0.3399 (0.3571) data time 0.0004 (0.0016) model time 0.3395 (0.3550) loss 3.5484 (3.0159) grad_norm 1.5926 (1.8534) loss_scale 1024.0000 (802.2158) mem 11224MB [2024-09-01 08:31:28 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 197 training takes 0:03:43 [2024-09-01 08:31:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:31:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:31:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.435 (0.435) Loss 0.5400 (0.5400) Acc@1 87.500 (87.500) Acc@5 98.291 (98.291) Mem 11224MB [2024-09-01 08:31:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.131) Loss 0.8804 (0.6581) Acc@1 78.809 (85.019) Acc@5 95.117 (97.346) Mem 11224MB [2024-09-01 08:31:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.0098 (0.7870) Acc@1 75.049 (81.927) Acc@5 94.092 (95.961) Mem 11224MB [2024-09-01 08:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.666 Acc@5 95.973 [2024-09-01 08:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.7% [2024-09-01 08:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.67% [2024-09-01 08:31:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:31:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:31:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.427 (0.427) Loss 0.4973 (0.4973) Acc@1 89.258 (89.258) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 08:31:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.126) Loss 0.7993 (0.6085) Acc@1 81.104 (86.452) Acc@5 96.094 (97.687) Mem 11224MB [2024-09-01 08:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.110) Loss 0.8936 (0.7204) Acc@1 77.588 (83.531) Acc@5 95.508 (96.568) Mem 11224MB [2024-09-01 08:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.223 Acc@5 96.595 [2024-09-01 08:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.2% [2024-09-01 08:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.22% [2024-09-01 08:31:35 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:31:36 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:31:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][0/625] eta 0:07:18 lr 0.000601 wd 0.0500 time 0.7023 (0.7023) data time 0.3918 (0.3918) model time 0.0000 (0.0000) loss 2.8607 (2.8607) grad_norm 1.8584 (1.8584) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:31:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][10/625] eta 0:03:53 lr 0.000600 wd 0.0500 time 0.3472 (0.3802) data time 0.0013 (0.0364) model time 0.0000 (0.0000) loss 2.7253 (2.9370) grad_norm 1.9766 (1.7434) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:31:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][20/625] eta 0:03:40 lr 0.000600 wd 0.0500 time 0.3473 (0.3641) data time 0.0008 (0.0194) model time 0.0000 (0.0000) loss 3.3194 (3.0191) grad_norm 1.5652 (1.7027) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:31:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][30/625] eta 0:03:33 lr 0.000600 wd 0.0500 time 0.3430 (0.3584) data time 0.0008 (0.0134) model time 0.0000 (0.0000) loss 3.3706 (3.0473) grad_norm 1.9789 (1.8117) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:31:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][40/625] eta 0:03:30 lr 0.000600 wd 0.0500 time 0.3435 (0.3602) data time 0.0008 (0.0104) model time 0.0000 (0.0000) loss 2.9463 (3.0197) grad_norm 1.4560 (1.7858) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:31:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][50/625] eta 0:03:25 lr 0.000600 wd 0.0500 time 0.3490 (0.3576) data time 0.0006 (0.0085) model time 0.0000 (0.0000) loss 2.0508 (2.9781) grad_norm 1.9511 (1.9056) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][60/625] eta 0:03:21 lr 0.000600 wd 0.0500 time 0.3527 (0.3561) data time 0.0008 (0.0072) model time 0.3519 (0.3476) loss 3.5089 (2.9792) grad_norm 1.2669 (1.8504) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][70/625] eta 0:03:20 lr 0.000599 wd 0.0500 time 0.4698 (0.3615) data time 0.0007 (0.0063) model time 0.4691 (0.3707) loss 2.4988 (2.9809) grad_norm 2.4149 (1.8382) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][80/625] eta 0:03:16 lr 0.000599 wd 0.0500 time 0.3476 (0.3598) data time 0.0008 (0.0056) model time 0.3467 (0.3627) loss 3.0080 (2.9884) grad_norm 4.6776 (2.0238) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][90/625] eta 0:03:11 lr 0.000599 wd 0.0500 time 0.3439 (0.3585) data time 0.0006 (0.0051) model time 0.3433 (0.3588) loss 2.9960 (3.0023) grad_norm 2.0717 (2.0422) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][100/625] eta 0:03:07 lr 0.000599 wd 0.0500 time 0.3476 (0.3573) data time 0.0008 (0.0047) model time 0.3468 (0.3561) loss 2.8485 (3.0080) grad_norm 1.5446 (2.0219) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][110/625] eta 0:03:03 lr 0.000599 wd 0.0500 time 0.3473 (0.3563) data time 0.0009 (0.0044) model time 0.3464 (0.3544) loss 2.4341 (2.9962) grad_norm 1.9511 (2.0005) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][120/625] eta 0:02:59 lr 0.000599 wd 0.0500 time 0.3451 (0.3556) data time 0.0007 (0.0041) model time 0.3444 (0.3533) loss 3.0002 (3.0050) grad_norm 1.9153 (2.0222) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][130/625] eta 0:02:55 lr 0.000598 wd 0.0500 time 0.3513 (0.3550) data time 0.0006 (0.0038) model time 0.3506 (0.3526) loss 3.9316 (3.0091) grad_norm 2.0895 (2.0597) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][140/625] eta 0:02:51 lr 0.000598 wd 0.0500 time 0.3447 (0.3545) data time 0.0009 (0.0036) model time 0.3438 (0.3519) loss 3.1236 (3.0191) grad_norm 1.6819 (2.0861) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][150/625] eta 0:02:48 lr 0.000598 wd 0.0500 time 0.3477 (0.3542) data time 0.0006 (0.0034) model time 0.3471 (0.3516) loss 3.5299 (3.0160) grad_norm 2.9892 (2.0892) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][160/625] eta 0:02:44 lr 0.000598 wd 0.0500 time 0.3522 (0.3538) data time 0.0006 (0.0033) model time 0.3515 (0.3513) loss 2.3421 (3.0075) grad_norm 1.5424 (2.0908) loss_scale 1024.0000 (1024.0000) mem 11224MB [2024-09-01 08:32:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][170/625] eta 0:02:42 lr 0.000598 wd 0.0500 time 0.3496 (0.3565) data time 0.0006 (0.0031) model time 0.3490 (0.3551) loss 2.5722 (3.0113) grad_norm 2.4919 (inf) loss_scale 512.0000 (1006.0351) mem 11224MB [2024-09-01 08:32:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][180/625] eta 0:02:38 lr 0.000598 wd 0.0500 time 0.3483 (0.3571) data time 0.0008 (0.0030) model time 0.3475 (0.3560) loss 3.1736 (3.0016) grad_norm 1.8418 (inf) loss_scale 512.0000 (978.7403) mem 11224MB [2024-09-01 08:32:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][190/625] eta 0:02:35 lr 0.000598 wd 0.0500 time 0.3480 (0.3566) data time 0.0006 (0.0029) model time 0.3474 (0.3554) loss 3.9987 (2.9967) grad_norm 2.2911 (inf) loss_scale 512.0000 (954.3037) mem 11224MB [2024-09-01 08:32:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][200/625] eta 0:02:31 lr 0.000597 wd 0.0500 time 0.3509 (0.3562) data time 0.0008 (0.0028) model time 0.3501 (0.3549) loss 2.2517 (2.9791) grad_norm 1.5142 (inf) loss_scale 512.0000 (932.2985) mem 11224MB [2024-09-01 08:32:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][210/625] eta 0:02:27 lr 0.000597 wd 0.0500 time 0.3560 (0.3559) data time 0.0006 (0.0027) model time 0.3554 (0.3546) loss 3.4633 (2.9857) grad_norm 2.1096 (inf) loss_scale 512.0000 (912.3791) mem 11224MB [2024-09-01 08:32:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][220/625] eta 0:02:24 lr 0.000597 wd 0.0500 time 0.3468 (0.3556) data time 0.0007 (0.0026) model time 0.3461 (0.3542) loss 2.9777 (2.9878) grad_norm 3.4109 (inf) loss_scale 512.0000 (894.2624) mem 11224MB [2024-09-01 08:32:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][230/625] eta 0:02:20 lr 0.000597 wd 0.0500 time 0.3485 (0.3553) data time 0.0007 (0.0025) model time 0.3478 (0.3538) loss 3.5147 (2.9909) grad_norm 1.8668 (inf) loss_scale 512.0000 (877.7143) mem 11224MB [2024-09-01 08:33:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][240/625] eta 0:02:16 lr 0.000597 wd 0.0500 time 0.3586 (0.3551) data time 0.0008 (0.0024) model time 0.3578 (0.3537) loss 2.9757 (2.9802) grad_norm 1.9584 (inf) loss_scale 512.0000 (862.5394) mem 11224MB [2024-09-01 08:33:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][250/625] eta 0:02:13 lr 0.000597 wd 0.0500 time 0.3511 (0.3557) data time 0.0008 (0.0024) model time 0.3503 (0.3543) loss 2.8404 (2.9806) grad_norm 1.7051 (inf) loss_scale 512.0000 (848.5737) mem 11224MB [2024-09-01 08:33:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][260/625] eta 0:02:09 lr 0.000596 wd 0.0500 time 0.3465 (0.3554) data time 0.0008 (0.0023) model time 0.3457 (0.3540) loss 2.3765 (2.9720) grad_norm 2.4517 (inf) loss_scale 512.0000 (835.6782) mem 11224MB [2024-09-01 08:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][270/625] eta 0:02:06 lr 0.000596 wd 0.0500 time 0.3486 (0.3551) data time 0.0006 (0.0023) model time 0.3480 (0.3537) loss 2.4638 (2.9595) grad_norm 1.6983 (inf) loss_scale 512.0000 (823.7343) mem 11224MB [2024-09-01 08:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][280/625] eta 0:02:03 lr 0.000596 wd 0.0500 time 0.5225 (0.3568) data time 0.0007 (0.0022) model time 0.5218 (0.3558) loss 2.3612 (2.9640) grad_norm 1.7062 (inf) loss_scale 512.0000 (812.6406) mem 11224MB [2024-09-01 08:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][290/625] eta 0:01:59 lr 0.000596 wd 0.0500 time 0.3451 (0.3579) data time 0.0006 (0.0022) model time 0.3445 (0.3571) loss 2.7979 (2.9693) grad_norm 1.9648 (inf) loss_scale 512.0000 (802.3093) mem 11224MB [2024-09-01 08:33:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][300/625] eta 0:01:56 lr 0.000596 wd 0.0500 time 0.3495 (0.3582) data time 0.0009 (0.0021) model time 0.3486 (0.3575) loss 3.0568 (2.9762) grad_norm 1.7638 (inf) loss_scale 512.0000 (792.6645) mem 11224MB [2024-09-01 08:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][310/625] eta 0:01:52 lr 0.000596 wd 0.0500 time 0.3505 (0.3579) data time 0.0005 (0.0021) model time 0.3500 (0.3572) loss 3.1052 (2.9728) grad_norm 1.5469 (inf) loss_scale 512.0000 (783.6399) mem 11224MB [2024-09-01 08:33:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][320/625] eta 0:01:49 lr 0.000595 wd 0.0500 time 0.3516 (0.3576) data time 0.0006 (0.0021) model time 0.3509 (0.3568) loss 3.6133 (2.9710) grad_norm 1.4022 (inf) loss_scale 512.0000 (775.1776) mem 11224MB [2024-09-01 08:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][330/625] eta 0:01:45 lr 0.000595 wd 0.0500 time 0.3510 (0.3573) data time 0.0006 (0.0020) model time 0.3504 (0.3564) loss 2.7067 (2.9668) grad_norm 1.4881 (inf) loss_scale 512.0000 (767.2266) mem 11224MB [2024-09-01 08:33:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][340/625] eta 0:01:41 lr 0.000595 wd 0.0500 time 0.3431 (0.3571) data time 0.0006 (0.0020) model time 0.3424 (0.3562) loss 3.2686 (2.9716) grad_norm 1.8050 (inf) loss_scale 512.0000 (759.7419) mem 11224MB [2024-09-01 08:33:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][350/625] eta 0:01:38 lr 0.000595 wd 0.0500 time 0.3452 (0.3570) data time 0.0006 (0.0020) model time 0.3445 (0.3561) loss 3.4956 (2.9764) grad_norm 2.0345 (inf) loss_scale 512.0000 (752.6838) mem 11224MB [2024-09-01 08:33:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][360/625] eta 0:01:34 lr 0.000595 wd 0.0500 time 0.3601 (0.3569) data time 0.0006 (0.0019) model time 0.3595 (0.3559) loss 3.2895 (2.9817) grad_norm 1.6290 (inf) loss_scale 512.0000 (746.0166) mem 11224MB [2024-09-01 08:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][370/625] eta 0:01:30 lr 0.000595 wd 0.0500 time 0.3520 (0.3566) data time 0.0006 (0.0019) model time 0.3513 (0.3556) loss 1.7187 (2.9826) grad_norm 2.0493 (inf) loss_scale 512.0000 (739.7089) mem 11224MB [2024-09-01 08:33:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][380/625] eta 0:01:27 lr 0.000594 wd 0.0500 time 0.3510 (0.3564) data time 0.0008 (0.0019) model time 0.3501 (0.3553) loss 3.7063 (2.9805) grad_norm 3.0534 (inf) loss_scale 512.0000 (733.7323) mem 11224MB [2024-09-01 08:33:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][390/625] eta 0:01:23 lr 0.000594 wd 0.0500 time 0.3525 (0.3562) data time 0.0009 (0.0019) model time 0.3516 (0.3551) loss 2.9120 (2.9831) grad_norm 1.3730 (inf) loss_scale 512.0000 (728.0614) mem 11224MB [2024-09-01 08:33:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][400/625] eta 0:01:20 lr 0.000594 wd 0.0500 time 0.3470 (0.3560) data time 0.0009 (0.0018) model time 0.3461 (0.3550) loss 2.2911 (2.9865) grad_norm 2.2351 (inf) loss_scale 512.0000 (722.6733) mem 11224MB [2024-09-01 08:34:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][410/625] eta 0:01:16 lr 0.000594 wd 0.0500 time 0.3468 (0.3559) data time 0.0008 (0.0018) model time 0.3460 (0.3548) loss 3.0615 (2.9922) grad_norm 1.5323 (inf) loss_scale 512.0000 (717.5474) mem 11224MB [2024-09-01 08:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][420/625] eta 0:01:12 lr 0.000594 wd 0.0500 time 0.3466 (0.3557) data time 0.0007 (0.0018) model time 0.3459 (0.3546) loss 2.6752 (2.9914) grad_norm 1.9165 (inf) loss_scale 512.0000 (712.6651) mem 11224MB [2024-09-01 08:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][430/625] eta 0:01:09 lr 0.000594 wd 0.0500 time 0.3479 (0.3559) data time 0.0007 (0.0018) model time 0.3472 (0.3548) loss 3.2066 (2.9899) grad_norm 2.0281 (inf) loss_scale 512.0000 (708.0093) mem 11224MB [2024-09-01 08:34:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][440/625] eta 0:01:05 lr 0.000593 wd 0.0500 time 0.3501 (0.3557) data time 0.0008 (0.0017) model time 0.3493 (0.3546) loss 3.6211 (2.9950) grad_norm 1.4947 (inf) loss_scale 512.0000 (703.5646) mem 11224MB [2024-09-01 08:34:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][450/625] eta 0:01:02 lr 0.000593 wd 0.0500 time 0.3429 (0.3555) data time 0.0007 (0.0017) model time 0.3422 (0.3544) loss 3.5600 (3.0002) grad_norm 2.9624 (inf) loss_scale 512.0000 (699.3171) mem 11224MB [2024-09-01 08:34:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][460/625] eta 0:00:58 lr 0.000593 wd 0.0500 time 0.3518 (0.3558) data time 0.0008 (0.0017) model time 0.3510 (0.3548) loss 3.5947 (3.0017) grad_norm 9.6323 (inf) loss_scale 512.0000 (695.2538) mem 11224MB [2024-09-01 08:34:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][470/625] eta 0:00:55 lr 0.000593 wd 0.0500 time 0.3469 (0.3563) data time 0.0007 (0.0017) model time 0.3462 (0.3553) loss 4.0241 (3.0084) grad_norm 1.6617 (inf) loss_scale 512.0000 (691.3631) mem 11224MB [2024-09-01 08:34:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][480/625] eta 0:00:51 lr 0.000593 wd 0.0500 time 0.3495 (0.3562) data time 0.0007 (0.0017) model time 0.3488 (0.3551) loss 2.8887 (3.0095) grad_norm 1.7363 (inf) loss_scale 512.0000 (687.6341) mem 11224MB [2024-09-01 08:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][490/625] eta 0:00:48 lr 0.000593 wd 0.0500 time 0.3480 (0.3560) data time 0.0009 (0.0017) model time 0.3471 (0.3550) loss 3.2942 (3.0087) grad_norm 4.7922 (inf) loss_scale 512.0000 (684.0570) mem 11224MB [2024-09-01 08:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][500/625] eta 0:00:44 lr 0.000593 wd 0.0500 time 0.5688 (0.3571) data time 0.0009 (0.0016) model time 0.5679 (0.3562) loss 2.7352 (3.0117) grad_norm 1.5430 (inf) loss_scale 512.0000 (680.6228) mem 11224MB [2024-09-01 08:34:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][510/625] eta 0:00:41 lr 0.000592 wd 0.0500 time 0.3471 (0.3569) data time 0.0006 (0.0016) model time 0.3465 (0.3560) loss 3.1931 (3.0091) grad_norm 1.6871 (inf) loss_scale 512.0000 (677.3229) mem 11224MB [2024-09-01 08:34:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][520/625] eta 0:00:37 lr 0.000592 wd 0.0500 time 0.3526 (0.3567) data time 0.0008 (0.0016) model time 0.3518 (0.3558) loss 2.4224 (3.0099) grad_norm 1.1507 (inf) loss_scale 512.0000 (674.1497) mem 11224MB [2024-09-01 08:34:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][530/625] eta 0:00:33 lr 0.000592 wd 0.0500 time 0.3483 (0.3566) data time 0.0008 (0.0016) model time 0.3475 (0.3556) loss 3.1230 (3.0106) grad_norm 1.6118 (inf) loss_scale 512.0000 (671.0960) mem 11224MB [2024-09-01 08:34:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][540/625] eta 0:00:30 lr 0.000592 wd 0.0500 time 0.3556 (0.3564) data time 0.0007 (0.0016) model time 0.3549 (0.3555) loss 3.2977 (3.0102) grad_norm 2.2077 (inf) loss_scale 512.0000 (668.1553) mem 11224MB [2024-09-01 08:34:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][550/625] eta 0:00:26 lr 0.000592 wd 0.0500 time 0.4200 (0.3568) data time 0.0008 (0.0016) model time 0.4192 (0.3558) loss 3.3097 (3.0130) grad_norm 1.7175 (inf) loss_scale 512.0000 (665.3212) mem 11224MB [2024-09-01 08:34:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][560/625] eta 0:00:23 lr 0.000592 wd 0.0500 time 0.3475 (0.3566) data time 0.0008 (0.0015) model time 0.3467 (0.3557) loss 2.9328 (3.0215) grad_norm 1.8817 (inf) loss_scale 512.0000 (662.5882) mem 11224MB [2024-09-01 08:35:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][570/625] eta 0:00:19 lr 0.000591 wd 0.0500 time 0.3492 (0.3565) data time 0.0008 (0.0015) model time 0.3484 (0.3555) loss 3.3036 (3.0242) grad_norm 1.6257 (inf) loss_scale 512.0000 (659.9510) mem 11224MB [2024-09-01 08:35:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][580/625] eta 0:00:16 lr 0.000591 wd 0.0500 time 0.3485 (0.3566) data time 0.0008 (0.0015) model time 0.3477 (0.3557) loss 3.2481 (3.0240) grad_norm 1.5679 (inf) loss_scale 512.0000 (657.4045) mem 11224MB [2024-09-01 08:35:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][590/625] eta 0:00:12 lr 0.000591 wd 0.0500 time 0.3478 (0.3568) data time 0.0006 (0.0015) model time 0.3472 (0.3559) loss 2.3072 (3.0237) grad_norm 1.8617 (inf) loss_scale 512.0000 (654.9442) mem 11224MB [2024-09-01 08:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][600/625] eta 0:00:08 lr 0.000591 wd 0.0500 time 0.3470 (0.3567) data time 0.0006 (0.0015) model time 0.3464 (0.3557) loss 2.2915 (3.0221) grad_norm 1.7019 (inf) loss_scale 512.0000 (652.5657) mem 11224MB [2024-09-01 08:35:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][610/625] eta 0:00:05 lr 0.000591 wd 0.0500 time 0.3377 (0.3565) data time 0.0004 (0.0015) model time 0.3373 (0.3555) loss 3.1834 (3.0185) grad_norm 1.5901 (inf) loss_scale 512.0000 (650.2651) mem 11224MB [2024-09-01 08:35:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [198/300][620/625] eta 0:00:01 lr 0.000591 wd 0.0500 time 0.3411 (0.3562) data time 0.0004 (0.0015) model time 0.3407 (0.3552) loss 1.9129 (3.0149) grad_norm 2.9588 (inf) loss_scale 512.0000 (648.0386) mem 11224MB [2024-09-01 08:35:19 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 198 training takes 0:03:42 [2024-09-01 08:35:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:35:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:35:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.452 (0.452) Loss 0.5664 (0.5664) Acc@1 88.184 (88.184) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 08:35:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.099 (0.131) Loss 0.8745 (0.6787) Acc@1 78.955 (85.134) Acc@5 95.557 (97.483) Mem 11224MB [2024-09-01 08:35:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 0.9917 (0.8052) Acc@1 75.195 (81.924) Acc@5 94.238 (95.994) Mem 11224MB [2024-09-01 08:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.724 Acc@5 95.967 [2024-09-01 08:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.7% [2024-09-01 08:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.72% [2024-09-01 08:35:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:35:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:35:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.432 (0.432) Loss 0.4963 (0.4963) Acc@1 89.502 (89.502) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 08:35:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.128) Loss 0.7988 (0.6082) Acc@1 80.957 (86.497) Acc@5 96.094 (97.670) Mem 11224MB [2024-09-01 08:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 0.8940 (0.7200) Acc@1 77.588 (83.575) Acc@5 95.459 (96.545) Mem 11224MB [2024-09-01 08:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.259 Acc@5 96.577 [2024-09-01 08:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 08:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.26% [2024-09-01 08:35:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:35:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:35:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][0/625] eta 0:07:22 lr 0.000590 wd 0.0500 time 0.7086 (0.7086) data time 0.3993 (0.3993) model time 0.0000 (0.0000) loss 1.8385 (1.8385) grad_norm 1.6589 (1.6589) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:35:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][10/625] eta 0:03:54 lr 0.000590 wd 0.0500 time 0.3545 (0.3813) data time 0.0008 (0.0370) model time 0.0000 (0.0000) loss 3.0779 (2.9536) grad_norm 1.7548 (1.8330) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:35:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][20/625] eta 0:03:41 lr 0.000590 wd 0.0500 time 0.3427 (0.3654) data time 0.0009 (0.0198) model time 0.0000 (0.0000) loss 2.5127 (2.8511) grad_norm 2.1152 (1.7157) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:35:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][30/625] eta 0:03:34 lr 0.000590 wd 0.0500 time 0.3449 (0.3603) data time 0.0008 (0.0137) model time 0.0000 (0.0000) loss 3.0403 (2.8795) grad_norm 1.3483 (1.7293) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:35:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][40/625] eta 0:03:28 lr 0.000590 wd 0.0500 time 0.3494 (0.3571) data time 0.0008 (0.0105) model time 0.0000 (0.0000) loss 3.1361 (2.9091) grad_norm 1.9051 (1.9381) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:35:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][50/625] eta 0:03:24 lr 0.000590 wd 0.0500 time 0.3447 (0.3551) data time 0.0008 (0.0086) model time 0.0000 (0.0000) loss 3.0763 (2.9613) grad_norm 2.1417 (2.0866) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:35:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][60/625] eta 0:03:23 lr 0.000590 wd 0.0500 time 0.3445 (0.3601) data time 0.0008 (0.0073) model time 0.3437 (0.3849) loss 3.5246 (2.9846) grad_norm 1.6419 (2.0363) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:35:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][70/625] eta 0:03:22 lr 0.000589 wd 0.0500 time 0.3515 (0.3644) data time 0.0006 (0.0064) model time 0.3508 (0.3872) loss 2.5101 (2.9728) grad_norm 1.1031 (1.9730) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:35:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][80/625] eta 0:03:17 lr 0.000589 wd 0.0500 time 0.3456 (0.3625) data time 0.0009 (0.0057) model time 0.3447 (0.3744) loss 3.2437 (3.0008) grad_norm 2.5757 (2.0144) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][90/625] eta 0:03:13 lr 0.000589 wd 0.0500 time 0.3409 (0.3608) data time 0.0008 (0.0052) model time 0.3401 (0.3673) loss 3.3363 (2.9671) grad_norm 1.7430 (2.0008) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][100/625] eta 0:03:10 lr 0.000589 wd 0.0500 time 0.3503 (0.3630) data time 0.0008 (0.0048) model time 0.3495 (0.3702) loss 2.8442 (2.9581) grad_norm 1.5194 (1.9644) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][110/625] eta 0:03:06 lr 0.000589 wd 0.0500 time 0.3493 (0.3619) data time 0.0006 (0.0044) model time 0.3487 (0.3668) loss 2.5868 (2.9460) grad_norm 2.0039 (1.9614) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][120/625] eta 0:03:02 lr 0.000589 wd 0.0500 time 0.3456 (0.3606) data time 0.0009 (0.0042) model time 0.3447 (0.3637) loss 3.1160 (2.9502) grad_norm 2.5144 (1.9712) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][130/625] eta 0:02:57 lr 0.000588 wd 0.0500 time 0.3440 (0.3594) data time 0.0007 (0.0039) model time 0.3433 (0.3613) loss 2.2261 (2.9457) grad_norm 1.2901 (1.9749) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][140/625] eta 0:02:53 lr 0.000588 wd 0.0500 time 0.3545 (0.3587) data time 0.0009 (0.0037) model time 0.3536 (0.3598) loss 2.7387 (2.9523) grad_norm 3.3448 (1.9525) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][150/625] eta 0:02:50 lr 0.000588 wd 0.0500 time 0.3515 (0.3580) data time 0.0010 (0.0035) model time 0.3506 (0.3585) loss 3.0211 (2.9663) grad_norm 1.3261 (1.9233) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][160/625] eta 0:02:46 lr 0.000588 wd 0.0500 time 0.3530 (0.3574) data time 0.0009 (0.0033) model time 0.3521 (0.3575) loss 3.2013 (2.9641) grad_norm 1.5566 (1.8975) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][170/625] eta 0:02:42 lr 0.000588 wd 0.0500 time 0.3453 (0.3569) data time 0.0006 (0.0032) model time 0.3447 (0.3567) loss 3.4296 (2.9902) grad_norm 1.9880 (1.9277) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][180/625] eta 0:02:38 lr 0.000588 wd 0.0500 time 0.3522 (0.3565) data time 0.0008 (0.0031) model time 0.3514 (0.3562) loss 2.4638 (2.9801) grad_norm 2.4284 (1.9449) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][190/625] eta 0:02:34 lr 0.000587 wd 0.0500 time 0.3518 (0.3561) data time 0.0009 (0.0029) model time 0.3510 (0.3555) loss 2.4348 (2.9755) grad_norm 1.3897 (1.9323) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][200/625] eta 0:02:31 lr 0.000587 wd 0.0500 time 0.3432 (0.3556) data time 0.0008 (0.0028) model time 0.3424 (0.3549) loss 3.1422 (2.9870) grad_norm 1.4887 (1.9106) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][210/625] eta 0:02:27 lr 0.000587 wd 0.0500 time 0.3481 (0.3552) data time 0.0006 (0.0028) model time 0.3474 (0.3543) loss 1.7513 (2.9775) grad_norm 1.4551 (1.9230) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][220/625] eta 0:02:23 lr 0.000587 wd 0.0500 time 0.3480 (0.3548) data time 0.0006 (0.0027) model time 0.3473 (0.3538) loss 3.2808 (2.9825) grad_norm 2.2786 (1.9261) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][230/625] eta 0:02:19 lr 0.000587 wd 0.0500 time 0.3441 (0.3544) data time 0.0006 (0.0026) model time 0.3434 (0.3533) loss 3.3711 (2.9823) grad_norm 1.3212 (1.9193) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][240/625] eta 0:02:16 lr 0.000587 wd 0.0500 time 0.3715 (0.3550) data time 0.0008 (0.0025) model time 0.3707 (0.3541) loss 3.5625 (2.9824) grad_norm 1.9548 (1.9103) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:36:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][250/625] eta 0:02:13 lr 0.000586 wd 0.0500 time 0.3474 (0.3569) data time 0.0006 (0.0024) model time 0.3468 (0.3565) loss 1.8348 (2.9725) grad_norm 1.9374 (1.9522) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:37:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][260/625] eta 0:02:10 lr 0.000586 wd 0.0500 time 0.3547 (0.3575) data time 0.0006 (0.0024) model time 0.3541 (0.3572) loss 2.5129 (2.9746) grad_norm 1.3587 (1.9475) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:37:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][270/625] eta 0:02:06 lr 0.000586 wd 0.0500 time 0.3473 (0.3572) data time 0.0007 (0.0023) model time 0.3467 (0.3568) loss 3.2078 (2.9680) grad_norm 1.7633 (1.9323) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:37:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][280/625] eta 0:02:03 lr 0.000586 wd 0.0500 time 0.3445 (0.3567) data time 0.0006 (0.0023) model time 0.3439 (0.3562) loss 2.5721 (2.9719) grad_norm 7.4793 (inf) loss_scale 256.0000 (506.5338) mem 11224MB [2024-09-01 08:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][290/625] eta 0:01:59 lr 0.000586 wd 0.0500 time 0.3502 (0.3564) data time 0.0008 (0.0022) model time 0.3494 (0.3559) loss 2.5502 (2.9766) grad_norm 1.6631 (inf) loss_scale 256.0000 (497.9244) mem 11224MB [2024-09-01 08:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][300/625] eta 0:01:55 lr 0.000586 wd 0.0500 time 0.3483 (0.3561) data time 0.0007 (0.0022) model time 0.3476 (0.3555) loss 2.3823 (2.9756) grad_norm 2.1243 (inf) loss_scale 256.0000 (489.8870) mem 11224MB [2024-09-01 08:37:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][310/625] eta 0:01:52 lr 0.000586 wd 0.0500 time 0.3455 (0.3565) data time 0.0010 (0.0021) model time 0.3446 (0.3559) loss 2.8053 (2.9614) grad_norm 1.9340 (inf) loss_scale 256.0000 (482.3666) mem 11224MB [2024-09-01 08:37:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][320/625] eta 0:01:48 lr 0.000585 wd 0.0500 time 0.3447 (0.3562) data time 0.0008 (0.0021) model time 0.3439 (0.3555) loss 3.1207 (2.9652) grad_norm 1.6232 (inf) loss_scale 256.0000 (475.3146) mem 11224MB [2024-09-01 08:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][330/625] eta 0:01:45 lr 0.000585 wd 0.0500 time 0.3458 (0.3560) data time 0.0007 (0.0021) model time 0.3451 (0.3553) loss 2.8407 (2.9605) grad_norm 2.3844 (inf) loss_scale 256.0000 (468.6888) mem 11224MB [2024-09-01 08:37:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][340/625] eta 0:01:41 lr 0.000585 wd 0.0500 time 0.3478 (0.3558) data time 0.0008 (0.0020) model time 0.3470 (0.3550) loss 3.2718 (2.9620) grad_norm 1.3668 (inf) loss_scale 256.0000 (462.4516) mem 11224MB [2024-09-01 08:37:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][350/625] eta 0:01:37 lr 0.000585 wd 0.0500 time 0.3509 (0.3555) data time 0.0008 (0.0020) model time 0.3501 (0.3548) loss 2.6901 (2.9578) grad_norm 1.6058 (inf) loss_scale 256.0000 (456.5698) mem 11224MB [2024-09-01 08:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][360/625] eta 0:01:34 lr 0.000585 wd 0.0500 time 0.3480 (0.3559) data time 0.0007 (0.0020) model time 0.3473 (0.3552) loss 1.9480 (2.9615) grad_norm 1.4620 (inf) loss_scale 256.0000 (451.0139) mem 11224MB [2024-09-01 08:37:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][370/625] eta 0:01:31 lr 0.000585 wd 0.0500 time 0.5699 (0.3579) data time 0.0006 (0.0019) model time 0.5692 (0.3575) loss 3.8996 (2.9687) grad_norm 1.7959 (inf) loss_scale 256.0000 (445.7574) mem 11224MB [2024-09-01 08:37:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][380/625] eta 0:01:27 lr 0.000584 wd 0.0500 time 0.3507 (0.3577) data time 0.0006 (0.0019) model time 0.3501 (0.3572) loss 3.5740 (2.9614) grad_norm 1.6651 (inf) loss_scale 256.0000 (440.7769) mem 11224MB [2024-09-01 08:37:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][390/625] eta 0:01:24 lr 0.000584 wd 0.0500 time 0.3469 (0.3575) data time 0.0007 (0.0019) model time 0.3462 (0.3570) loss 3.3201 (2.9666) grad_norm 1.1504 (inf) loss_scale 256.0000 (436.0512) mem 11224MB [2024-09-01 08:37:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][400/625] eta 0:01:20 lr 0.000584 wd 0.0500 time 0.3535 (0.3573) data time 0.0006 (0.0018) model time 0.3529 (0.3567) loss 3.7156 (2.9727) grad_norm 2.1255 (inf) loss_scale 256.0000 (431.5611) mem 11224MB [2024-09-01 08:37:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][410/625] eta 0:01:16 lr 0.000584 wd 0.0500 time 0.3472 (0.3571) data time 0.0007 (0.0018) model time 0.3466 (0.3565) loss 3.7199 (2.9763) grad_norm 2.0039 (inf) loss_scale 256.0000 (427.2895) mem 11224MB [2024-09-01 08:37:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][420/625] eta 0:01:13 lr 0.000584 wd 0.0500 time 0.3474 (0.3569) data time 0.0007 (0.0018) model time 0.3467 (0.3562) loss 2.6562 (2.9816) grad_norm 2.4642 (inf) loss_scale 256.0000 (423.2209) mem 11224MB [2024-09-01 08:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][430/625] eta 0:01:09 lr 0.000584 wd 0.0500 time 0.3487 (0.3567) data time 0.0007 (0.0018) model time 0.3480 (0.3560) loss 3.0637 (2.9847) grad_norm 1.1762 (inf) loss_scale 256.0000 (419.3411) mem 11224MB [2024-09-01 08:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][440/625] eta 0:01:05 lr 0.000583 wd 0.0500 time 0.3468 (0.3565) data time 0.0008 (0.0017) model time 0.3459 (0.3559) loss 3.1897 (2.9835) grad_norm 2.0976 (inf) loss_scale 256.0000 (415.6372) mem 11224MB [2024-09-01 08:38:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][450/625] eta 0:01:02 lr 0.000583 wd 0.0500 time 0.3503 (0.3564) data time 0.0006 (0.0017) model time 0.3496 (0.3557) loss 2.3454 (2.9830) grad_norm 1.4495 (inf) loss_scale 256.0000 (412.0976) mem 11224MB [2024-09-01 08:38:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][460/625] eta 0:00:58 lr 0.000583 wd 0.0500 time 0.3420 (0.3562) data time 0.0007 (0.0017) model time 0.3413 (0.3555) loss 3.5994 (2.9781) grad_norm 2.2670 (inf) loss_scale 256.0000 (408.7115) mem 11224MB [2024-09-01 08:38:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][470/625] eta 0:00:55 lr 0.000583 wd 0.0500 time 0.3458 (0.3560) data time 0.0008 (0.0017) model time 0.3450 (0.3552) loss 3.7190 (2.9803) grad_norm 2.7080 (inf) loss_scale 256.0000 (405.4692) mem 11224MB [2024-09-01 08:38:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][480/625] eta 0:00:51 lr 0.000583 wd 0.0500 time 0.3457 (0.3558) data time 0.0009 (0.0017) model time 0.3449 (0.3550) loss 2.9782 (2.9813) grad_norm 2.3110 (inf) loss_scale 256.0000 (402.3617) mem 11224MB [2024-09-01 08:38:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][490/625] eta 0:00:48 lr 0.000583 wd 0.0500 time 0.3495 (0.3560) data time 0.0007 (0.0017) model time 0.3488 (0.3552) loss 3.0375 (2.9835) grad_norm 1.4818 (inf) loss_scale 256.0000 (399.3809) mem 11224MB [2024-09-01 08:38:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][500/625] eta 0:00:44 lr 0.000582 wd 0.0500 time 0.3464 (0.3558) data time 0.0009 (0.0016) model time 0.3456 (0.3550) loss 3.1000 (2.9804) grad_norm 2.9725 (inf) loss_scale 256.0000 (396.5190) mem 11224MB [2024-09-01 08:38:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][510/625] eta 0:00:40 lr 0.000582 wd 0.0500 time 0.3438 (0.3557) data time 0.0008 (0.0016) model time 0.3430 (0.3549) loss 2.2706 (2.9780) grad_norm 1.4880 (inf) loss_scale 256.0000 (393.7691) mem 11224MB [2024-09-01 08:38:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][520/625] eta 0:00:37 lr 0.000582 wd 0.0500 time 0.3551 (0.3555) data time 0.0008 (0.0016) model time 0.3543 (0.3547) loss 2.5569 (2.9771) grad_norm 1.4320 (inf) loss_scale 256.0000 (391.1248) mem 11224MB [2024-09-01 08:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][530/625] eta 0:00:33 lr 0.000582 wd 0.0500 time 0.3523 (0.3554) data time 0.0008 (0.0016) model time 0.3514 (0.3546) loss 3.0081 (2.9781) grad_norm 1.5709 (inf) loss_scale 256.0000 (388.5800) mem 11224MB [2024-09-01 08:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][540/625] eta 0:00:30 lr 0.000582 wd 0.0500 time 0.3492 (0.3553) data time 0.0008 (0.0016) model time 0.3484 (0.3545) loss 3.3232 (2.9787) grad_norm 1.6387 (inf) loss_scale 256.0000 (386.1294) mem 11224MB [2024-09-01 08:38:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][550/625] eta 0:00:26 lr 0.000582 wd 0.0500 time 0.3383 (0.3552) data time 0.0009 (0.0016) model time 0.3374 (0.3543) loss 3.0328 (2.9816) grad_norm 1.6595 (inf) loss_scale 256.0000 (383.7677) mem 11224MB [2024-09-01 08:38:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][560/625] eta 0:00:23 lr 0.000581 wd 0.0500 time 0.3567 (0.3551) data time 0.0008 (0.0016) model time 0.3559 (0.3542) loss 3.3419 (2.9824) grad_norm 1.7302 (inf) loss_scale 256.0000 (381.4902) mem 11224MB [2024-09-01 08:38:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][570/625] eta 0:00:19 lr 0.000581 wd 0.0500 time 0.5563 (0.3556) data time 0.0006 (0.0015) model time 0.5556 (0.3548) loss 2.4919 (2.9796) grad_norm 1.8838 (inf) loss_scale 256.0000 (379.2925) mem 11224MB [2024-09-01 08:38:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][580/625] eta 0:00:16 lr 0.000581 wd 0.0500 time 0.3488 (0.3561) data time 0.0008 (0.0015) model time 0.3480 (0.3554) loss 3.2907 (2.9830) grad_norm 1.7260 (inf) loss_scale 256.0000 (377.1704) mem 11224MB [2024-09-01 08:38:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][590/625] eta 0:00:12 lr 0.000581 wd 0.0500 time 0.3465 (0.3564) data time 0.0006 (0.0015) model time 0.3459 (0.3557) loss 2.4853 (2.9824) grad_norm 2.0593 (inf) loss_scale 256.0000 (375.1201) mem 11224MB [2024-09-01 08:39:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][600/625] eta 0:00:08 lr 0.000581 wd 0.0500 time 0.3475 (0.3563) data time 0.0009 (0.0015) model time 0.3467 (0.3555) loss 2.2761 (2.9825) grad_norm 1.4211 (inf) loss_scale 256.0000 (373.1381) mem 11224MB [2024-09-01 08:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][610/625] eta 0:00:05 lr 0.000581 wd 0.0500 time 0.3412 (0.3565) data time 0.0006 (0.0015) model time 0.3406 (0.3557) loss 2.8677 (2.9802) grad_norm 2.0509 (inf) loss_scale 256.0000 (371.2209) mem 11224MB [2024-09-01 08:39:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [199/300][620/625] eta 0:00:01 lr 0.000581 wd 0.0500 time 0.3419 (0.3562) data time 0.0004 (0.0015) model time 0.3414 (0.3555) loss 2.1979 (2.9778) grad_norm 1.9411 (inf) loss_scale 256.0000 (369.3655) mem 11224MB [2024-09-01 08:39:10 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 199 training takes 0:03:42 [2024-09-01 08:39:10 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:39:11 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:39:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.440 (0.440) Loss 0.5469 (0.5469) Acc@1 88.477 (88.477) Acc@5 98.291 (98.291) Mem 11224MB [2024-09-01 08:39:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.133) Loss 0.8862 (0.6673) Acc@1 78.613 (85.116) Acc@5 95.508 (97.461) Mem 11224MB [2024-09-01 08:39:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.114) Loss 0.9492 (0.7908) Acc@1 76.416 (81.899) Acc@5 94.580 (96.059) Mem 11224MB [2024-09-01 08:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.676 Acc@5 96.051 [2024-09-01 08:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.7% [2024-09-01 08:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.763 (0.763) Loss 0.4956 (0.4956) Acc@1 89.600 (89.600) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 08:39:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.165) Loss 0.7974 (0.6074) Acc@1 81.201 (86.501) Acc@5 96.191 (97.723) Mem 11224MB [2024-09-01 08:39:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.130) Loss 0.8921 (0.7190) Acc@1 77.637 (83.577) Acc@5 95.312 (96.568) Mem 11224MB [2024-09-01 08:39:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.275 Acc@5 96.595 [2024-09-01 08:39:17 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 08:39:17 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.28% [2024-09-01 08:39:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:39:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][0/625] eta 0:07:02 lr 0.000580 wd 0.0500 time 0.6753 (0.6753) data time 0.3565 (0.3565) model time 0.0000 (0.0000) loss 3.1594 (3.1594) grad_norm 2.3373 (2.3373) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][10/625] eta 0:03:51 lr 0.000580 wd 0.0500 time 0.3394 (0.3757) data time 0.0008 (0.0331) model time 0.0000 (0.0000) loss 3.2443 (3.2699) grad_norm 2.3208 (2.3381) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][20/625] eta 0:03:39 lr 0.000580 wd 0.0500 time 0.3471 (0.3628) data time 0.0008 (0.0177) model time 0.0000 (0.0000) loss 2.4957 (3.1435) grad_norm 2.1009 (2.2568) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][30/625] eta 0:03:32 lr 0.000580 wd 0.0500 time 0.3451 (0.3576) data time 0.0007 (0.0123) model time 0.0000 (0.0000) loss 2.1543 (3.0167) grad_norm 3.1756 (2.2074) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][40/625] eta 0:03:28 lr 0.000580 wd 0.0500 time 0.3456 (0.3556) data time 0.0007 (0.0095) model time 0.0000 (0.0000) loss 3.1886 (2.9615) grad_norm 1.8142 (2.1411) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][50/625] eta 0:03:23 lr 0.000580 wd 0.0500 time 0.3484 (0.3542) data time 0.0006 (0.0078) model time 0.0000 (0.0000) loss 3.3002 (2.9885) grad_norm 2.8302 (2.0971) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][60/625] eta 0:03:19 lr 0.000579 wd 0.0500 time 0.3504 (0.3533) data time 0.0007 (0.0066) model time 0.3498 (0.3481) loss 2.1573 (2.9742) grad_norm 2.5498 (2.0387) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][70/625] eta 0:03:17 lr 0.000579 wd 0.0500 time 0.3470 (0.3555) data time 0.0008 (0.0058) model time 0.3461 (0.3580) loss 3.1680 (2.9421) grad_norm 1.5337 (1.9933) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][80/625] eta 0:03:13 lr 0.000579 wd 0.0500 time 0.3459 (0.3544) data time 0.0007 (0.0052) model time 0.3452 (0.3539) loss 2.2115 (2.8998) grad_norm 2.8033 (2.0457) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][90/625] eta 0:03:09 lr 0.000579 wd 0.0500 time 0.3507 (0.3538) data time 0.0009 (0.0047) model time 0.3499 (0.3525) loss 1.9862 (2.8965) grad_norm 2.0808 (2.0583) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][100/625] eta 0:03:05 lr 0.000579 wd 0.0500 time 0.3416 (0.3531) data time 0.0009 (0.0043) model time 0.3407 (0.3511) loss 2.7479 (2.8731) grad_norm 1.7560 (2.0716) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:39:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][110/625] eta 0:03:01 lr 0.000579 wd 0.0500 time 0.3454 (0.3526) data time 0.0009 (0.0041) model time 0.3444 (0.3503) loss 2.1022 (2.8612) grad_norm 1.4184 (2.0748) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][120/625] eta 0:02:57 lr 0.000579 wd 0.0500 time 0.3238 (0.3523) data time 0.0011 (0.0038) model time 0.3227 (0.3501) loss 2.1372 (2.8646) grad_norm 2.1960 (2.0686) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][130/625] eta 0:02:54 lr 0.000578 wd 0.0500 time 0.3489 (0.3520) data time 0.0008 (0.0036) model time 0.3482 (0.3497) loss 3.7168 (2.8732) grad_norm 1.4685 (2.0437) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][140/625] eta 0:02:50 lr 0.000578 wd 0.0500 time 0.3501 (0.3517) data time 0.0008 (0.0034) model time 0.3492 (0.3494) loss 2.8275 (2.8715) grad_norm 1.5606 (2.0184) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][150/625] eta 0:02:47 lr 0.000578 wd 0.0500 time 0.3462 (0.3535) data time 0.0009 (0.0032) model time 0.3453 (0.3523) loss 3.2222 (2.8844) grad_norm 1.6791 (1.9997) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][160/625] eta 0:02:45 lr 0.000578 wd 0.0500 time 0.3445 (0.3555) data time 0.0007 (0.0031) model time 0.3438 (0.3552) loss 2.5893 (2.8998) grad_norm 2.2630 (2.0028) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][170/625] eta 0:02:42 lr 0.000578 wd 0.0500 time 0.3503 (0.3563) data time 0.0006 (0.0029) model time 0.3497 (0.3564) loss 2.1538 (2.8969) grad_norm 1.9998 (1.9887) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][180/625] eta 0:02:38 lr 0.000578 wd 0.0500 time 0.3456 (0.3558) data time 0.0008 (0.0028) model time 0.3448 (0.3556) loss 3.4326 (2.9142) grad_norm 2.3674 (1.9909) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][190/625] eta 0:02:35 lr 0.000577 wd 0.0500 time 0.3489 (0.3564) data time 0.0007 (0.0027) model time 0.3482 (0.3563) loss 3.3337 (2.9207) grad_norm 2.3078 (2.0081) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][200/625] eta 0:02:31 lr 0.000577 wd 0.0500 time 0.3538 (0.3560) data time 0.0007 (0.0026) model time 0.3531 (0.3558) loss 2.8721 (2.9223) grad_norm 2.6163 (2.0058) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][210/625] eta 0:02:27 lr 0.000577 wd 0.0500 time 0.3475 (0.3556) data time 0.0008 (0.0025) model time 0.3467 (0.3551) loss 1.8482 (2.9121) grad_norm 1.4869 (1.9949) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][220/625] eta 0:02:23 lr 0.000577 wd 0.0500 time 0.3480 (0.3552) data time 0.0006 (0.0024) model time 0.3474 (0.3546) loss 3.3933 (2.9218) grad_norm 3.3466 (2.0606) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][230/625] eta 0:02:20 lr 0.000577 wd 0.0500 time 0.3494 (0.3549) data time 0.0008 (0.0024) model time 0.3486 (0.3542) loss 2.6472 (2.9209) grad_norm 9.5642 (2.0909) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][240/625] eta 0:02:16 lr 0.000577 wd 0.0500 time 0.3538 (0.3546) data time 0.0007 (0.0023) model time 0.3531 (0.3539) loss 3.4226 (2.9187) grad_norm 1.7371 (2.0881) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][250/625] eta 0:02:12 lr 0.000576 wd 0.0500 time 0.3452 (0.3544) data time 0.0008 (0.0023) model time 0.3443 (0.3536) loss 2.0322 (2.9129) grad_norm 1.2104 (2.0731) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][260/625] eta 0:02:09 lr 0.000576 wd 0.0500 time 0.3466 (0.3541) data time 0.0008 (0.0022) model time 0.3458 (0.3532) loss 2.3685 (2.9058) grad_norm 1.3229 (2.0707) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][270/625] eta 0:02:05 lr 0.000576 wd 0.0500 time 0.3504 (0.3539) data time 0.0006 (0.0022) model time 0.3498 (0.3530) loss 3.6689 (2.9123) grad_norm 1.4563 (2.0842) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:40:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][280/625] eta 0:02:02 lr 0.000576 wd 0.0500 time 0.3418 (0.3537) data time 0.0009 (0.0021) model time 0.3409 (0.3527) loss 3.2805 (2.9171) grad_norm 1.5622 (2.0870) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][290/625] eta 0:01:58 lr 0.000576 wd 0.0500 time 0.3486 (0.3541) data time 0.0007 (0.0021) model time 0.3479 (0.3532) loss 3.4068 (2.9231) grad_norm 1.9651 (2.0744) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][300/625] eta 0:01:55 lr 0.000576 wd 0.0500 time 0.3475 (0.3539) data time 0.0006 (0.0020) model time 0.3469 (0.3530) loss 3.6239 (2.9251) grad_norm 2.5197 (2.0687) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][310/625] eta 0:01:51 lr 0.000575 wd 0.0500 time 0.3496 (0.3537) data time 0.0008 (0.0020) model time 0.3488 (0.3528) loss 2.7393 (2.9320) grad_norm 2.1919 (2.0724) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][320/625] eta 0:01:47 lr 0.000575 wd 0.0500 time 0.3467 (0.3535) data time 0.0007 (0.0019) model time 0.3460 (0.3525) loss 3.2831 (2.9266) grad_norm 1.5445 (2.0743) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][330/625] eta 0:01:44 lr 0.000575 wd 0.0500 time 0.3499 (0.3533) data time 0.0008 (0.0019) model time 0.3491 (0.3523) loss 2.2229 (2.9301) grad_norm 1.6024 (2.0704) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][340/625] eta 0:01:41 lr 0.000575 wd 0.0500 time 0.3490 (0.3546) data time 0.0006 (0.0019) model time 0.3484 (0.3539) loss 2.4809 (2.9249) grad_norm 1.8439 (2.0645) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][350/625] eta 0:01:37 lr 0.000575 wd 0.0500 time 0.3519 (0.3550) data time 0.0006 (0.0018) model time 0.3513 (0.3543) loss 3.4319 (2.9286) grad_norm 1.4677 (2.0549) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][360/625] eta 0:01:34 lr 0.000575 wd 0.0500 time 0.3481 (0.3548) data time 0.0007 (0.0018) model time 0.3474 (0.3541) loss 3.1739 (2.9317) grad_norm 1.2700 (2.0630) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][370/625] eta 0:01:30 lr 0.000575 wd 0.0500 time 0.3407 (0.3546) data time 0.0007 (0.0018) model time 0.3400 (0.3539) loss 3.6371 (2.9333) grad_norm 2.1718 (2.1073) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][380/625] eta 0:01:26 lr 0.000574 wd 0.0500 time 0.3496 (0.3545) data time 0.0007 (0.0018) model time 0.3490 (0.3537) loss 2.3773 (2.9374) grad_norm 1.5170 (2.1013) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][390/625] eta 0:01:23 lr 0.000574 wd 0.0500 time 0.3412 (0.3544) data time 0.0009 (0.0017) model time 0.3402 (0.3536) loss 2.5222 (2.9387) grad_norm 2.1530 (2.1398) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][400/625] eta 0:01:19 lr 0.000574 wd 0.0500 time 0.3443 (0.3547) data time 0.0010 (0.0017) model time 0.3433 (0.3539) loss 3.3366 (2.9394) grad_norm 1.9688 (2.1348) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][410/625] eta 0:01:16 lr 0.000574 wd 0.0500 time 0.3422 (0.3544) data time 0.0007 (0.0017) model time 0.3414 (0.3537) loss 2.2001 (2.9373) grad_norm 2.7322 (2.1270) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][420/625] eta 0:01:12 lr 0.000574 wd 0.0500 time 0.3485 (0.3543) data time 0.0008 (0.0017) model time 0.3477 (0.3534) loss 3.3488 (2.9449) grad_norm 1.7318 (2.1189) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][430/625] eta 0:01:09 lr 0.000574 wd 0.0500 time 0.3562 (0.3541) data time 0.0006 (0.0017) model time 0.3556 (0.3533) loss 3.2905 (2.9460) grad_norm 1.9324 (2.1082) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][440/625] eta 0:01:05 lr 0.000573 wd 0.0500 time 0.3459 (0.3539) data time 0.0010 (0.0016) model time 0.3449 (0.3530) loss 2.9112 (2.9489) grad_norm 2.5681 (2.1066) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:41:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][450/625] eta 0:01:01 lr 0.000573 wd 0.0500 time 0.3486 (0.3538) data time 0.0008 (0.0016) model time 0.3477 (0.3529) loss 2.3388 (2.9405) grad_norm 1.7359 (2.1020) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][460/625] eta 0:00:58 lr 0.000573 wd 0.0500 time 0.3482 (0.3545) data time 0.0010 (0.0016) model time 0.3472 (0.3537) loss 3.3928 (2.9374) grad_norm 2.6268 (2.0977) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][470/625] eta 0:00:54 lr 0.000573 wd 0.0500 time 0.3481 (0.3548) data time 0.0006 (0.0016) model time 0.3475 (0.3540) loss 3.0963 (2.9360) grad_norm 1.5587 (2.0920) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][480/625] eta 0:00:51 lr 0.000573 wd 0.0500 time 0.3462 (0.3546) data time 0.0009 (0.0016) model time 0.3452 (0.3538) loss 3.1601 (2.9376) grad_norm 1.6104 (2.0864) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][490/625] eta 0:00:47 lr 0.000573 wd 0.0500 time 0.3467 (0.3545) data time 0.0008 (0.0016) model time 0.3459 (0.3537) loss 2.9577 (2.9355) grad_norm 1.5549 (2.0804) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][500/625] eta 0:00:44 lr 0.000572 wd 0.0500 time 0.3473 (0.3547) data time 0.0006 (0.0016) model time 0.3467 (0.3540) loss 3.5586 (2.9359) grad_norm 3.2733 (2.0876) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][510/625] eta 0:00:40 lr 0.000572 wd 0.0500 time 0.3485 (0.3546) data time 0.0006 (0.0015) model time 0.3479 (0.3538) loss 3.3414 (2.9378) grad_norm 1.5285 (2.0814) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][520/625] eta 0:00:37 lr 0.000572 wd 0.0500 time 0.3514 (0.3545) data time 0.0006 (0.0015) model time 0.3508 (0.3537) loss 2.6523 (2.9321) grad_norm 1.4460 (2.0722) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][530/625] eta 0:00:33 lr 0.000572 wd 0.0500 time 0.3465 (0.3544) data time 0.0007 (0.0015) model time 0.3458 (0.3535) loss 2.8451 (2.9337) grad_norm 1.4448 (2.0663) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][540/625] eta 0:00:30 lr 0.000572 wd 0.0500 time 0.3187 (0.3543) data time 0.0008 (0.0015) model time 0.3179 (0.3534) loss 3.0931 (2.9347) grad_norm 1.2260 (2.0620) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][550/625] eta 0:00:26 lr 0.000572 wd 0.0500 time 0.3437 (0.3542) data time 0.0006 (0.0015) model time 0.3431 (0.3533) loss 2.9741 (2.9325) grad_norm 1.5381 (2.0659) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][560/625] eta 0:00:23 lr 0.000572 wd 0.0500 time 0.3549 (0.3541) data time 0.0007 (0.0015) model time 0.3542 (0.3532) loss 1.8729 (2.9286) grad_norm 1.7190 (2.0652) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][570/625] eta 0:00:19 lr 0.000571 wd 0.0500 time 0.3437 (0.3540) data time 0.0009 (0.0015) model time 0.3428 (0.3531) loss 3.2840 (2.9267) grad_norm 1.3250 (2.0588) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][580/625] eta 0:00:15 lr 0.000571 wd 0.0500 time 0.3435 (0.3541) data time 0.0008 (0.0015) model time 0.3427 (0.3532) loss 1.9345 (2.9285) grad_norm 2.2125 (2.0544) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][590/625] eta 0:00:12 lr 0.000571 wd 0.0500 time 0.3525 (0.3540) data time 0.0008 (0.0014) model time 0.3517 (0.3531) loss 2.2668 (2.9271) grad_norm 2.1987 (2.0529) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][600/625] eta 0:00:08 lr 0.000571 wd 0.0500 time 0.3491 (0.3539) data time 0.0006 (0.0014) model time 0.3485 (0.3530) loss 3.0011 (2.9293) grad_norm 2.7440 (2.0581) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][610/625] eta 0:00:05 lr 0.000571 wd 0.0500 time 0.3393 (0.3538) data time 0.0006 (0.0014) model time 0.3387 (0.3529) loss 2.6529 (2.9310) grad_norm 1.1649 (2.0546) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [200/300][620/625] eta 0:00:01 lr 0.000571 wd 0.0500 time 0.3425 (0.3536) data time 0.0004 (0.0014) model time 0.3421 (0.3527) loss 2.6582 (2.9287) grad_norm 1.2962 (2.0469) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:42:59 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 200 training takes 0:03:40 [2024-09-01 08:42:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:43:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:43:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.462 (0.462) Loss 0.5605 (0.5605) Acc@1 88.135 (88.135) Acc@5 98.291 (98.291) Mem 11224MB [2024-09-01 08:43:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 0.8477 (0.6632) Acc@1 78.906 (85.405) Acc@5 95.654 (97.408) Mem 11224MB [2024-09-01 08:43:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.0156 (0.7909) Acc@1 75.391 (82.185) Acc@5 93.945 (96.029) Mem 11224MB [2024-09-01 08:43:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.828 Acc@5 96.013 [2024-09-01 08:43:03 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.8% [2024-09-01 08:43:03 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 81.83% [2024-09-01 08:43:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:43:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:43:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.447 (0.447) Loss 0.4951 (0.4951) Acc@1 89.600 (89.600) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 08:43:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.7969 (0.6066) Acc@1 81.152 (86.581) Acc@5 96.240 (97.723) Mem 11224MB [2024-09-01 08:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 0.8901 (0.7181) Acc@1 77.539 (83.617) Acc@5 95.361 (96.589) Mem 11224MB [2024-09-01 08:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.307 Acc@5 96.613 [2024-09-01 08:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 08:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.31% [2024-09-01 08:43:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:43:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:43:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][0/625] eta 0:07:23 lr 0.000570 wd 0.0500 time 0.7101 (0.7101) data time 0.3988 (0.3988) model time 0.0000 (0.0000) loss 3.5743 (3.5743) grad_norm 1.5036 (1.5036) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][10/625] eta 0:03:54 lr 0.000570 wd 0.0500 time 0.3503 (0.3811) data time 0.0006 (0.0370) model time 0.0000 (0.0000) loss 2.6893 (2.8481) grad_norm 1.7810 (1.8292) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][20/625] eta 0:03:46 lr 0.000570 wd 0.0500 time 0.3420 (0.3741) data time 0.0006 (0.0197) model time 0.0000 (0.0000) loss 2.2332 (2.8775) grad_norm 1.6741 (1.9488) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][30/625] eta 0:03:44 lr 0.000570 wd 0.0500 time 0.3472 (0.3779) data time 0.0008 (0.0136) model time 0.0000 (0.0000) loss 3.1329 (2.9422) grad_norm 1.5028 (1.8660) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][40/625] eta 0:03:36 lr 0.000570 wd 0.0500 time 0.3487 (0.3704) data time 0.0008 (0.0105) model time 0.0000 (0.0000) loss 3.2310 (2.9363) grad_norm 1.4067 (1.8474) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][50/625] eta 0:03:30 lr 0.000570 wd 0.0500 time 0.3441 (0.3656) data time 0.0008 (0.0086) model time 0.0000 (0.0000) loss 2.7692 (2.9418) grad_norm 1.8941 (1.8754) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][60/625] eta 0:03:26 lr 0.000570 wd 0.0500 time 0.3517 (0.3659) data time 0.0006 (0.0073) model time 0.3512 (0.3662) loss 2.0055 (2.9225) grad_norm 1.9522 (1.8959) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][70/625] eta 0:03:21 lr 0.000569 wd 0.0500 time 0.3485 (0.3634) data time 0.0007 (0.0064) model time 0.3477 (0.3568) loss 2.7500 (2.9416) grad_norm 2.5653 (1.9169) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][80/625] eta 0:03:17 lr 0.000569 wd 0.0500 time 0.3448 (0.3616) data time 0.0008 (0.0057) model time 0.3440 (0.3539) loss 2.5389 (2.9467) grad_norm 1.8263 (1.9963) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][90/625] eta 0:03:12 lr 0.000569 wd 0.0500 time 0.3469 (0.3601) data time 0.0009 (0.0052) model time 0.3460 (0.3523) loss 3.5218 (2.9405) grad_norm 1.7737 (1.9978) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][100/625] eta 0:03:08 lr 0.000569 wd 0.0500 time 0.3474 (0.3589) data time 0.0006 (0.0047) model time 0.3468 (0.3513) loss 3.7623 (2.9267) grad_norm 2.3826 (2.0157) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][110/625] eta 0:03:04 lr 0.000569 wd 0.0500 time 0.3435 (0.3579) data time 0.0006 (0.0044) model time 0.3429 (0.3505) loss 3.3543 (2.9278) grad_norm 1.6295 (2.1087) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][120/625] eta 0:03:00 lr 0.000569 wd 0.0500 time 0.3553 (0.3572) data time 0.0009 (0.0041) model time 0.3544 (0.3502) loss 3.4683 (2.9405) grad_norm 1.2776 (2.0876) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][130/625] eta 0:02:56 lr 0.000568 wd 0.0500 time 0.3479 (0.3566) data time 0.0008 (0.0039) model time 0.3471 (0.3499) loss 2.8992 (2.9392) grad_norm 1.3667 (2.0575) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][140/625] eta 0:02:52 lr 0.000568 wd 0.0500 time 0.3459 (0.3559) data time 0.0007 (0.0037) model time 0.3452 (0.3495) loss 2.1637 (2.9321) grad_norm 1.2866 (2.0334) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][150/625] eta 0:02:48 lr 0.000568 wd 0.0500 time 0.3440 (0.3554) data time 0.0008 (0.0035) model time 0.3433 (0.3493) loss 2.9885 (2.9240) grad_norm 2.5266 (2.0673) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][160/625] eta 0:02:45 lr 0.000568 wd 0.0500 time 0.3458 (0.3551) data time 0.0008 (0.0033) model time 0.3449 (0.3493) loss 2.3909 (2.9312) grad_norm 1.2627 (2.0558) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][170/625] eta 0:02:41 lr 0.000568 wd 0.0500 time 0.3497 (0.3547) data time 0.0007 (0.0032) model time 0.3490 (0.3492) loss 1.9849 (2.9276) grad_norm 5.5197 (2.0594) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][180/625] eta 0:02:37 lr 0.000568 wd 0.0500 time 0.3445 (0.3543) data time 0.0009 (0.0030) model time 0.3436 (0.3490) loss 3.2606 (2.9259) grad_norm 2.2551 (2.0583) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][190/625] eta 0:02:33 lr 0.000567 wd 0.0500 time 0.3490 (0.3540) data time 0.0009 (0.0029) model time 0.3481 (0.3489) loss 3.2256 (2.9341) grad_norm 1.4797 (2.0424) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][200/625] eta 0:02:30 lr 0.000567 wd 0.0500 time 0.3404 (0.3536) data time 0.0007 (0.0028) model time 0.3397 (0.3486) loss 4.1179 (2.9515) grad_norm 1.5171 (2.0956) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][210/625] eta 0:02:26 lr 0.000567 wd 0.0500 time 0.3468 (0.3533) data time 0.0007 (0.0027) model time 0.3461 (0.3485) loss 2.6018 (2.9556) grad_norm 2.7194 (2.0843) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][220/625] eta 0:02:23 lr 0.000567 wd 0.0500 time 0.3524 (0.3531) data time 0.0009 (0.0026) model time 0.3515 (0.3485) loss 2.3834 (2.9617) grad_norm 1.7191 (2.0631) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][230/625] eta 0:02:19 lr 0.000567 wd 0.0500 time 0.5524 (0.3538) data time 0.0009 (0.0026) model time 0.5515 (0.3496) loss 3.4982 (2.9777) grad_norm 1.5539 (2.0400) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][240/625] eta 0:02:16 lr 0.000567 wd 0.0500 time 0.3469 (0.3551) data time 0.0007 (0.0025) model time 0.3462 (0.3515) loss 3.2850 (2.9714) grad_norm 2.3094 (2.0195) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][250/625] eta 0:02:13 lr 0.000566 wd 0.0500 time 0.3543 (0.3563) data time 0.0006 (0.0024) model time 0.3538 (0.3530) loss 3.3627 (2.9732) grad_norm 1.4959 (2.0044) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][260/625] eta 0:02:09 lr 0.000566 wd 0.0500 time 0.3573 (0.3560) data time 0.0006 (0.0024) model time 0.3567 (0.3528) loss 3.3533 (2.9670) grad_norm 1.2437 (1.9847) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][270/625] eta 0:02:06 lr 0.000566 wd 0.0500 time 0.3526 (0.3558) data time 0.0006 (0.0023) model time 0.3520 (0.3526) loss 3.8830 (2.9722) grad_norm 1.7378 (1.9772) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][280/625] eta 0:02:02 lr 0.000566 wd 0.0500 time 0.3501 (0.3555) data time 0.0007 (0.0023) model time 0.3494 (0.3524) loss 3.5196 (2.9702) grad_norm 2.4935 (1.9706) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][290/625] eta 0:01:59 lr 0.000566 wd 0.0500 time 0.3492 (0.3553) data time 0.0008 (0.0022) model time 0.3484 (0.3522) loss 2.8301 (2.9750) grad_norm 1.5777 (1.9654) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][300/625] eta 0:01:55 lr 0.000566 wd 0.0500 time 0.3454 (0.3550) data time 0.0006 (0.0022) model time 0.3448 (0.3520) loss 3.6206 (2.9718) grad_norm 1.5648 (1.9653) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:44:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][310/625] eta 0:01:51 lr 0.000566 wd 0.0500 time 0.3497 (0.3548) data time 0.0009 (0.0021) model time 0.3488 (0.3518) loss 3.4008 (2.9723) grad_norm 1.7353 (1.9646) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][320/625] eta 0:01:48 lr 0.000565 wd 0.0500 time 0.3508 (0.3546) data time 0.0007 (0.0021) model time 0.3501 (0.3517) loss 2.5690 (2.9721) grad_norm 2.1111 (1.9564) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][330/625] eta 0:01:44 lr 0.000565 wd 0.0500 time 0.3475 (0.3545) data time 0.0006 (0.0020) model time 0.3468 (0.3517) loss 2.4607 (2.9738) grad_norm 2.0125 (1.9706) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][340/625] eta 0:01:40 lr 0.000565 wd 0.0500 time 0.3488 (0.3543) data time 0.0008 (0.0020) model time 0.3479 (0.3515) loss 3.0470 (2.9739) grad_norm 2.5601 (1.9854) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][350/625] eta 0:01:37 lr 0.000565 wd 0.0500 time 0.3456 (0.3542) data time 0.0007 (0.0020) model time 0.3450 (0.3514) loss 3.3449 (2.9688) grad_norm 1.4688 (1.9851) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][360/625] eta 0:01:33 lr 0.000565 wd 0.0500 time 0.3443 (0.3540) data time 0.0008 (0.0019) model time 0.3435 (0.3512) loss 3.1855 (2.9708) grad_norm 3.1016 (1.9890) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][370/625] eta 0:01:30 lr 0.000565 wd 0.0500 time 0.3478 (0.3548) data time 0.0008 (0.0019) model time 0.3471 (0.3522) loss 3.2362 (2.9663) grad_norm 2.2697 (2.0039) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][380/625] eta 0:01:26 lr 0.000564 wd 0.0500 time 0.3502 (0.3546) data time 0.0008 (0.0019) model time 0.3494 (0.3521) loss 3.5095 (2.9716) grad_norm 2.6911 (1.9996) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][390/625] eta 0:01:23 lr 0.000564 wd 0.0500 time 0.3513 (0.3544) data time 0.0006 (0.0018) model time 0.3506 (0.3519) loss 3.2717 (2.9719) grad_norm 1.9801 (1.9974) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][400/625] eta 0:01:19 lr 0.000564 wd 0.0500 time 0.3516 (0.3543) data time 0.0007 (0.0018) model time 0.3510 (0.3518) loss 3.0567 (2.9751) grad_norm 1.4037 (1.9860) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][410/625] eta 0:01:16 lr 0.000564 wd 0.0500 time 0.3444 (0.3541) data time 0.0007 (0.0018) model time 0.3437 (0.3516) loss 3.0937 (2.9777) grad_norm 2.3734 (1.9824) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][420/625] eta 0:01:12 lr 0.000564 wd 0.0500 time 0.3410 (0.3556) data time 0.0009 (0.0018) model time 0.3402 (0.3534) loss 3.3003 (2.9706) grad_norm 1.3862 (1.9786) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][430/625] eta 0:01:09 lr 0.000564 wd 0.0500 time 0.3477 (0.3555) data time 0.0009 (0.0018) model time 0.3468 (0.3533) loss 2.7862 (2.9706) grad_norm 1.7777 (1.9701) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][440/625] eta 0:01:05 lr 0.000563 wd 0.0500 time 0.3543 (0.3553) data time 0.0006 (0.0017) model time 0.3537 (0.3531) loss 3.4495 (2.9688) grad_norm 3.3119 (1.9733) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][450/625] eta 0:01:02 lr 0.000563 wd 0.0500 time 0.3384 (0.3551) data time 0.0009 (0.0017) model time 0.3375 (0.3529) loss 3.0398 (2.9699) grad_norm 2.3196 (1.9699) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][460/625] eta 0:00:58 lr 0.000563 wd 0.0500 time 0.3428 (0.3550) data time 0.0010 (0.0017) model time 0.3418 (0.3528) loss 3.5360 (2.9685) grad_norm 2.3043 (1.9769) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][470/625] eta 0:00:54 lr 0.000563 wd 0.0500 time 0.3487 (0.3548) data time 0.0008 (0.0017) model time 0.3479 (0.3527) loss 3.3857 (2.9730) grad_norm 1.9614 (1.9859) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:45:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][480/625] eta 0:00:51 lr 0.000563 wd 0.0500 time 0.3479 (0.3546) data time 0.0006 (0.0017) model time 0.3473 (0.3525) loss 3.4328 (2.9771) grad_norm 1.7011 (1.9842) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][490/625] eta 0:00:47 lr 0.000563 wd 0.0500 time 0.3452 (0.3545) data time 0.0009 (0.0016) model time 0.3443 (0.3524) loss 3.3426 (2.9836) grad_norm 2.0694 (1.9887) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][500/625] eta 0:00:44 lr 0.000563 wd 0.0500 time 0.3441 (0.3544) data time 0.0007 (0.0016) model time 0.3434 (0.3523) loss 3.1381 (2.9803) grad_norm 2.5295 (1.9889) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][510/625] eta 0:00:40 lr 0.000562 wd 0.0500 time 0.3469 (0.3542) data time 0.0006 (0.0016) model time 0.3463 (0.3521) loss 3.3484 (2.9809) grad_norm 1.5255 (1.9896) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][520/625] eta 0:00:37 lr 0.000562 wd 0.0500 time 0.3438 (0.3541) data time 0.0008 (0.0016) model time 0.3431 (0.3520) loss 2.7846 (2.9813) grad_norm 2.0880 (1.9846) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][530/625] eta 0:00:33 lr 0.000562 wd 0.0500 time 0.3576 (0.3539) data time 0.0007 (0.0016) model time 0.3569 (0.3519) loss 2.5403 (2.9813) grad_norm 1.3693 (1.9788) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][540/625] eta 0:00:30 lr 0.000562 wd 0.0500 time 0.3490 (0.3550) data time 0.0009 (0.0016) model time 0.3481 (0.3530) loss 3.1683 (2.9816) grad_norm 1.4245 (1.9776) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][550/625] eta 0:00:26 lr 0.000562 wd 0.0500 time 0.3478 (0.3548) data time 0.0009 (0.0016) model time 0.3469 (0.3529) loss 3.2404 (2.9814) grad_norm 1.4641 (1.9789) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][560/625] eta 0:00:23 lr 0.000562 wd 0.0500 time 0.3327 (0.3547) data time 0.0010 (0.0015) model time 0.3317 (0.3528) loss 2.4352 (2.9769) grad_norm 1.6886 (1.9742) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][570/625] eta 0:00:19 lr 0.000561 wd 0.0500 time 0.5503 (0.3550) data time 0.0008 (0.0015) model time 0.5495 (0.3531) loss 2.6028 (2.9743) grad_norm 2.4179 (1.9751) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][580/625] eta 0:00:15 lr 0.000561 wd 0.0500 time 0.3478 (0.3552) data time 0.0006 (0.0015) model time 0.3472 (0.3533) loss 2.0393 (2.9708) grad_norm 1.9610 (1.9724) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][590/625] eta 0:00:12 lr 0.000561 wd 0.0500 time 0.3484 (0.3551) data time 0.0008 (0.0015) model time 0.3476 (0.3532) loss 3.5190 (2.9701) grad_norm 1.7656 (1.9694) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][600/625] eta 0:00:08 lr 0.000561 wd 0.0500 time 0.3540 (0.3550) data time 0.0007 (0.0015) model time 0.3533 (0.3531) loss 3.7503 (2.9771) grad_norm 3.1451 (1.9707) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][610/625] eta 0:00:05 lr 0.000561 wd 0.0500 time 0.3411 (0.3549) data time 0.0006 (0.0015) model time 0.3405 (0.3530) loss 2.8740 (2.9808) grad_norm 1.9576 (1.9703) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [201/300][620/625] eta 0:00:01 lr 0.000561 wd 0.0500 time 0.3372 (0.3546) data time 0.0004 (0.0015) model time 0.3368 (0.3528) loss 3.5697 (2.9817) grad_norm 1.5239 (1.9657) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:46:49 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 201 training takes 0:03:41 [2024-09-01 08:46:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:46:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:46:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.603 (0.603) Loss 0.5312 (0.5312) Acc@1 88.232 (88.232) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 08:46:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.146) Loss 0.9268 (0.6646) Acc@1 77.588 (85.489) Acc@5 95.166 (97.408) Mem 11224MB [2024-09-01 08:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.120) Loss 0.9761 (0.7897) Acc@1 77.002 (82.275) Acc@5 94.141 (96.054) Mem 11224MB [2024-09-01 08:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.000 Acc@5 96.041 [2024-09-01 08:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.0% [2024-09-01 08:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.00% [2024-09-01 08:46:53 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:46:54 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:46:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.491 (0.491) Loss 0.4934 (0.4934) Acc@1 89.551 (89.551) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 08:46:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.133) Loss 0.7964 (0.6060) Acc@1 81.104 (86.590) Acc@5 96.143 (97.714) Mem 11224MB [2024-09-01 08:46:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.113) Loss 0.8887 (0.7171) Acc@1 77.686 (83.612) Acc@5 95.312 (96.594) Mem 11224MB [2024-09-01 08:46:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.285 Acc@5 96.615 [2024-09-01 08:46:57 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 08:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][0/625] eta 0:12:13 lr 0.000561 wd 0.0500 time 1.1730 (1.1730) data time 0.6197 (0.6197) model time 0.0000 (0.0000) loss 2.6560 (2.6560) grad_norm 1.5748 (1.5748) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][10/625] eta 0:04:20 lr 0.000560 wd 0.0500 time 0.3417 (0.4237) data time 0.0006 (0.0570) model time 0.0000 (0.0000) loss 3.4920 (2.9513) grad_norm 1.6725 (1.6249) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][20/625] eta 0:03:54 lr 0.000560 wd 0.0500 time 0.3468 (0.3873) data time 0.0006 (0.0304) model time 0.0000 (0.0000) loss 3.0387 (2.9901) grad_norm 1.9488 (1.7250) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][30/625] eta 0:03:42 lr 0.000560 wd 0.0500 time 0.3438 (0.3742) data time 0.0009 (0.0208) model time 0.0000 (0.0000) loss 3.1833 (3.0112) grad_norm 1.8893 (1.7758) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][40/625] eta 0:03:35 lr 0.000560 wd 0.0500 time 0.3478 (0.3681) data time 0.0009 (0.0160) model time 0.0000 (0.0000) loss 2.6706 (3.0149) grad_norm 2.1410 (1.8394) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][50/625] eta 0:03:29 lr 0.000560 wd 0.0500 time 0.3498 (0.3641) data time 0.0009 (0.0130) model time 0.0000 (0.0000) loss 3.0374 (3.0139) grad_norm 3.1041 (1.8389) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][60/625] eta 0:03:25 lr 0.000560 wd 0.0500 time 0.3506 (0.3643) data time 0.0007 (0.0110) model time 0.3499 (0.3646) loss 3.6551 (3.0218) grad_norm 1.6013 (1.8470) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][70/625] eta 0:03:23 lr 0.000559 wd 0.0500 time 0.3433 (0.3665) data time 0.0008 (0.0096) model time 0.3425 (0.3719) loss 2.5252 (3.0097) grad_norm 3.2878 (1.9362) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][80/625] eta 0:03:19 lr 0.000559 wd 0.0500 time 0.3405 (0.3666) data time 0.0008 (0.0085) model time 0.3396 (0.3700) loss 2.9000 (2.9745) grad_norm 1.6133 (1.9515) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][90/625] eta 0:03:15 lr 0.000559 wd 0.0500 time 0.3514 (0.3648) data time 0.0006 (0.0076) model time 0.3509 (0.3649) loss 3.3126 (2.9691) grad_norm 3.1295 (1.9471) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][100/625] eta 0:03:10 lr 0.000559 wd 0.0500 time 0.3501 (0.3632) data time 0.0008 (0.0070) model time 0.3493 (0.3615) loss 3.3125 (2.9954) grad_norm 1.9275 (1.9696) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][110/625] eta 0:03:06 lr 0.000559 wd 0.0500 time 0.3439 (0.3618) data time 0.0009 (0.0064) model time 0.3430 (0.3590) loss 2.9575 (2.9932) grad_norm 1.3726 (1.9880) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][120/625] eta 0:03:02 lr 0.000559 wd 0.0500 time 0.3557 (0.3606) data time 0.0007 (0.0059) model time 0.3550 (0.3572) loss 3.6400 (3.0013) grad_norm 1.7711 (2.0006) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][130/625] eta 0:02:57 lr 0.000558 wd 0.0500 time 0.3444 (0.3596) data time 0.0009 (0.0055) model time 0.3435 (0.3559) loss 3.3652 (3.0004) grad_norm 1.8981 (1.9982) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][140/625] eta 0:02:54 lr 0.000558 wd 0.0500 time 0.3477 (0.3588) data time 0.0009 (0.0052) model time 0.3468 (0.3550) loss 3.4196 (3.0026) grad_norm 1.9554 (1.9754) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][150/625] eta 0:02:51 lr 0.000558 wd 0.0500 time 0.3453 (0.3604) data time 0.0008 (0.0049) model time 0.3444 (0.3577) loss 2.1841 (2.9887) grad_norm 1.7426 (1.9484) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][160/625] eta 0:02:47 lr 0.000558 wd 0.0500 time 0.3481 (0.3596) data time 0.0008 (0.0047) model time 0.3472 (0.3567) loss 2.7820 (2.9812) grad_norm 1.5529 (1.9419) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:47:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][170/625] eta 0:02:43 lr 0.000558 wd 0.0500 time 0.3422 (0.3591) data time 0.0008 (0.0044) model time 0.3414 (0.3561) loss 2.8758 (2.9933) grad_norm 1.2984 (1.9304) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][180/625] eta 0:02:39 lr 0.000558 wd 0.0500 time 0.3652 (0.3586) data time 0.0008 (0.0043) model time 0.3644 (0.3556) loss 2.9253 (3.0010) grad_norm 1.8214 (1.9575) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][190/625] eta 0:02:35 lr 0.000558 wd 0.0500 time 0.3507 (0.3580) data time 0.0008 (0.0041) model time 0.3499 (0.3550) loss 3.2354 (3.0055) grad_norm 1.6738 (1.9431) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][200/625] eta 0:02:31 lr 0.000557 wd 0.0500 time 0.3465 (0.3575) data time 0.0006 (0.0039) model time 0.3458 (0.3544) loss 3.0906 (3.0090) grad_norm 2.1065 (1.9377) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][210/625] eta 0:02:28 lr 0.000557 wd 0.0500 time 0.3493 (0.3571) data time 0.0009 (0.0038) model time 0.3484 (0.3540) loss 2.8823 (2.9945) grad_norm 1.5343 (1.9288) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][220/625] eta 0:02:24 lr 0.000557 wd 0.0500 time 0.3524 (0.3566) data time 0.0007 (0.0036) model time 0.3517 (0.3536) loss 3.7607 (3.0019) grad_norm 3.9352 (1.9354) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][230/625] eta 0:02:20 lr 0.000557 wd 0.0500 time 0.3417 (0.3563) data time 0.0006 (0.0035) model time 0.3411 (0.3532) loss 1.6423 (2.9913) grad_norm 3.5758 (1.9571) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][240/625] eta 0:02:17 lr 0.000557 wd 0.0500 time 0.3436 (0.3566) data time 0.0009 (0.0034) model time 0.3427 (0.3537) loss 2.9002 (2.9950) grad_norm 2.3092 (1.9694) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][250/625] eta 0:02:13 lr 0.000557 wd 0.0500 time 0.3454 (0.3563) data time 0.0009 (0.0033) model time 0.3445 (0.3535) loss 2.4888 (2.9853) grad_norm 1.6205 (1.9577) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][260/625] eta 0:02:09 lr 0.000556 wd 0.0500 time 0.3546 (0.3561) data time 0.0006 (0.0032) model time 0.3540 (0.3533) loss 3.5596 (2.9891) grad_norm 1.6882 (1.9523) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][270/625] eta 0:02:06 lr 0.000556 wd 0.0500 time 0.3452 (0.3559) data time 0.0006 (0.0031) model time 0.3446 (0.3532) loss 3.4855 (2.9941) grad_norm 1.5645 (1.9436) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][280/625] eta 0:02:03 lr 0.000556 wd 0.0500 time 0.5010 (0.3568) data time 0.0007 (0.0030) model time 0.5003 (0.3544) loss 3.2298 (2.9939) grad_norm 1.5693 (1.9289) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][290/625] eta 0:01:59 lr 0.000556 wd 0.0500 time 0.3448 (0.3572) data time 0.0007 (0.0030) model time 0.3441 (0.3550) loss 3.5504 (2.9932) grad_norm 1.5099 (1.9198) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][300/625] eta 0:01:56 lr 0.000556 wd 0.0500 time 0.3429 (0.3570) data time 0.0007 (0.0029) model time 0.3423 (0.3547) loss 2.9340 (2.9865) grad_norm 1.8167 (1.9073) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][310/625] eta 0:01:52 lr 0.000556 wd 0.0500 time 0.3413 (0.3567) data time 0.0006 (0.0028) model time 0.3407 (0.3544) loss 3.5930 (2.9878) grad_norm 3.4080 (1.9152) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][320/625] eta 0:01:48 lr 0.000555 wd 0.0500 time 0.3429 (0.3564) data time 0.0008 (0.0028) model time 0.3421 (0.3541) loss 2.7040 (2.9864) grad_norm 1.7587 (1.9168) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][330/625] eta 0:01:45 lr 0.000555 wd 0.0500 time 0.3461 (0.3561) data time 0.0007 (0.0027) model time 0.3455 (0.3539) loss 2.4620 (2.9826) grad_norm 1.9162 (1.9565) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:48:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][340/625] eta 0:01:41 lr 0.000555 wd 0.0500 time 0.3466 (0.3567) data time 0.0009 (0.0026) model time 0.3457 (0.3546) loss 3.4123 (2.9833) grad_norm 2.1641 (1.9604) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:49:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][350/625] eta 0:01:38 lr 0.000555 wd 0.0500 time 0.3484 (0.3565) data time 0.0006 (0.0026) model time 0.3478 (0.3544) loss 2.0183 (2.9831) grad_norm 2.0480 (2.0027) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:49:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][360/625] eta 0:01:34 lr 0.000555 wd 0.0500 time 0.3538 (0.3568) data time 0.0006 (0.0025) model time 0.3532 (0.3548) loss 3.5114 (2.9841) grad_norm 2.1598 (1.9975) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:49:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][370/625] eta 0:01:30 lr 0.000555 wd 0.0500 time 0.3433 (0.3565) data time 0.0007 (0.0025) model time 0.3426 (0.3545) loss 2.6043 (2.9866) grad_norm 1.6882 (1.9941) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][380/625] eta 0:01:27 lr 0.000555 wd 0.0500 time 0.3499 (0.3563) data time 0.0008 (0.0025) model time 0.3491 (0.3543) loss 2.5530 (2.9838) grad_norm 1.4854 (1.9869) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:49:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][390/625] eta 0:01:23 lr 0.000554 wd 0.0500 time 0.3456 (0.3561) data time 0.0006 (0.0024) model time 0.3449 (0.3541) loss 2.2633 (2.9765) grad_norm 1.6889 (1.9836) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 08:49:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][400/625] eta 0:01:20 lr 0.000554 wd 0.0500 time 0.3427 (0.3559) data time 0.0007 (0.0024) model time 0.3420 (0.3539) loss 2.7690 (2.9819) grad_norm 1.6812 (1.9850) loss_scale 512.0000 (256.6384) mem 11224MB [2024-09-01 08:49:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][410/625] eta 0:01:16 lr 0.000554 wd 0.0500 time 0.3480 (0.3557) data time 0.0009 (0.0023) model time 0.3471 (0.3537) loss 3.2861 (2.9897) grad_norm 1.8077 (1.9798) loss_scale 512.0000 (262.8516) mem 11224MB [2024-09-01 08:49:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][420/625] eta 0:01:12 lr 0.000554 wd 0.0500 time 0.3457 (0.3556) data time 0.0008 (0.0023) model time 0.3450 (0.3536) loss 3.0201 (2.9872) grad_norm 1.1495 (1.9760) loss_scale 512.0000 (268.7696) mem 11224MB [2024-09-01 08:49:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][430/625] eta 0:01:09 lr 0.000554 wd 0.0500 time 0.3508 (0.3554) data time 0.0007 (0.0023) model time 0.3501 (0.3535) loss 2.7450 (2.9907) grad_norm 4.8218 (1.9831) loss_scale 512.0000 (274.4130) mem 11224MB [2024-09-01 08:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][440/625] eta 0:01:05 lr 0.000554 wd 0.0500 time 0.3487 (0.3553) data time 0.0006 (0.0022) model time 0.3481 (0.3533) loss 3.4465 (2.9949) grad_norm 1.7448 (1.9779) loss_scale 512.0000 (279.8005) mem 11224MB [2024-09-01 08:49:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][450/625] eta 0:01:02 lr 0.000553 wd 0.0500 time 0.3471 (0.3551) data time 0.0006 (0.0022) model time 0.3464 (0.3532) loss 3.4399 (2.9956) grad_norm 1.6997 (1.9734) loss_scale 512.0000 (284.9490) mem 11224MB [2024-09-01 08:49:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][460/625] eta 0:00:58 lr 0.000553 wd 0.0500 time 0.5144 (0.3558) data time 0.0005 (0.0022) model time 0.5138 (0.3539) loss 1.9601 (2.9908) grad_norm 1.6398 (1.9641) loss_scale 512.0000 (289.8742) mem 11224MB [2024-09-01 08:49:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][470/625] eta 0:00:55 lr 0.000553 wd 0.0500 time 0.3482 (0.3571) data time 0.0009 (0.0021) model time 0.3474 (0.3554) loss 3.1579 (2.9946) grad_norm 1.6447 (1.9583) loss_scale 512.0000 (294.5902) mem 11224MB [2024-09-01 08:49:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][480/625] eta 0:00:51 lr 0.000553 wd 0.0500 time 0.3473 (0.3569) data time 0.0006 (0.0021) model time 0.3467 (0.3552) loss 2.0208 (2.9940) grad_norm 1.7251 (1.9503) loss_scale 512.0000 (299.1102) mem 11224MB [2024-09-01 08:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][490/625] eta 0:00:48 lr 0.000553 wd 0.0500 time 0.3493 (0.3567) data time 0.0007 (0.0021) model time 0.3486 (0.3550) loss 3.5493 (2.9932) grad_norm 1.9565 (1.9486) loss_scale 512.0000 (303.4460) mem 11224MB [2024-09-01 08:49:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][500/625] eta 0:00:44 lr 0.000553 wd 0.0500 time 0.3498 (0.3566) data time 0.0006 (0.0021) model time 0.3492 (0.3549) loss 1.8695 (2.9920) grad_norm 1.8582 (1.9496) loss_scale 512.0000 (307.6088) mem 11224MB [2024-09-01 08:49:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][510/625] eta 0:00:40 lr 0.000552 wd 0.0500 time 0.3477 (0.3565) data time 0.0007 (0.0020) model time 0.3470 (0.3548) loss 2.7058 (2.9919) grad_norm 4.1778 (1.9627) loss_scale 512.0000 (311.6086) mem 11224MB [2024-09-01 08:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][520/625] eta 0:00:37 lr 0.000552 wd 0.0500 time 0.3525 (0.3563) data time 0.0008 (0.0020) model time 0.3517 (0.3547) loss 3.7956 (2.9942) grad_norm 1.9792 (1.9719) loss_scale 512.0000 (315.4549) mem 11224MB [2024-09-01 08:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][530/625] eta 0:00:33 lr 0.000552 wd 0.0500 time 0.3461 (0.3562) data time 0.0006 (0.0020) model time 0.3455 (0.3546) loss 3.2063 (2.9908) grad_norm 1.5556 (1.9693) loss_scale 512.0000 (319.1563) mem 11224MB [2024-09-01 08:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][540/625] eta 0:00:30 lr 0.000552 wd 0.0500 time 0.3497 (0.3561) data time 0.0006 (0.0020) model time 0.3492 (0.3545) loss 3.5142 (2.9874) grad_norm 1.5890 (1.9663) loss_scale 512.0000 (322.7209) mem 11224MB [2024-09-01 08:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][550/625] eta 0:00:26 lr 0.000552 wd 0.0500 time 0.3440 (0.3560) data time 0.0007 (0.0019) model time 0.3433 (0.3543) loss 2.7203 (2.9833) grad_norm 2.7183 (1.9662) loss_scale 512.0000 (326.1561) mem 11224MB [2024-09-01 08:50:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][560/625] eta 0:00:23 lr 0.000552 wd 0.0500 time 0.3480 (0.3558) data time 0.0006 (0.0019) model time 0.3474 (0.3542) loss 2.3291 (2.9822) grad_norm 2.4437 (1.9838) loss_scale 512.0000 (329.4688) mem 11224MB [2024-09-01 08:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][570/625] eta 0:00:19 lr 0.000552 wd 0.0500 time 0.3459 (0.3561) data time 0.0006 (0.0019) model time 0.3452 (0.3545) loss 3.5797 (2.9812) grad_norm 1.7041 (1.9831) loss_scale 512.0000 (332.6655) mem 11224MB [2024-09-01 08:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][580/625] eta 0:00:16 lr 0.000551 wd 0.0500 time 0.5490 (0.3566) data time 0.0007 (0.0019) model time 0.5483 (0.3551) loss 3.3938 (2.9846) grad_norm 1.3650 (1.9788) loss_scale 512.0000 (335.7522) mem 11224MB [2024-09-01 08:50:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][590/625] eta 0:00:12 lr 0.000551 wd 0.0500 time 0.3446 (0.3571) data time 0.0007 (0.0019) model time 0.3439 (0.3556) loss 3.5521 (2.9860) grad_norm 1.3176 (1.9785) loss_scale 512.0000 (338.7343) mem 11224MB [2024-09-01 08:50:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][600/625] eta 0:00:08 lr 0.000551 wd 0.0500 time 0.3445 (0.3569) data time 0.0009 (0.0018) model time 0.3436 (0.3555) loss 3.0310 (2.9825) grad_norm 1.3651 (1.9723) loss_scale 512.0000 (341.6173) mem 11224MB [2024-09-01 08:50:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][610/625] eta 0:00:05 lr 0.000551 wd 0.0500 time 0.3388 (0.3568) data time 0.0006 (0.0018) model time 0.3382 (0.3553) loss 3.1015 (2.9824) grad_norm 1.1769 (1.9652) loss_scale 512.0000 (344.4059) mem 11224MB [2024-09-01 08:50:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [202/300][620/625] eta 0:00:01 lr 0.000551 wd 0.0500 time 0.3383 (0.3565) data time 0.0004 (0.0018) model time 0.3379 (0.3550) loss 3.0924 (2.9806) grad_norm 1.9064 (1.9731) loss_scale 512.0000 (347.1047) mem 11224MB [2024-09-01 08:50:39 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 202 training takes 0:03:42 [2024-09-01 08:50:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:50:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:50:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.466 (0.466) Loss 0.5327 (0.5327) Acc@1 87.891 (87.891) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 08:50:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 0.8564 (0.6499) Acc@1 78.906 (85.547) Acc@5 95.703 (97.554) Mem 11224MB [2024-09-01 08:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 1.0127 (0.7798) Acc@1 74.902 (82.310) Acc@5 94.531 (96.101) Mem 11224MB [2024-09-01 08:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 81.926 Acc@5 96.079 [2024-09-01 08:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 81.9% [2024-09-01 08:50:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.749 (0.749) Loss 0.4927 (0.4927) Acc@1 89.551 (89.551) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 08:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.161) Loss 0.7949 (0.6048) Acc@1 81.055 (86.594) Acc@5 96.143 (97.723) Mem 11224MB [2024-09-01 08:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.129) Loss 0.8892 (0.7160) Acc@1 77.686 (83.638) Acc@5 95.264 (96.603) Mem 11224MB [2024-09-01 08:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.319 Acc@5 96.643 [2024-09-01 08:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 08:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.32% [2024-09-01 08:50:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:50:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:50:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][0/625] eta 0:07:16 lr 0.000551 wd 0.0500 time 0.6981 (0.6981) data time 0.3878 (0.3878) model time 0.0000 (0.0000) loss 2.9165 (2.9165) grad_norm 1.4661 (1.4661) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:50:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][10/625] eta 0:03:54 lr 0.000551 wd 0.0500 time 0.3501 (0.3809) data time 0.0006 (0.0360) model time 0.0000 (0.0000) loss 3.5663 (3.0456) grad_norm 1.1702 (1.6988) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:50:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][20/625] eta 0:03:46 lr 0.000550 wd 0.0500 time 0.3454 (0.3744) data time 0.0008 (0.0192) model time 0.0000 (0.0000) loss 2.9198 (2.9637) grad_norm 1.9092 (2.0044) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:50:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][30/625] eta 0:03:37 lr 0.000550 wd 0.0500 time 0.3471 (0.3654) data time 0.0007 (0.0133) model time 0.0000 (0.0000) loss 2.8202 (2.9538) grad_norm 1.3530 (2.0179) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][40/625] eta 0:03:31 lr 0.000550 wd 0.0500 time 0.3465 (0.3615) data time 0.0008 (0.0103) model time 0.0000 (0.0000) loss 2.7333 (2.9678) grad_norm 2.1049 (1.9697) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][50/625] eta 0:03:26 lr 0.000550 wd 0.0500 time 0.3476 (0.3588) data time 0.0006 (0.0084) model time 0.0000 (0.0000) loss 3.3005 (2.9167) grad_norm 2.0839 (1.9258) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][60/625] eta 0:03:21 lr 0.000550 wd 0.0500 time 0.3485 (0.3570) data time 0.0009 (0.0072) model time 0.3476 (0.3470) loss 2.5207 (2.9063) grad_norm 1.8379 (1.8831) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][70/625] eta 0:03:17 lr 0.000550 wd 0.0500 time 0.3492 (0.3557) data time 0.0006 (0.0063) model time 0.3486 (0.3471) loss 2.1775 (2.9299) grad_norm 2.5543 (1.8909) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][80/625] eta 0:03:14 lr 0.000549 wd 0.0500 time 0.3485 (0.3571) data time 0.0008 (0.0056) model time 0.3477 (0.3533) loss 3.4130 (2.9426) grad_norm 2.7583 (1.9455) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][90/625] eta 0:03:10 lr 0.000549 wd 0.0500 time 0.3494 (0.3560) data time 0.0008 (0.0051) model time 0.3486 (0.3517) loss 3.2045 (2.9464) grad_norm 2.1644 (1.9465) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][100/625] eta 0:03:09 lr 0.000549 wd 0.0500 time 0.3497 (0.3608) data time 0.0007 (0.0047) model time 0.3489 (0.3621) loss 3.3913 (2.9404) grad_norm 1.7158 (1.9550) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][110/625] eta 0:03:05 lr 0.000549 wd 0.0500 time 0.3431 (0.3595) data time 0.0008 (0.0043) model time 0.3422 (0.3593) loss 2.9604 (2.9425) grad_norm 1.6979 (2.0109) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][120/625] eta 0:03:00 lr 0.000549 wd 0.0500 time 0.3443 (0.3584) data time 0.0008 (0.0040) model time 0.3435 (0.3573) loss 2.7458 (2.9428) grad_norm 1.8616 (1.9958) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][130/625] eta 0:02:56 lr 0.000549 wd 0.0500 time 0.3467 (0.3575) data time 0.0007 (0.0038) model time 0.3460 (0.3559) loss 3.1056 (2.9381) grad_norm 2.6093 (1.9869) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][140/625] eta 0:02:53 lr 0.000548 wd 0.0500 time 0.3479 (0.3568) data time 0.0007 (0.0036) model time 0.3472 (0.3549) loss 3.4343 (2.9306) grad_norm 2.3891 (1.9862) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][150/625] eta 0:02:49 lr 0.000548 wd 0.0500 time 0.3529 (0.3562) data time 0.0008 (0.0034) model time 0.3520 (0.3540) loss 3.0043 (2.9274) grad_norm 1.7978 (1.9823) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][160/625] eta 0:02:45 lr 0.000548 wd 0.0500 time 0.3495 (0.3557) data time 0.0008 (0.0032) model time 0.3487 (0.3535) loss 2.3701 (2.9180) grad_norm 1.7582 (1.9908) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][170/625] eta 0:02:41 lr 0.000548 wd 0.0500 time 0.3473 (0.3552) data time 0.0006 (0.0031) model time 0.3468 (0.3528) loss 3.0614 (2.9276) grad_norm 1.8053 (1.9830) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][180/625] eta 0:02:37 lr 0.000548 wd 0.0500 time 0.3427 (0.3547) data time 0.0007 (0.0030) model time 0.3420 (0.3522) loss 2.9762 (2.9318) grad_norm 2.1610 (1.9958) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][190/625] eta 0:02:34 lr 0.000548 wd 0.0500 time 0.3471 (0.3543) data time 0.0006 (0.0029) model time 0.3465 (0.3518) loss 3.2934 (2.9364) grad_norm 3.3550 (2.0067) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:51:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][200/625] eta 0:02:30 lr 0.000548 wd 0.0500 time 0.3490 (0.3547) data time 0.0006 (0.0028) model time 0.3484 (0.3525) loss 2.6424 (2.9288) grad_norm 1.4699 (2.0439) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][210/625] eta 0:02:27 lr 0.000547 wd 0.0500 time 0.3450 (0.3543) data time 0.0007 (0.0027) model time 0.3443 (0.3521) loss 2.3159 (2.9307) grad_norm 1.5629 (2.0293) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][220/625] eta 0:02:23 lr 0.000547 wd 0.0500 time 0.3486 (0.3541) data time 0.0008 (0.0026) model time 0.3477 (0.3518) loss 2.7730 (2.9351) grad_norm 2.3865 (2.0659) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][230/625] eta 0:02:20 lr 0.000547 wd 0.0500 time 0.5362 (0.3547) data time 0.0006 (0.0025) model time 0.5356 (0.3528) loss 3.5996 (2.9452) grad_norm 1.3798 (2.0597) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][240/625] eta 0:02:16 lr 0.000547 wd 0.0500 time 0.3430 (0.3544) data time 0.0009 (0.0024) model time 0.3420 (0.3524) loss 3.2214 (2.9484) grad_norm 1.8030 (2.0611) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][250/625] eta 0:02:12 lr 0.000547 wd 0.0500 time 0.3495 (0.3542) data time 0.0009 (0.0024) model time 0.3486 (0.3521) loss 1.7660 (2.9498) grad_norm 1.6255 (2.0601) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][260/625] eta 0:02:09 lr 0.000547 wd 0.0500 time 0.3477 (0.3540) data time 0.0006 (0.0023) model time 0.3472 (0.3519) loss 1.9453 (2.9443) grad_norm 1.6758 (2.0659) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][270/625] eta 0:02:05 lr 0.000546 wd 0.0500 time 0.3469 (0.3537) data time 0.0009 (0.0023) model time 0.3461 (0.3517) loss 2.9047 (2.9433) grad_norm 2.6921 (2.1278) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][280/625] eta 0:02:01 lr 0.000546 wd 0.0500 time 0.3482 (0.3534) data time 0.0008 (0.0022) model time 0.3474 (0.3514) loss 3.2747 (2.9442) grad_norm 2.5635 (2.1593) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][290/625] eta 0:01:58 lr 0.000546 wd 0.0500 time 0.3499 (0.3532) data time 0.0007 (0.0022) model time 0.3493 (0.3512) loss 2.8804 (2.9421) grad_norm 2.0936 (2.1593) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][300/625] eta 0:01:54 lr 0.000546 wd 0.0500 time 0.3472 (0.3530) data time 0.0006 (0.0021) model time 0.3466 (0.3510) loss 2.0458 (2.9424) grad_norm 2.3943 (2.1625) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][310/625] eta 0:01:51 lr 0.000546 wd 0.0500 time 0.3482 (0.3547) data time 0.0009 (0.0021) model time 0.3473 (0.3531) loss 3.3019 (2.9381) grad_norm 2.0190 (2.1700) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][320/625] eta 0:01:48 lr 0.000546 wd 0.0500 time 0.3438 (0.3545) data time 0.0006 (0.0020) model time 0.3432 (0.3528) loss 3.9467 (2.9436) grad_norm 2.1815 (2.1670) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][330/625] eta 0:01:44 lr 0.000545 wd 0.0500 time 0.3562 (0.3544) data time 0.0007 (0.0020) model time 0.3556 (0.3527) loss 3.2688 (2.9497) grad_norm 1.8171 (2.1714) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][340/625] eta 0:01:40 lr 0.000545 wd 0.0500 time 0.3444 (0.3541) data time 0.0006 (0.0020) model time 0.3438 (0.3524) loss 3.2848 (2.9488) grad_norm 1.6628 (2.1577) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][350/625] eta 0:01:37 lr 0.000545 wd 0.0500 time 0.3619 (0.3540) data time 0.0008 (0.0019) model time 0.3612 (0.3523) loss 3.4801 (2.9542) grad_norm 1.6459 (2.1594) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][360/625] eta 0:01:33 lr 0.000545 wd 0.0500 time 0.3486 (0.3538) data time 0.0008 (0.0019) model time 0.3477 (0.3521) loss 2.4848 (2.9563) grad_norm 1.8608 (2.1587) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][370/625] eta 0:01:30 lr 0.000545 wd 0.0500 time 0.3453 (0.3537) data time 0.0009 (0.0019) model time 0.3444 (0.3520) loss 3.0882 (2.9496) grad_norm 1.7629 (2.1493) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][380/625] eta 0:01:26 lr 0.000545 wd 0.0500 time 0.3486 (0.3535) data time 0.0008 (0.0019) model time 0.3478 (0.3518) loss 3.0324 (2.9478) grad_norm 2.4931 (2.1460) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][390/625] eta 0:01:23 lr 0.000545 wd 0.0500 time 0.3458 (0.3533) data time 0.0011 (0.0018) model time 0.3447 (0.3516) loss 3.2548 (2.9493) grad_norm 3.1397 (2.1607) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][400/625] eta 0:01:19 lr 0.000544 wd 0.0500 time 0.3448 (0.3531) data time 0.0009 (0.0018) model time 0.3439 (0.3515) loss 3.0158 (2.9527) grad_norm 1.5755 (2.1508) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][410/625] eta 0:01:15 lr 0.000544 wd 0.0500 time 0.3478 (0.3534) data time 0.0008 (0.0018) model time 0.3470 (0.3517) loss 2.7887 (2.9476) grad_norm 1.9500 (2.1401) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][420/625] eta 0:01:12 lr 0.000544 wd 0.0500 time 0.3453 (0.3537) data time 0.0006 (0.0018) model time 0.3448 (0.3521) loss 2.9403 (2.9502) grad_norm 1.6708 (2.1320) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][430/625] eta 0:01:08 lr 0.000544 wd 0.0500 time 0.3420 (0.3535) data time 0.0007 (0.0017) model time 0.3413 (0.3519) loss 1.9474 (2.9451) grad_norm 1.5357 (2.1191) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][440/625] eta 0:01:05 lr 0.000544 wd 0.0500 time 0.3504 (0.3534) data time 0.0006 (0.0017) model time 0.3498 (0.3518) loss 2.2534 (2.9470) grad_norm 1.6878 (2.1106) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][450/625] eta 0:01:01 lr 0.000544 wd 0.0500 time 0.3490 (0.3532) data time 0.0009 (0.0017) model time 0.3480 (0.3517) loss 3.3805 (2.9430) grad_norm 1.4587 (2.1030) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][460/625] eta 0:00:58 lr 0.000543 wd 0.0500 time 0.3436 (0.3531) data time 0.0008 (0.0017) model time 0.3429 (0.3516) loss 3.6552 (2.9425) grad_norm 1.5884 (2.0964) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][470/625] eta 0:00:54 lr 0.000543 wd 0.0500 time 0.3469 (0.3530) data time 0.0006 (0.0017) model time 0.3463 (0.3515) loss 2.8394 (2.9445) grad_norm 1.6532 (2.0885) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][480/625] eta 0:00:51 lr 0.000543 wd 0.0500 time 0.3506 (0.3529) data time 0.0009 (0.0016) model time 0.3497 (0.3514) loss 3.1660 (2.9453) grad_norm 2.1356 (2.0825) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][490/625] eta 0:00:47 lr 0.000543 wd 0.0500 time 0.5736 (0.3541) data time 0.0006 (0.0016) model time 0.5729 (0.3527) loss 3.4115 (2.9432) grad_norm 2.3571 (2.0904) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][500/625] eta 0:00:44 lr 0.000543 wd 0.0500 time 0.3450 (0.3540) data time 0.0007 (0.0016) model time 0.3443 (0.3526) loss 3.9078 (2.9481) grad_norm 2.3448 (2.0888) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][510/625] eta 0:00:40 lr 0.000543 wd 0.0500 time 0.3501 (0.3539) data time 0.0007 (0.0016) model time 0.3494 (0.3525) loss 3.6006 (2.9474) grad_norm 2.7141 (2.0922) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][520/625] eta 0:00:37 lr 0.000543 wd 0.0500 time 0.3435 (0.3538) data time 0.0008 (0.0016) model time 0.3427 (0.3524) loss 3.4050 (2.9489) grad_norm 2.2174 (2.0911) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][530/625] eta 0:00:33 lr 0.000542 wd 0.0500 time 0.3483 (0.3537) data time 0.0006 (0.0016) model time 0.3477 (0.3523) loss 3.1839 (2.9480) grad_norm 1.6396 (2.0890) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][540/625] eta 0:00:30 lr 0.000542 wd 0.0500 time 0.3499 (0.3539) data time 0.0009 (0.0016) model time 0.3490 (0.3526) loss 3.4693 (2.9538) grad_norm 1.7522 (2.0905) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][550/625] eta 0:00:26 lr 0.000542 wd 0.0500 time 0.3395 (0.3538) data time 0.0007 (0.0016) model time 0.3388 (0.3525) loss 3.4519 (2.9589) grad_norm 1.2583 (2.0855) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][560/625] eta 0:00:22 lr 0.000542 wd 0.0500 time 0.3522 (0.3538) data time 0.0006 (0.0015) model time 0.3516 (0.3524) loss 2.4290 (2.9617) grad_norm 1.4109 (2.0792) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][570/625] eta 0:00:19 lr 0.000542 wd 0.0500 time 0.3499 (0.3536) data time 0.0006 (0.0015) model time 0.3492 (0.3523) loss 3.2217 (2.9626) grad_norm 2.1324 (2.0794) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][580/625] eta 0:00:15 lr 0.000542 wd 0.0500 time 0.3479 (0.3536) data time 0.0008 (0.0015) model time 0.3471 (0.3522) loss 3.2764 (2.9607) grad_norm 1.5839 (2.0729) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][590/625] eta 0:00:12 lr 0.000541 wd 0.0500 time 0.3442 (0.3536) data time 0.0007 (0.0015) model time 0.3435 (0.3523) loss 2.8444 (2.9581) grad_norm 1.4563 (2.0650) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][600/625] eta 0:00:08 lr 0.000541 wd 0.0500 time 0.3461 (0.3536) data time 0.0007 (0.0015) model time 0.3455 (0.3522) loss 2.6538 (2.9551) grad_norm 1.7552 (2.0596) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][610/625] eta 0:00:05 lr 0.000541 wd 0.0500 time 0.3458 (0.3544) data time 0.0006 (0.0015) model time 0.3452 (0.3531) loss 3.1663 (2.9547) grad_norm 1.9843 (2.0512) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [203/300][620/625] eta 0:00:01 lr 0.000541 wd 0.0500 time 0.3413 (0.3541) data time 0.0006 (0.0015) model time 0.3407 (0.3528) loss 2.7997 (2.9568) grad_norm 1.9636 (2.0435) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:29 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 203 training takes 0:03:41 [2024-09-01 08:54:29 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:54:30 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:54:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.434 (0.434) Loss 0.5166 (0.5166) Acc@1 89.062 (89.062) Acc@5 98.340 (98.340) Mem 11224MB [2024-09-01 08:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.132) Loss 0.8691 (0.6526) Acc@1 79.932 (85.622) Acc@5 95.508 (97.412) Mem 11224MB [2024-09-01 08:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.114) Loss 0.9849 (0.7778) Acc@1 76.074 (82.394) Acc@5 94.189 (96.050) Mem 11224MB [2024-09-01 08:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.048 Acc@5 96.041 [2024-09-01 08:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.0% [2024-09-01 08:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.05% [2024-09-01 08:54:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:54:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:54:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.442 (0.442) Loss 0.4922 (0.4922) Acc@1 89.551 (89.551) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 08:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.128) Loss 0.7949 (0.6042) Acc@1 81.104 (86.674) Acc@5 96.045 (97.732) Mem 11224MB [2024-09-01 08:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 0.8896 (0.7152) Acc@1 77.637 (83.696) Acc@5 95.215 (96.629) Mem 11224MB [2024-09-01 08:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.371 Acc@5 96.663 [2024-09-01 08:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 08:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.37% [2024-09-01 08:54:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:54:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:54:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][0/625] eta 0:07:02 lr 0.000541 wd 0.0500 time 0.6754 (0.6754) data time 0.3658 (0.3658) model time 0.0000 (0.0000) loss 3.1131 (3.1131) grad_norm 1.5844 (1.5844) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][10/625] eta 0:03:52 lr 0.000541 wd 0.0500 time 0.3486 (0.3786) data time 0.0007 (0.0340) model time 0.0000 (0.0000) loss 3.3441 (2.7070) grad_norm 1.4855 (1.8996) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][20/625] eta 0:03:41 lr 0.000541 wd 0.0500 time 0.3452 (0.3653) data time 0.0007 (0.0182) model time 0.0000 (0.0000) loss 1.9398 (2.7721) grad_norm 1.5189 (1.9213) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][30/625] eta 0:03:33 lr 0.000540 wd 0.0500 time 0.3508 (0.3596) data time 0.0007 (0.0126) model time 0.0000 (0.0000) loss 3.0678 (2.8265) grad_norm 1.3846 (1.8446) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][40/625] eta 0:03:31 lr 0.000540 wd 0.0500 time 0.3541 (0.3613) data time 0.0007 (0.0097) model time 0.0000 (0.0000) loss 3.4397 (2.9133) grad_norm 2.2965 (1.8412) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][50/625] eta 0:03:26 lr 0.000540 wd 0.0500 time 0.3499 (0.3585) data time 0.0006 (0.0080) model time 0.0000 (0.0000) loss 3.2960 (2.8768) grad_norm 1.6247 (1.8641) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:54:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][60/625] eta 0:03:21 lr 0.000540 wd 0.0500 time 0.3423 (0.3569) data time 0.0008 (0.0068) model time 0.3415 (0.3478) loss 3.0810 (2.8837) grad_norm 1.5045 (1.8448) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][70/625] eta 0:03:18 lr 0.000540 wd 0.0500 time 0.3550 (0.3580) data time 0.0007 (0.0060) model time 0.3544 (0.3558) loss 2.6157 (2.8916) grad_norm 2.3082 (1.8589) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][80/625] eta 0:03:14 lr 0.000540 wd 0.0500 time 0.3560 (0.3567) data time 0.0009 (0.0053) model time 0.3551 (0.3529) loss 2.7863 (2.8860) grad_norm 2.1100 (1.8770) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][90/625] eta 0:03:10 lr 0.000539 wd 0.0500 time 0.3439 (0.3558) data time 0.0008 (0.0048) model time 0.3431 (0.3514) loss 2.8942 (2.8859) grad_norm 1.8515 (1.8819) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][100/625] eta 0:03:06 lr 0.000539 wd 0.0500 time 0.3407 (0.3551) data time 0.0009 (0.0044) model time 0.3399 (0.3508) loss 3.2625 (2.8869) grad_norm 2.0343 (1.8873) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][110/625] eta 0:03:02 lr 0.000539 wd 0.0500 time 0.3460 (0.3546) data time 0.0007 (0.0041) model time 0.3453 (0.3505) loss 3.7126 (2.9102) grad_norm 1.9105 (1.9480) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][120/625] eta 0:02:58 lr 0.000539 wd 0.0500 time 0.3403 (0.3541) data time 0.0007 (0.0038) model time 0.3396 (0.3500) loss 3.2022 (2.9011) grad_norm 1.8662 (1.9342) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][130/625] eta 0:02:55 lr 0.000539 wd 0.0500 time 0.3443 (0.3536) data time 0.0007 (0.0036) model time 0.3436 (0.3496) loss 2.9403 (2.8915) grad_norm 2.0770 (1.9326) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][140/625] eta 0:02:51 lr 0.000539 wd 0.0500 time 0.3437 (0.3531) data time 0.0008 (0.0034) model time 0.3429 (0.3492) loss 3.0555 (2.8835) grad_norm 1.3684 (1.9759) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][150/625] eta 0:02:47 lr 0.000539 wd 0.0500 time 0.3492 (0.3527) data time 0.0006 (0.0033) model time 0.3485 (0.3489) loss 3.4364 (2.9099) grad_norm 2.2263 (1.9703) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][160/625] eta 0:02:43 lr 0.000538 wd 0.0500 time 0.3453 (0.3525) data time 0.0008 (0.0031) model time 0.3445 (0.3489) loss 3.1525 (2.9298) grad_norm 1.5971 (1.9696) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][170/625] eta 0:02:42 lr 0.000538 wd 0.0500 time 0.5417 (0.3564) data time 0.0006 (0.0030) model time 0.5411 (0.3546) loss 2.2338 (2.8983) grad_norm 1.5844 (1.9858) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][180/625] eta 0:02:38 lr 0.000538 wd 0.0500 time 0.3452 (0.3559) data time 0.0008 (0.0029) model time 0.3444 (0.3541) loss 2.0812 (2.9138) grad_norm 2.1498 (1.9987) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][190/625] eta 0:02:34 lr 0.000538 wd 0.0500 time 0.3497 (0.3555) data time 0.0009 (0.0027) model time 0.3488 (0.3535) loss 3.0590 (2.9179) grad_norm 3.2369 (2.0035) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][200/625] eta 0:02:30 lr 0.000538 wd 0.0500 time 0.3507 (0.3551) data time 0.0008 (0.0026) model time 0.3499 (0.3531) loss 1.9660 (2.9004) grad_norm 2.6377 (2.0140) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][210/625] eta 0:02:27 lr 0.000538 wd 0.0500 time 0.3470 (0.3547) data time 0.0007 (0.0026) model time 0.3463 (0.3526) loss 3.4406 (2.9058) grad_norm 2.1471 (2.0121) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][220/625] eta 0:02:23 lr 0.000537 wd 0.0500 time 0.3461 (0.3543) data time 0.0009 (0.0025) model time 0.3452 (0.3522) loss 3.5531 (2.9238) grad_norm 2.3441 (2.0144) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:55:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][230/625] eta 0:02:19 lr 0.000537 wd 0.0500 time 0.3462 (0.3541) data time 0.0008 (0.0024) model time 0.3454 (0.3520) loss 3.2878 (2.9185) grad_norm 1.6529 (2.0116) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][240/625] eta 0:02:16 lr 0.000537 wd 0.0500 time 0.3464 (0.3539) data time 0.0006 (0.0023) model time 0.3457 (0.3518) loss 2.9678 (2.9312) grad_norm 2.2636 (2.1559) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][250/625] eta 0:02:12 lr 0.000537 wd 0.0500 time 0.3471 (0.3543) data time 0.0008 (0.0023) model time 0.3463 (0.3524) loss 3.3336 (2.9382) grad_norm 1.4653 (2.1552) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][260/625] eta 0:02:09 lr 0.000537 wd 0.0500 time 0.3470 (0.3540) data time 0.0006 (0.0022) model time 0.3464 (0.3521) loss 2.2489 (2.9266) grad_norm 1.4727 (2.1312) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][270/625] eta 0:02:05 lr 0.000537 wd 0.0500 time 0.3524 (0.3538) data time 0.0006 (0.0022) model time 0.3518 (0.3518) loss 3.4955 (2.9361) grad_norm 2.0306 (2.1231) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][280/625] eta 0:02:02 lr 0.000536 wd 0.0500 time 0.3496 (0.3543) data time 0.0006 (0.0021) model time 0.3490 (0.3525) loss 2.0988 (2.9322) grad_norm 2.3122 (2.1108) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][290/625] eta 0:01:59 lr 0.000536 wd 0.0500 time 0.3494 (0.3560) data time 0.0006 (0.0021) model time 0.3488 (0.3546) loss 3.4985 (2.9336) grad_norm 1.5385 (2.0978) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][300/625] eta 0:01:55 lr 0.000536 wd 0.0500 time 0.3468 (0.3557) data time 0.0007 (0.0021) model time 0.3461 (0.3543) loss 2.0274 (2.9346) grad_norm 2.1515 (2.0871) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][310/625] eta 0:01:51 lr 0.000536 wd 0.0500 time 0.3510 (0.3554) data time 0.0006 (0.0020) model time 0.3503 (0.3540) loss 2.9135 (2.9291) grad_norm 2.7333 (2.0752) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][320/625] eta 0:01:48 lr 0.000536 wd 0.0500 time 0.3467 (0.3551) data time 0.0006 (0.0020) model time 0.3462 (0.3537) loss 3.2930 (2.9339) grad_norm 2.4409 (2.0769) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][330/625] eta 0:01:44 lr 0.000536 wd 0.0500 time 0.3488 (0.3549) data time 0.0008 (0.0019) model time 0.3481 (0.3534) loss 3.0539 (2.9423) grad_norm 1.8373 (2.0707) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][340/625] eta 0:01:41 lr 0.000536 wd 0.0500 time 0.3465 (0.3547) data time 0.0006 (0.0019) model time 0.3459 (0.3531) loss 2.8236 (2.9424) grad_norm 2.3429 (2.0769) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][350/625] eta 0:01:37 lr 0.000535 wd 0.0500 time 0.3521 (0.3545) data time 0.0009 (0.0019) model time 0.3512 (0.3529) loss 2.5931 (2.9399) grad_norm 1.8445 (2.0834) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][360/625] eta 0:01:33 lr 0.000535 wd 0.0500 time 0.3428 (0.3543) data time 0.0007 (0.0019) model time 0.3422 (0.3527) loss 2.0294 (2.9379) grad_norm 1.7283 (2.0729) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][370/625] eta 0:01:30 lr 0.000535 wd 0.0500 time 0.3437 (0.3541) data time 0.0008 (0.0018) model time 0.3429 (0.3526) loss 2.8438 (2.9358) grad_norm 1.4780 (2.0689) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][380/625] eta 0:01:26 lr 0.000535 wd 0.0500 time 0.3525 (0.3541) data time 0.0008 (0.0018) model time 0.3518 (0.3525) loss 2.4712 (2.9403) grad_norm 1.5168 (2.0550) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][390/625] eta 0:01:23 lr 0.000535 wd 0.0500 time 0.3509 (0.3539) data time 0.0006 (0.0018) model time 0.3503 (0.3524) loss 2.0391 (2.9365) grad_norm 1.4121 (2.0477) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][400/625] eta 0:01:19 lr 0.000535 wd 0.0500 time 0.3440 (0.3537) data time 0.0008 (0.0018) model time 0.3431 (0.3522) loss 3.1634 (2.9345) grad_norm 3.0215 (2.0385) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][410/625] eta 0:01:16 lr 0.000534 wd 0.0500 time 0.3444 (0.3536) data time 0.0007 (0.0017) model time 0.3437 (0.3521) loss 2.7261 (2.9316) grad_norm 3.8794 (2.0554) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][420/625] eta 0:01:12 lr 0.000534 wd 0.0500 time 0.3534 (0.3535) data time 0.0006 (0.0017) model time 0.3528 (0.3520) loss 3.7278 (2.9299) grad_norm 2.7256 (2.0622) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][430/625] eta 0:01:08 lr 0.000534 wd 0.0500 time 0.3535 (0.3534) data time 0.0008 (0.0017) model time 0.3526 (0.3519) loss 2.8077 (2.9316) grad_norm 1.7570 (2.0586) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][440/625] eta 0:01:05 lr 0.000534 wd 0.0500 time 0.3491 (0.3537) data time 0.0006 (0.0017) model time 0.3485 (0.3522) loss 3.4609 (2.9340) grad_norm 1.3456 (2.0474) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][450/625] eta 0:01:01 lr 0.000534 wd 0.0500 time 0.3554 (0.3536) data time 0.0007 (0.0017) model time 0.3547 (0.3521) loss 1.9251 (2.9278) grad_norm 1.9159 (2.0427) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][460/625] eta 0:00:58 lr 0.000534 wd 0.0500 time 0.3504 (0.3535) data time 0.0008 (0.0017) model time 0.3496 (0.3520) loss 3.2234 (2.9272) grad_norm 1.6093 (2.0400) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][470/625] eta 0:00:54 lr 0.000534 wd 0.0500 time 0.3555 (0.3539) data time 0.0006 (0.0016) model time 0.3549 (0.3524) loss 3.5277 (2.9269) grad_norm 2.2440 (2.0390) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][480/625] eta 0:00:51 lr 0.000533 wd 0.0500 time 0.3450 (0.3538) data time 0.0008 (0.0016) model time 0.3442 (0.3523) loss 3.2844 (2.9259) grad_norm 1.5967 (2.0292) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][490/625] eta 0:00:47 lr 0.000533 wd 0.0500 time 0.3469 (0.3536) data time 0.0009 (0.0016) model time 0.3459 (0.3522) loss 2.8814 (2.9240) grad_norm 1.6861 (2.0316) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][500/625] eta 0:00:44 lr 0.000533 wd 0.0500 time 0.5581 (0.3548) data time 0.0006 (0.0016) model time 0.5575 (0.3535) loss 2.4546 (2.9260) grad_norm 1.4936 (2.0295) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][510/625] eta 0:00:40 lr 0.000533 wd 0.0500 time 0.3422 (0.3546) data time 0.0008 (0.0016) model time 0.3414 (0.3534) loss 2.8687 (2.9262) grad_norm 1.9937 (2.0320) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][520/625] eta 0:00:37 lr 0.000533 wd 0.0500 time 0.3520 (0.3545) data time 0.0008 (0.0016) model time 0.3512 (0.3532) loss 3.1170 (2.9266) grad_norm 1.5442 (2.0262) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][530/625] eta 0:00:33 lr 0.000533 wd 0.0500 time 0.3474 (0.3544) data time 0.0006 (0.0015) model time 0.3468 (0.3531) loss 1.9510 (2.9256) grad_norm 1.5479 (2.0204) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][540/625] eta 0:00:30 lr 0.000532 wd 0.0500 time 0.3470 (0.3543) data time 0.0008 (0.0015) model time 0.3461 (0.3530) loss 2.9334 (2.9292) grad_norm 2.7936 (2.0222) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][550/625] eta 0:00:26 lr 0.000532 wd 0.0500 time 0.3113 (0.3545) data time 0.0007 (0.0015) model time 0.3106 (0.3532) loss 2.6427 (2.9263) grad_norm 1.9394 (2.0289) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][560/625] eta 0:00:23 lr 0.000532 wd 0.0500 time 0.3431 (0.3544) data time 0.0008 (0.0015) model time 0.3423 (0.3531) loss 2.5744 (2.9278) grad_norm 1.6709 (2.0243) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:57:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][570/625] eta 0:00:19 lr 0.000532 wd 0.0500 time 0.3461 (0.3542) data time 0.0007 (0.0015) model time 0.3454 (0.3530) loss 3.2725 (2.9283) grad_norm 1.9452 (2.0163) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][580/625] eta 0:00:15 lr 0.000532 wd 0.0500 time 0.5285 (0.3544) data time 0.0008 (0.0015) model time 0.5278 (0.3532) loss 2.8269 (2.9296) grad_norm 1.2028 (2.0110) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][590/625] eta 0:00:12 lr 0.000532 wd 0.0500 time 0.3466 (0.3544) data time 0.0007 (0.0015) model time 0.3460 (0.3531) loss 2.1193 (2.9294) grad_norm 1.3384 (2.0108) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][600/625] eta 0:00:08 lr 0.000532 wd 0.0500 time 0.3609 (0.3543) data time 0.0006 (0.0015) model time 0.3604 (0.3530) loss 3.1128 (2.9315) grad_norm 2.0693 (2.0065) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][610/625] eta 0:00:05 lr 0.000531 wd 0.0500 time 0.3465 (0.3542) data time 0.0004 (0.0015) model time 0.3461 (0.3529) loss 3.5470 (2.9345) grad_norm 3.7062 (2.0043) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [204/300][620/625] eta 0:00:01 lr 0.000531 wd 0.0500 time 0.3428 (0.3540) data time 0.0006 (0.0015) model time 0.3422 (0.3528) loss 2.9480 (2.9336) grad_norm 1.8204 (2.0027) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:18 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 204 training takes 0:03:41 [2024-09-01 08:58:18 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 08:58:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 08:58:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.438 (0.438) Loss 0.5513 (0.5513) Acc@1 88.135 (88.135) Acc@5 98.291 (98.291) Mem 11224MB [2024-09-01 08:58:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.9038 (0.6591) Acc@1 79.150 (85.587) Acc@5 95.361 (97.412) Mem 11224MB [2024-09-01 08:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 0.9785 (0.7850) Acc@1 75.635 (82.327) Acc@5 94.482 (96.087) Mem 11224MB [2024-09-01 08:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.068 Acc@5 96.093 [2024-09-01 08:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-09-01 08:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.07% [2024-09-01 08:58:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 08:58:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 08:58:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.450 (0.450) Loss 0.4907 (0.4907) Acc@1 89.453 (89.453) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 08:58:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.131) Loss 0.7939 (0.6032) Acc@1 81.201 (86.657) Acc@5 96.094 (97.772) Mem 11224MB [2024-09-01 08:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.8887 (0.7144) Acc@1 77.686 (83.680) Acc@5 95.166 (96.663) Mem 11224MB [2024-09-01 08:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.381 Acc@5 96.689 [2024-09-01 08:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 08:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.38% [2024-09-01 08:58:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 08:58:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 08:58:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][0/625] eta 0:07:29 lr 0.000531 wd 0.0500 time 0.7186 (0.7186) data time 0.4067 (0.4067) model time 0.0000 (0.0000) loss 3.9083 (3.9083) grad_norm 2.2916 (2.2916) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][10/625] eta 0:03:54 lr 0.000531 wd 0.0500 time 0.3459 (0.3808) data time 0.0009 (0.0377) model time 0.0000 (0.0000) loss 3.0297 (3.0808) grad_norm 3.1132 (2.0245) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][20/625] eta 0:03:40 lr 0.000531 wd 0.0500 time 0.3450 (0.3647) data time 0.0006 (0.0202) model time 0.0000 (0.0000) loss 3.5166 (2.9993) grad_norm 1.4100 (2.6783) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][30/625] eta 0:03:33 lr 0.000531 wd 0.0500 time 0.3435 (0.3591) data time 0.0008 (0.0139) model time 0.0000 (0.0000) loss 3.5966 (3.0031) grad_norm 1.7500 (2.3350) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][40/625] eta 0:03:28 lr 0.000530 wd 0.0500 time 0.3465 (0.3560) data time 0.0006 (0.0107) model time 0.0000 (0.0000) loss 3.1119 (3.0055) grad_norm 1.8399 (2.2379) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][50/625] eta 0:03:24 lr 0.000530 wd 0.0500 time 0.3449 (0.3564) data time 0.0006 (0.0087) model time 0.0000 (0.0000) loss 3.6656 (2.9953) grad_norm 2.0279 (2.1531) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][60/625] eta 0:03:20 lr 0.000530 wd 0.0500 time 0.3555 (0.3550) data time 0.0006 (0.0074) model time 0.3549 (0.3472) loss 3.1250 (3.0270) grad_norm 1.5975 (2.0759) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][70/625] eta 0:03:21 lr 0.000530 wd 0.0500 time 0.3460 (0.3623) data time 0.0006 (0.0065) model time 0.3453 (0.3765) loss 3.6640 (3.0201) grad_norm 2.0616 (2.0279) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:58:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][80/625] eta 0:03:16 lr 0.000530 wd 0.0500 time 0.3446 (0.3605) data time 0.0006 (0.0058) model time 0.3440 (0.3666) loss 3.5295 (3.0198) grad_norm 2.4640 (2.3893) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][90/625] eta 0:03:12 lr 0.000530 wd 0.0500 time 0.3471 (0.3590) data time 0.0007 (0.0053) model time 0.3464 (0.3616) loss 3.5288 (3.0061) grad_norm 1.7203 (2.4162) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][100/625] eta 0:03:08 lr 0.000530 wd 0.0500 time 0.3444 (0.3595) data time 0.0006 (0.0048) model time 0.3439 (0.3619) loss 3.3955 (3.0119) grad_norm 1.4240 (2.3526) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][110/625] eta 0:03:04 lr 0.000529 wd 0.0500 time 0.3423 (0.3586) data time 0.0008 (0.0045) model time 0.3415 (0.3597) loss 3.2681 (3.0069) grad_norm 2.6258 (2.3038) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][120/625] eta 0:03:00 lr 0.000529 wd 0.0500 time 0.3561 (0.3578) data time 0.0008 (0.0041) model time 0.3554 (0.3579) loss 3.2288 (3.0074) grad_norm 1.6888 (2.2718) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][130/625] eta 0:02:56 lr 0.000529 wd 0.0500 time 0.3486 (0.3574) data time 0.0008 (0.0039) model time 0.3478 (0.3573) loss 1.8732 (2.9941) grad_norm 1.9462 (2.2536) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][140/625] eta 0:02:53 lr 0.000529 wd 0.0500 time 0.3412 (0.3567) data time 0.0006 (0.0037) model time 0.3405 (0.3561) loss 3.3387 (3.0058) grad_norm 1.6633 (2.2039) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][150/625] eta 0:02:49 lr 0.000529 wd 0.0500 time 0.3496 (0.3562) data time 0.0006 (0.0035) model time 0.3489 (0.3553) loss 2.3499 (2.9957) grad_norm 2.0158 (2.1727) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][160/625] eta 0:02:45 lr 0.000529 wd 0.0500 time 0.3451 (0.3557) data time 0.0009 (0.0033) model time 0.3442 (0.3546) loss 1.8792 (2.9956) grad_norm 1.5694 (2.1469) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][170/625] eta 0:02:41 lr 0.000528 wd 0.0500 time 0.3531 (0.3554) data time 0.0008 (0.0032) model time 0.3523 (0.3541) loss 2.6484 (2.9999) grad_norm 1.7385 (2.1179) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][180/625] eta 0:02:37 lr 0.000528 wd 0.0500 time 0.3467 (0.3550) data time 0.0009 (0.0030) model time 0.3458 (0.3537) loss 3.1382 (3.0073) grad_norm 2.6164 (2.1024) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][190/625] eta 0:02:34 lr 0.000528 wd 0.0500 time 0.3467 (0.3546) data time 0.0009 (0.0029) model time 0.3458 (0.3532) loss 3.0324 (3.0077) grad_norm 2.0095 (2.0855) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][200/625] eta 0:02:30 lr 0.000528 wd 0.0500 time 0.3490 (0.3543) data time 0.0009 (0.0028) model time 0.3481 (0.3527) loss 2.9448 (2.9999) grad_norm 2.1688 (2.0859) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][210/625] eta 0:02:26 lr 0.000528 wd 0.0500 time 0.3442 (0.3540) data time 0.0008 (0.0027) model time 0.3434 (0.3524) loss 2.2275 (2.9855) grad_norm 1.9801 (2.0760) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][220/625] eta 0:02:23 lr 0.000528 wd 0.0500 time 0.3517 (0.3538) data time 0.0009 (0.0026) model time 0.3509 (0.3521) loss 3.3507 (2.9797) grad_norm 1.9298 (2.0683) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][230/625] eta 0:02:19 lr 0.000528 wd 0.0500 time 0.3483 (0.3539) data time 0.0006 (0.0026) model time 0.3477 (0.3524) loss 3.6015 (2.9794) grad_norm 2.3163 (2.0641) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][240/625] eta 0:02:16 lr 0.000527 wd 0.0500 time 0.3557 (0.3537) data time 0.0006 (0.0025) model time 0.3551 (0.3521) loss 2.6274 (2.9728) grad_norm 1.4413 (2.0602) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 08:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][250/625] eta 0:02:13 lr 0.000527 wd 0.0500 time 0.5176 (0.3549) data time 0.0008 (0.0024) model time 0.5168 (0.3537) loss 2.1542 (2.9730) grad_norm 1.6777 (2.0501) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][260/625] eta 0:02:09 lr 0.000527 wd 0.0500 time 0.3496 (0.3555) data time 0.0008 (0.0024) model time 0.3488 (0.3545) loss 2.5493 (2.9729) grad_norm 1.9922 (2.0473) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][270/625] eta 0:02:06 lr 0.000527 wd 0.0500 time 0.3513 (0.3552) data time 0.0008 (0.0023) model time 0.3505 (0.3541) loss 2.9669 (2.9702) grad_norm 2.2571 (2.0393) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][280/625] eta 0:02:02 lr 0.000527 wd 0.0500 time 0.3525 (0.3550) data time 0.0008 (0.0023) model time 0.3517 (0.3539) loss 3.1790 (2.9616) grad_norm 1.6449 (2.0447) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][290/625] eta 0:01:58 lr 0.000527 wd 0.0500 time 0.3455 (0.3549) data time 0.0006 (0.0022) model time 0.3449 (0.3537) loss 3.6561 (2.9612) grad_norm 2.2808 (2.0553) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][300/625] eta 0:01:55 lr 0.000526 wd 0.0500 time 0.3423 (0.3546) data time 0.0007 (0.0022) model time 0.3416 (0.3534) loss 3.1598 (2.9586) grad_norm 1.8584 (2.0500) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][310/625] eta 0:01:51 lr 0.000526 wd 0.0500 time 0.3452 (0.3550) data time 0.0009 (0.0021) model time 0.3443 (0.3539) loss 1.8011 (2.9643) grad_norm 1.9249 (2.0441) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][320/625] eta 0:01:48 lr 0.000526 wd 0.0500 time 0.3483 (0.3548) data time 0.0008 (0.0021) model time 0.3475 (0.3537) loss 3.0782 (2.9714) grad_norm 2.6400 (2.0393) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][330/625] eta 0:01:44 lr 0.000526 wd 0.0500 time 0.3448 (0.3546) data time 0.0008 (0.0020) model time 0.3440 (0.3534) loss 2.4695 (2.9631) grad_norm 2.3021 (2.0400) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][340/625] eta 0:01:40 lr 0.000526 wd 0.0500 time 0.3527 (0.3544) data time 0.0009 (0.0020) model time 0.3518 (0.3532) loss 2.8557 (2.9530) grad_norm 1.4399 (2.0392) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][350/625] eta 0:01:37 lr 0.000526 wd 0.0500 time 0.3414 (0.3546) data time 0.0006 (0.0020) model time 0.3407 (0.3535) loss 3.2456 (2.9531) grad_norm 2.1637 (2.0333) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][360/625] eta 0:01:33 lr 0.000526 wd 0.0500 time 0.3559 (0.3545) data time 0.0008 (0.0019) model time 0.3551 (0.3533) loss 3.0499 (2.9484) grad_norm 1.8007 (2.0255) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][370/625] eta 0:01:30 lr 0.000525 wd 0.0500 time 0.5555 (0.3553) data time 0.0007 (0.0019) model time 0.5548 (0.3544) loss 3.0153 (2.9425) grad_norm 1.5155 (2.0203) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][380/625] eta 0:01:27 lr 0.000525 wd 0.0500 time 0.3470 (0.3557) data time 0.0007 (0.0019) model time 0.3462 (0.3547) loss 3.3524 (2.9410) grad_norm 1.8508 (2.0227) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][390/625] eta 0:01:23 lr 0.000525 wd 0.0500 time 0.3526 (0.3554) data time 0.0009 (0.0019) model time 0.3517 (0.3545) loss 2.5300 (2.9334) grad_norm 1.7809 (2.0452) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][400/625] eta 0:01:19 lr 0.000525 wd 0.0500 time 0.3496 (0.3552) data time 0.0007 (0.0018) model time 0.3489 (0.3542) loss 3.4063 (2.9342) grad_norm 2.4480 (2.0896) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][410/625] eta 0:01:16 lr 0.000525 wd 0.0500 time 0.3582 (0.3551) data time 0.0008 (0.0018) model time 0.3574 (0.3540) loss 2.8141 (2.9349) grad_norm 2.4452 (2.1089) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][420/625] eta 0:01:12 lr 0.000525 wd 0.0500 time 0.3496 (0.3549) data time 0.0007 (0.0018) model time 0.3489 (0.3539) loss 3.3753 (2.9378) grad_norm 1.7177 (2.1249) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][430/625] eta 0:01:09 lr 0.000524 wd 0.0500 time 0.3546 (0.3547) data time 0.0009 (0.0018) model time 0.3538 (0.3537) loss 3.1409 (2.9428) grad_norm 2.0758 (2.1255) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][440/625] eta 0:01:05 lr 0.000524 wd 0.0500 time 0.3476 (0.3546) data time 0.0007 (0.0017) model time 0.3470 (0.3535) loss 1.6512 (2.9397) grad_norm 2.0624 (2.1290) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][450/625] eta 0:01:02 lr 0.000524 wd 0.0500 time 0.3486 (0.3544) data time 0.0009 (0.0017) model time 0.3477 (0.3534) loss 3.3341 (2.9438) grad_norm 2.4920 (2.1315) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][460/625] eta 0:00:58 lr 0.000524 wd 0.0500 time 0.3462 (0.3543) data time 0.0008 (0.0017) model time 0.3454 (0.3532) loss 3.3913 (2.9447) grad_norm 1.7688 (2.1267) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][470/625] eta 0:00:54 lr 0.000524 wd 0.0500 time 0.3505 (0.3541) data time 0.0007 (0.0017) model time 0.3498 (0.3530) loss 3.6589 (2.9468) grad_norm 1.7335 (2.1243) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][480/625] eta 0:00:51 lr 0.000524 wd 0.0500 time 0.3473 (0.3540) data time 0.0008 (0.0017) model time 0.3465 (0.3529) loss 3.1520 (2.9446) grad_norm 16.1397 (2.1642) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][490/625] eta 0:00:47 lr 0.000524 wd 0.0500 time 0.3482 (0.3543) data time 0.0008 (0.0017) model time 0.3474 (0.3533) loss 2.4311 (2.9433) grad_norm 1.5181 (2.1604) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][500/625] eta 0:00:44 lr 0.000523 wd 0.0500 time 0.3448 (0.3542) data time 0.0008 (0.0016) model time 0.3440 (0.3531) loss 2.5322 (2.9439) grad_norm 1.2519 (2.1493) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][510/625] eta 0:00:40 lr 0.000523 wd 0.0500 time 0.3488 (0.3541) data time 0.0009 (0.0016) model time 0.3480 (0.3530) loss 3.0563 (2.9459) grad_norm 1.8210 (2.1525) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][520/625] eta 0:00:37 lr 0.000523 wd 0.0500 time 0.3602 (0.3540) data time 0.0006 (0.0016) model time 0.3596 (0.3529) loss 3.3176 (2.9465) grad_norm 1.5850 (2.1521) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 09:01:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][530/625] eta 0:00:33 lr 0.000523 wd 0.0500 time 0.3476 (0.3539) data time 0.0008 (0.0016) model time 0.3468 (0.3528) loss 3.3793 (2.9486) grad_norm 2.2616 (2.1489) loss_scale 1024.0000 (517.7853) mem 11224MB [2024-09-01 09:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][540/625] eta 0:00:30 lr 0.000523 wd 0.0500 time 0.3526 (0.3537) data time 0.0009 (0.0016) model time 0.3518 (0.3526) loss 2.6807 (2.9464) grad_norm 1.7791 (2.1416) loss_scale 1024.0000 (527.1423) mem 11224MB [2024-09-01 09:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][550/625] eta 0:00:26 lr 0.000523 wd 0.0500 time 0.3478 (0.3537) data time 0.0009 (0.0016) model time 0.3469 (0.3525) loss 3.1938 (2.9472) grad_norm 2.4507 (2.1518) loss_scale 1024.0000 (536.1597) mem 11224MB [2024-09-01 09:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][560/625] eta 0:00:22 lr 0.000522 wd 0.0500 time 0.3453 (0.3538) data time 0.0006 (0.0016) model time 0.3447 (0.3527) loss 2.3720 (2.9471) grad_norm 1.5127 (2.1515) loss_scale 1024.0000 (544.8556) mem 11224MB [2024-09-01 09:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][570/625] eta 0:00:19 lr 0.000522 wd 0.0500 time 0.3488 (0.3537) data time 0.0006 (0.0015) model time 0.3482 (0.3526) loss 2.4799 (2.9444) grad_norm 2.2619 (2.1609) loss_scale 1024.0000 (553.2469) mem 11224MB [2024-09-01 09:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][580/625] eta 0:00:15 lr 0.000522 wd 0.0500 time 0.3511 (0.3543) data time 0.0006 (0.0015) model time 0.3505 (0.3533) loss 3.5326 (2.9475) grad_norm 2.6742 (2.1643) loss_scale 1024.0000 (561.3494) mem 11224MB [2024-09-01 09:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][590/625] eta 0:00:12 lr 0.000522 wd 0.0500 time 0.3466 (0.3546) data time 0.0006 (0.0015) model time 0.3460 (0.3536) loss 3.6414 (2.9484) grad_norm 2.5240 (2.1669) loss_scale 1024.0000 (569.1777) mem 11224MB [2024-09-01 09:02:00 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 09:02:00 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:02:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:03:47 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 09:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 09:04:02 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 09:04:16 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 09:04:16 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 09:04:18 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 09:04:20 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 09:04:20 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 205) [2024-09-01 09:04:20 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 09:04:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][600/625] eta 0:03:04 lr 0.000522 wd 0.0500 time 0.5265 (7.3806) data time 0.0011 (0.3739) model time 0.5254 (7.0067) loss 3.5835 (3.8784) grad_norm 1.5083 (1.5608) loss_scale 1024.0000 (1024.0000) mem 11217MB [2024-09-01 09:04:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][610/625] eta 0:00:23 lr 0.000522 wd 0.0500 time 0.3587 (1.5358) data time 0.0006 (0.0636) model time 0.3581 (1.4722) loss 2.6032 (3.3163) grad_norm 1.8765 (1.6889) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 09:04:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [205/300][620/625] eta 0:00:05 lr 0.000522 wd 0.0500 time 0.3604 (1.0001) data time 0.0008 (0.0351) model time 0.3596 (0.9650) loss 3.4986 (3.2793) grad_norm 2.5447 (1.7960) loss_scale 1024.0000 (1024.0000) mem 11219MB [2024-09-01 09:04:49 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 205 training takes 0:00:23 [2024-09-01 09:04:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:04:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.493 (0.493) Loss 0.5264 (0.5264) Acc@1 88.428 (88.428) Acc@5 98.438 (98.438) Mem 11219MB [2024-09-01 09:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 0.9268 (0.6628) Acc@1 78.125 (85.440) Acc@5 95.117 (97.452) Mem 11219MB [2024-09-01 09:04:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9722 (0.7890) Acc@1 76.123 (82.292) Acc@5 94.531 (96.029) Mem 11219MB [2024-09-01 09:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.048 Acc@5 96.057 [2024-09-01 09:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.0% [2024-09-01 09:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.822 (0.822) Loss 0.4902 (0.4902) Acc@1 89.453 (89.453) Acc@5 98.535 (98.535) Mem 11219MB [2024-09-01 09:04:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.175) Loss 0.7939 (0.6025) Acc@1 81.152 (86.634) Acc@5 96.094 (97.767) Mem 11219MB [2024-09-01 09:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.8887 (0.7138) Acc@1 77.930 (83.670) Acc@5 95.068 (96.659) Mem 11219MB [2024-09-01 09:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.373 Acc@5 96.687 [2024-09-01 09:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 09:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.37% [2024-09-01 09:04:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:05:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:05:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][0/625] eta 0:09:54 lr 0.000521 wd 0.0500 time 0.9519 (0.9519) data time 0.5209 (0.5209) model time 0.0000 (0.0000) loss 3.6154 (3.6154) grad_norm 2.8036 (2.8036) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][10/625] eta 0:04:36 lr 0.000521 wd 0.0500 time 0.3716 (0.4500) data time 0.0011 (0.0485) model time 0.0000 (0.0000) loss 2.6322 (3.0395) grad_norm 2.1360 (2.3743) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][20/625] eta 0:04:09 lr 0.000521 wd 0.0500 time 0.3647 (0.4128) data time 0.0010 (0.0260) model time 0.0000 (0.0000) loss 3.1467 (3.0159) grad_norm 1.6743 (2.1552) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][30/625] eta 0:03:58 lr 0.000521 wd 0.0500 time 0.3688 (0.4001) data time 0.0012 (0.0180) model time 0.0000 (0.0000) loss 2.8768 (3.0162) grad_norm 1.4097 (2.1024) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][40/625] eta 0:03:49 lr 0.000521 wd 0.0500 time 0.3719 (0.3931) data time 0.0012 (0.0139) model time 0.0000 (0.0000) loss 3.4042 (2.9785) grad_norm 1.6206 (2.0439) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][50/625] eta 0:03:43 lr 0.000521 wd 0.0500 time 0.3727 (0.3887) data time 0.0012 (0.0114) model time 0.0000 (0.0000) loss 2.9837 (2.9449) grad_norm 1.9139 (2.0054) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][60/625] eta 0:03:38 lr 0.000520 wd 0.0500 time 0.3667 (0.3859) data time 0.0011 (0.0097) model time 0.3655 (0.3707) loss 2.9613 (2.9268) grad_norm 2.1643 (1.9890) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][70/625] eta 0:03:32 lr 0.000520 wd 0.0500 time 0.3664 (0.3836) data time 0.0012 (0.0085) model time 0.3652 (0.3695) loss 3.1921 (2.9459) grad_norm 1.3825 (1.9540) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][80/625] eta 0:03:28 lr 0.000520 wd 0.0500 time 0.3730 (0.3820) data time 0.0009 (0.0076) model time 0.3721 (0.3696) loss 3.3519 (2.9800) grad_norm 1.8228 (1.9533) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][90/625] eta 0:03:24 lr 0.000520 wd 0.0500 time 0.3659 (0.3830) data time 0.0013 (0.0069) model time 0.3646 (0.3747) loss 3.4194 (2.9704) grad_norm 1.6546 (1.9205) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][100/625] eta 0:03:20 lr 0.000520 wd 0.0500 time 0.3712 (0.3819) data time 0.0012 (0.0063) model time 0.3699 (0.3738) loss 3.3061 (2.9687) grad_norm 2.0011 (1.9007) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][110/625] eta 0:03:16 lr 0.000520 wd 0.0500 time 0.3718 (0.3808) data time 0.0009 (0.0059) model time 0.3708 (0.3730) loss 2.3027 (2.9666) grad_norm 3.2123 (1.9123) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][120/625] eta 0:03:11 lr 0.000520 wd 0.0500 time 0.3692 (0.3798) data time 0.0011 (0.0055) model time 0.3681 (0.3721) loss 2.8981 (2.9634) grad_norm 2.8217 (1.9747) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][130/625] eta 0:03:07 lr 0.000519 wd 0.0500 time 0.3626 (0.3791) data time 0.0011 (0.0052) model time 0.3615 (0.3718) loss 2.5207 (2.9604) grad_norm 1.4661 (1.9828) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][140/625] eta 0:03:03 lr 0.000519 wd 0.0500 time 0.3731 (0.3787) data time 0.0012 (0.0049) model time 0.3719 (0.3718) loss 3.0022 (2.9741) grad_norm 1.6250 (1.9618) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][150/625] eta 0:02:59 lr 0.000519 wd 0.0500 time 0.3720 (0.3782) data time 0.0011 (0.0047) model time 0.3709 (0.3717) loss 3.0312 (2.9604) grad_norm 1.1338 (1.9645) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][160/625] eta 0:02:55 lr 0.000519 wd 0.0500 time 0.3696 (0.3776) data time 0.0011 (0.0045) model time 0.3685 (0.3713) loss 2.7988 (2.9502) grad_norm 1.8723 (1.9609) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][170/625] eta 0:02:51 lr 0.000519 wd 0.0500 time 0.3703 (0.3771) data time 0.0009 (0.0043) model time 0.3694 (0.3710) loss 3.3049 (2.9451) grad_norm 1.3986 (1.9663) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][180/625] eta 0:02:47 lr 0.000519 wd 0.0500 time 0.3741 (0.3766) data time 0.0008 (0.0041) model time 0.3733 (0.3707) loss 2.9488 (2.9279) grad_norm 1.9657 (1.9906) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][190/625] eta 0:02:43 lr 0.000518 wd 0.0500 time 0.3693 (0.3763) data time 0.0010 (0.0039) model time 0.3683 (0.3705) loss 3.0677 (2.9253) grad_norm 2.2344 (1.9849) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][200/625] eta 0:02:39 lr 0.000518 wd 0.0500 time 0.3761 (0.3760) data time 0.0011 (0.0038) model time 0.3750 (0.3705) loss 2.8603 (2.9272) grad_norm 1.1503 (1.9888) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][210/625] eta 0:02:35 lr 0.000518 wd 0.0500 time 0.3709 (0.3758) data time 0.0009 (0.0037) model time 0.3700 (0.3705) loss 3.2776 (2.9200) grad_norm 1.9356 (1.9915) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][220/625] eta 0:02:32 lr 0.000518 wd 0.0500 time 0.3695 (0.3765) data time 0.0011 (0.0036) model time 0.3683 (0.3717) loss 3.5225 (2.9181) grad_norm 2.2466 (1.9823) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][230/625] eta 0:02:28 lr 0.000518 wd 0.0500 time 0.3695 (0.3763) data time 0.0008 (0.0035) model time 0.3687 (0.3716) loss 2.2749 (2.9097) grad_norm 1.6581 (1.9760) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][240/625] eta 0:02:24 lr 0.000518 wd 0.0500 time 0.3840 (0.3761) data time 0.0011 (0.0034) model time 0.3829 (0.3716) loss 2.4509 (2.9077) grad_norm 1.3878 (1.9751) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][250/625] eta 0:02:20 lr 0.000518 wd 0.0500 time 0.3709 (0.3760) data time 0.0011 (0.0033) model time 0.3698 (0.3715) loss 3.0747 (2.9138) grad_norm 1.2644 (1.9719) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][260/625] eta 0:02:17 lr 0.000517 wd 0.0500 time 0.3658 (0.3758) data time 0.0008 (0.0032) model time 0.3650 (0.3714) loss 3.4930 (2.9087) grad_norm 1.2751 (1.9717) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][270/625] eta 0:02:13 lr 0.000517 wd 0.0500 time 0.3738 (0.3757) data time 0.0014 (0.0031) model time 0.3724 (0.3714) loss 2.1650 (2.8985) grad_norm 2.1601 (1.9662) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][280/625] eta 0:02:09 lr 0.000517 wd 0.0500 time 0.3693 (0.3755) data time 0.0009 (0.0031) model time 0.3684 (0.3714) loss 3.3582 (2.8989) grad_norm 2.2947 (1.9812) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][290/625] eta 0:02:05 lr 0.000517 wd 0.0500 time 0.3742 (0.3755) data time 0.0011 (0.0030) model time 0.3731 (0.3714) loss 3.0540 (2.9069) grad_norm 1.9196 (1.9770) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][300/625] eta 0:02:02 lr 0.000517 wd 0.0500 time 0.3709 (0.3760) data time 0.0009 (0.0030) model time 0.3700 (0.3722) loss 2.1422 (2.9154) grad_norm 1.8690 (1.9753) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][310/625] eta 0:01:58 lr 0.000517 wd 0.0500 time 0.3695 (0.3759) data time 0.0011 (0.0029) model time 0.3684 (0.3721) loss 2.4663 (2.9131) grad_norm 1.6764 (1.9680) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][320/625] eta 0:01:54 lr 0.000516 wd 0.0500 time 0.3783 (0.3757) data time 0.0009 (0.0028) model time 0.3773 (0.3720) loss 2.4938 (2.9147) grad_norm 1.5735 (1.9741) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][330/625] eta 0:01:50 lr 0.000516 wd 0.0500 time 0.3660 (0.3756) data time 0.0011 (0.0028) model time 0.3649 (0.3720) loss 2.2706 (2.9148) grad_norm 2.9294 (1.9825) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][340/625] eta 0:01:47 lr 0.000516 wd 0.0500 time 0.3613 (0.3754) data time 0.0010 (0.0027) model time 0.3603 (0.3719) loss 2.3181 (2.9176) grad_norm 1.2518 (1.9949) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][350/625] eta 0:01:43 lr 0.000516 wd 0.0500 time 0.3712 (0.3753) data time 0.0011 (0.0027) model time 0.3700 (0.3718) loss 3.1369 (2.9138) grad_norm 1.3647 (1.9852) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][360/625] eta 0:01:39 lr 0.000516 wd 0.0500 time 0.3754 (0.3751) data time 0.0009 (0.0027) model time 0.3745 (0.3717) loss 3.3619 (2.9110) grad_norm 1.6473 (1.9742) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][370/625] eta 0:01:35 lr 0.000516 wd 0.0500 time 0.3756 (0.3750) data time 0.0011 (0.0026) model time 0.3745 (0.3716) loss 3.1463 (2.9115) grad_norm 1.6645 (1.9762) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][380/625] eta 0:01:31 lr 0.000516 wd 0.0500 time 0.3761 (0.3749) data time 0.0011 (0.0026) model time 0.3750 (0.3716) loss 2.7706 (2.9126) grad_norm 1.6593 (1.9692) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][390/625] eta 0:01:28 lr 0.000515 wd 0.0500 time 0.3703 (0.3749) data time 0.0009 (0.0026) model time 0.3694 (0.3716) loss 2.1324 (2.9139) grad_norm 2.8459 (2.0045) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][400/625] eta 0:01:24 lr 0.000515 wd 0.0500 time 0.3775 (0.3748) data time 0.0014 (0.0025) model time 0.3761 (0.3716) loss 3.3758 (2.9202) grad_norm 2.2270 (2.0020) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:07:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][410/625] eta 0:01:20 lr 0.000515 wd 0.0500 time 0.3723 (0.3757) data time 0.0012 (0.0025) model time 0.3711 (0.3727) loss 3.5619 (2.9297) grad_norm 1.8033 (inf) loss_scale 512.0000 (1014.0341) mem 11225MB [2024-09-01 09:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][420/625] eta 0:01:16 lr 0.000515 wd 0.0500 time 0.3666 (0.3756) data time 0.0011 (0.0025) model time 0.3655 (0.3726) loss 2.3360 (2.9303) grad_norm 1.5397 (inf) loss_scale 512.0000 (1002.1093) mem 11225MB [2024-09-01 09:07:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][430/625] eta 0:01:13 lr 0.000515 wd 0.0500 time 0.3782 (0.3755) data time 0.0012 (0.0024) model time 0.3770 (0.3725) loss 1.6604 (2.9263) grad_norm 1.5080 (inf) loss_scale 512.0000 (990.7378) mem 11225MB [2024-09-01 09:07:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][440/625] eta 0:01:09 lr 0.000515 wd 0.0500 time 0.3671 (0.3755) data time 0.0011 (0.0024) model time 0.3660 (0.3725) loss 2.6494 (2.9221) grad_norm 1.7216 (inf) loss_scale 512.0000 (979.8821) mem 11225MB [2024-09-01 09:07:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][450/625] eta 0:01:05 lr 0.000514 wd 0.0500 time 0.3753 (0.3754) data time 0.0009 (0.0024) model time 0.3745 (0.3725) loss 3.1513 (2.9195) grad_norm 1.7537 (inf) loss_scale 512.0000 (969.5078) mem 11225MB [2024-09-01 09:07:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][460/625] eta 0:01:01 lr 0.000514 wd 0.0500 time 0.3720 (0.3753) data time 0.0011 (0.0023) model time 0.3709 (0.3724) loss 2.6595 (2.9221) grad_norm 2.5275 (inf) loss_scale 512.0000 (959.5835) mem 11225MB [2024-09-01 09:07:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][470/625] eta 0:00:58 lr 0.000514 wd 0.0500 time 0.3720 (0.3753) data time 0.0009 (0.0023) model time 0.3710 (0.3725) loss 2.9862 (2.9221) grad_norm 1.6563 (inf) loss_scale 512.0000 (950.0807) mem 11225MB [2024-09-01 09:08:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][480/625] eta 0:00:54 lr 0.000514 wd 0.0500 time 0.3794 (0.3755) data time 0.0012 (0.0023) model time 0.3783 (0.3727) loss 2.6761 (2.9242) grad_norm 1.6856 (inf) loss_scale 512.0000 (940.9730) mem 11225MB [2024-09-01 09:08:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][490/625] eta 0:00:50 lr 0.000514 wd 0.0500 time 0.3741 (0.3755) data time 0.0011 (0.0023) model time 0.3730 (0.3728) loss 2.9940 (2.9273) grad_norm 2.2822 (inf) loss_scale 512.0000 (932.2363) mem 11225MB [2024-09-01 09:08:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][500/625] eta 0:00:46 lr 0.000514 wd 0.0500 time 0.3754 (0.3755) data time 0.0011 (0.0023) model time 0.3742 (0.3727) loss 3.1058 (2.9235) grad_norm 1.9738 (inf) loss_scale 512.0000 (923.8483) mem 11225MB [2024-09-01 09:08:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][510/625] eta 0:00:43 lr 0.000514 wd 0.0500 time 0.3922 (0.3754) data time 0.0009 (0.0022) model time 0.3913 (0.3728) loss 1.9249 (2.9213) grad_norm 2.4590 (inf) loss_scale 512.0000 (915.7886) mem 11225MB [2024-09-01 09:08:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][520/625] eta 0:00:39 lr 0.000513 wd 0.0500 time 0.5779 (0.3757) data time 0.0009 (0.0022) model time 0.5771 (0.3731) loss 3.6212 (2.9238) grad_norm 1.7925 (inf) loss_scale 512.0000 (908.0384) mem 11225MB [2024-09-01 09:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][530/625] eta 0:00:35 lr 0.000513 wd 0.0500 time 0.3710 (0.3761) data time 0.0009 (0.0022) model time 0.3701 (0.3736) loss 1.6440 (2.9267) grad_norm 7.6583 (inf) loss_scale 512.0000 (900.5800) mem 11225MB [2024-09-01 09:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][540/625] eta 0:00:31 lr 0.000513 wd 0.0500 time 0.3717 (0.3761) data time 0.0012 (0.0022) model time 0.3704 (0.3736) loss 3.5321 (2.9311) grad_norm 2.8249 (inf) loss_scale 512.0000 (893.3974) mem 11225MB [2024-09-01 09:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][550/625] eta 0:00:28 lr 0.000513 wd 0.0500 time 0.3653 (0.3760) data time 0.0010 (0.0022) model time 0.3643 (0.3734) loss 3.4508 (2.9320) grad_norm 1.5133 (inf) loss_scale 512.0000 (886.4755) mem 11225MB [2024-09-01 09:08:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][560/625] eta 0:00:24 lr 0.000513 wd 0.0500 time 0.3692 (0.3759) data time 0.0009 (0.0021) model time 0.3682 (0.3734) loss 3.4248 (2.9351) grad_norm 1.8540 (inf) loss_scale 512.0000 (879.8004) mem 11225MB [2024-09-01 09:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][570/625] eta 0:00:20 lr 0.000513 wd 0.0500 time 0.3623 (0.3758) data time 0.0010 (0.0021) model time 0.3613 (0.3734) loss 3.4138 (2.9364) grad_norm 1.7806 (inf) loss_scale 512.0000 (873.3590) mem 11225MB [2024-09-01 09:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][580/625] eta 0:00:16 lr 0.000512 wd 0.0500 time 0.3782 (0.3758) data time 0.0011 (0.0021) model time 0.3771 (0.3733) loss 3.1393 (2.9344) grad_norm 1.3782 (inf) loss_scale 512.0000 (867.1394) mem 11225MB [2024-09-01 09:08:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][590/625] eta 0:00:13 lr 0.000512 wd 0.0500 time 0.3664 (0.3757) data time 0.0011 (0.0021) model time 0.3653 (0.3733) loss 3.0556 (2.9364) grad_norm 2.1071 (inf) loss_scale 512.0000 (861.1303) mem 11225MB [2024-09-01 09:08:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][600/625] eta 0:00:09 lr 0.000512 wd 0.0500 time 0.3738 (0.3760) data time 0.0008 (0.0021) model time 0.3729 (0.3736) loss 2.7773 (2.9398) grad_norm 1.6359 (inf) loss_scale 512.0000 (855.3211) mem 11225MB [2024-09-01 09:08:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][610/625] eta 0:00:05 lr 0.000512 wd 0.0500 time 0.3598 (0.3759) data time 0.0006 (0.0021) model time 0.3592 (0.3735) loss 2.8667 (2.9406) grad_norm 2.2002 (inf) loss_scale 512.0000 (849.7021) mem 11225MB [2024-09-01 09:08:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [206/300][620/625] eta 0:00:01 lr 0.000512 wd 0.0500 time 0.3638 (0.3756) data time 0.0006 (0.0021) model time 0.3632 (0.3732) loss 2.8361 (2.9369) grad_norm 1.3859 (inf) loss_scale 512.0000 (844.2641) mem 11225MB [2024-09-01 09:08:57 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 206 training takes 0:03:54 [2024-09-01 09:08:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:08:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.5498 (0.5498) Acc@1 88.672 (88.672) Acc@5 98.438 (98.438) Mem 11225MB [2024-09-01 09:09:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 0.9004 (0.6674) Acc@1 78.857 (85.520) Acc@5 95.312 (97.492) Mem 11225MB [2024-09-01 09:09:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 0.9561 (0.7867) Acc@1 77.197 (82.396) Acc@5 94.775 (96.161) Mem 11225MB [2024-09-01 09:09:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.034 Acc@5 96.133 [2024-09-01 09:09:01 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.0% [2024-09-01 09:09:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.861 (0.861) Loss 0.4900 (0.4900) Acc@1 89.502 (89.502) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.177) Loss 0.7949 (0.6019) Acc@1 81.348 (86.683) Acc@5 96.045 (97.749) Mem 11225MB [2024-09-01 09:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8877 (0.7129) Acc@1 77.881 (83.731) Acc@5 95.166 (96.636) Mem 11225MB [2024-09-01 09:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.441 Acc@5 96.677 [2024-09-01 09:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 09:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.44% [2024-09-01 09:09:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:09:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][0/625] eta 0:08:00 lr 0.000512 wd 0.0500 time 0.7688 (0.7688) data time 0.4260 (0.4260) model time 0.0000 (0.0000) loss 2.2176 (2.2176) grad_norm 1.8439 (1.8439) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][10/625] eta 0:04:09 lr 0.000512 wd 0.0500 time 0.3675 (0.4058) data time 0.0012 (0.0398) model time 0.0000 (0.0000) loss 3.1413 (2.7563) grad_norm 1.7320 (1.6191) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][20/625] eta 0:03:56 lr 0.000511 wd 0.0500 time 0.3699 (0.3906) data time 0.0011 (0.0214) model time 0.0000 (0.0000) loss 2.9129 (2.9146) grad_norm 4.1410 (1.9147) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][30/625] eta 0:03:48 lr 0.000511 wd 0.0500 time 0.3676 (0.3845) data time 0.0012 (0.0149) model time 0.0000 (0.0000) loss 3.0185 (2.9497) grad_norm 1.7772 (2.0198) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][40/625] eta 0:03:43 lr 0.000511 wd 0.0500 time 0.3683 (0.3813) data time 0.0011 (0.0116) model time 0.0000 (0.0000) loss 3.0105 (2.9082) grad_norm 1.7775 (1.9494) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][50/625] eta 0:03:38 lr 0.000511 wd 0.0500 time 0.3699 (0.3796) data time 0.0009 (0.0095) model time 0.0000 (0.0000) loss 2.6172 (2.8754) grad_norm 2.2345 (1.8983) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][60/625] eta 0:03:37 lr 0.000511 wd 0.0500 time 0.5836 (0.3855) data time 0.0009 (0.0081) model time 0.5827 (0.4141) loss 2.1472 (2.8504) grad_norm 1.5633 (1.9096) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][70/625] eta 0:03:34 lr 0.000511 wd 0.0500 time 0.3326 (0.3864) data time 0.0010 (0.0072) model time 0.3316 (0.4024) loss 3.1005 (2.8469) grad_norm 2.1327 (2.0737) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][80/625] eta 0:03:29 lr 0.000511 wd 0.0500 time 0.3647 (0.3847) data time 0.0013 (0.0064) model time 0.3634 (0.3922) loss 3.1653 (2.8711) grad_norm 1.3851 (2.0750) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][90/625] eta 0:03:25 lr 0.000510 wd 0.0500 time 0.3746 (0.3833) data time 0.0012 (0.0059) model time 0.3735 (0.3868) loss 2.2765 (2.9024) grad_norm 1.8938 (2.0713) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][100/625] eta 0:03:20 lr 0.000510 wd 0.0500 time 0.3851 (0.3823) data time 0.0009 (0.0054) model time 0.3842 (0.3838) loss 3.4304 (2.9013) grad_norm 1.6379 (2.0598) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][110/625] eta 0:03:16 lr 0.000510 wd 0.0500 time 0.3693 (0.3813) data time 0.0009 (0.0050) model time 0.3684 (0.3815) loss 3.4100 (2.9078) grad_norm 1.8045 (2.0566) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][120/625] eta 0:03:12 lr 0.000510 wd 0.0500 time 0.3616 (0.3805) data time 0.0009 (0.0047) model time 0.3607 (0.3799) loss 2.9891 (2.9215) grad_norm 1.9782 (2.0426) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][130/625] eta 0:03:07 lr 0.000510 wd 0.0500 time 0.3733 (0.3798) data time 0.0009 (0.0045) model time 0.3724 (0.3787) loss 2.7376 (2.9221) grad_norm 1.9090 (2.0195) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:09:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][140/625] eta 0:03:03 lr 0.000510 wd 0.0500 time 0.3723 (0.3794) data time 0.0011 (0.0042) model time 0.3712 (0.3780) loss 3.4515 (2.9246) grad_norm 2.3610 (2.0286) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][150/625] eta 0:03:00 lr 0.000509 wd 0.0500 time 0.3692 (0.3790) data time 0.0009 (0.0040) model time 0.3683 (0.3775) loss 1.8671 (2.9255) grad_norm 1.4610 (2.0097) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][160/625] eta 0:02:56 lr 0.000509 wd 0.0500 time 0.3727 (0.3786) data time 0.0011 (0.0038) model time 0.3716 (0.3769) loss 3.3545 (2.9044) grad_norm 1.9479 (2.0128) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][170/625] eta 0:02:52 lr 0.000509 wd 0.0500 time 0.3706 (0.3784) data time 0.0009 (0.0037) model time 0.3697 (0.3767) loss 2.6039 (2.9109) grad_norm 1.3844 (2.0073) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][180/625] eta 0:02:48 lr 0.000509 wd 0.0500 time 0.3622 (0.3780) data time 0.0009 (0.0035) model time 0.3614 (0.3762) loss 2.7639 (2.8952) grad_norm 1.6828 (1.9949) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][190/625] eta 0:02:44 lr 0.000509 wd 0.0500 time 0.3752 (0.3778) data time 0.0011 (0.0034) model time 0.3741 (0.3760) loss 2.3791 (2.8962) grad_norm 1.5853 (1.9991) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][200/625] eta 0:02:40 lr 0.000509 wd 0.0500 time 0.3677 (0.3775) data time 0.0008 (0.0033) model time 0.3669 (0.3756) loss 2.6658 (2.9013) grad_norm 1.9268 (1.9976) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][210/625] eta 0:02:36 lr 0.000509 wd 0.0500 time 0.3749 (0.3773) data time 0.0013 (0.0032) model time 0.3736 (0.3753) loss 3.0404 (2.9076) grad_norm 2.6169 (2.0128) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][220/625] eta 0:02:32 lr 0.000508 wd 0.0500 time 0.3661 (0.3771) data time 0.0009 (0.0031) model time 0.3652 (0.3751) loss 2.0793 (2.9085) grad_norm 2.2422 (2.0109) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][230/625] eta 0:02:28 lr 0.000508 wd 0.0500 time 0.3705 (0.3769) data time 0.0011 (0.0030) model time 0.3695 (0.3749) loss 2.0887 (2.9026) grad_norm 2.3414 (2.0188) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][240/625] eta 0:02:25 lr 0.000508 wd 0.0500 time 0.3705 (0.3768) data time 0.0011 (0.0030) model time 0.3694 (0.3748) loss 3.1215 (2.9025) grad_norm 1.9855 (2.0829) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][250/625] eta 0:02:21 lr 0.000508 wd 0.0500 time 0.3722 (0.3766) data time 0.0008 (0.0029) model time 0.3714 (0.3747) loss 3.5632 (2.9053) grad_norm 1.6374 (2.0678) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][260/625] eta 0:02:17 lr 0.000508 wd 0.0500 time 0.3754 (0.3766) data time 0.0009 (0.0028) model time 0.3745 (0.3747) loss 1.9805 (2.9026) grad_norm 1.2858 (2.0599) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][270/625] eta 0:02:13 lr 0.000508 wd 0.0500 time 0.3685 (0.3774) data time 0.0014 (0.0028) model time 0.3670 (0.3757) loss 2.9177 (2.9089) grad_norm 1.8808 (2.0713) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][280/625] eta 0:02:10 lr 0.000508 wd 0.0500 time 0.5926 (0.3789) data time 0.0009 (0.0027) model time 0.5917 (0.3775) loss 3.6360 (2.9225) grad_norm 1.5725 (2.0814) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][290/625] eta 0:02:06 lr 0.000507 wd 0.0500 time 0.3709 (0.3785) data time 0.0011 (0.0027) model time 0.3698 (0.3771) loss 2.1728 (2.9170) grad_norm 1.4410 (2.0780) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:10:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][300/625] eta 0:02:02 lr 0.000507 wd 0.0500 time 0.3862 (0.3783) data time 0.0012 (0.0026) model time 0.3851 (0.3769) loss 3.1861 (2.9059) grad_norm 1.6340 (2.0699) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][310/625] eta 0:01:59 lr 0.000507 wd 0.0500 time 0.3656 (0.3781) data time 0.0011 (0.0026) model time 0.3645 (0.3766) loss 2.6282 (2.9038) grad_norm 1.5594 (2.0781) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][320/625] eta 0:01:55 lr 0.000507 wd 0.0500 time 0.3746 (0.3780) data time 0.0010 (0.0025) model time 0.3736 (0.3765) loss 3.8062 (2.9099) grad_norm 2.1048 (2.0739) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][330/625] eta 0:01:51 lr 0.000507 wd 0.0500 time 0.3679 (0.3778) data time 0.0009 (0.0025) model time 0.3669 (0.3762) loss 3.4390 (2.9102) grad_norm 10.3754 (2.0926) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][340/625] eta 0:01:47 lr 0.000507 wd 0.0500 time 0.3667 (0.3776) data time 0.0013 (0.0024) model time 0.3654 (0.3761) loss 3.5061 (2.9144) grad_norm 2.0037 (2.0933) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][350/625] eta 0:01:43 lr 0.000506 wd 0.0500 time 0.3662 (0.3774) data time 0.0013 (0.0024) model time 0.3650 (0.3758) loss 3.0767 (2.9094) grad_norm 1.4982 (2.0924) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][360/625] eta 0:01:39 lr 0.000506 wd 0.0500 time 0.3733 (0.3772) data time 0.0009 (0.0024) model time 0.3724 (0.3756) loss 2.9737 (2.9161) grad_norm 2.0107 (2.0863) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][370/625] eta 0:01:36 lr 0.000506 wd 0.0500 time 0.3681 (0.3771) data time 0.0011 (0.0023) model time 0.3669 (0.3755) loss 3.0495 (2.9153) grad_norm 2.4774 (2.0845) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][380/625] eta 0:01:32 lr 0.000506 wd 0.0500 time 0.3782 (0.3771) data time 0.0008 (0.0023) model time 0.3773 (0.3755) loss 3.2442 (2.9078) grad_norm 1.7617 (2.0757) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][390/625] eta 0:01:28 lr 0.000506 wd 0.0500 time 0.3755 (0.3771) data time 0.0009 (0.0023) model time 0.3746 (0.3756) loss 3.3064 (2.9123) grad_norm 2.6260 (2.0737) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][400/625] eta 0:01:24 lr 0.000506 wd 0.0500 time 0.3729 (0.3771) data time 0.0011 (0.0023) model time 0.3718 (0.3755) loss 3.6278 (2.9143) grad_norm 1.8101 (2.0643) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][410/625] eta 0:01:21 lr 0.000506 wd 0.0500 time 0.3755 (0.3771) data time 0.0009 (0.0022) model time 0.3746 (0.3755) loss 3.3865 (2.9198) grad_norm 2.0158 (2.0742) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][420/625] eta 0:01:17 lr 0.000505 wd 0.0500 time 0.3758 (0.3771) data time 0.0011 (0.0022) model time 0.3747 (0.3756) loss 1.8476 (2.9184) grad_norm 1.9390 (2.0731) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][430/625] eta 0:01:13 lr 0.000505 wd 0.0500 time 0.3833 (0.3771) data time 0.0012 (0.0022) model time 0.3821 (0.3756) loss 3.0039 (2.9196) grad_norm 1.5502 (2.0648) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][440/625] eta 0:01:09 lr 0.000505 wd 0.0500 time 0.3767 (0.3771) data time 0.0011 (0.0022) model time 0.3756 (0.3756) loss 2.3327 (2.9167) grad_norm 1.6650 (2.0576) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:11:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][450/625] eta 0:01:05 lr 0.000505 wd 0.0500 time 0.3756 (0.3771) data time 0.0011 (0.0021) model time 0.3745 (0.3755) loss 2.5329 (2.9176) grad_norm 1.6341 (2.0506) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][460/625] eta 0:01:02 lr 0.000505 wd 0.0500 time 0.3702 (0.3782) data time 0.0012 (0.0021) model time 0.3690 (0.3769) loss 3.3326 (2.9220) grad_norm 2.9703 (2.0525) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][470/625] eta 0:00:58 lr 0.000505 wd 0.0500 time 0.3701 (0.3785) data time 0.0009 (0.0021) model time 0.3692 (0.3771) loss 2.7065 (2.9239) grad_norm 2.1748 (2.0509) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][480/625] eta 0:00:54 lr 0.000504 wd 0.0500 time 0.3786 (0.3784) data time 0.0009 (0.0021) model time 0.3777 (0.3771) loss 1.9727 (2.9251) grad_norm 3.1815 (2.0717) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][490/625] eta 0:00:51 lr 0.000504 wd 0.0500 time 0.3702 (0.3783) data time 0.0014 (0.0021) model time 0.3688 (0.3770) loss 3.3906 (2.9240) grad_norm 1.2245 (2.0650) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][500/625] eta 0:00:47 lr 0.000504 wd 0.0500 time 0.3680 (0.3783) data time 0.0010 (0.0020) model time 0.3670 (0.3770) loss 3.2872 (2.9249) grad_norm 3.0946 (2.0657) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][510/625] eta 0:00:43 lr 0.000504 wd 0.0500 time 0.3738 (0.3783) data time 0.0008 (0.0020) model time 0.3730 (0.3770) loss 2.5028 (2.9223) grad_norm 1.3997 (2.0592) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][520/625] eta 0:00:39 lr 0.000504 wd 0.0500 time 0.3751 (0.3783) data time 0.0008 (0.0020) model time 0.3743 (0.3770) loss 2.6647 (2.9236) grad_norm 1.6424 (2.0546) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][530/625] eta 0:00:35 lr 0.000504 wd 0.0500 time 0.3752 (0.3783) data time 0.0010 (0.0020) model time 0.3743 (0.3769) loss 3.6724 (2.9313) grad_norm 2.9675 (2.0816) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][540/625] eta 0:00:32 lr 0.000504 wd 0.0500 time 0.3741 (0.3782) data time 0.0012 (0.0020) model time 0.3729 (0.3769) loss 2.5746 (2.9348) grad_norm 1.5144 (2.0766) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][550/625] eta 0:00:28 lr 0.000503 wd 0.0500 time 0.3730 (0.3781) data time 0.0013 (0.0020) model time 0.3717 (0.3768) loss 1.8943 (2.9367) grad_norm 2.7339 (2.0834) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][560/625] eta 0:00:24 lr 0.000503 wd 0.0500 time 0.3802 (0.3781) data time 0.0009 (0.0020) model time 0.3793 (0.3768) loss 3.6994 (2.9417) grad_norm 1.9118 (2.0876) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][570/625] eta 0:00:20 lr 0.000503 wd 0.0500 time 0.5776 (0.3784) data time 0.0011 (0.0019) model time 0.5765 (0.3771) loss 3.6018 (2.9407) grad_norm 1.4848 (2.0858) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][580/625] eta 0:00:17 lr 0.000503 wd 0.0500 time 0.3750 (0.3791) data time 0.0008 (0.0019) model time 0.3742 (0.3778) loss 2.4810 (2.9390) grad_norm 2.3021 (2.1019) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][590/625] eta 0:00:13 lr 0.000503 wd 0.0500 time 0.3720 (0.3793) data time 0.0009 (0.0019) model time 0.3711 (0.3780) loss 3.7560 (2.9426) grad_norm 1.4363 (2.0965) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][600/625] eta 0:00:09 lr 0.000503 wd 0.0500 time 0.3716 (0.3792) data time 0.0011 (0.0019) model time 0.3705 (0.3779) loss 3.0763 (2.9395) grad_norm 2.2485 (2.0926) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][610/625] eta 0:00:05 lr 0.000502 wd 0.0500 time 0.3582 (0.3790) data time 0.0006 (0.0019) model time 0.3576 (0.3777) loss 2.8785 (2.9397) grad_norm 1.6930 (2.0880) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [207/300][620/625] eta 0:00:01 lr 0.000502 wd 0.0500 time 0.3649 (0.3787) data time 0.0008 (0.0019) model time 0.3641 (0.3774) loss 2.7162 (2.9407) grad_norm 2.0609 (2.0793) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:02 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 207 training takes 0:03:56 [2024-09-01 09:13:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:13:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.519 (0.519) Loss 0.5625 (0.5625) Acc@1 87.988 (87.988) Acc@5 98.193 (98.193) Mem 11225MB [2024-09-01 09:13:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 0.8901 (0.6625) Acc@1 79.053 (85.489) Acc@5 95.703 (97.465) Mem 11225MB [2024-09-01 09:13:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.9829 (0.7831) Acc@1 76.855 (82.366) Acc@5 94.727 (96.166) Mem 11225MB [2024-09-01 09:13:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.058 Acc@5 96.171 [2024-09-01 09:13:06 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-09-01 09:13:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.803 (0.803) Loss 0.4897 (0.4897) Acc@1 89.648 (89.648) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 09:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.172) Loss 0.7944 (0.6013) Acc@1 81.250 (86.705) Acc@5 96.191 (97.781) Mem 11225MB [2024-09-01 09:13:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.135) Loss 0.8887 (0.7125) Acc@1 78.027 (83.770) Acc@5 95.117 (96.645) Mem 11225MB [2024-09-01 09:13:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.477 Acc@5 96.683 [2024-09-01 09:13:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:13:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.48% [2024-09-01 09:13:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:13:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][0/625] eta 0:08:25 lr 0.000502 wd 0.0500 time 0.8089 (0.8089) data time 0.4765 (0.4765) model time 0.0000 (0.0000) loss 3.1250 (3.1250) grad_norm 2.4173 (2.4173) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][10/625] eta 0:04:14 lr 0.000502 wd 0.0500 time 0.3639 (0.4131) data time 0.0014 (0.0444) model time 0.0000 (0.0000) loss 2.8162 (3.1334) grad_norm 1.9898 (2.2676) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][20/625] eta 0:03:58 lr 0.000502 wd 0.0500 time 0.3746 (0.3949) data time 0.0009 (0.0238) model time 0.0000 (0.0000) loss 3.0804 (3.0606) grad_norm 1.5633 (2.1315) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][30/625] eta 0:03:51 lr 0.000502 wd 0.0500 time 0.3744 (0.3890) data time 0.0009 (0.0166) model time 0.0000 (0.0000) loss 3.0067 (3.0174) grad_norm 2.1036 (2.0563) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][40/625] eta 0:03:45 lr 0.000502 wd 0.0500 time 0.3787 (0.3851) data time 0.0010 (0.0128) model time 0.0000 (0.0000) loss 2.6692 (2.9276) grad_norm 1.8403 (1.9943) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][50/625] eta 0:03:40 lr 0.000501 wd 0.0500 time 0.3740 (0.3831) data time 0.0008 (0.0106) model time 0.0000 (0.0000) loss 2.5548 (2.8952) grad_norm 1.8123 (1.9774) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][60/625] eta 0:03:35 lr 0.000501 wd 0.0500 time 0.3757 (0.3817) data time 0.0011 (0.0090) model time 0.3746 (0.3732) loss 3.2097 (2.8992) grad_norm 2.0770 (1.9768) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][70/625] eta 0:03:31 lr 0.000501 wd 0.0500 time 0.3750 (0.3806) data time 0.0012 (0.0079) model time 0.3738 (0.3730) loss 3.0035 (2.9172) grad_norm 3.0349 (1.9768) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][80/625] eta 0:03:27 lr 0.000501 wd 0.0500 time 0.3821 (0.3802) data time 0.0009 (0.0071) model time 0.3812 (0.3741) loss 3.5818 (2.9425) grad_norm 5.0398 (2.0010) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][90/625] eta 0:03:25 lr 0.000501 wd 0.0500 time 0.3764 (0.3841) data time 0.0012 (0.0065) model time 0.3752 (0.3840) loss 3.1422 (2.9672) grad_norm 1.6685 (1.9947) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][100/625] eta 0:03:21 lr 0.000501 wd 0.0500 time 0.3680 (0.3829) data time 0.0013 (0.0060) model time 0.3667 (0.3814) loss 2.4592 (2.9708) grad_norm 1.6796 (1.9696) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][110/625] eta 0:03:17 lr 0.000501 wd 0.0500 time 0.3742 (0.3834) data time 0.0011 (0.0055) model time 0.3731 (0.3823) loss 3.4146 (2.9883) grad_norm 1.4932 (1.9597) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][120/625] eta 0:03:13 lr 0.000500 wd 0.0500 time 0.3706 (0.3825) data time 0.0012 (0.0052) model time 0.3694 (0.3808) loss 3.1245 (2.9738) grad_norm 1.4175 (1.9351) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][130/625] eta 0:03:08 lr 0.000500 wd 0.0500 time 0.3704 (0.3817) data time 0.0010 (0.0049) model time 0.3694 (0.3796) loss 3.1011 (2.9904) grad_norm 2.0564 (1.9133) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][140/625] eta 0:03:05 lr 0.000500 wd 0.0500 time 0.3853 (0.3815) data time 0.0011 (0.0046) model time 0.3842 (0.3794) loss 2.3056 (2.9812) grad_norm 1.6157 (1.9337) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][150/625] eta 0:03:00 lr 0.000500 wd 0.0500 time 0.3830 (0.3809) data time 0.0009 (0.0044) model time 0.3821 (0.3786) loss 2.3953 (2.9820) grad_norm 1.4351 (1.9250) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][160/625] eta 0:02:56 lr 0.000500 wd 0.0500 time 0.3792 (0.3805) data time 0.0009 (0.0042) model time 0.3783 (0.3780) loss 2.8502 (2.9859) grad_norm 2.3025 (1.9502) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][170/625] eta 0:02:52 lr 0.000500 wd 0.0500 time 0.3791 (0.3801) data time 0.0009 (0.0040) model time 0.3782 (0.3775) loss 3.7544 (2.9936) grad_norm 2.0452 (1.9748) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][180/625] eta 0:02:49 lr 0.000499 wd 0.0500 time 0.3856 (0.3799) data time 0.0011 (0.0039) model time 0.3845 (0.3774) loss 3.3415 (2.9891) grad_norm 1.4875 (1.9811) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][190/625] eta 0:02:45 lr 0.000499 wd 0.0500 time 0.3834 (0.3797) data time 0.0012 (0.0037) model time 0.3822 (0.3772) loss 2.5349 (2.9860) grad_norm 1.8455 (1.9999) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][200/625] eta 0:02:41 lr 0.000499 wd 0.0500 time 0.3656 (0.3795) data time 0.0012 (0.0036) model time 0.3644 (0.3770) loss 2.0731 (2.9776) grad_norm 2.4278 (2.0010) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][210/625] eta 0:02:37 lr 0.000499 wd 0.0500 time 0.3678 (0.3792) data time 0.0008 (0.0035) model time 0.3670 (0.3766) loss 2.4989 (2.9619) grad_norm 1.7423 (2.0235) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][220/625] eta 0:02:33 lr 0.000499 wd 0.0500 time 0.3677 (0.3789) data time 0.0009 (0.0034) model time 0.3668 (0.3764) loss 2.5731 (2.9559) grad_norm 2.0352 (2.0169) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][230/625] eta 0:02:29 lr 0.000499 wd 0.0500 time 0.3761 (0.3789) data time 0.0008 (0.0033) model time 0.3753 (0.3764) loss 3.7141 (2.9655) grad_norm 3.4732 (2.0379) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][240/625] eta 0:02:25 lr 0.000499 wd 0.0500 time 0.3837 (0.3787) data time 0.0011 (0.0032) model time 0.3825 (0.3763) loss 2.7951 (2.9624) grad_norm 2.7109 (2.0394) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][250/625] eta 0:02:21 lr 0.000498 wd 0.0500 time 0.3774 (0.3786) data time 0.0011 (0.0031) model time 0.3763 (0.3762) loss 2.6201 (2.9699) grad_norm 1.5438 (2.0329) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][260/625] eta 0:02:18 lr 0.000498 wd 0.0500 time 0.3789 (0.3785) data time 0.0011 (0.0031) model time 0.3779 (0.3761) loss 2.9133 (2.9585) grad_norm 1.4910 (2.0270) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][270/625] eta 0:02:14 lr 0.000498 wd 0.0500 time 0.3731 (0.3784) data time 0.0011 (0.0030) model time 0.3720 (0.3760) loss 3.3281 (2.9571) grad_norm 1.6426 (2.0319) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:14:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][280/625] eta 0:02:10 lr 0.000498 wd 0.0500 time 0.3735 (0.3783) data time 0.0011 (0.0029) model time 0.3724 (0.3760) loss 3.1886 (2.9579) grad_norm 2.2300 (2.0237) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][290/625] eta 0:02:06 lr 0.000498 wd 0.0500 time 0.3773 (0.3788) data time 0.0009 (0.0029) model time 0.3764 (0.3766) loss 3.2194 (2.9456) grad_norm 2.0243 (2.0240) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][300/625] eta 0:02:03 lr 0.000498 wd 0.0500 time 0.3695 (0.3800) data time 0.0008 (0.0028) model time 0.3687 (0.3781) loss 2.5478 (2.9398) grad_norm 1.9315 (2.0288) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][310/625] eta 0:01:59 lr 0.000498 wd 0.0500 time 0.3687 (0.3797) data time 0.0008 (0.0028) model time 0.3679 (0.3778) loss 2.9479 (2.9425) grad_norm 2.0633 (2.0212) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][320/625] eta 0:01:55 lr 0.000497 wd 0.0500 time 0.3668 (0.3796) data time 0.0009 (0.0027) model time 0.3659 (0.3777) loss 3.1886 (2.9530) grad_norm 1.5444 (2.0182) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][330/625] eta 0:01:51 lr 0.000497 wd 0.0500 time 0.3591 (0.3794) data time 0.0009 (0.0027) model time 0.3582 (0.3775) loss 3.5775 (2.9571) grad_norm 2.0898 (2.0095) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][340/625] eta 0:01:48 lr 0.000497 wd 0.0500 time 0.3755 (0.3792) data time 0.0009 (0.0026) model time 0.3746 (0.3773) loss 3.2579 (2.9576) grad_norm 1.8723 (2.0101) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][350/625] eta 0:01:44 lr 0.000497 wd 0.0500 time 0.3750 (0.3791) data time 0.0011 (0.0026) model time 0.3739 (0.3772) loss 3.1591 (2.9534) grad_norm 1.7472 (2.0071) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][360/625] eta 0:01:40 lr 0.000497 wd 0.0500 time 0.3765 (0.3791) data time 0.0012 (0.0025) model time 0.3754 (0.3772) loss 3.1233 (2.9596) grad_norm 2.5535 (2.0066) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][370/625] eta 0:01:36 lr 0.000497 wd 0.0500 time 0.3771 (0.3791) data time 0.0008 (0.0025) model time 0.3762 (0.3772) loss 1.8366 (2.9447) grad_norm 1.8464 (2.0041) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][380/625] eta 0:01:32 lr 0.000496 wd 0.0500 time 0.3742 (0.3789) data time 0.0011 (0.0025) model time 0.3731 (0.3771) loss 3.0330 (2.9418) grad_norm 2.4118 (2.0034) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][390/625] eta 0:01:29 lr 0.000496 wd 0.0500 time 0.3709 (0.3788) data time 0.0008 (0.0024) model time 0.3700 (0.3770) loss 3.5246 (2.9395) grad_norm 3.8947 (2.0155) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][400/625] eta 0:01:25 lr 0.000496 wd 0.0500 time 0.3765 (0.3787) data time 0.0011 (0.0024) model time 0.3754 (0.3768) loss 2.1358 (2.9389) grad_norm 4.6707 (2.0382) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][410/625] eta 0:01:21 lr 0.000496 wd 0.0500 time 0.3731 (0.3790) data time 0.0011 (0.0024) model time 0.3720 (0.3773) loss 3.3295 (2.9430) grad_norm 1.4544 (2.0468) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][420/625] eta 0:01:17 lr 0.000496 wd 0.0500 time 0.3824 (0.3790) data time 0.0013 (0.0023) model time 0.3812 (0.3772) loss 2.7320 (2.9434) grad_norm 1.7182 (2.0420) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][430/625] eta 0:01:13 lr 0.000496 wd 0.0500 time 0.3785 (0.3789) data time 0.0011 (0.0023) model time 0.3774 (0.3771) loss 2.6910 (2.9502) grad_norm 1.8577 (2.0604) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:15:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][440/625] eta 0:01:10 lr 0.000496 wd 0.0500 time 0.3703 (0.3788) data time 0.0012 (0.0023) model time 0.3691 (0.3770) loss 3.4058 (2.9513) grad_norm 1.7040 (2.0510) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][450/625] eta 0:01:06 lr 0.000495 wd 0.0500 time 0.3762 (0.3787) data time 0.0012 (0.0023) model time 0.3750 (0.3769) loss 2.0308 (2.9470) grad_norm 2.1008 (2.0438) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][460/625] eta 0:01:02 lr 0.000495 wd 0.0500 time 0.3738 (0.3786) data time 0.0009 (0.0022) model time 0.3729 (0.3769) loss 3.0073 (2.9495) grad_norm 1.8528 (2.0391) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][470/625] eta 0:00:58 lr 0.000495 wd 0.0500 time 0.3832 (0.3786) data time 0.0012 (0.0022) model time 0.3820 (0.3768) loss 2.6190 (2.9472) grad_norm 2.1771 (2.0402) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][480/625] eta 0:00:54 lr 0.000495 wd 0.0500 time 0.5529 (0.3789) data time 0.0012 (0.0022) model time 0.5517 (0.3772) loss 3.1707 (2.9414) grad_norm 2.4702 (2.0428) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][490/625] eta 0:00:51 lr 0.000495 wd 0.0500 time 0.3740 (0.3796) data time 0.0010 (0.0022) model time 0.3730 (0.3779) loss 2.4989 (2.9448) grad_norm 1.5969 (2.0366) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][500/625] eta 0:00:47 lr 0.000495 wd 0.0500 time 0.3638 (0.3794) data time 0.0012 (0.0022) model time 0.3627 (0.3777) loss 3.2180 (2.9497) grad_norm 1.8541 (2.0289) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][510/625] eta 0:00:43 lr 0.000494 wd 0.0500 time 0.3727 (0.3792) data time 0.0009 (0.0021) model time 0.3718 (0.3776) loss 2.8162 (2.9447) grad_norm 1.3770 (2.0259) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][520/625] eta 0:00:39 lr 0.000494 wd 0.0500 time 0.3790 (0.3791) data time 0.0010 (0.0021) model time 0.3780 (0.3774) loss 3.5685 (2.9483) grad_norm 2.4792 (2.0227) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][530/625] eta 0:00:36 lr 0.000494 wd 0.0500 time 0.3723 (0.3790) data time 0.0011 (0.0021) model time 0.3712 (0.3774) loss 3.2332 (2.9482) grad_norm 1.9677 (2.0153) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][540/625] eta 0:00:32 lr 0.000494 wd 0.0500 time 0.3695 (0.3789) data time 0.0009 (0.0021) model time 0.3686 (0.3772) loss 3.3647 (2.9466) grad_norm 1.2945 (2.0091) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][550/625] eta 0:00:28 lr 0.000494 wd 0.0500 time 0.3809 (0.3788) data time 0.0008 (0.0021) model time 0.3801 (0.3772) loss 2.4830 (2.9474) grad_norm 7.3907 (2.0361) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][560/625] eta 0:00:24 lr 0.000494 wd 0.0500 time 0.3769 (0.3787) data time 0.0011 (0.0021) model time 0.3759 (0.3771) loss 3.0742 (2.9509) grad_norm 1.4384 (2.0325) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][570/625] eta 0:00:20 lr 0.000494 wd 0.0500 time 0.3864 (0.3787) data time 0.0009 (0.0020) model time 0.3855 (0.3771) loss 2.0928 (2.9458) grad_norm 1.9814 (2.0339) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][580/625] eta 0:00:17 lr 0.000493 wd 0.0500 time 0.3777 (0.3787) data time 0.0009 (0.0020) model time 0.3768 (0.3771) loss 3.1828 (2.9424) grad_norm 2.1978 (2.0322) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][590/625] eta 0:00:13 lr 0.000493 wd 0.0500 time 0.3802 (0.3786) data time 0.0009 (0.0020) model time 0.3793 (0.3770) loss 2.0399 (2.9393) grad_norm 1.5168 (2.0286) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][600/625] eta 0:00:09 lr 0.000493 wd 0.0500 time 0.5833 (0.3789) data time 0.0008 (0.0020) model time 0.5825 (0.3773) loss 3.3076 (2.9420) grad_norm 1.9215 (2.0268) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][610/625] eta 0:00:05 lr 0.000493 wd 0.0500 time 0.3661 (0.3791) data time 0.0008 (0.0020) model time 0.3653 (0.3775) loss 2.8478 (2.9441) grad_norm 1.6733 (2.0265) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [208/300][620/625] eta 0:00:01 lr 0.000493 wd 0.0500 time 0.3587 (0.3792) data time 0.0006 (0.0020) model time 0.3581 (0.3776) loss 2.1703 (2.9416) grad_norm 3.3185 (2.0527) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:07 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 208 training takes 0:03:56 [2024-09-01 09:17:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:17:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.510 (0.510) Loss 0.5508 (0.5508) Acc@1 88.672 (88.672) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:17:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 0.8691 (0.6657) Acc@1 79.932 (85.622) Acc@5 95.459 (97.439) Mem 11225MB [2024-09-01 09:17:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9834 (0.7904) Acc@1 76.367 (82.322) Acc@5 94.482 (96.119) Mem 11225MB [2024-09-01 09:17:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.096 Acc@5 96.139 [2024-09-01 09:17:11 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-09-01 09:17:11 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.10% [2024-09-01 09:17:11 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 09:17:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 09:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.492 (0.492) Loss 0.4900 (0.4900) Acc@1 89.844 (89.844) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:17:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.139) Loss 0.7925 (0.6007) Acc@1 81.299 (86.728) Acc@5 96.289 (97.758) Mem 11225MB [2024-09-01 09:17:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.118) Loss 0.8882 (0.7117) Acc@1 77.783 (83.791) Acc@5 95.264 (96.638) Mem 11225MB [2024-09-01 09:17:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.483 Acc@5 96.675 [2024-09-01 09:17:19 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:17:19 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.48% [2024-09-01 09:17:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:17:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][0/625] eta 0:07:46 lr 0.000493 wd 0.0500 time 0.7470 (0.7470) data time 0.4043 (0.4043) model time 0.0000 (0.0000) loss 2.8741 (2.8741) grad_norm 2.5163 (2.5163) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][10/625] eta 0:04:09 lr 0.000493 wd 0.0500 time 0.3621 (0.4050) data time 0.0011 (0.0379) model time 0.0000 (0.0000) loss 2.7338 (3.0207) grad_norm 1.2023 (1.7434) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][20/625] eta 0:03:54 lr 0.000492 wd 0.0500 time 0.3701 (0.3878) data time 0.0011 (0.0204) model time 0.0000 (0.0000) loss 3.0080 (2.7839) grad_norm 3.0530 (1.8235) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][30/625] eta 0:03:46 lr 0.000492 wd 0.0500 time 0.3730 (0.3812) data time 0.0008 (0.0142) model time 0.0000 (0.0000) loss 1.9712 (2.7852) grad_norm 2.3383 (1.7863) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][40/625] eta 0:03:41 lr 0.000492 wd 0.0500 time 0.3816 (0.3786) data time 0.0011 (0.0110) model time 0.0000 (0.0000) loss 2.7812 (2.7950) grad_norm 2.0440 (1.8615) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][50/625] eta 0:03:36 lr 0.000492 wd 0.0500 time 0.3668 (0.3767) data time 0.0011 (0.0091) model time 0.0000 (0.0000) loss 2.8227 (2.7897) grad_norm 1.3428 (1.9395) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][60/625] eta 0:03:32 lr 0.000492 wd 0.0500 time 0.3705 (0.3762) data time 0.0009 (0.0078) model time 0.3696 (0.3719) loss 3.7717 (2.8076) grad_norm 2.4097 (1.9980) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][70/625] eta 0:03:29 lr 0.000492 wd 0.0500 time 0.3788 (0.3778) data time 0.0011 (0.0069) model time 0.3777 (0.3792) loss 3.3813 (2.7715) grad_norm 2.4233 (1.9742) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][80/625] eta 0:03:25 lr 0.000492 wd 0.0500 time 0.3823 (0.3772) data time 0.0011 (0.0062) model time 0.3812 (0.3768) loss 2.1351 (2.7680) grad_norm 2.3551 (1.9521) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][90/625] eta 0:03:21 lr 0.000491 wd 0.0500 time 0.3693 (0.3769) data time 0.0012 (0.0056) model time 0.3681 (0.3760) loss 2.8481 (2.7793) grad_norm 2.1574 (1.9517) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:17:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][100/625] eta 0:03:17 lr 0.000491 wd 0.0500 time 0.3700 (0.3763) data time 0.0008 (0.0052) model time 0.3691 (0.3746) loss 3.0129 (2.7784) grad_norm 1.8638 (1.9409) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][110/625] eta 0:03:13 lr 0.000491 wd 0.0500 time 0.3689 (0.3758) data time 0.0012 (0.0049) model time 0.3677 (0.3738) loss 2.9792 (2.8142) grad_norm 1.6218 (1.9293) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][120/625] eta 0:03:09 lr 0.000491 wd 0.0500 time 0.3655 (0.3753) data time 0.0012 (0.0046) model time 0.3643 (0.3730) loss 2.6827 (2.8216) grad_norm 2.6446 (1.9809) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][130/625] eta 0:03:05 lr 0.000491 wd 0.0500 time 0.3706 (0.3750) data time 0.0012 (0.0043) model time 0.3694 (0.3726) loss 3.0994 (2.8402) grad_norm 1.9255 (2.0196) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][140/625] eta 0:03:01 lr 0.000491 wd 0.0500 time 0.3696 (0.3748) data time 0.0012 (0.0041) model time 0.3684 (0.3725) loss 2.7716 (2.8478) grad_norm 1.4676 (2.0071) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][150/625] eta 0:02:57 lr 0.000490 wd 0.0500 time 0.3648 (0.3746) data time 0.0011 (0.0039) model time 0.3637 (0.3722) loss 2.8854 (2.8478) grad_norm 1.9147 (1.9998) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][160/625] eta 0:02:54 lr 0.000490 wd 0.0500 time 0.5803 (0.3756) data time 0.0008 (0.0037) model time 0.5794 (0.3739) loss 2.1431 (2.8380) grad_norm 1.5646 (1.9904) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][170/625] eta 0:02:51 lr 0.000490 wd 0.0500 time 0.3706 (0.3779) data time 0.0011 (0.0036) model time 0.3695 (0.3771) loss 3.4497 (2.8601) grad_norm 1.5302 (1.9864) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][180/625] eta 0:02:48 lr 0.000490 wd 0.0500 time 0.3729 (0.3776) data time 0.0008 (0.0034) model time 0.3721 (0.3767) loss 1.6657 (2.8508) grad_norm 1.6930 (1.9869) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][190/625] eta 0:02:44 lr 0.000490 wd 0.0500 time 0.3673 (0.3772) data time 0.0011 (0.0033) model time 0.3662 (0.3762) loss 2.0515 (2.8663) grad_norm 3.1226 (1.9840) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][200/625] eta 0:02:40 lr 0.000490 wd 0.0500 time 0.3691 (0.3769) data time 0.0009 (0.0032) model time 0.3682 (0.3757) loss 3.5071 (2.8620) grad_norm 1.5860 (1.9720) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][210/625] eta 0:02:36 lr 0.000490 wd 0.0500 time 0.3711 (0.3766) data time 0.0012 (0.0031) model time 0.3699 (0.3753) loss 2.2809 (2.8677) grad_norm 1.1195 (1.9642) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][220/625] eta 0:02:32 lr 0.000489 wd 0.0500 time 0.3662 (0.3763) data time 0.0009 (0.0030) model time 0.3654 (0.3750) loss 3.1832 (2.8729) grad_norm 1.5352 (1.9697) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][230/625] eta 0:02:28 lr 0.000489 wd 0.0500 time 0.3642 (0.3762) data time 0.0009 (0.0030) model time 0.3633 (0.3748) loss 2.6494 (2.8795) grad_norm 2.3474 (1.9574) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][240/625] eta 0:02:24 lr 0.000489 wd 0.0500 time 0.3697 (0.3761) data time 0.0011 (0.0029) model time 0.3686 (0.3747) loss 3.3500 (2.8873) grad_norm 1.6026 (1.9576) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][250/625] eta 0:02:20 lr 0.000489 wd 0.0500 time 0.3670 (0.3759) data time 0.0013 (0.0028) model time 0.3658 (0.3745) loss 3.7711 (2.8903) grad_norm 7.6412 (1.9901) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:18:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][260/625] eta 0:02:17 lr 0.000489 wd 0.0500 time 0.3732 (0.3757) data time 0.0011 (0.0028) model time 0.3721 (0.3743) loss 2.4832 (2.8946) grad_norm 1.8095 (2.0040) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][270/625] eta 0:02:13 lr 0.000489 wd 0.0500 time 0.3805 (0.3757) data time 0.0009 (0.0027) model time 0.3796 (0.3742) loss 2.2730 (2.9031) grad_norm 2.3866 (2.0022) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][280/625] eta 0:02:10 lr 0.000488 wd 0.0500 time 0.5858 (0.3777) data time 0.0012 (0.0026) model time 0.5846 (0.3767) loss 3.0890 (2.9056) grad_norm 1.7848 (2.0006) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][290/625] eta 0:02:06 lr 0.000488 wd 0.0500 time 0.3773 (0.3782) data time 0.0011 (0.0026) model time 0.3761 (0.3774) loss 2.2615 (2.9104) grad_norm 1.6874 (1.9898) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][300/625] eta 0:02:02 lr 0.000488 wd 0.0500 time 0.3689 (0.3779) data time 0.0012 (0.0025) model time 0.3677 (0.3770) loss 3.0722 (2.9046) grad_norm 1.3325 (1.9873) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][310/625] eta 0:01:58 lr 0.000488 wd 0.0500 time 0.3653 (0.3777) data time 0.0010 (0.0025) model time 0.3643 (0.3767) loss 3.3365 (2.9029) grad_norm 1.7319 (1.9913) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][320/625] eta 0:01:55 lr 0.000488 wd 0.0500 time 0.3800 (0.3775) data time 0.0011 (0.0025) model time 0.3789 (0.3764) loss 2.5931 (2.8963) grad_norm 1.5272 (1.9935) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][330/625] eta 0:01:51 lr 0.000488 wd 0.0500 time 0.3734 (0.3773) data time 0.0009 (0.0024) model time 0.3725 (0.3762) loss 2.0301 (2.8898) grad_norm 2.0002 (1.9875) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][340/625] eta 0:01:47 lr 0.000488 wd 0.0500 time 0.3692 (0.3771) data time 0.0012 (0.0024) model time 0.3680 (0.3760) loss 3.3103 (2.8910) grad_norm 2.0223 (1.9741) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][350/625] eta 0:01:43 lr 0.000487 wd 0.0500 time 0.3737 (0.3770) data time 0.0009 (0.0024) model time 0.3728 (0.3758) loss 3.5931 (2.8894) grad_norm 1.8497 (1.9667) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][360/625] eta 0:01:39 lr 0.000487 wd 0.0500 time 0.3750 (0.3769) data time 0.0011 (0.0023) model time 0.3739 (0.3757) loss 2.9458 (2.8886) grad_norm 1.5881 (1.9606) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][370/625] eta 0:01:36 lr 0.000487 wd 0.0500 time 0.3722 (0.3767) data time 0.0010 (0.0023) model time 0.3712 (0.3755) loss 3.3071 (2.8903) grad_norm 1.9921 (1.9729) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][380/625] eta 0:01:32 lr 0.000487 wd 0.0500 time 0.3735 (0.3766) data time 0.0012 (0.0023) model time 0.3723 (0.3754) loss 3.0482 (2.8953) grad_norm 2.1695 (1.9750) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][390/625] eta 0:01:28 lr 0.000487 wd 0.0500 time 0.3711 (0.3765) data time 0.0009 (0.0022) model time 0.3702 (0.3753) loss 3.4671 (2.8960) grad_norm 2.0294 (1.9702) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][400/625] eta 0:01:24 lr 0.000487 wd 0.0500 time 0.3757 (0.3765) data time 0.0010 (0.0022) model time 0.3747 (0.3753) loss 2.0420 (2.8917) grad_norm 1.5819 (2.0086) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][410/625] eta 0:01:20 lr 0.000487 wd 0.0500 time 0.3709 (0.3764) data time 0.0012 (0.0022) model time 0.3698 (0.3752) loss 3.0394 (2.8915) grad_norm 8.3604 (2.0178) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][420/625] eta 0:01:17 lr 0.000486 wd 0.0500 time 0.3669 (0.3763) data time 0.0013 (0.0022) model time 0.3655 (0.3750) loss 3.2167 (2.8985) grad_norm 1.6807 (2.0329) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][430/625] eta 0:01:13 lr 0.000486 wd 0.0500 time 0.3707 (0.3761) data time 0.0009 (0.0021) model time 0.3698 (0.3748) loss 3.3563 (2.9041) grad_norm 2.4433 (2.0489) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][440/625] eta 0:01:09 lr 0.000486 wd 0.0500 time 0.3719 (0.3760) data time 0.0011 (0.0021) model time 0.3708 (0.3746) loss 3.3260 (2.9039) grad_norm 2.4107 (2.0508) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][450/625] eta 0:01:05 lr 0.000486 wd 0.0500 time 0.3899 (0.3759) data time 0.0009 (0.0021) model time 0.3890 (0.3746) loss 3.0091 (2.9058) grad_norm 4.0933 (2.0607) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][460/625] eta 0:01:02 lr 0.000486 wd 0.0500 time 0.3690 (0.3762) data time 0.0009 (0.0021) model time 0.3681 (0.3749) loss 3.2281 (2.9040) grad_norm 2.0133 (2.0585) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][470/625] eta 0:00:58 lr 0.000486 wd 0.0500 time 0.3678 (0.3760) data time 0.0009 (0.0021) model time 0.3668 (0.3748) loss 3.3408 (2.9043) grad_norm 8.7652 (2.0741) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][480/625] eta 0:00:54 lr 0.000485 wd 0.0500 time 0.3694 (0.3759) data time 0.0011 (0.0020) model time 0.3683 (0.3746) loss 3.2026 (2.9061) grad_norm 1.8134 (2.0833) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][490/625] eta 0:00:50 lr 0.000485 wd 0.0500 time 0.3663 (0.3762) data time 0.0012 (0.0020) model time 0.3651 (0.3750) loss 2.4526 (2.9022) grad_norm 1.7165 (2.0860) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][500/625] eta 0:00:47 lr 0.000485 wd 0.0500 time 0.3655 (0.3770) data time 0.0011 (0.0020) model time 0.3644 (0.3758) loss 3.1389 (2.9037) grad_norm 1.8165 (2.0842) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][510/625] eta 0:00:43 lr 0.000485 wd 0.0500 time 0.3686 (0.3770) data time 0.0009 (0.0020) model time 0.3677 (0.3758) loss 2.3436 (2.9047) grad_norm 1.5758 (2.0780) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][520/625] eta 0:00:39 lr 0.000485 wd 0.0500 time 0.3739 (0.3769) data time 0.0009 (0.0020) model time 0.3730 (0.3757) loss 1.9698 (2.8989) grad_norm 1.4656 (2.0745) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][530/625] eta 0:00:35 lr 0.000485 wd 0.0500 time 0.3701 (0.3768) data time 0.0011 (0.0020) model time 0.3691 (0.3756) loss 2.5811 (2.9002) grad_norm 1.3698 (2.0676) loss_scale 1024.0000 (514.8927) mem 11225MB [2024-09-01 09:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][540/625] eta 0:00:32 lr 0.000485 wd 0.0500 time 0.3703 (0.3767) data time 0.0013 (0.0019) model time 0.3690 (0.3755) loss 2.7165 (2.9037) grad_norm 1.5159 (2.0610) loss_scale 1024.0000 (524.3031) mem 11225MB [2024-09-01 09:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][550/625] eta 0:00:28 lr 0.000484 wd 0.0500 time 0.3725 (0.3767) data time 0.0009 (0.0019) model time 0.3716 (0.3755) loss 3.6698 (2.9058) grad_norm 2.7997 (2.0749) loss_scale 1024.0000 (533.3721) mem 11225MB [2024-09-01 09:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][560/625] eta 0:00:24 lr 0.000484 wd 0.0500 time 0.3734 (0.3766) data time 0.0009 (0.0019) model time 0.3725 (0.3754) loss 1.7939 (2.9067) grad_norm 2.9396 (2.0788) loss_scale 1024.0000 (542.1176) mem 11225MB [2024-09-01 09:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][570/625] eta 0:00:20 lr 0.000484 wd 0.0500 time 0.3717 (0.3765) data time 0.0011 (0.0019) model time 0.3706 (0.3753) loss 2.7317 (2.9075) grad_norm 2.3880 (2.0775) loss_scale 1024.0000 (550.5569) mem 11225MB [2024-09-01 09:20:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][580/625] eta 0:00:16 lr 0.000484 wd 0.0500 time 0.3671 (0.3768) data time 0.0009 (0.0019) model time 0.3662 (0.3756) loss 2.6004 (2.9074) grad_norm 1.6222 (2.0778) loss_scale 1024.0000 (558.7057) mem 11225MB [2024-09-01 09:21:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][590/625] eta 0:00:13 lr 0.000484 wd 0.0500 time 0.3699 (0.3767) data time 0.0011 (0.0019) model time 0.3688 (0.3755) loss 2.7937 (2.9100) grad_norm 2.0108 (2.0723) loss_scale 1024.0000 (566.5787) mem 11225MB [2024-09-01 09:21:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][600/625] eta 0:00:09 lr 0.000484 wd 0.0500 time 0.3594 (0.3765) data time 0.0011 (0.0019) model time 0.3583 (0.3753) loss 1.9102 (2.9067) grad_norm 1.7327 (2.0751) loss_scale 1024.0000 (574.1897) mem 11225MB [2024-09-01 09:21:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][610/625] eta 0:00:05 lr 0.000484 wd 0.0500 time 0.3592 (0.3764) data time 0.0006 (0.0019) model time 0.3586 (0.3751) loss 3.1554 (2.9070) grad_norm 2.2200 (2.0736) loss_scale 1024.0000 (581.5516) mem 11225MB [2024-09-01 09:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [209/300][620/625] eta 0:00:01 lr 0.000483 wd 0.0500 time 0.3531 (0.3761) data time 0.0008 (0.0019) model time 0.3523 (0.3748) loss 2.9037 (2.9094) grad_norm 1.8565 (2.0698) loss_scale 1024.0000 (588.6763) mem 11225MB [2024-09-01 09:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 209 training takes 0:03:54 [2024-09-01 09:21:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:21:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:21:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.504 (0.504) Loss 0.5596 (0.5596) Acc@1 88.086 (88.086) Acc@5 98.242 (98.242) Mem 11225MB [2024-09-01 09:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.142) Loss 0.8877 (0.6606) Acc@1 79.346 (85.418) Acc@5 95.752 (97.456) Mem 11225MB [2024-09-01 09:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9434 (0.7790) Acc@1 77.783 (82.347) Acc@5 94.336 (96.084) Mem 11225MB [2024-09-01 09:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.110 Acc@5 96.087 [2024-09-01 09:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-09-01 09:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.11% [2024-09-01 09:21:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 09:21:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 09:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 0.4900 (0.4900) Acc@1 89.844 (89.844) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 09:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 0.7910 (0.5999) Acc@1 81.494 (86.710) Acc@5 96.289 (97.776) Mem 11225MB [2024-09-01 09:21:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.8872 (0.7111) Acc@1 77.881 (83.787) Acc@5 95.215 (96.636) Mem 11225MB [2024-09-01 09:21:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.471 Acc@5 96.667 [2024-09-01 09:21:23 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:21:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][0/625] eta 0:13:12 lr 0.000483 wd 0.0500 time 1.2686 (1.2686) data time 0.7706 (0.7706) model time 0.0000 (0.0000) loss 2.8724 (2.8724) grad_norm 1.8830 (1.8830) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][10/625] eta 0:04:38 lr 0.000483 wd 0.0500 time 0.3745 (0.4530) data time 0.0011 (0.0711) model time 0.0000 (0.0000) loss 3.1947 (2.9747) grad_norm 1.7701 (1.8191) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][20/625] eta 0:04:10 lr 0.000483 wd 0.0500 time 0.3706 (0.4140) data time 0.0010 (0.0377) model time 0.0000 (0.0000) loss 3.7537 (2.9242) grad_norm 2.1192 (1.9265) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][30/625] eta 0:03:58 lr 0.000483 wd 0.0500 time 0.3693 (0.4001) data time 0.0014 (0.0260) model time 0.0000 (0.0000) loss 2.8956 (2.9836) grad_norm 1.4641 (1.8801) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][40/625] eta 0:03:50 lr 0.000483 wd 0.0500 time 0.3720 (0.3942) data time 0.0011 (0.0200) model time 0.0000 (0.0000) loss 2.9877 (2.9899) grad_norm 2.0597 (1.9945) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][50/625] eta 0:03:43 lr 0.000483 wd 0.0500 time 0.3704 (0.3896) data time 0.0013 (0.0163) model time 0.0000 (0.0000) loss 2.4260 (2.9846) grad_norm 1.7589 (1.9853) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][60/625] eta 0:03:44 lr 0.000482 wd 0.0500 time 0.3679 (0.3966) data time 0.0011 (0.0138) model time 0.3667 (0.4310) loss 2.9404 (2.9652) grad_norm 2.6921 (1.9778) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][70/625] eta 0:03:38 lr 0.000482 wd 0.0500 time 0.3762 (0.3929) data time 0.0011 (0.0120) model time 0.3751 (0.4002) loss 2.7551 (2.9385) grad_norm 2.1384 (2.0432) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][80/625] eta 0:03:32 lr 0.000482 wd 0.0500 time 0.3685 (0.3899) data time 0.0012 (0.0107) model time 0.3673 (0.3892) loss 3.3348 (2.9532) grad_norm 3.0429 (2.0472) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:21:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][90/625] eta 0:03:27 lr 0.000482 wd 0.0500 time 0.3747 (0.3878) data time 0.0011 (0.0097) model time 0.3735 (0.3844) loss 3.1972 (2.9372) grad_norm 1.7223 (2.0364) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][100/625] eta 0:03:23 lr 0.000482 wd 0.0500 time 0.3697 (0.3875) data time 0.0009 (0.0088) model time 0.3688 (0.3842) loss 3.5857 (2.9456) grad_norm 1.5054 (2.0102) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][110/625] eta 0:03:18 lr 0.000482 wd 0.0500 time 0.3706 (0.3862) data time 0.0009 (0.0081) model time 0.3697 (0.3821) loss 2.3065 (2.9457) grad_norm 1.7967 (1.9723) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][120/625] eta 0:03:14 lr 0.000481 wd 0.0500 time 0.3677 (0.3849) data time 0.0009 (0.0076) model time 0.3668 (0.3803) loss 3.3563 (2.9638) grad_norm 1.4276 (1.9659) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][130/625] eta 0:03:09 lr 0.000481 wd 0.0500 time 0.3668 (0.3838) data time 0.0011 (0.0071) model time 0.3657 (0.3789) loss 3.0825 (2.9531) grad_norm 2.5859 (1.9415) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][140/625] eta 0:03:05 lr 0.000481 wd 0.0500 time 0.3680 (0.3831) data time 0.0012 (0.0066) model time 0.3668 (0.3782) loss 2.6815 (2.9613) grad_norm 1.8596 (1.9176) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][150/625] eta 0:03:01 lr 0.000481 wd 0.0500 time 0.3738 (0.3823) data time 0.0011 (0.0063) model time 0.3727 (0.3774) loss 3.1795 (2.9648) grad_norm 1.7622 (1.9098) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][160/625] eta 0:02:57 lr 0.000481 wd 0.0500 time 0.3701 (0.3817) data time 0.0009 (0.0060) model time 0.3692 (0.3768) loss 3.3898 (2.9673) grad_norm 1.6237 (1.9078) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][170/625] eta 0:02:53 lr 0.000481 wd 0.0500 time 0.3724 (0.3811) data time 0.0009 (0.0057) model time 0.3716 (0.3763) loss 3.3178 (2.9731) grad_norm 3.6815 (1.9204) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][180/625] eta 0:02:49 lr 0.000481 wd 0.0500 time 0.3712 (0.3807) data time 0.0009 (0.0054) model time 0.3703 (0.3760) loss 3.5387 (2.9888) grad_norm 2.1874 (1.9269) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][190/625] eta 0:02:45 lr 0.000480 wd 0.0500 time 0.3720 (0.3802) data time 0.0009 (0.0052) model time 0.3711 (0.3757) loss 2.2293 (2.9884) grad_norm 1.8921 (1.9236) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][200/625] eta 0:02:41 lr 0.000480 wd 0.0500 time 0.3642 (0.3797) data time 0.0008 (0.0050) model time 0.3633 (0.3752) loss 3.5648 (2.9964) grad_norm 1.7290 (1.9209) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][210/625] eta 0:02:37 lr 0.000480 wd 0.0500 time 0.3717 (0.3793) data time 0.0012 (0.0048) model time 0.3705 (0.3749) loss 2.9566 (2.9910) grad_norm 1.7298 (1.9308) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][220/625] eta 0:02:33 lr 0.000480 wd 0.0500 time 0.3673 (0.3789) data time 0.0009 (0.0047) model time 0.3663 (0.3745) loss 2.0673 (2.9827) grad_norm 2.5026 (1.9295) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][230/625] eta 0:02:29 lr 0.000480 wd 0.0500 time 0.3776 (0.3786) data time 0.0008 (0.0045) model time 0.3768 (0.3743) loss 2.5481 (2.9712) grad_norm 1.8496 (1.9505) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][240/625] eta 0:02:25 lr 0.000480 wd 0.0500 time 0.3760 (0.3791) data time 0.0011 (0.0044) model time 0.3749 (0.3751) loss 3.1031 (2.9737) grad_norm 1.4793 (1.9591) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:22:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][250/625] eta 0:02:22 lr 0.000480 wd 0.0500 time 0.3708 (0.3812) data time 0.0010 (0.0043) model time 0.3698 (0.3778) loss 3.8184 (2.9719) grad_norm 1.9931 (1.9645) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][260/625] eta 0:02:18 lr 0.000479 wd 0.0500 time 0.3710 (0.3808) data time 0.0012 (0.0041) model time 0.3699 (0.3774) loss 2.2179 (2.9612) grad_norm 2.3629 (1.9918) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][270/625] eta 0:02:15 lr 0.000479 wd 0.0500 time 0.3709 (0.3804) data time 0.0011 (0.0040) model time 0.3699 (0.3771) loss 3.0638 (2.9610) grad_norm 1.8401 (1.9954) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][280/625] eta 0:02:11 lr 0.000479 wd 0.0500 time 0.3696 (0.3807) data time 0.0010 (0.0039) model time 0.3686 (0.3775) loss 3.2016 (2.9639) grad_norm 1.7077 (1.9863) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][290/625] eta 0:02:07 lr 0.000479 wd 0.0500 time 0.3645 (0.3804) data time 0.0010 (0.0038) model time 0.3635 (0.3772) loss 3.4700 (2.9677) grad_norm 2.1659 (1.9893) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][300/625] eta 0:02:03 lr 0.000479 wd 0.0500 time 0.3671 (0.3801) data time 0.0009 (0.0038) model time 0.3663 (0.3770) loss 2.2436 (2.9604) grad_norm 2.1613 (1.9902) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][310/625] eta 0:01:59 lr 0.000479 wd 0.0500 time 0.3680 (0.3798) data time 0.0009 (0.0037) model time 0.3671 (0.3767) loss 2.8709 (2.9600) grad_norm 1.5096 (1.9840) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][320/625] eta 0:01:55 lr 0.000478 wd 0.0500 time 0.3702 (0.3795) data time 0.0009 (0.0036) model time 0.3693 (0.3764) loss 2.4146 (2.9515) grad_norm 2.0633 (2.0095) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][330/625] eta 0:01:51 lr 0.000478 wd 0.0500 time 0.3682 (0.3792) data time 0.0011 (0.0035) model time 0.3671 (0.3761) loss 2.4433 (2.9451) grad_norm 2.5854 (2.0203) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][340/625] eta 0:01:48 lr 0.000478 wd 0.0500 time 0.3683 (0.3790) data time 0.0009 (0.0035) model time 0.3674 (0.3759) loss 2.4651 (2.9356) grad_norm 1.9228 (2.0151) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][350/625] eta 0:01:44 lr 0.000478 wd 0.0500 time 0.3700 (0.3787) data time 0.0011 (0.0034) model time 0.3689 (0.3757) loss 3.1491 (2.9370) grad_norm 1.9570 (2.0058) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][360/625] eta 0:01:40 lr 0.000478 wd 0.0500 time 0.3749 (0.3791) data time 0.0012 (0.0033) model time 0.3737 (0.3762) loss 2.0740 (2.9263) grad_norm 1.6628 (2.0146) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][370/625] eta 0:01:36 lr 0.000478 wd 0.0500 time 0.3677 (0.3801) data time 0.0012 (0.0033) model time 0.3665 (0.3773) loss 3.7235 (2.9297) grad_norm 2.8141 (2.0148) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][380/625] eta 0:01:33 lr 0.000478 wd 0.0500 time 0.3767 (0.3799) data time 0.0009 (0.0032) model time 0.3758 (0.3772) loss 3.2423 (2.9295) grad_norm 2.0277 (2.0198) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][390/625] eta 0:01:29 lr 0.000477 wd 0.0500 time 0.3692 (0.3797) data time 0.0011 (0.0032) model time 0.3681 (0.3770) loss 3.4792 (2.9312) grad_norm 1.9645 (2.0369) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][400/625] eta 0:01:25 lr 0.000477 wd 0.0500 time 0.3712 (0.3800) data time 0.0011 (0.0031) model time 0.3700 (0.3774) loss 3.2503 (2.9276) grad_norm 1.8287 (2.0415) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:23:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][410/625] eta 0:01:21 lr 0.000477 wd 0.0500 time 0.3695 (0.3799) data time 0.0013 (0.0031) model time 0.3682 (0.3773) loss 3.1675 (2.9338) grad_norm 1.4556 (2.0353) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][420/625] eta 0:01:17 lr 0.000477 wd 0.0500 time 0.3722 (0.3797) data time 0.0009 (0.0030) model time 0.3713 (0.3772) loss 2.3422 (2.9327) grad_norm 3.8332 (2.0312) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][430/625] eta 0:01:14 lr 0.000477 wd 0.0500 time 0.3676 (0.3796) data time 0.0009 (0.0030) model time 0.3668 (0.3770) loss 2.5989 (2.9275) grad_norm 1.5501 (2.0277) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][440/625] eta 0:01:10 lr 0.000477 wd 0.0500 time 0.3747 (0.3794) data time 0.0012 (0.0030) model time 0.3735 (0.3769) loss 2.8737 (2.9252) grad_norm 1.8493 (2.0227) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][450/625] eta 0:01:06 lr 0.000477 wd 0.0500 time 0.3733 (0.3793) data time 0.0011 (0.0029) model time 0.3721 (0.3768) loss 3.0461 (2.9314) grad_norm 1.4521 (2.0200) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][460/625] eta 0:01:02 lr 0.000476 wd 0.0500 time 0.3695 (0.3792) data time 0.0012 (0.0029) model time 0.3683 (0.3767) loss 3.0669 (2.9258) grad_norm 2.0491 (2.0153) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][470/625] eta 0:00:58 lr 0.000476 wd 0.0500 time 0.3726 (0.3791) data time 0.0009 (0.0028) model time 0.3717 (0.3766) loss 3.0455 (2.9266) grad_norm 1.3811 (2.0146) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][480/625] eta 0:00:54 lr 0.000476 wd 0.0500 time 0.3778 (0.3790) data time 0.0009 (0.0028) model time 0.3769 (0.3765) loss 3.0995 (2.9259) grad_norm 1.4028 (2.0044) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][490/625] eta 0:00:51 lr 0.000476 wd 0.0500 time 0.3693 (0.3789) data time 0.0011 (0.0028) model time 0.3683 (0.3764) loss 2.5225 (2.9271) grad_norm 1.6176 (1.9983) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][500/625] eta 0:00:47 lr 0.000476 wd 0.0500 time 0.3703 (0.3787) data time 0.0008 (0.0027) model time 0.3694 (0.3763) loss 2.2789 (2.9243) grad_norm 2.4397 (1.9988) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][510/625] eta 0:00:43 lr 0.000476 wd 0.0500 time 0.3673 (0.3786) data time 0.0011 (0.0027) model time 0.3662 (0.3762) loss 3.4780 (2.9290) grad_norm 1.6555 (1.9952) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][520/625] eta 0:00:39 lr 0.000475 wd 0.0500 time 0.3769 (0.3785) data time 0.0011 (0.0027) model time 0.3758 (0.3761) loss 2.2731 (2.9261) grad_norm 1.6116 (1.9968) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][530/625] eta 0:00:35 lr 0.000475 wd 0.0500 time 0.3722 (0.3784) data time 0.0011 (0.0027) model time 0.3711 (0.3760) loss 3.2742 (2.9286) grad_norm 1.1865 (1.9947) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][540/625] eta 0:00:32 lr 0.000475 wd 0.0500 time 0.3633 (0.3783) data time 0.0008 (0.0026) model time 0.3625 (0.3759) loss 2.5693 (2.9237) grad_norm 2.9857 (2.0003) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][550/625] eta 0:00:28 lr 0.000475 wd 0.0500 time 0.3654 (0.3782) data time 0.0008 (0.0026) model time 0.3646 (0.3758) loss 2.2584 (2.9240) grad_norm 2.1042 (1.9982) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][560/625] eta 0:00:24 lr 0.000475 wd 0.0500 time 0.3779 (0.3780) data time 0.0008 (0.0026) model time 0.3770 (0.3757) loss 2.8818 (2.9205) grad_norm 1.5776 (2.0001) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:24:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][570/625] eta 0:00:20 lr 0.000475 wd 0.0500 time 0.5998 (0.3787) data time 0.0019 (0.0025) model time 0.5978 (0.3764) loss 3.0055 (2.9235) grad_norm 2.0300 (1.9988) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][580/625] eta 0:00:17 lr 0.000475 wd 0.0500 time 0.3713 (0.3789) data time 0.0011 (0.0025) model time 0.3702 (0.3766) loss 2.9003 (2.9198) grad_norm 2.1900 (1.9996) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][590/625] eta 0:00:13 lr 0.000474 wd 0.0500 time 0.3720 (0.3787) data time 0.0008 (0.0025) model time 0.3712 (0.3765) loss 2.6182 (2.9207) grad_norm 1.7174 (2.0011) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][600/625] eta 0:00:09 lr 0.000474 wd 0.0500 time 0.3931 (0.3786) data time 0.0012 (0.0025) model time 0.3920 (0.3764) loss 2.7024 (2.9219) grad_norm 1.7774 (2.0119) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][610/625] eta 0:00:05 lr 0.000474 wd 0.0500 time 0.3552 (0.3788) data time 0.0006 (0.0025) model time 0.3546 (0.3766) loss 1.8749 (2.9224) grad_norm 1.5379 (2.0121) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [210/300][620/625] eta 0:00:01 lr 0.000474 wd 0.0500 time 0.3620 (0.3785) data time 0.0006 (0.0024) model time 0.3614 (0.3763) loss 3.1735 (2.9189) grad_norm 1.5965 (2.0054) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:19 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 210 training takes 0:03:56 [2024-09-01 09:25:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:25:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:25:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.518 (0.518) Loss 0.5371 (0.5371) Acc@1 87.695 (87.695) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:25:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.8525 (0.6408) Acc@1 79.443 (85.578) Acc@5 95.801 (97.572) Mem 11225MB [2024-09-01 09:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.9844 (0.7672) Acc@1 75.879 (82.331) Acc@5 94.092 (96.159) Mem 11225MB [2024-09-01 09:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.124 Acc@5 96.171 [2024-09-01 09:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.1% [2024-09-01 09:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.12% [2024-09-01 09:25:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 09:25:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 09:25:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.562 (0.562) Loss 0.4905 (0.4905) Acc@1 89.893 (89.893) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 09:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.146) Loss 0.7900 (0.5998) Acc@1 81.641 (86.732) Acc@5 96.338 (97.758) Mem 11225MB [2024-09-01 09:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 0.8867 (0.7109) Acc@1 77.930 (83.798) Acc@5 95.215 (96.626) Mem 11225MB [2024-09-01 09:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.485 Acc@5 96.663 [2024-09-01 09:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.49% [2024-09-01 09:25:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:25:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:25:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][0/625] eta 0:08:30 lr 0.000474 wd 0.0500 time 0.8170 (0.8170) data time 0.4828 (0.4828) model time 0.0000 (0.0000) loss 2.8007 (2.8007) grad_norm 1.7133 (1.7133) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][10/625] eta 0:04:13 lr 0.000474 wd 0.0500 time 0.3710 (0.4125) data time 0.0012 (0.0450) model time 0.0000 (0.0000) loss 2.5794 (2.7989) grad_norm 1.6668 (1.5424) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][20/625] eta 0:03:58 lr 0.000474 wd 0.0500 time 0.3765 (0.3940) data time 0.0011 (0.0242) model time 0.0000 (0.0000) loss 2.8190 (2.8726) grad_norm 1.6375 (1.6004) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][30/625] eta 0:03:50 lr 0.000473 wd 0.0500 time 0.3672 (0.3878) data time 0.0012 (0.0168) model time 0.0000 (0.0000) loss 3.0285 (2.9572) grad_norm 1.5179 (1.5928) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][40/625] eta 0:03:44 lr 0.000473 wd 0.0500 time 0.3788 (0.3845) data time 0.0010 (0.0130) model time 0.0000 (0.0000) loss 1.9871 (2.8933) grad_norm 1.7839 (1.6888) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][50/625] eta 0:03:39 lr 0.000473 wd 0.0500 time 0.3718 (0.3818) data time 0.0009 (0.0107) model time 0.0000 (0.0000) loss 2.5992 (2.9537) grad_norm 1.7968 (1.8046) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][60/625] eta 0:03:34 lr 0.000473 wd 0.0500 time 0.3675 (0.3800) data time 0.0009 (0.0091) model time 0.3666 (0.3697) loss 3.5584 (2.9549) grad_norm 2.4680 (1.9699) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][70/625] eta 0:03:30 lr 0.000473 wd 0.0500 time 0.3718 (0.3789) data time 0.0014 (0.0080) model time 0.3704 (0.3702) loss 2.2518 (2.9641) grad_norm 1.8626 (1.9428) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][80/625] eta 0:03:26 lr 0.000473 wd 0.0500 time 0.3856 (0.3781) data time 0.0012 (0.0072) model time 0.3843 (0.3706) loss 3.3681 (2.9833) grad_norm 2.2147 (1.9355) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][90/625] eta 0:03:21 lr 0.000473 wd 0.0500 time 0.3695 (0.3773) data time 0.0011 (0.0065) model time 0.3685 (0.3702) loss 3.2191 (3.0158) grad_norm 2.0901 (1.9491) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][100/625] eta 0:03:17 lr 0.000472 wd 0.0500 time 0.3804 (0.3770) data time 0.0011 (0.0060) model time 0.3793 (0.3709) loss 3.5649 (3.0133) grad_norm 2.8869 (1.9653) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][110/625] eta 0:03:14 lr 0.000472 wd 0.0500 time 0.3697 (0.3785) data time 0.0008 (0.0056) model time 0.3689 (0.3744) loss 2.4066 (2.9978) grad_norm 2.4136 (1.9665) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][120/625] eta 0:03:10 lr 0.000472 wd 0.0500 time 0.3656 (0.3780) data time 0.0012 (0.0052) model time 0.3644 (0.3740) loss 2.5382 (2.9839) grad_norm 1.3941 (1.9451) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][130/625] eta 0:03:06 lr 0.000472 wd 0.0500 time 0.3653 (0.3778) data time 0.0013 (0.0049) model time 0.3640 (0.3740) loss 2.3403 (2.9667) grad_norm 1.2283 (1.9437) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][140/625] eta 0:03:02 lr 0.000472 wd 0.0500 time 0.3637 (0.3772) data time 0.0009 (0.0046) model time 0.3629 (0.3735) loss 2.8890 (2.9741) grad_norm 1.3096 (1.9322) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][150/625] eta 0:03:00 lr 0.000472 wd 0.0500 time 0.3786 (0.3810) data time 0.0009 (0.0044) model time 0.3777 (0.3795) loss 3.4941 (2.9761) grad_norm 1.7992 (1.9424) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][160/625] eta 0:02:56 lr 0.000472 wd 0.0500 time 0.3680 (0.3805) data time 0.0012 (0.0042) model time 0.3669 (0.3787) loss 2.7728 (2.9853) grad_norm 1.4663 (1.9473) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][170/625] eta 0:02:52 lr 0.000471 wd 0.0500 time 0.3681 (0.3800) data time 0.0013 (0.0040) model time 0.3668 (0.3781) loss 2.4831 (2.9882) grad_norm 2.8476 (1.9639) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][180/625] eta 0:02:48 lr 0.000471 wd 0.0500 time 0.3706 (0.3795) data time 0.0014 (0.0039) model time 0.3692 (0.3774) loss 2.5881 (2.9629) grad_norm 1.4905 (1.9585) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][190/625] eta 0:02:44 lr 0.000471 wd 0.0500 time 0.3762 (0.3792) data time 0.0011 (0.0037) model time 0.3750 (0.3770) loss 2.8203 (2.9467) grad_norm 1.6390 (1.9564) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][200/625] eta 0:02:40 lr 0.000471 wd 0.0500 time 0.3695 (0.3787) data time 0.0011 (0.0036) model time 0.3684 (0.3765) loss 3.4526 (2.9471) grad_norm 2.1921 (1.9629) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][210/625] eta 0:02:36 lr 0.000471 wd 0.0500 time 0.3737 (0.3783) data time 0.0008 (0.0035) model time 0.3728 (0.3760) loss 3.2085 (2.9509) grad_norm 3.2759 (1.9808) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][220/625] eta 0:02:33 lr 0.000471 wd 0.0500 time 0.3584 (0.3779) data time 0.0009 (0.0034) model time 0.3575 (0.3755) loss 2.6516 (2.9523) grad_norm inf (inf) loss_scale 512.0000 (1021.6833) mem 11225MB [2024-09-01 09:26:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][230/625] eta 0:02:29 lr 0.000470 wd 0.0500 time 0.3756 (0.3776) data time 0.0009 (0.0033) model time 0.3747 (0.3752) loss 1.8150 (2.9503) grad_norm 2.5938 (inf) loss_scale 512.0000 (999.6190) mem 11225MB [2024-09-01 09:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][240/625] eta 0:02:25 lr 0.000470 wd 0.0500 time 0.3637 (0.3773) data time 0.0012 (0.0032) model time 0.3625 (0.3749) loss 2.9036 (2.9578) grad_norm 2.4025 (inf) loss_scale 512.0000 (979.3859) mem 11225MB [2024-09-01 09:27:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][250/625] eta 0:02:21 lr 0.000470 wd 0.0500 time 0.3872 (0.3771) data time 0.0012 (0.0031) model time 0.3861 (0.3747) loss 2.8284 (2.9440) grad_norm 1.7651 (inf) loss_scale 512.0000 (960.7649) mem 11225MB [2024-09-01 09:27:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][260/625] eta 0:02:17 lr 0.000470 wd 0.0500 time 0.3681 (0.3769) data time 0.0012 (0.0030) model time 0.3669 (0.3745) loss 2.3400 (2.9443) grad_norm 1.7661 (inf) loss_scale 512.0000 (943.5709) mem 11225MB [2024-09-01 09:27:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][270/625] eta 0:02:13 lr 0.000470 wd 0.0500 time 0.3662 (0.3767) data time 0.0011 (0.0030) model time 0.3651 (0.3743) loss 2.9182 (2.9458) grad_norm 2.0280 (inf) loss_scale 512.0000 (927.6458) mem 11225MB [2024-09-01 09:27:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][280/625] eta 0:02:09 lr 0.000470 wd 0.0500 time 0.3649 (0.3765) data time 0.0014 (0.0029) model time 0.3635 (0.3742) loss 2.7260 (2.9425) grad_norm 2.0606 (inf) loss_scale 512.0000 (912.8541) mem 11225MB [2024-09-01 09:27:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][290/625] eta 0:02:06 lr 0.000470 wd 0.0500 time 0.3701 (0.3771) data time 0.0009 (0.0029) model time 0.3693 (0.3749) loss 3.5801 (2.9397) grad_norm 1.7740 (inf) loss_scale 512.0000 (899.0790) mem 11225MB [2024-09-01 09:27:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][300/625] eta 0:02:02 lr 0.000469 wd 0.0500 time 0.3665 (0.3769) data time 0.0013 (0.0028) model time 0.3652 (0.3747) loss 3.0262 (2.9361) grad_norm 1.4214 (inf) loss_scale 512.0000 (886.2193) mem 11225MB [2024-09-01 09:27:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][310/625] eta 0:01:58 lr 0.000469 wd 0.0500 time 0.3743 (0.3768) data time 0.0011 (0.0027) model time 0.3732 (0.3746) loss 2.7812 (2.9368) grad_norm 1.2618 (inf) loss_scale 512.0000 (874.1865) mem 11225MB [2024-09-01 09:27:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][320/625] eta 0:01:54 lr 0.000469 wd 0.0500 time 0.3704 (0.3767) data time 0.0011 (0.0027) model time 0.3694 (0.3745) loss 2.8283 (2.9278) grad_norm 3.7078 (inf) loss_scale 512.0000 (862.9034) mem 11225MB [2024-09-01 09:27:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][330/625] eta 0:01:51 lr 0.000469 wd 0.0500 time 0.3619 (0.3771) data time 0.0010 (0.0027) model time 0.3609 (0.3751) loss 2.8222 (2.9314) grad_norm 2.6698 (inf) loss_scale 512.0000 (852.3021) mem 11225MB [2024-09-01 09:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][340/625] eta 0:01:47 lr 0.000469 wd 0.0500 time 0.3721 (0.3783) data time 0.0008 (0.0026) model time 0.3713 (0.3764) loss 3.2177 (2.9276) grad_norm 2.3906 (inf) loss_scale 512.0000 (842.3226) mem 11225MB [2024-09-01 09:27:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][350/625] eta 0:01:43 lr 0.000469 wd 0.0500 time 0.3643 (0.3781) data time 0.0011 (0.0026) model time 0.3632 (0.3763) loss 3.4653 (2.9277) grad_norm 3.0329 (inf) loss_scale 512.0000 (832.9117) mem 11225MB [2024-09-01 09:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][360/625] eta 0:01:40 lr 0.000469 wd 0.0500 time 0.3815 (0.3779) data time 0.0011 (0.0025) model time 0.3804 (0.3761) loss 2.9611 (2.9297) grad_norm 1.6892 (inf) loss_scale 512.0000 (824.0222) mem 11225MB [2024-09-01 09:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][370/625] eta 0:01:36 lr 0.000468 wd 0.0500 time 0.3810 (0.3778) data time 0.0013 (0.0025) model time 0.3797 (0.3760) loss 2.8106 (2.9272) grad_norm 1.3570 (inf) loss_scale 512.0000 (815.6119) mem 11225MB [2024-09-01 09:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][380/625] eta 0:01:32 lr 0.000468 wd 0.0500 time 0.3724 (0.3777) data time 0.0012 (0.0025) model time 0.3712 (0.3758) loss 3.2519 (2.9270) grad_norm 1.2860 (inf) loss_scale 512.0000 (807.6430) mem 11225MB [2024-09-01 09:27:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][390/625] eta 0:01:28 lr 0.000468 wd 0.0500 time 0.3788 (0.3775) data time 0.0011 (0.0024) model time 0.3777 (0.3757) loss 2.6523 (2.9251) grad_norm 1.7844 (inf) loss_scale 512.0000 (800.0818) mem 11225MB [2024-09-01 09:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][400/625] eta 0:01:24 lr 0.000468 wd 0.0500 time 0.3728 (0.3774) data time 0.0011 (0.0024) model time 0.3716 (0.3755) loss 2.7488 (2.9186) grad_norm 1.7556 (inf) loss_scale 512.0000 (792.8978) mem 11225MB [2024-09-01 09:28:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][410/625] eta 0:01:21 lr 0.000468 wd 0.0500 time 0.3703 (0.3778) data time 0.0011 (0.0024) model time 0.3692 (0.3760) loss 2.7271 (2.9257) grad_norm 1.6723 (inf) loss_scale 512.0000 (786.0633) mem 11225MB [2024-09-01 09:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][420/625] eta 0:01:17 lr 0.000468 wd 0.0500 time 0.3673 (0.3776) data time 0.0011 (0.0023) model time 0.3662 (0.3758) loss 3.1045 (2.9267) grad_norm 1.9472 (inf) loss_scale 512.0000 (779.5534) mem 11225MB [2024-09-01 09:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][430/625] eta 0:01:13 lr 0.000468 wd 0.0500 time 0.3739 (0.3775) data time 0.0013 (0.0023) model time 0.3726 (0.3758) loss 2.3199 (2.9281) grad_norm 3.8027 (inf) loss_scale 512.0000 (773.3457) mem 11225MB [2024-09-01 09:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][440/625] eta 0:01:09 lr 0.000467 wd 0.0500 time 0.3685 (0.3774) data time 0.0009 (0.0023) model time 0.3676 (0.3756) loss 2.7927 (2.9317) grad_norm 2.2814 (inf) loss_scale 512.0000 (767.4195) mem 11225MB [2024-09-01 09:28:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][450/625] eta 0:01:06 lr 0.000467 wd 0.0500 time 0.3782 (0.3778) data time 0.0009 (0.0023) model time 0.3773 (0.3761) loss 2.8381 (2.9296) grad_norm 1.5929 (inf) loss_scale 512.0000 (761.7561) mem 11225MB [2024-09-01 09:28:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][460/625] eta 0:01:02 lr 0.000467 wd 0.0500 time 0.3710 (0.3786) data time 0.0009 (0.0022) model time 0.3701 (0.3770) loss 3.7439 (2.9318) grad_norm 1.9808 (inf) loss_scale 512.0000 (756.3384) mem 11225MB [2024-09-01 09:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][470/625] eta 0:00:58 lr 0.000467 wd 0.0500 time 0.3725 (0.3785) data time 0.0009 (0.0022) model time 0.3716 (0.3769) loss 3.0752 (2.9334) grad_norm 2.1400 (inf) loss_scale 512.0000 (751.1507) mem 11225MB [2024-09-01 09:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][480/625] eta 0:00:54 lr 0.000467 wd 0.0500 time 0.3706 (0.3783) data time 0.0013 (0.0022) model time 0.3693 (0.3767) loss 3.3778 (2.9381) grad_norm 2.0115 (inf) loss_scale 512.0000 (746.1788) mem 11225MB [2024-09-01 09:28:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][490/625] eta 0:00:51 lr 0.000467 wd 0.0500 time 0.3673 (0.3782) data time 0.0012 (0.0022) model time 0.3661 (0.3766) loss 2.9288 (2.9365) grad_norm 1.4636 (inf) loss_scale 512.0000 (741.4094) mem 11225MB [2024-09-01 09:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][500/625] eta 0:00:47 lr 0.000466 wd 0.0500 time 0.3798 (0.3781) data time 0.0012 (0.0022) model time 0.3786 (0.3765) loss 3.5527 (2.9402) grad_norm 1.8475 (inf) loss_scale 512.0000 (736.8303) mem 11225MB [2024-09-01 09:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][510/625] eta 0:00:43 lr 0.000466 wd 0.0500 time 0.3689 (0.3780) data time 0.0011 (0.0021) model time 0.3678 (0.3763) loss 3.0311 (2.9385) grad_norm 2.4362 (inf) loss_scale 512.0000 (732.4305) mem 11225MB [2024-09-01 09:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][520/625] eta 0:00:39 lr 0.000466 wd 0.0500 time 0.3859 (0.3779) data time 0.0011 (0.0021) model time 0.3847 (0.3763) loss 3.0412 (2.9418) grad_norm 2.8652 (inf) loss_scale 512.0000 (728.1996) mem 11225MB [2024-09-01 09:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][530/625] eta 0:00:35 lr 0.000466 wd 0.0500 time 0.3643 (0.3778) data time 0.0009 (0.0021) model time 0.3634 (0.3762) loss 2.9768 (2.9416) grad_norm 2.0443 (inf) loss_scale 512.0000 (724.1281) mem 11225MB [2024-09-01 09:28:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][540/625] eta 0:00:32 lr 0.000466 wd 0.0500 time 0.3716 (0.3777) data time 0.0011 (0.0021) model time 0.3705 (0.3761) loss 3.0383 (2.9421) grad_norm 2.4534 (inf) loss_scale 512.0000 (720.2070) mem 11225MB [2024-09-01 09:28:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][550/625] eta 0:00:28 lr 0.000466 wd 0.0500 time 0.3700 (0.3776) data time 0.0014 (0.0021) model time 0.3686 (0.3759) loss 3.1204 (2.9398) grad_norm 1.4500 (inf) loss_scale 512.0000 (716.4283) mem 11225MB [2024-09-01 09:29:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][560/625] eta 0:00:24 lr 0.000466 wd 0.0500 time 0.3604 (0.3775) data time 0.0011 (0.0021) model time 0.3593 (0.3758) loss 2.1642 (2.9385) grad_norm 1.8199 (inf) loss_scale 512.0000 (712.7843) mem 11225MB [2024-09-01 09:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][570/625] eta 0:00:20 lr 0.000465 wd 0.0500 time 0.3707 (0.3773) data time 0.0013 (0.0020) model time 0.3694 (0.3757) loss 3.3690 (2.9415) grad_norm 1.8122 (inf) loss_scale 512.0000 (709.2680) mem 11225MB [2024-09-01 09:29:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][580/625] eta 0:00:16 lr 0.000465 wd 0.0500 time 0.3755 (0.3772) data time 0.0009 (0.0020) model time 0.3746 (0.3756) loss 3.6826 (2.9452) grad_norm 5.0181 (inf) loss_scale 512.0000 (705.8726) mem 11225MB [2024-09-01 09:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][590/625] eta 0:00:13 lr 0.000465 wd 0.0500 time 0.3667 (0.3771) data time 0.0012 (0.0020) model time 0.3655 (0.3755) loss 2.9967 (2.9478) grad_norm 2.0881 (inf) loss_scale 512.0000 (702.5922) mem 11225MB [2024-09-01 09:29:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][600/625] eta 0:00:09 lr 0.000465 wd 0.0500 time 0.3752 (0.3770) data time 0.0012 (0.0020) model time 0.3740 (0.3754) loss 3.1114 (2.9461) grad_norm 1.4359 (inf) loss_scale 512.0000 (699.4210) mem 11225MB [2024-09-01 09:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][610/625] eta 0:00:05 lr 0.000465 wd 0.0500 time 0.3568 (0.3769) data time 0.0006 (0.0020) model time 0.3563 (0.3752) loss 3.4859 (2.9472) grad_norm 2.1467 (inf) loss_scale 512.0000 (696.3535) mem 11225MB [2024-09-01 09:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [211/300][620/625] eta 0:00:01 lr 0.000465 wd 0.0500 time 0.3563 (0.3770) data time 0.0006 (0.0020) model time 0.3557 (0.3753) loss 3.2156 (2.9494) grad_norm 1.6518 (inf) loss_scale 512.0000 (693.3849) mem 11225MB [2024-09-01 09:29:24 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 211 training takes 0:03:55 [2024-09-01 09:29:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:29:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:29:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.5254 (0.5254) Acc@1 88.477 (88.477) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:29:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.141) Loss 0.9312 (0.6564) Acc@1 77.930 (85.480) Acc@5 94.824 (97.377) Mem 11225MB [2024-09-01 09:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9224 (0.7734) Acc@1 78.369 (82.596) Acc@5 94.775 (96.143) Mem 11225MB [2024-09-01 09:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.310 Acc@5 96.191 [2024-09-01 09:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.3% [2024-09-01 09:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.31% [2024-09-01 09:29:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 09:29:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 09:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.518 (0.518) Loss 0.4915 (0.4915) Acc@1 89.795 (89.795) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 09:29:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.7896 (0.5997) Acc@1 81.299 (86.732) Acc@5 96.387 (97.754) Mem 11225MB [2024-09-01 09:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.8872 (0.7108) Acc@1 77.979 (83.796) Acc@5 95.264 (96.645) Mem 11225MB [2024-09-01 09:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.487 Acc@5 96.679 [2024-09-01 09:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:29:32 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.49% [2024-09-01 09:29:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:29:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:29:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][0/625] eta 0:08:02 lr 0.000465 wd 0.0500 time 0.7713 (0.7713) data time 0.4419 (0.4419) model time 0.0000 (0.0000) loss 3.1195 (3.1195) grad_norm 1.7296 (1.7296) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:29:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][10/625] eta 0:04:09 lr 0.000464 wd 0.0500 time 0.3724 (0.4062) data time 0.0009 (0.0412) model time 0.0000 (0.0000) loss 2.7539 (2.8475) grad_norm 2.6867 (2.8984) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:29:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][20/625] eta 0:04:08 lr 0.000464 wd 0.0500 time 0.3710 (0.4112) data time 0.0011 (0.0221) model time 0.0000 (0.0000) loss 3.2381 (2.9178) grad_norm 1.9443 (2.4431) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:29:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][30/625] eta 0:04:00 lr 0.000464 wd 0.0500 time 0.3642 (0.4048) data time 0.0009 (0.0154) model time 0.0000 (0.0000) loss 2.8850 (2.8940) grad_norm 1.2983 (2.3433) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:29:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][40/625] eta 0:03:52 lr 0.000464 wd 0.0500 time 0.3749 (0.3973) data time 0.0012 (0.0120) model time 0.0000 (0.0000) loss 3.7911 (2.8515) grad_norm 1.8224 (2.2323) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][50/625] eta 0:03:45 lr 0.000464 wd 0.0500 time 0.3705 (0.3924) data time 0.0015 (0.0099) model time 0.0000 (0.0000) loss 2.9393 (2.8805) grad_norm 1.6178 (2.2275) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][60/625] eta 0:03:41 lr 0.000464 wd 0.0500 time 0.3352 (0.3923) data time 0.0012 (0.0085) model time 0.3341 (0.3909) loss 3.4021 (2.8761) grad_norm 1.8661 (2.2585) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][70/625] eta 0:03:36 lr 0.000464 wd 0.0500 time 0.3956 (0.3895) data time 0.0012 (0.0074) model time 0.3944 (0.3810) loss 3.2024 (2.8463) grad_norm 4.1745 (2.3406) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][80/625] eta 0:03:30 lr 0.000463 wd 0.0500 time 0.3668 (0.3871) data time 0.0009 (0.0067) model time 0.3659 (0.3769) loss 2.1922 (2.8592) grad_norm 1.3922 (2.2976) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][90/625] eta 0:03:26 lr 0.000463 wd 0.0500 time 0.3573 (0.3852) data time 0.0011 (0.0061) model time 0.3563 (0.3749) loss 2.9537 (2.8435) grad_norm 1.6294 (2.2377) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][100/625] eta 0:03:21 lr 0.000463 wd 0.0500 time 0.3793 (0.3838) data time 0.0009 (0.0056) model time 0.3784 (0.3740) loss 1.7148 (2.8529) grad_norm 1.2645 (2.1805) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][110/625] eta 0:03:16 lr 0.000463 wd 0.0500 time 0.3669 (0.3825) data time 0.0008 (0.0052) model time 0.3660 (0.3728) loss 2.7965 (2.8436) grad_norm 1.6057 (2.1855) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][120/625] eta 0:03:12 lr 0.000463 wd 0.0500 time 0.3699 (0.3814) data time 0.0008 (0.0049) model time 0.3690 (0.3723) loss 3.2514 (2.8433) grad_norm 3.1304 (2.2216) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][130/625] eta 0:03:08 lr 0.000463 wd 0.0500 time 0.3707 (0.3808) data time 0.0016 (0.0046) model time 0.3692 (0.3722) loss 2.7069 (2.8326) grad_norm 1.6540 (2.1985) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][140/625] eta 0:03:04 lr 0.000463 wd 0.0500 time 0.3700 (0.3801) data time 0.0011 (0.0043) model time 0.3689 (0.3720) loss 2.9065 (2.8484) grad_norm 2.0872 (2.1981) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][150/625] eta 0:03:00 lr 0.000462 wd 0.0500 time 0.3716 (0.3798) data time 0.0011 (0.0041) model time 0.3705 (0.3722) loss 3.2054 (2.8453) grad_norm 1.5867 (2.1615) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][160/625] eta 0:02:56 lr 0.000462 wd 0.0500 time 0.3820 (0.3793) data time 0.0009 (0.0039) model time 0.3811 (0.3721) loss 1.8308 (2.8457) grad_norm 3.2829 (2.1642) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][170/625] eta 0:02:52 lr 0.000462 wd 0.0500 time 0.3675 (0.3787) data time 0.0010 (0.0038) model time 0.3665 (0.3718) loss 2.1474 (2.8464) grad_norm 1.6299 (2.1452) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][180/625] eta 0:02:48 lr 0.000462 wd 0.0500 time 0.3631 (0.3783) data time 0.0009 (0.0036) model time 0.3622 (0.3716) loss 3.6638 (2.8572) grad_norm 1.4965 (2.1157) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][190/625] eta 0:02:44 lr 0.000462 wd 0.0500 time 0.3736 (0.3782) data time 0.0013 (0.0035) model time 0.3723 (0.3718) loss 2.8200 (2.8619) grad_norm 1.6980 (2.1007) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][200/625] eta 0:02:40 lr 0.000462 wd 0.0500 time 0.3687 (0.3777) data time 0.0009 (0.0034) model time 0.3679 (0.3716) loss 3.1936 (2.8709) grad_norm 2.2994 (2.0869) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][210/625] eta 0:02:36 lr 0.000462 wd 0.0500 time 0.3734 (0.3774) data time 0.0009 (0.0033) model time 0.3725 (0.3714) loss 2.3048 (2.8740) grad_norm 1.9725 (2.1005) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][220/625] eta 0:02:32 lr 0.000461 wd 0.0500 time 0.3621 (0.3771) data time 0.0011 (0.0032) model time 0.3610 (0.3713) loss 3.2194 (2.8814) grad_norm 1.3136 (2.0977) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][230/625] eta 0:02:29 lr 0.000461 wd 0.0500 time 0.5844 (0.3786) data time 0.0012 (0.0031) model time 0.5832 (0.3735) loss 1.9157 (2.8727) grad_norm 1.7632 (2.0906) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][240/625] eta 0:02:25 lr 0.000461 wd 0.0500 time 0.3729 (0.3791) data time 0.0009 (0.0030) model time 0.3720 (0.3744) loss 3.2242 (2.8714) grad_norm 1.8700 (2.0921) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][250/625] eta 0:02:22 lr 0.000461 wd 0.0500 time 0.3727 (0.3788) data time 0.0009 (0.0030) model time 0.3719 (0.3742) loss 2.5266 (2.8731) grad_norm 10.4902 (2.1379) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][260/625] eta 0:02:18 lr 0.000461 wd 0.0500 time 0.3687 (0.3785) data time 0.0008 (0.0029) model time 0.3679 (0.3739) loss 3.0601 (2.8749) grad_norm 2.0750 (2.1567) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][270/625] eta 0:02:14 lr 0.000461 wd 0.0500 time 0.6114 (0.3790) data time 0.0011 (0.0028) model time 0.6103 (0.3747) loss 3.5076 (2.8866) grad_norm 1.4874 (2.1449) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][280/625] eta 0:02:10 lr 0.000460 wd 0.0500 time 0.3758 (0.3786) data time 0.0012 (0.0028) model time 0.3745 (0.3744) loss 3.2796 (2.8972) grad_norm 1.3960 (2.1296) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][290/625] eta 0:02:06 lr 0.000460 wd 0.0500 time 0.3661 (0.3785) data time 0.0013 (0.0027) model time 0.3649 (0.3743) loss 2.0360 (2.8908) grad_norm 1.9162 (2.1292) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][300/625] eta 0:02:02 lr 0.000460 wd 0.0500 time 0.3708 (0.3782) data time 0.0009 (0.0027) model time 0.3698 (0.3741) loss 3.6102 (2.8894) grad_norm 2.0395 (2.1182) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][310/625] eta 0:01:59 lr 0.000460 wd 0.0500 time 0.3683 (0.3779) data time 0.0009 (0.0026) model time 0.3674 (0.3738) loss 2.6462 (2.8778) grad_norm 1.5062 (2.1078) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][320/625] eta 0:01:55 lr 0.000460 wd 0.0500 time 0.3726 (0.3776) data time 0.0009 (0.0026) model time 0.3716 (0.3736) loss 2.6452 (2.8809) grad_norm 2.8388 (2.1102) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][330/625] eta 0:01:51 lr 0.000460 wd 0.0500 time 0.3669 (0.3774) data time 0.0011 (0.0025) model time 0.3658 (0.3735) loss 2.9399 (2.8724) grad_norm 1.2207 (2.1059) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][340/625] eta 0:01:47 lr 0.000460 wd 0.0500 time 0.3706 (0.3772) data time 0.0011 (0.0025) model time 0.3695 (0.3733) loss 2.7040 (2.8738) grad_norm 2.1179 (2.1050) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][350/625] eta 0:01:43 lr 0.000459 wd 0.0500 time 0.3634 (0.3770) data time 0.0008 (0.0025) model time 0.3626 (0.3731) loss 3.2114 (2.8733) grad_norm 2.3580 (2.0952) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][360/625] eta 0:01:39 lr 0.000459 wd 0.0500 time 0.3713 (0.3768) data time 0.0011 (0.0024) model time 0.3702 (0.3731) loss 2.9823 (2.8735) grad_norm 1.4850 (2.0822) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][370/625] eta 0:01:36 lr 0.000459 wd 0.0500 time 0.3700 (0.3766) data time 0.0009 (0.0024) model time 0.3692 (0.3729) loss 2.6139 (2.8720) grad_norm 1.7680 (2.0845) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:31:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][380/625] eta 0:01:32 lr 0.000459 wd 0.0500 time 0.3679 (0.3765) data time 0.0011 (0.0024) model time 0.3668 (0.3729) loss 3.2870 (2.8771) grad_norm 1.7512 (2.0778) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][390/625] eta 0:01:28 lr 0.000459 wd 0.0500 time 0.3618 (0.3764) data time 0.0011 (0.0023) model time 0.3607 (0.3728) loss 3.1254 (2.8788) grad_norm 2.8055 (2.1063) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][400/625] eta 0:01:24 lr 0.000459 wd 0.0500 time 0.3755 (0.3762) data time 0.0009 (0.0023) model time 0.3746 (0.3727) loss 3.0720 (2.8857) grad_norm 1.8878 (2.1061) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][410/625] eta 0:01:21 lr 0.000459 wd 0.0500 time 0.5986 (0.3770) data time 0.0009 (0.0023) model time 0.5977 (0.3736) loss 2.6132 (2.8831) grad_norm 1.3973 (2.0981) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][420/625] eta 0:01:17 lr 0.000458 wd 0.0500 time 0.3767 (0.3779) data time 0.0009 (0.0022) model time 0.3758 (0.3747) loss 3.4596 (2.8818) grad_norm 2.5202 (2.0915) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][430/625] eta 0:01:13 lr 0.000458 wd 0.0500 time 0.3726 (0.3777) data time 0.0011 (0.0022) model time 0.3715 (0.3746) loss 2.8219 (2.8827) grad_norm 1.6024 (2.0878) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][440/625] eta 0:01:09 lr 0.000458 wd 0.0500 time 0.3809 (0.3776) data time 0.0014 (0.0022) model time 0.3795 (0.3745) loss 3.0808 (2.8831) grad_norm 1.6867 (2.0836) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][450/625] eta 0:01:06 lr 0.000458 wd 0.0500 time 0.3665 (0.3774) data time 0.0012 (0.0022) model time 0.3652 (0.3743) loss 3.0729 (2.8821) grad_norm 2.1723 (2.0907) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][460/625] eta 0:01:02 lr 0.000458 wd 0.0500 time 0.3698 (0.3776) data time 0.0011 (0.0022) model time 0.3687 (0.3746) loss 2.9884 (2.8792) grad_norm 1.4547 (2.0907) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][470/625] eta 0:00:58 lr 0.000458 wd 0.0500 time 0.3729 (0.3775) data time 0.0009 (0.0021) model time 0.3720 (0.3745) loss 3.6329 (2.8825) grad_norm 2.3217 (2.0841) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][480/625] eta 0:00:54 lr 0.000458 wd 0.0500 time 0.3851 (0.3775) data time 0.0013 (0.0021) model time 0.3838 (0.3745) loss 3.0022 (2.8865) grad_norm 1.6611 (2.0797) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][490/625] eta 0:00:50 lr 0.000457 wd 0.0500 time 0.3674 (0.3773) data time 0.0014 (0.0021) model time 0.3660 (0.3744) loss 2.7379 (2.8825) grad_norm 1.6108 (2.0745) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][500/625] eta 0:00:47 lr 0.000457 wd 0.0500 time 0.3720 (0.3773) data time 0.0011 (0.0021) model time 0.3709 (0.3743) loss 2.2532 (2.8811) grad_norm 1.5446 (2.0710) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][510/625] eta 0:00:43 lr 0.000457 wd 0.0500 time 0.3702 (0.3771) data time 0.0009 (0.0021) model time 0.3693 (0.3742) loss 3.4629 (2.8831) grad_norm 1.9999 (2.0653) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][520/625] eta 0:00:39 lr 0.000457 wd 0.0500 time 0.3684 (0.3770) data time 0.0009 (0.0021) model time 0.3675 (0.3741) loss 3.5270 (2.8846) grad_norm 2.3533 (2.0670) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][530/625] eta 0:00:35 lr 0.000457 wd 0.0500 time 0.3694 (0.3773) data time 0.0012 (0.0020) model time 0.3683 (0.3745) loss 3.0781 (2.8805) grad_norm 4.2975 (2.0784) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:32:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][540/625] eta 0:00:32 lr 0.000457 wd 0.0500 time 0.3694 (0.3776) data time 0.0011 (0.0020) model time 0.3683 (0.3748) loss 3.3546 (2.8857) grad_norm 2.4376 (2.1008) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][550/625] eta 0:00:28 lr 0.000456 wd 0.0500 time 0.3677 (0.3774) data time 0.0009 (0.0020) model time 0.3668 (0.3747) loss 2.1896 (2.8889) grad_norm 1.3293 (2.0970) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][560/625] eta 0:00:24 lr 0.000456 wd 0.0500 time 0.3700 (0.3773) data time 0.0009 (0.0020) model time 0.3690 (0.3745) loss 1.6924 (2.8868) grad_norm 1.1715 (2.0914) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][570/625] eta 0:00:20 lr 0.000456 wd 0.0500 time 0.3655 (0.3771) data time 0.0011 (0.0020) model time 0.3644 (0.3744) loss 3.5109 (2.8837) grad_norm 1.8630 (2.0907) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][580/625] eta 0:00:16 lr 0.000456 wd 0.0500 time 0.3693 (0.3774) data time 0.0009 (0.0020) model time 0.3684 (0.3747) loss 3.2300 (2.8863) grad_norm 2.2883 (2.0916) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][590/625] eta 0:00:13 lr 0.000456 wd 0.0500 time 0.3738 (0.3773) data time 0.0013 (0.0020) model time 0.3725 (0.3746) loss 3.0536 (2.8856) grad_norm 1.8862 (2.0899) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][600/625] eta 0:00:09 lr 0.000456 wd 0.0500 time 0.3686 (0.3772) data time 0.0008 (0.0019) model time 0.3678 (0.3746) loss 2.8218 (2.8888) grad_norm 1.9071 (2.0928) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][610/625] eta 0:00:05 lr 0.000456 wd 0.0500 time 0.3595 (0.3771) data time 0.0008 (0.0019) model time 0.3587 (0.3745) loss 2.6727 (2.8871) grad_norm 1.7180 (2.0962) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [212/300][620/625] eta 0:00:01 lr 0.000455 wd 0.0500 time 0.3537 (0.3768) data time 0.0006 (0.0019) model time 0.3531 (0.3742) loss 2.9564 (2.8885) grad_norm 2.3372 (2.0999) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:29 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 212 training takes 0:03:55 [2024-09-01 09:33:29 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:33:30 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:33:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.549 (0.549) Loss 0.5288 (0.5288) Acc@1 88.574 (88.574) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 09:33:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 0.8486 (0.6376) Acc@1 80.029 (85.622) Acc@5 95.410 (97.536) Mem 11225MB [2024-09-01 09:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9102 (0.7610) Acc@1 78.516 (82.545) Acc@5 94.727 (96.215) Mem 11225MB [2024-09-01 09:33:33 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.308 Acc@5 96.225 [2024-09-01 09:33:33 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.3% [2024-09-01 09:33:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.872 (0.872) Loss 0.4912 (0.4912) Acc@1 89.844 (89.844) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 09:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.180) Loss 0.7905 (0.5991) Acc@1 81.152 (86.688) Acc@5 96.436 (97.758) Mem 11225MB [2024-09-01 09:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.139) Loss 0.8862 (0.7100) Acc@1 78.027 (83.801) Acc@5 95.312 (96.642) Mem 11225MB [2024-09-01 09:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.499 Acc@5 96.675 [2024-09-01 09:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.50% [2024-09-01 09:33:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:33:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:33:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][0/625] eta 0:08:13 lr 0.000455 wd 0.0500 time 0.7889 (0.7889) data time 0.4600 (0.4600) model time 0.0000 (0.0000) loss 3.0263 (3.0263) grad_norm 1.5757 (1.5757) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][10/625] eta 0:04:11 lr 0.000455 wd 0.0500 time 0.3857 (0.4083) data time 0.0009 (0.0430) model time 0.0000 (0.0000) loss 2.4543 (2.9611) grad_norm 3.0773 (2.7742) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][20/625] eta 0:03:55 lr 0.000455 wd 0.0500 time 0.3763 (0.3897) data time 0.0008 (0.0231) model time 0.0000 (0.0000) loss 3.1524 (2.9100) grad_norm 2.4323 (2.9279) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][30/625] eta 0:03:48 lr 0.000455 wd 0.0500 time 0.3728 (0.3841) data time 0.0011 (0.0160) model time 0.0000 (0.0000) loss 2.8951 (2.8639) grad_norm 1.7479 (2.6484) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][40/625] eta 0:03:42 lr 0.000455 wd 0.0500 time 0.3645 (0.3804) data time 0.0010 (0.0124) model time 0.0000 (0.0000) loss 2.0051 (2.8812) grad_norm 1.4892 (2.3893) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:33:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][50/625] eta 0:03:37 lr 0.000455 wd 0.0500 time 0.3667 (0.3785) data time 0.0011 (0.0102) model time 0.0000 (0.0000) loss 2.1397 (2.8991) grad_norm 1.6382 (2.2804) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][60/625] eta 0:03:33 lr 0.000454 wd 0.0500 time 0.3798 (0.3773) data time 0.0011 (0.0087) model time 0.3787 (0.3700) loss 2.7797 (2.8761) grad_norm 1.5602 (2.2061) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][70/625] eta 0:03:31 lr 0.000454 wd 0.0500 time 0.3637 (0.3817) data time 0.0012 (0.0077) model time 0.3625 (0.3884) loss 2.8294 (2.9364) grad_norm 1.7321 (2.1630) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][80/625] eta 0:03:27 lr 0.000454 wd 0.0500 time 0.3726 (0.3806) data time 0.0009 (0.0069) model time 0.3717 (0.3828) loss 3.4354 (2.9386) grad_norm 3.2300 (2.1793) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][90/625] eta 0:03:23 lr 0.000454 wd 0.0500 time 0.3725 (0.3799) data time 0.0011 (0.0063) model time 0.3714 (0.3804) loss 3.1500 (2.9020) grad_norm 1.6401 (2.1823) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][100/625] eta 0:03:19 lr 0.000454 wd 0.0500 time 0.3459 (0.3808) data time 0.0008 (0.0058) model time 0.3451 (0.3819) loss 2.8100 (2.9072) grad_norm 1.9058 (2.1732) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][110/625] eta 0:03:15 lr 0.000454 wd 0.0500 time 0.3772 (0.3803) data time 0.0010 (0.0054) model time 0.3763 (0.3805) loss 2.0275 (2.9210) grad_norm 1.6102 (2.1460) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][120/625] eta 0:03:11 lr 0.000454 wd 0.0500 time 0.3759 (0.3795) data time 0.0011 (0.0050) model time 0.3747 (0.3790) loss 2.7242 (2.9169) grad_norm 1.8075 (2.1454) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][130/625] eta 0:03:07 lr 0.000453 wd 0.0500 time 0.3696 (0.3787) data time 0.0017 (0.0047) model time 0.3679 (0.3776) loss 3.1758 (2.9150) grad_norm 1.8634 (2.1393) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][140/625] eta 0:03:03 lr 0.000453 wd 0.0500 time 0.3669 (0.3784) data time 0.0012 (0.0045) model time 0.3656 (0.3771) loss 3.3990 (2.9127) grad_norm 1.4773 (2.1109) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][150/625] eta 0:02:59 lr 0.000453 wd 0.0500 time 0.3641 (0.3779) data time 0.0015 (0.0043) model time 0.3626 (0.3764) loss 2.2718 (2.9182) grad_norm 1.4764 (2.0980) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][160/625] eta 0:02:55 lr 0.000453 wd 0.0500 time 0.3806 (0.3775) data time 0.0009 (0.0041) model time 0.3797 (0.3758) loss 3.5189 (2.9306) grad_norm 1.7258 (2.0744) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][170/625] eta 0:02:51 lr 0.000453 wd 0.0500 time 0.3709 (0.3771) data time 0.0008 (0.0039) model time 0.3701 (0.3752) loss 3.5997 (2.9366) grad_norm 1.9386 (2.0749) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][180/625] eta 0:02:47 lr 0.000453 wd 0.0500 time 0.3680 (0.3768) data time 0.0011 (0.0038) model time 0.3668 (0.3749) loss 3.5735 (2.9419) grad_norm 1.5732 (2.0624) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][190/625] eta 0:02:43 lr 0.000453 wd 0.0500 time 0.3678 (0.3766) data time 0.0008 (0.0036) model time 0.3669 (0.3746) loss 3.4908 (2.9485) grad_norm 1.6190 (2.0485) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][200/625] eta 0:02:39 lr 0.000452 wd 0.0500 time 0.3686 (0.3763) data time 0.0011 (0.0035) model time 0.3674 (0.3743) loss 2.7834 (2.9422) grad_norm 1.4519 (2.0402) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:34:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][210/625] eta 0:02:36 lr 0.000452 wd 0.0500 time 0.3662 (0.3761) data time 0.0009 (0.0034) model time 0.3653 (0.3741) loss 2.4356 (2.9300) grad_norm 1.4822 (2.0334) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][220/625] eta 0:02:32 lr 0.000452 wd 0.0500 time 0.3697 (0.3759) data time 0.0011 (0.0033) model time 0.3686 (0.3739) loss 2.9385 (2.9340) grad_norm 1.6971 (2.0239) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][230/625] eta 0:02:28 lr 0.000452 wd 0.0500 time 0.3741 (0.3758) data time 0.0014 (0.0032) model time 0.3727 (0.3738) loss 2.9865 (2.9318) grad_norm 3.8003 (2.0466) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][240/625] eta 0:02:24 lr 0.000452 wd 0.0500 time 0.3633 (0.3756) data time 0.0012 (0.0031) model time 0.3621 (0.3735) loss 3.5049 (2.9395) grad_norm 1.5748 (2.0536) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][250/625] eta 0:02:20 lr 0.000452 wd 0.0500 time 0.3641 (0.3754) data time 0.0010 (0.0030) model time 0.3631 (0.3734) loss 3.5628 (2.9381) grad_norm 1.8063 (2.0617) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][260/625] eta 0:02:16 lr 0.000452 wd 0.0500 time 0.3710 (0.3753) data time 0.0009 (0.0030) model time 0.3701 (0.3733) loss 3.5691 (2.9529) grad_norm 1.7780 (2.0441) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][270/625] eta 0:02:13 lr 0.000451 wd 0.0500 time 0.3850 (0.3752) data time 0.0011 (0.0029) model time 0.3839 (0.3732) loss 3.0783 (2.9571) grad_norm 1.8240 (2.0634) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][280/625] eta 0:02:09 lr 0.000451 wd 0.0500 time 0.3827 (0.3767) data time 0.0010 (0.0028) model time 0.3817 (0.3750) loss 2.8151 (2.9571) grad_norm 2.2079 (2.0652) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][290/625] eta 0:02:06 lr 0.000451 wd 0.0500 time 0.3742 (0.3770) data time 0.0009 (0.0028) model time 0.3733 (0.3755) loss 3.5210 (2.9623) grad_norm 2.2487 (2.0835) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][300/625] eta 0:02:02 lr 0.000451 wd 0.0500 time 0.3745 (0.3770) data time 0.0009 (0.0027) model time 0.3736 (0.3755) loss 3.4964 (2.9718) grad_norm 2.8408 (2.1076) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][310/625] eta 0:01:58 lr 0.000451 wd 0.0500 time 0.3880 (0.3770) data time 0.0014 (0.0027) model time 0.3866 (0.3755) loss 2.3925 (2.9654) grad_norm 1.9027 (2.1130) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][320/625] eta 0:01:54 lr 0.000451 wd 0.0500 time 0.3874 (0.3769) data time 0.0011 (0.0026) model time 0.3862 (0.3754) loss 3.8754 (2.9689) grad_norm 1.3739 (2.1062) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][330/625] eta 0:01:51 lr 0.000451 wd 0.0500 time 0.3671 (0.3768) data time 0.0008 (0.0026) model time 0.3662 (0.3752) loss 3.3421 (2.9619) grad_norm 2.0033 (2.0946) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][340/625] eta 0:01:47 lr 0.000450 wd 0.0500 time 0.3708 (0.3767) data time 0.0011 (0.0026) model time 0.3697 (0.3751) loss 2.9815 (2.9638) grad_norm 1.6490 (2.0865) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][350/625] eta 0:01:43 lr 0.000450 wd 0.0500 time 0.3735 (0.3767) data time 0.0009 (0.0025) model time 0.3726 (0.3751) loss 2.0426 (2.9570) grad_norm 2.0457 (2.0904) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][360/625] eta 0:01:39 lr 0.000450 wd 0.0500 time 0.3656 (0.3765) data time 0.0011 (0.0025) model time 0.3645 (0.3749) loss 3.5000 (2.9538) grad_norm 1.6280 (2.0808) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:35:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][370/625] eta 0:01:35 lr 0.000450 wd 0.0500 time 0.3754 (0.3764) data time 0.0011 (0.0024) model time 0.3743 (0.3748) loss 3.1217 (2.9482) grad_norm 1.6150 (2.1173) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][380/625] eta 0:01:32 lr 0.000450 wd 0.0500 time 0.3758 (0.3762) data time 0.0011 (0.0024) model time 0.3747 (0.3747) loss 3.1077 (2.9472) grad_norm 1.7987 (2.1247) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][390/625] eta 0:01:28 lr 0.000450 wd 0.0500 time 0.3741 (0.3762) data time 0.0011 (0.0024) model time 0.3730 (0.3746) loss 2.1931 (2.9429) grad_norm 2.1283 (2.1205) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][400/625] eta 0:01:24 lr 0.000450 wd 0.0500 time 0.3695 (0.3761) data time 0.0012 (0.0024) model time 0.3684 (0.3745) loss 2.8138 (2.9438) grad_norm 1.6168 (2.1136) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][410/625] eta 0:01:20 lr 0.000449 wd 0.0500 time 0.3691 (0.3765) data time 0.0012 (0.0023) model time 0.3680 (0.3750) loss 3.1807 (2.9434) grad_norm 2.4904 (2.1099) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][420/625] eta 0:01:17 lr 0.000449 wd 0.0500 time 0.3749 (0.3764) data time 0.0011 (0.0023) model time 0.3737 (0.3749) loss 2.0699 (2.9407) grad_norm 1.5855 (2.1039) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][430/625] eta 0:01:13 lr 0.000449 wd 0.0500 time 0.3743 (0.3763) data time 0.0010 (0.0023) model time 0.3733 (0.3748) loss 3.0444 (2.9394) grad_norm 3.1860 (2.1092) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][440/625] eta 0:01:09 lr 0.000449 wd 0.0500 time 0.3701 (0.3762) data time 0.0009 (0.0023) model time 0.3692 (0.3747) loss 3.1616 (2.9431) grad_norm 2.0807 (2.1078) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][450/625] eta 0:01:05 lr 0.000449 wd 0.0500 time 0.3655 (0.3760) data time 0.0009 (0.0022) model time 0.3647 (0.3745) loss 2.3306 (2.9392) grad_norm 1.8580 (2.1034) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][460/625] eta 0:01:02 lr 0.000449 wd 0.0500 time 0.3700 (0.3769) data time 0.0010 (0.0022) model time 0.3690 (0.3755) loss 2.1300 (2.9342) grad_norm 2.3083 (2.1013) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][470/625] eta 0:00:58 lr 0.000448 wd 0.0500 time 0.3689 (0.3773) data time 0.0011 (0.0022) model time 0.3678 (0.3759) loss 2.1622 (2.9293) grad_norm 1.6915 (2.0911) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][480/625] eta 0:00:54 lr 0.000448 wd 0.0500 time 0.3666 (0.3771) data time 0.0013 (0.0022) model time 0.3653 (0.3757) loss 3.5862 (2.9300) grad_norm 1.6215 (2.0928) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][490/625] eta 0:00:50 lr 0.000448 wd 0.0500 time 0.3713 (0.3770) data time 0.0011 (0.0021) model time 0.3703 (0.3756) loss 2.9435 (2.9285) grad_norm 1.6209 (2.0990) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][500/625] eta 0:00:47 lr 0.000448 wd 0.0500 time 0.3748 (0.3769) data time 0.0011 (0.0021) model time 0.3737 (0.3755) loss 3.3511 (2.9283) grad_norm 2.7111 (2.1077) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][510/625] eta 0:00:43 lr 0.000448 wd 0.0500 time 0.3683 (0.3769) data time 0.0013 (0.0021) model time 0.3670 (0.3755) loss 3.3627 (2.9264) grad_norm 1.6259 (2.1061) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][520/625] eta 0:00:39 lr 0.000448 wd 0.0500 time 0.3728 (0.3767) data time 0.0009 (0.0021) model time 0.3719 (0.3753) loss 3.1728 (2.9280) grad_norm 1.6048 (2.1040) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:36:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][530/625] eta 0:00:35 lr 0.000448 wd 0.0500 time 0.3676 (0.3767) data time 0.0011 (0.0021) model time 0.3665 (0.3753) loss 3.2653 (2.9268) grad_norm 2.0745 (2.1131) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][540/625] eta 0:00:32 lr 0.000447 wd 0.0500 time 0.3780 (0.3766) data time 0.0011 (0.0021) model time 0.3769 (0.3752) loss 2.5559 (2.9288) grad_norm 1.2577 (2.1081) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][550/625] eta 0:00:28 lr 0.000447 wd 0.0500 time 0.3702 (0.3766) data time 0.0009 (0.0020) model time 0.3692 (0.3752) loss 2.9836 (2.9284) grad_norm 2.4453 (2.1032) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][560/625] eta 0:00:24 lr 0.000447 wd 0.0500 time 0.3682 (0.3765) data time 0.0009 (0.0020) model time 0.3673 (0.3751) loss 2.2398 (2.9268) grad_norm 1.8866 (2.0976) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][570/625] eta 0:00:20 lr 0.000447 wd 0.0500 time 0.3693 (0.3764) data time 0.0009 (0.0020) model time 0.3684 (0.3750) loss 3.8499 (2.9265) grad_norm 2.7393 (2.0976) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][580/625] eta 0:00:16 lr 0.000447 wd 0.0500 time 0.5988 (0.3774) data time 0.0011 (0.0020) model time 0.5977 (0.3760) loss 3.3071 (2.9314) grad_norm 1.5473 (2.0983) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][590/625] eta 0:00:13 lr 0.000447 wd 0.0500 time 0.3719 (0.3773) data time 0.0012 (0.0020) model time 0.3707 (0.3759) loss 2.6794 (2.9308) grad_norm 1.4429 (2.0990) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][600/625] eta 0:00:09 lr 0.000447 wd 0.0500 time 0.3814 (0.3772) data time 0.0012 (0.0020) model time 0.3802 (0.3758) loss 3.1783 (2.9317) grad_norm 2.2623 (2.0949) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][610/625] eta 0:00:05 lr 0.000446 wd 0.0500 time 0.3646 (0.3770) data time 0.0006 (0.0020) model time 0.3640 (0.3757) loss 3.0412 (2.9311) grad_norm 1.6999 (2.0895) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [213/300][620/625] eta 0:00:01 lr 0.000446 wd 0.0500 time 0.3629 (0.3772) data time 0.0008 (0.0020) model time 0.3621 (0.3758) loss 2.3360 (2.9322) grad_norm 2.0556 (2.1059) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:33 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 213 training takes 0:03:55 [2024-09-01 09:37:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:37:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:37:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 0.5420 (0.5420) Acc@1 88.623 (88.623) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 09:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.099 (0.143) Loss 0.8867 (0.6500) Acc@1 79.492 (85.747) Acc@5 95.410 (97.492) Mem 11225MB [2024-09-01 09:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9478 (0.7705) Acc@1 76.367 (82.603) Acc@5 94.678 (96.166) Mem 11225MB [2024-09-01 09:37:37 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.294 Acc@5 96.163 [2024-09-01 09:37:37 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.3% [2024-09-01 09:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.876 (0.876) Loss 0.4917 (0.4917) Acc@1 89.746 (89.746) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 09:37:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.182) Loss 0.7905 (0.5988) Acc@1 81.152 (86.674) Acc@5 96.387 (97.776) Mem 11225MB [2024-09-01 09:37:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.140) Loss 0.8857 (0.7096) Acc@1 78.125 (83.803) Acc@5 95.312 (96.659) Mem 11225MB [2024-09-01 09:37:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.509 Acc@5 96.693 [2024-09-01 09:37:40 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:37:40 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.51% [2024-09-01 09:37:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:37:41 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][0/625] eta 0:08:28 lr 0.000446 wd 0.0500 time 0.8135 (0.8135) data time 0.4810 (0.4810) model time 0.0000 (0.0000) loss 3.0500 (3.0500) grad_norm 1.8828 (1.8828) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][10/625] eta 0:04:12 lr 0.000446 wd 0.0500 time 0.3681 (0.4108) data time 0.0012 (0.0453) model time 0.0000 (0.0000) loss 3.0753 (2.7390) grad_norm 1.6895 (2.0373) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][20/625] eta 0:03:57 lr 0.000446 wd 0.0500 time 0.3819 (0.3929) data time 0.0009 (0.0243) model time 0.0000 (0.0000) loss 2.2805 (2.8119) grad_norm 1.4126 (1.8339) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][30/625] eta 0:03:50 lr 0.000446 wd 0.0500 time 0.3724 (0.3870) data time 0.0011 (0.0168) model time 0.0000 (0.0000) loss 3.1189 (2.8380) grad_norm 7.7567 (2.0704) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:37:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][40/625] eta 0:03:44 lr 0.000446 wd 0.0500 time 0.3673 (0.3839) data time 0.0009 (0.0130) model time 0.0000 (0.0000) loss 3.1226 (2.8627) grad_norm 2.2842 (2.0655) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][50/625] eta 0:03:39 lr 0.000445 wd 0.0500 time 0.3672 (0.3812) data time 0.0011 (0.0107) model time 0.0000 (0.0000) loss 2.3273 (2.8138) grad_norm 2.6610 (2.0339) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][60/625] eta 0:03:34 lr 0.000445 wd 0.0500 time 0.3816 (0.3802) data time 0.0008 (0.0091) model time 0.3807 (0.3739) loss 3.4695 (2.8147) grad_norm 2.6600 (2.0469) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][70/625] eta 0:03:30 lr 0.000445 wd 0.0500 time 0.3657 (0.3788) data time 0.0011 (0.0080) model time 0.3646 (0.3716) loss 3.5770 (2.8049) grad_norm 3.4049 (2.4876) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][80/625] eta 0:03:27 lr 0.000445 wd 0.0500 time 0.3702 (0.3805) data time 0.0011 (0.0072) model time 0.3691 (0.3780) loss 3.0277 (2.8201) grad_norm 1.7356 (2.4722) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][90/625] eta 0:03:24 lr 0.000445 wd 0.0500 time 0.5687 (0.3817) data time 0.0008 (0.0065) model time 0.5679 (0.3811) loss 2.3148 (2.8256) grad_norm 2.1259 (2.4414) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][100/625] eta 0:03:22 lr 0.000445 wd 0.0500 time 0.3658 (0.3849) data time 0.0009 (0.0060) model time 0.3649 (0.3874) loss 3.0141 (2.8350) grad_norm 1.6952 (2.3779) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][110/625] eta 0:03:17 lr 0.000445 wd 0.0500 time 0.3717 (0.3837) data time 0.0010 (0.0056) model time 0.3707 (0.3847) loss 2.9964 (2.8495) grad_norm 4.7833 (2.5124) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][120/625] eta 0:03:13 lr 0.000444 wd 0.0500 time 0.3824 (0.3827) data time 0.0009 (0.0052) model time 0.3815 (0.3826) loss 2.3359 (2.8670) grad_norm 2.4689 (2.4764) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][130/625] eta 0:03:08 lr 0.000444 wd 0.0500 time 0.3766 (0.3818) data time 0.0011 (0.0049) model time 0.3755 (0.3810) loss 3.0540 (2.8766) grad_norm 1.7243 (2.4201) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][140/625] eta 0:03:04 lr 0.000444 wd 0.0500 time 0.3666 (0.3809) data time 0.0011 (0.0046) model time 0.3654 (0.3795) loss 3.0869 (2.8867) grad_norm 1.8918 (2.3769) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][150/625] eta 0:03:00 lr 0.000444 wd 0.0500 time 0.3607 (0.3802) data time 0.0012 (0.0044) model time 0.3595 (0.3785) loss 2.2132 (2.8775) grad_norm 1.5587 (2.3410) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][160/625] eta 0:02:56 lr 0.000444 wd 0.0500 time 0.3744 (0.3796) data time 0.0012 (0.0042) model time 0.3731 (0.3776) loss 2.3372 (2.8820) grad_norm 1.8743 (2.3360) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][170/625] eta 0:02:52 lr 0.000444 wd 0.0500 time 0.3716 (0.3791) data time 0.0009 (0.0041) model time 0.3707 (0.3770) loss 3.5884 (2.8837) grad_norm 1.3562 (2.3339) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][180/625] eta 0:02:48 lr 0.000444 wd 0.0500 time 0.3691 (0.3786) data time 0.0009 (0.0039) model time 0.3681 (0.3763) loss 3.9960 (2.8869) grad_norm 1.7496 (2.3075) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][190/625] eta 0:02:44 lr 0.000443 wd 0.0500 time 0.3759 (0.3783) data time 0.0010 (0.0038) model time 0.3749 (0.3760) loss 3.1731 (2.9031) grad_norm 1.8031 (2.2830) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][200/625] eta 0:02:40 lr 0.000443 wd 0.0500 time 0.3843 (0.3781) data time 0.0012 (0.0036) model time 0.3831 (0.3758) loss 2.2111 (2.8970) grad_norm 7.1173 (2.3046) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][210/625] eta 0:02:36 lr 0.000443 wd 0.0500 time 0.3696 (0.3777) data time 0.0011 (0.0035) model time 0.3685 (0.3754) loss 3.1026 (2.9072) grad_norm 1.8768 (2.2858) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][220/625] eta 0:02:32 lr 0.000443 wd 0.0500 time 0.3661 (0.3775) data time 0.0008 (0.0034) model time 0.3653 (0.3752) loss 2.8106 (2.9158) grad_norm 1.4668 (2.2607) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][230/625] eta 0:02:29 lr 0.000443 wd 0.0500 time 0.3711 (0.3772) data time 0.0012 (0.0033) model time 0.3699 (0.3749) loss 3.3503 (2.9151) grad_norm 1.6466 (2.2457) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][240/625] eta 0:02:25 lr 0.000443 wd 0.0500 time 0.3765 (0.3769) data time 0.0011 (0.0032) model time 0.3754 (0.3745) loss 3.0679 (2.9172) grad_norm 1.9050 (2.2226) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][250/625] eta 0:02:21 lr 0.000443 wd 0.0500 time 0.3755 (0.3766) data time 0.0011 (0.0031) model time 0.3744 (0.3742) loss 1.8697 (2.9099) grad_norm 1.6102 (2.2138) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][260/625] eta 0:02:17 lr 0.000442 wd 0.0500 time 0.3724 (0.3764) data time 0.0009 (0.0031) model time 0.3714 (0.3740) loss 3.3126 (2.9202) grad_norm 1.7245 (2.2040) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][270/625] eta 0:02:13 lr 0.000442 wd 0.0500 time 0.3602 (0.3762) data time 0.0011 (0.0030) model time 0.3590 (0.3739) loss 3.1463 (2.9220) grad_norm 1.4579 (2.2067) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][280/625] eta 0:02:09 lr 0.000442 wd 0.0500 time 0.3805 (0.3761) data time 0.0009 (0.0029) model time 0.3796 (0.3737) loss 3.2848 (2.9263) grad_norm 2.2056 (2.1954) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][290/625] eta 0:02:06 lr 0.000442 wd 0.0500 time 0.3688 (0.3766) data time 0.0009 (0.0029) model time 0.3680 (0.3744) loss 2.7063 (2.9212) grad_norm 1.4059 (2.1761) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][300/625] eta 0:02:02 lr 0.000442 wd 0.0500 time 0.3616 (0.3771) data time 0.0009 (0.0028) model time 0.3607 (0.3750) loss 3.3733 (2.9201) grad_norm 1.4734 (2.1650) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][310/625] eta 0:01:59 lr 0.000442 wd 0.0500 time 0.3710 (0.3783) data time 0.0011 (0.0028) model time 0.3699 (0.3765) loss 2.9999 (2.9151) grad_norm 1.6490 (2.1575) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][320/625] eta 0:01:55 lr 0.000442 wd 0.0500 time 0.3770 (0.3780) data time 0.0008 (0.0027) model time 0.3762 (0.3762) loss 3.9354 (2.9220) grad_norm 2.5202 (2.1571) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][330/625] eta 0:01:51 lr 0.000441 wd 0.0500 time 0.3684 (0.3779) data time 0.0012 (0.0027) model time 0.3672 (0.3761) loss 3.0308 (2.9198) grad_norm 2.3110 (2.1552) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][340/625] eta 0:01:47 lr 0.000441 wd 0.0500 time 0.3726 (0.3777) data time 0.0009 (0.0026) model time 0.3717 (0.3758) loss 2.2388 (2.9144) grad_norm 2.1753 (2.1582) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:39:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][350/625] eta 0:01:43 lr 0.000441 wd 0.0500 time 0.3869 (0.3776) data time 0.0009 (0.0026) model time 0.3861 (0.3758) loss 3.2836 (2.9132) grad_norm 1.9917 (2.1720) loss_scale 1024.0000 (520.7521) mem 11225MB [2024-09-01 09:39:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][360/625] eta 0:01:40 lr 0.000441 wd 0.0500 time 0.3784 (0.3775) data time 0.0010 (0.0026) model time 0.3774 (0.3757) loss 2.3783 (2.9088) grad_norm 1.4690 (2.1656) loss_scale 1024.0000 (534.6925) mem 11225MB [2024-09-01 09:40:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][370/625] eta 0:01:36 lr 0.000441 wd 0.0500 time 0.3719 (0.3773) data time 0.0008 (0.0025) model time 0.3711 (0.3754) loss 2.9633 (2.9111) grad_norm 2.5762 (2.1618) loss_scale 1024.0000 (547.8814) mem 11225MB [2024-09-01 09:40:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][380/625] eta 0:01:32 lr 0.000441 wd 0.0500 time 0.3812 (0.3771) data time 0.0009 (0.0025) model time 0.3803 (0.3753) loss 2.3898 (2.9116) grad_norm 1.7908 (2.1584) loss_scale 1024.0000 (560.3780) mem 11225MB [2024-09-01 09:40:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][390/625] eta 0:01:28 lr 0.000441 wd 0.0500 time 0.3737 (0.3770) data time 0.0011 (0.0024) model time 0.3726 (0.3752) loss 3.1923 (2.9118) grad_norm 3.4608 (2.1617) loss_scale 1024.0000 (572.2353) mem 11225MB [2024-09-01 09:40:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][400/625] eta 0:01:24 lr 0.000440 wd 0.0500 time 0.3747 (0.3769) data time 0.0009 (0.0024) model time 0.3738 (0.3751) loss 3.1844 (2.9052) grad_norm 1.5510 (2.1570) loss_scale 1024.0000 (583.5012) mem 11225MB [2024-09-01 09:40:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][410/625] eta 0:01:21 lr 0.000440 wd 0.0500 time 0.3700 (0.3768) data time 0.0013 (0.0024) model time 0.3688 (0.3750) loss 2.5879 (2.9072) grad_norm 2.1742 (2.1735) loss_scale 1024.0000 (594.2190) mem 11225MB [2024-09-01 09:40:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][420/625] eta 0:01:17 lr 0.000440 wd 0.0500 time 0.3655 (0.3767) data time 0.0009 (0.0024) model time 0.3646 (0.3749) loss 2.0379 (2.9077) grad_norm 1.6258 (2.1773) loss_scale 1024.0000 (604.4276) mem 11225MB [2024-09-01 09:40:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][430/625] eta 0:01:13 lr 0.000440 wd 0.0500 time 0.3561 (0.3765) data time 0.0009 (0.0023) model time 0.3553 (0.3747) loss 2.2196 (2.9091) grad_norm 4.5675 (2.1838) loss_scale 1024.0000 (614.1624) mem 11225MB [2024-09-01 09:40:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][440/625] eta 0:01:09 lr 0.000440 wd 0.0500 time 0.3747 (0.3764) data time 0.0011 (0.0023) model time 0.3736 (0.3746) loss 3.3846 (2.9134) grad_norm 2.3918 (2.1897) loss_scale 1024.0000 (623.4558) mem 11225MB [2024-09-01 09:40:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][450/625] eta 0:01:05 lr 0.000440 wd 0.0500 time 0.3741 (0.3763) data time 0.0009 (0.0023) model time 0.3732 (0.3745) loss 2.9701 (2.9130) grad_norm 3.2242 (2.1883) loss_scale 1024.0000 (632.3370) mem 11225MB [2024-09-01 09:40:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][460/625] eta 0:01:02 lr 0.000440 wd 0.0500 time 0.3719 (0.3762) data time 0.0011 (0.0023) model time 0.3708 (0.3743) loss 2.9355 (2.9087) grad_norm 3.0821 (2.1881) loss_scale 1024.0000 (640.8330) mem 11225MB [2024-09-01 09:40:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][470/625] eta 0:00:58 lr 0.000439 wd 0.0500 time 0.6043 (0.3766) data time 0.0011 (0.0022) model time 0.6033 (0.3748) loss 2.2721 (2.9110) grad_norm 2.8605 (2.1908) loss_scale 1024.0000 (648.9682) mem 11225MB [2024-09-01 09:40:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][480/625] eta 0:00:54 lr 0.000439 wd 0.0500 time 0.3674 (0.3764) data time 0.0009 (0.0022) model time 0.3666 (0.3746) loss 1.9500 (2.9091) grad_norm 2.2830 (2.1918) loss_scale 1024.0000 (656.7651) mem 11225MB [2024-09-01 09:40:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][490/625] eta 0:00:51 lr 0.000439 wd 0.0500 time 0.3732 (0.3780) data time 0.0011 (0.0022) model time 0.3721 (0.3764) loss 3.1069 (2.9082) grad_norm 2.3979 (2.1918) loss_scale 1024.0000 (664.2444) mem 11225MB [2024-09-01 09:40:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][500/625] eta 0:00:47 lr 0.000439 wd 0.0500 time 0.3676 (0.3778) data time 0.0009 (0.0022) model time 0.3667 (0.3763) loss 2.7214 (2.9101) grad_norm 1.7174 (2.2151) loss_scale 1024.0000 (671.4251) mem 11225MB [2024-09-01 09:40:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][510/625] eta 0:00:43 lr 0.000439 wd 0.0500 time 0.3682 (0.3777) data time 0.0011 (0.0022) model time 0.3671 (0.3762) loss 3.3047 (2.9065) grad_norm 1.8378 (2.2113) loss_scale 1024.0000 (678.3249) mem 11225MB [2024-09-01 09:40:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][520/625] eta 0:00:39 lr 0.000439 wd 0.0500 time 0.3885 (0.3777) data time 0.0014 (0.0021) model time 0.3871 (0.3761) loss 3.3457 (2.9021) grad_norm 1.5518 (2.2103) loss_scale 1024.0000 (684.9597) mem 11225MB [2024-09-01 09:41:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][530/625] eta 0:00:35 lr 0.000438 wd 0.0500 time 0.3748 (0.3776) data time 0.0013 (0.0021) model time 0.3735 (0.3760) loss 3.0919 (2.9034) grad_norm 2.7552 (2.2171) loss_scale 1024.0000 (691.3446) mem 11225MB [2024-09-01 09:41:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][540/625] eta 0:00:32 lr 0.000438 wd 0.0500 time 0.3763 (0.3775) data time 0.0011 (0.0021) model time 0.3752 (0.3759) loss 3.0319 (2.9046) grad_norm 1.6027 (2.2177) loss_scale 1024.0000 (697.4935) mem 11225MB [2024-09-01 09:41:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][550/625] eta 0:00:28 lr 0.000438 wd 0.0500 time 0.3713 (0.3774) data time 0.0012 (0.0021) model time 0.3701 (0.3758) loss 3.2039 (2.9012) grad_norm 2.0700 (2.2126) loss_scale 1024.0000 (703.4192) mem 11225MB [2024-09-01 09:41:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][560/625] eta 0:00:24 lr 0.000438 wd 0.0500 time 0.3666 (0.3772) data time 0.0008 (0.0021) model time 0.3657 (0.3756) loss 3.4015 (2.9034) grad_norm 2.0553 (2.2062) loss_scale 1024.0000 (709.1337) mem 11225MB [2024-09-01 09:41:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][570/625] eta 0:00:20 lr 0.000438 wd 0.0500 time 0.3757 (0.3772) data time 0.0011 (0.0020) model time 0.3746 (0.3756) loss 2.6921 (2.9035) grad_norm 2.1485 (2.2079) loss_scale 1024.0000 (714.6480) mem 11225MB [2024-09-01 09:41:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][580/625] eta 0:00:16 lr 0.000438 wd 0.0500 time 0.3750 (0.3771) data time 0.0011 (0.0020) model time 0.3739 (0.3755) loss 3.1202 (2.9011) grad_norm 1.6277 (2.2036) loss_scale 1024.0000 (719.9725) mem 11225MB [2024-09-01 09:41:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][590/625] eta 0:00:13 lr 0.000438 wd 0.0500 time 0.3334 (0.3774) data time 0.0012 (0.0020) model time 0.3322 (0.3758) loss 2.3396 (2.8997) grad_norm 1.5862 (2.1953) loss_scale 1024.0000 (725.1168) mem 11225MB [2024-09-01 09:41:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][600/625] eta 0:00:09 lr 0.000437 wd 0.0500 time 0.3731 (0.3772) data time 0.0008 (0.0020) model time 0.3722 (0.3757) loss 3.4153 (2.9035) grad_norm 2.2008 (2.1894) loss_scale 1024.0000 (730.0899) mem 11225MB [2024-09-01 09:41:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][610/625] eta 0:00:05 lr 0.000437 wd 0.0500 time 0.3583 (0.3782) data time 0.0006 (0.0020) model time 0.3577 (0.3767) loss 2.5849 (2.9075) grad_norm 1.8502 (2.1879) loss_scale 1024.0000 (734.9002) mem 11225MB [2024-09-01 09:41:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [214/300][620/625] eta 0:00:01 lr 0.000437 wd 0.0500 time 0.3677 (0.3779) data time 0.0008 (0.0020) model time 0.3669 (0.3764) loss 2.3696 (2.9052) grad_norm 1.5136 (2.1805) loss_scale 1024.0000 (739.5556) mem 11225MB [2024-09-01 09:41:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 214 training takes 0:03:56 [2024-09-01 09:41:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:41:39 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:41:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.513 (0.513) Loss 0.5488 (0.5488) Acc@1 88.721 (88.721) Acc@5 98.193 (98.193) Mem 11225MB [2024-09-01 09:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.142) Loss 0.8833 (0.6522) Acc@1 78.271 (85.578) Acc@5 95.020 (97.452) Mem 11225MB [2024-09-01 09:41:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9810 (0.7721) Acc@1 76.807 (82.575) Acc@5 94.189 (96.194) Mem 11225MB [2024-09-01 09:41:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.356 Acc@5 96.285 [2024-09-01 09:41:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.4% [2024-09-01 09:41:41 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.36% [2024-09-01 09:41:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 09:41:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 09:41:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.481 (0.481) Loss 0.4917 (0.4917) Acc@1 89.746 (89.746) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.139) Loss 0.7910 (0.5989) Acc@1 81.152 (86.648) Acc@5 96.484 (97.816) Mem 11225MB [2024-09-01 09:41:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.117) Loss 0.8853 (0.7093) Acc@1 78.223 (83.780) Acc@5 95.361 (96.694) Mem 11225MB [2024-09-01 09:41:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.489 Acc@5 96.717 [2024-09-01 09:41:45 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][0/625] eta 0:13:05 lr 0.000437 wd 0.0500 time 1.2572 (1.2572) data time 0.5464 (0.5464) model time 0.0000 (0.0000) loss 3.2512 (3.2512) grad_norm 2.1093 (2.1093) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:41:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][10/625] eta 0:04:38 lr 0.000437 wd 0.0500 time 0.3741 (0.4524) data time 0.0010 (0.0507) model time 0.0000 (0.0000) loss 2.7078 (2.8732) grad_norm 1.9055 (1.7404) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:41:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][20/625] eta 0:04:11 lr 0.000437 wd 0.0500 time 0.3780 (0.4153) data time 0.0011 (0.0272) model time 0.0000 (0.0000) loss 2.4275 (2.7734) grad_norm 1.5614 (1.6772) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:41:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][30/625] eta 0:03:59 lr 0.000437 wd 0.0500 time 0.3710 (0.4022) data time 0.0013 (0.0188) model time 0.0000 (0.0000) loss 3.2779 (2.8347) grad_norm 1.6489 (1.7946) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][40/625] eta 0:03:51 lr 0.000437 wd 0.0500 time 0.3653 (0.3956) data time 0.0009 (0.0145) model time 0.0000 (0.0000) loss 2.9674 (2.8672) grad_norm 1.5876 (1.7862) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][50/625] eta 0:03:44 lr 0.000436 wd 0.0500 time 0.3772 (0.3910) data time 0.0009 (0.0119) model time 0.0000 (0.0000) loss 3.3616 (2.8988) grad_norm 1.7437 (1.7668) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][60/625] eta 0:03:38 lr 0.000436 wd 0.0500 time 0.3664 (0.3875) data time 0.0012 (0.0101) model time 0.3652 (0.3684) loss 2.9508 (2.8801) grad_norm 1.6212 (1.7573) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][70/625] eta 0:03:33 lr 0.000436 wd 0.0500 time 0.3725 (0.3850) data time 0.0009 (0.0089) model time 0.3716 (0.3685) loss 2.9182 (2.8775) grad_norm 3.1360 (1.8220) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][80/625] eta 0:03:29 lr 0.000436 wd 0.0500 time 0.3836 (0.3837) data time 0.0011 (0.0079) model time 0.3825 (0.3700) loss 3.1904 (2.8531) grad_norm 1.7281 (1.8156) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][90/625] eta 0:03:24 lr 0.000436 wd 0.0500 time 0.3699 (0.3822) data time 0.0009 (0.0072) model time 0.3690 (0.3699) loss 3.3810 (2.8725) grad_norm 1.8440 (1.8359) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][100/625] eta 0:03:20 lr 0.000436 wd 0.0500 time 0.3717 (0.3816) data time 0.0013 (0.0066) model time 0.3704 (0.3709) loss 2.8634 (2.8775) grad_norm 2.2711 (1.8846) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][110/625] eta 0:03:16 lr 0.000436 wd 0.0500 time 0.3684 (0.3807) data time 0.0014 (0.0061) model time 0.3671 (0.3708) loss 3.3155 (2.8942) grad_norm 2.3536 (1.9327) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][120/625] eta 0:03:12 lr 0.000435 wd 0.0500 time 0.3711 (0.3811) data time 0.0008 (0.0057) model time 0.3703 (0.3727) loss 3.4724 (2.8955) grad_norm 1.6324 (1.9174) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][130/625] eta 0:03:08 lr 0.000435 wd 0.0500 time 0.3699 (0.3803) data time 0.0011 (0.0054) model time 0.3688 (0.3722) loss 3.1466 (2.8891) grad_norm 2.2147 (1.9124) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][140/625] eta 0:03:04 lr 0.000435 wd 0.0500 time 0.3885 (0.3801) data time 0.0009 (0.0050) model time 0.3876 (0.3728) loss 2.1545 (2.8826) grad_norm 1.7522 (1.8908) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 09:42:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][150/625] eta 0:03:00 lr 0.000435 wd 0.0500 time 0.3540 (0.3794) data time 0.0014 (0.0048) model time 0.3526 (0.3722) loss 3.0926 (2.8865) grad_norm inf (inf) loss_scale 512.0000 (1020.6093) mem 11225MB [2024-09-01 09:42:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][160/625] eta 0:02:56 lr 0.000435 wd 0.0500 time 0.3827 (0.3790) data time 0.0009 (0.0046) model time 0.3819 (0.3723) loss 2.4141 (2.8910) grad_norm 3.1575 (inf) loss_scale 512.0000 (989.0186) mem 11225MB [2024-09-01 09:42:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][170/625] eta 0:02:53 lr 0.000435 wd 0.0500 time 0.3855 (0.3824) data time 0.0011 (0.0044) model time 0.3843 (0.3776) loss 3.1154 (2.8882) grad_norm 1.3714 (inf) loss_scale 512.0000 (961.1228) mem 11225MB [2024-09-01 09:42:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][180/625] eta 0:02:49 lr 0.000435 wd 0.0500 time 0.3640 (0.3819) data time 0.0011 (0.0042) model time 0.3629 (0.3771) loss 2.3483 (2.8959) grad_norm 1.8733 (inf) loss_scale 512.0000 (936.3094) mem 11225MB [2024-09-01 09:42:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][190/625] eta 0:02:45 lr 0.000434 wd 0.0500 time 0.3880 (0.3816) data time 0.0011 (0.0040) model time 0.3869 (0.3769) loss 2.6157 (2.8941) grad_norm 2.4850 (inf) loss_scale 512.0000 (914.0942) mem 11225MB [2024-09-01 09:43:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][200/625] eta 0:02:41 lr 0.000434 wd 0.0500 time 0.3713 (0.3811) data time 0.0008 (0.0039) model time 0.3704 (0.3765) loss 3.2516 (2.8904) grad_norm 2.2744 (inf) loss_scale 512.0000 (894.0896) mem 11225MB [2024-09-01 09:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][210/625] eta 0:02:37 lr 0.000434 wd 0.0500 time 0.3687 (0.3805) data time 0.0009 (0.0038) model time 0.3678 (0.3760) loss 3.2878 (2.8976) grad_norm 1.9440 (inf) loss_scale 512.0000 (875.9810) mem 11225MB [2024-09-01 09:43:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][220/625] eta 0:02:33 lr 0.000434 wd 0.0500 time 0.3491 (0.3802) data time 0.0009 (0.0036) model time 0.3483 (0.3758) loss 3.3184 (2.9130) grad_norm 2.7717 (inf) loss_scale 512.0000 (859.5113) mem 11225MB [2024-09-01 09:43:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][230/625] eta 0:02:30 lr 0.000434 wd 0.0500 time 0.3679 (0.3799) data time 0.0009 (0.0035) model time 0.3670 (0.3755) loss 3.0662 (2.9182) grad_norm 2.7911 (inf) loss_scale 512.0000 (844.4675) mem 11225MB [2024-09-01 09:43:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][240/625] eta 0:02:26 lr 0.000434 wd 0.0500 time 0.3730 (0.3796) data time 0.0009 (0.0034) model time 0.3721 (0.3753) loss 1.7628 (2.9173) grad_norm 3.0383 (inf) loss_scale 512.0000 (830.6722) mem 11225MB [2024-09-01 09:43:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][250/625] eta 0:02:22 lr 0.000433 wd 0.0500 time 0.3739 (0.3792) data time 0.0011 (0.0034) model time 0.3728 (0.3750) loss 3.1450 (2.9034) grad_norm 1.5144 (inf) loss_scale 512.0000 (817.9761) mem 11225MB [2024-09-01 09:43:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][260/625] eta 0:02:18 lr 0.000433 wd 0.0500 time 0.3725 (0.3789) data time 0.0012 (0.0033) model time 0.3713 (0.3748) loss 2.8989 (2.9071) grad_norm 1.3448 (inf) loss_scale 512.0000 (806.2529) mem 11225MB [2024-09-01 09:43:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][270/625] eta 0:02:14 lr 0.000433 wd 0.0500 time 0.3750 (0.3787) data time 0.0011 (0.0032) model time 0.3739 (0.3746) loss 3.1141 (2.9147) grad_norm 2.1367 (inf) loss_scale 512.0000 (795.3948) mem 11225MB [2024-09-01 09:43:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][280/625] eta 0:02:10 lr 0.000433 wd 0.0500 time 0.3664 (0.3784) data time 0.0012 (0.0031) model time 0.3652 (0.3745) loss 3.0816 (2.9145) grad_norm 1.5326 (inf) loss_scale 512.0000 (785.3096) mem 11225MB [2024-09-01 09:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][290/625] eta 0:02:07 lr 0.000433 wd 0.0500 time 0.3742 (0.3803) data time 0.0008 (0.0031) model time 0.3734 (0.3768) loss 3.0288 (2.9185) grad_norm 2.7327 (inf) loss_scale 512.0000 (775.9175) mem 11225MB [2024-09-01 09:43:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][300/625] eta 0:02:03 lr 0.000433 wd 0.0500 time 0.3354 (0.3805) data time 0.0011 (0.0030) model time 0.3343 (0.3772) loss 2.4135 (2.9189) grad_norm 3.6393 (inf) loss_scale 512.0000 (767.1495) mem 11225MB [2024-09-01 09:43:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][310/625] eta 0:01:59 lr 0.000433 wd 0.0500 time 0.3722 (0.3801) data time 0.0010 (0.0029) model time 0.3711 (0.3768) loss 2.2730 (2.9118) grad_norm 1.9300 (inf) loss_scale 512.0000 (758.9453) mem 11225MB [2024-09-01 09:43:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][320/625] eta 0:01:55 lr 0.000432 wd 0.0500 time 0.3688 (0.3799) data time 0.0008 (0.0029) model time 0.3680 (0.3766) loss 3.3303 (2.9202) grad_norm 1.8870 (inf) loss_scale 512.0000 (751.2523) mem 11225MB [2024-09-01 09:43:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][330/625] eta 0:01:51 lr 0.000432 wd 0.0500 time 0.3700 (0.3796) data time 0.0010 (0.0028) model time 0.3690 (0.3764) loss 3.2675 (2.9249) grad_norm 1.8852 (inf) loss_scale 512.0000 (744.0242) mem 11225MB [2024-09-01 09:43:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][340/625] eta 0:01:48 lr 0.000432 wd 0.0500 time 0.3749 (0.3794) data time 0.0011 (0.0028) model time 0.3738 (0.3762) loss 3.2599 (2.9242) grad_norm 2.0917 (inf) loss_scale 512.0000 (737.2199) mem 11225MB [2024-09-01 09:43:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][350/625] eta 0:01:44 lr 0.000432 wd 0.0500 time 0.3706 (0.3791) data time 0.0010 (0.0027) model time 0.3695 (0.3759) loss 2.5565 (2.9237) grad_norm 1.7022 (inf) loss_scale 512.0000 (730.8034) mem 11225MB [2024-09-01 09:44:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][360/625] eta 0:01:40 lr 0.000432 wd 0.0500 time 0.3726 (0.3789) data time 0.0009 (0.0027) model time 0.3716 (0.3758) loss 3.5983 (2.9247) grad_norm 2.2041 (inf) loss_scale 512.0000 (724.7424) mem 11225MB [2024-09-01 09:44:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][370/625] eta 0:01:36 lr 0.000432 wd 0.0500 time 0.3715 (0.3788) data time 0.0009 (0.0027) model time 0.3706 (0.3757) loss 2.4222 (2.9234) grad_norm 1.9139 (inf) loss_scale 512.0000 (719.0081) mem 11225MB [2024-09-01 09:44:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][380/625] eta 0:01:32 lr 0.000432 wd 0.0500 time 0.3719 (0.3786) data time 0.0009 (0.0026) model time 0.3710 (0.3755) loss 1.8474 (2.9190) grad_norm 2.3174 (inf) loss_scale 512.0000 (713.5748) mem 11225MB [2024-09-01 09:44:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][390/625] eta 0:01:28 lr 0.000431 wd 0.0500 time 0.3716 (0.3785) data time 0.0011 (0.0026) model time 0.3705 (0.3754) loss 2.8172 (2.9146) grad_norm 1.6706 (inf) loss_scale 512.0000 (708.4194) mem 11225MB [2024-09-01 09:44:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][400/625] eta 0:01:25 lr 0.000431 wd 0.0500 time 0.3688 (0.3783) data time 0.0009 (0.0026) model time 0.3678 (0.3752) loss 2.9991 (2.9125) grad_norm 1.7225 (inf) loss_scale 512.0000 (703.5212) mem 11225MB [2024-09-01 09:44:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][410/625] eta 0:01:21 lr 0.000431 wd 0.0500 time 0.3654 (0.3781) data time 0.0009 (0.0025) model time 0.3645 (0.3751) loss 3.1906 (2.9126) grad_norm 1.5207 (inf) loss_scale 512.0000 (698.8613) mem 11225MB [2024-09-01 09:44:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][420/625] eta 0:01:17 lr 0.000431 wd 0.0500 time 0.3341 (0.3784) data time 0.0009 (0.0025) model time 0.3332 (0.3754) loss 2.8139 (2.9151) grad_norm 6.7276 (inf) loss_scale 512.0000 (694.4228) mem 11225MB [2024-09-01 09:44:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][430/625] eta 0:01:13 lr 0.000431 wd 0.0500 time 0.3776 (0.3782) data time 0.0009 (0.0025) model time 0.3767 (0.3752) loss 2.7423 (2.9150) grad_norm 1.9404 (inf) loss_scale 512.0000 (690.1903) mem 11225MB [2024-09-01 09:44:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][440/625] eta 0:01:09 lr 0.000431 wd 0.0500 time 0.3663 (0.3780) data time 0.0011 (0.0024) model time 0.3652 (0.3751) loss 2.6122 (2.9183) grad_norm 2.9104 (inf) loss_scale 512.0000 (686.1497) mem 11225MB [2024-09-01 09:44:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][450/625] eta 0:01:06 lr 0.000431 wd 0.0500 time 0.3810 (0.3778) data time 0.0010 (0.0024) model time 0.3800 (0.3750) loss 2.6192 (2.9135) grad_norm 2.8703 (inf) loss_scale 512.0000 (682.2882) mem 11225MB [2024-09-01 09:44:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][460/625] eta 0:01:02 lr 0.000430 wd 0.0500 time 0.3750 (0.3777) data time 0.0013 (0.0024) model time 0.3737 (0.3749) loss 2.6338 (2.9104) grad_norm 1.8581 (inf) loss_scale 512.0000 (678.5944) mem 11225MB [2024-09-01 09:44:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][470/625] eta 0:00:58 lr 0.000430 wd 0.0500 time 0.3756 (0.3776) data time 0.0009 (0.0024) model time 0.3747 (0.3748) loss 2.0551 (2.9089) grad_norm 1.6455 (inf) loss_scale 512.0000 (675.0573) mem 11225MB [2024-09-01 09:44:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][480/625] eta 0:00:54 lr 0.000430 wd 0.0500 time 0.3675 (0.3775) data time 0.0009 (0.0023) model time 0.3665 (0.3747) loss 3.3537 (2.9082) grad_norm 1.9166 (inf) loss_scale 512.0000 (671.6674) mem 11225MB [2024-09-01 09:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][490/625] eta 0:00:50 lr 0.000430 wd 0.0500 time 0.3762 (0.3775) data time 0.0012 (0.0023) model time 0.3750 (0.3747) loss 2.0975 (2.9028) grad_norm 1.7331 (inf) loss_scale 512.0000 (668.4155) mem 11225MB [2024-09-01 09:44:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][500/625] eta 0:00:47 lr 0.000430 wd 0.0500 time 0.3732 (0.3786) data time 0.0012 (0.0023) model time 0.3720 (0.3760) loss 3.0330 (2.9039) grad_norm 2.2229 (inf) loss_scale 512.0000 (665.2934) mem 11225MB [2024-09-01 09:44:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][510/625] eta 0:00:43 lr 0.000430 wd 0.0500 time 0.3668 (0.3784) data time 0.0011 (0.0023) model time 0.3657 (0.3758) loss 2.6895 (2.9036) grad_norm 1.6405 (inf) loss_scale 512.0000 (662.2935) mem 11225MB [2024-09-01 09:45:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][520/625] eta 0:00:39 lr 0.000430 wd 0.0500 time 0.3709 (0.3783) data time 0.0008 (0.0022) model time 0.3701 (0.3757) loss 3.3163 (2.9049) grad_norm 1.5282 (inf) loss_scale 512.0000 (659.4088) mem 11225MB [2024-09-01 09:45:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][530/625] eta 0:00:35 lr 0.000429 wd 0.0500 time 0.3718 (0.3781) data time 0.0011 (0.0022) model time 0.3707 (0.3756) loss 3.0584 (2.9092) grad_norm 1.7694 (inf) loss_scale 512.0000 (656.6328) mem 11225MB [2024-09-01 09:45:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][540/625] eta 0:00:32 lr 0.000429 wd 0.0500 time 0.3733 (0.3780) data time 0.0011 (0.0022) model time 0.3721 (0.3754) loss 3.3406 (2.9098) grad_norm 2.2370 (inf) loss_scale 512.0000 (653.9593) mem 11225MB [2024-09-01 09:45:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][550/625] eta 0:00:28 lr 0.000429 wd 0.0500 time 0.3715 (0.3779) data time 0.0010 (0.0022) model time 0.3705 (0.3753) loss 3.1969 (2.9096) grad_norm 1.7831 (inf) loss_scale 512.0000 (651.3829) mem 11225MB [2024-09-01 09:45:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][560/625] eta 0:00:24 lr 0.000429 wd 0.0500 time 0.3809 (0.3778) data time 0.0011 (0.0022) model time 0.3797 (0.3753) loss 2.0265 (2.9037) grad_norm 2.5295 (inf) loss_scale 512.0000 (648.8984) mem 11225MB [2024-09-01 09:45:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][570/625] eta 0:00:20 lr 0.000429 wd 0.0500 time 0.3678 (0.3777) data time 0.0010 (0.0021) model time 0.3668 (0.3752) loss 3.4151 (2.9051) grad_norm 2.4068 (inf) loss_scale 512.0000 (646.5009) mem 11225MB [2024-09-01 09:45:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][580/625] eta 0:00:16 lr 0.000429 wd 0.0500 time 0.3804 (0.3776) data time 0.0009 (0.0021) model time 0.3794 (0.3751) loss 2.3780 (2.9010) grad_norm 1.7784 (inf) loss_scale 512.0000 (644.1859) mem 11225MB [2024-09-01 09:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][590/625] eta 0:00:13 lr 0.000429 wd 0.0500 time 0.3763 (0.3775) data time 0.0013 (0.0021) model time 0.3750 (0.3751) loss 3.2309 (2.9046) grad_norm 1.3970 (inf) loss_scale 512.0000 (641.9492) mem 11225MB [2024-09-01 09:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][600/625] eta 0:00:09 lr 0.000428 wd 0.0500 time 0.3580 (0.3774) data time 0.0011 (0.0021) model time 0.3569 (0.3750) loss 3.0693 (2.9072) grad_norm 2.0585 (inf) loss_scale 512.0000 (639.7870) mem 11225MB [2024-09-01 09:45:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][610/625] eta 0:00:05 lr 0.000428 wd 0.0500 time 0.3643 (0.3773) data time 0.0006 (0.0021) model time 0.3638 (0.3749) loss 3.2446 (2.9072) grad_norm 2.0575 (inf) loss_scale 512.0000 (637.6956) mem 11225MB [2024-09-01 09:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [215/300][620/625] eta 0:00:01 lr 0.000428 wd 0.0500 time 0.3657 (0.3771) data time 0.0008 (0.0021) model time 0.3649 (0.3747) loss 2.3033 (2.9091) grad_norm 2.1224 (inf) loss_scale 512.0000 (635.6715) mem 11225MB [2024-09-01 09:45:41 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 215 training takes 0:03:55 [2024-09-01 09:45:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:45:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:45:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 0.5625 (0.5625) Acc@1 88.135 (88.135) Acc@5 98.340 (98.340) Mem 11225MB [2024-09-01 09:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 0.8965 (0.6632) Acc@1 78.809 (85.676) Acc@5 95.459 (97.505) Mem 11225MB [2024-09-01 09:45:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9897 (0.7813) Acc@1 77.197 (82.640) Acc@5 94.385 (96.205) Mem 11225MB [2024-09-01 09:45:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.308 Acc@5 96.213 [2024-09-01 09:45:45 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.3% [2024-09-01 09:45:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.843 (0.843) Loss 0.4922 (0.4922) Acc@1 89.648 (89.648) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:45:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.175) Loss 0.7910 (0.5984) Acc@1 81.006 (86.630) Acc@5 96.289 (97.812) Mem 11225MB [2024-09-01 09:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.8838 (0.7088) Acc@1 78.076 (83.770) Acc@5 95.459 (96.689) Mem 11225MB [2024-09-01 09:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.493 Acc@5 96.717 [2024-09-01 09:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:45:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][0/625] eta 0:12:51 lr 0.000428 wd 0.0500 time 1.2347 (1.2347) data time 0.5513 (0.5513) model time 0.0000 (0.0000) loss 3.2484 (3.2484) grad_norm 2.3416 (2.3416) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][10/625] eta 0:04:35 lr 0.000428 wd 0.0500 time 0.3672 (0.4483) data time 0.0009 (0.0513) model time 0.0000 (0.0000) loss 3.7361 (3.0035) grad_norm 4.1101 (2.0364) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:45:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][20/625] eta 0:04:08 lr 0.000428 wd 0.0500 time 0.3716 (0.4109) data time 0.0011 (0.0275) model time 0.0000 (0.0000) loss 2.9513 (3.0383) grad_norm 2.2027 (2.1619) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][30/625] eta 0:03:57 lr 0.000428 wd 0.0500 time 0.3880 (0.3993) data time 0.0009 (0.0190) model time 0.0000 (0.0000) loss 3.2455 (3.0760) grad_norm 1.6816 (2.1889) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][40/625] eta 0:03:49 lr 0.000428 wd 0.0500 time 0.3741 (0.3926) data time 0.0011 (0.0147) model time 0.0000 (0.0000) loss 3.4026 (3.0625) grad_norm 1.7914 (2.1119) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][50/625] eta 0:03:43 lr 0.000427 wd 0.0500 time 0.3728 (0.3893) data time 0.0008 (0.0120) model time 0.0000 (0.0000) loss 3.3136 (3.0489) grad_norm 1.5859 (2.0239) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][60/625] eta 0:03:40 lr 0.000427 wd 0.0500 time 0.5793 (0.3901) data time 0.0008 (0.0102) model time 0.5785 (0.3929) loss 2.9802 (3.0032) grad_norm 2.7078 (2.0091) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][70/625] eta 0:03:38 lr 0.000427 wd 0.0500 time 0.3691 (0.3937) data time 0.0011 (0.0090) model time 0.3680 (0.4039) loss 3.0848 (2.9913) grad_norm 1.9157 (2.2564) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][80/625] eta 0:03:33 lr 0.000427 wd 0.0500 time 0.3714 (0.3910) data time 0.0012 (0.0080) model time 0.3702 (0.3928) loss 3.2386 (2.9682) grad_norm 2.2297 (2.3020) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][90/625] eta 0:03:28 lr 0.000427 wd 0.0500 time 0.3684 (0.3890) data time 0.0012 (0.0073) model time 0.3672 (0.3874) loss 2.1449 (2.9691) grad_norm 2.4952 (2.2775) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][100/625] eta 0:03:23 lr 0.000427 wd 0.0500 time 0.3698 (0.3878) data time 0.0011 (0.0067) model time 0.3687 (0.3851) loss 2.9640 (2.9761) grad_norm 1.5808 (2.2098) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][110/625] eta 0:03:19 lr 0.000427 wd 0.0500 time 0.3858 (0.3866) data time 0.0011 (0.0062) model time 0.3847 (0.3831) loss 2.7413 (2.9743) grad_norm 2.6653 (2.2042) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][120/625] eta 0:03:14 lr 0.000426 wd 0.0500 time 0.3744 (0.3855) data time 0.0011 (0.0058) model time 0.3733 (0.3815) loss 3.2783 (2.9791) grad_norm 1.8211 (2.2135) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][130/625] eta 0:03:10 lr 0.000426 wd 0.0500 time 0.3810 (0.3845) data time 0.0011 (0.0054) model time 0.3799 (0.3803) loss 2.0154 (2.9699) grad_norm 1.9585 (2.2090) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][140/625] eta 0:03:06 lr 0.000426 wd 0.0500 time 0.3663 (0.3837) data time 0.0009 (0.0051) model time 0.3654 (0.3794) loss 2.4082 (2.9646) grad_norm 1.8427 (2.2101) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][150/625] eta 0:03:01 lr 0.000426 wd 0.0500 time 0.3670 (0.3830) data time 0.0012 (0.0049) model time 0.3658 (0.3787) loss 3.1943 (2.9684) grad_norm 1.8020 (2.1967) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][160/625] eta 0:02:57 lr 0.000426 wd 0.0500 time 0.3698 (0.3822) data time 0.0015 (0.0046) model time 0.3683 (0.3777) loss 2.7382 (2.9504) grad_norm 1.4948 (2.1726) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][170/625] eta 0:02:54 lr 0.000426 wd 0.0500 time 0.3784 (0.3828) data time 0.0011 (0.0044) model time 0.3772 (0.3788) loss 2.0420 (2.9361) grad_norm 1.7834 (2.1531) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][180/625] eta 0:02:50 lr 0.000426 wd 0.0500 time 0.3692 (0.3822) data time 0.0009 (0.0043) model time 0.3683 (0.3783) loss 2.7636 (2.9284) grad_norm 6.4101 (2.1587) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][190/625] eta 0:02:46 lr 0.000425 wd 0.0500 time 0.3754 (0.3818) data time 0.0011 (0.0041) model time 0.3743 (0.3779) loss 2.1745 (2.9146) grad_norm 1.6481 (2.1701) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][200/625] eta 0:02:42 lr 0.000425 wd 0.0500 time 0.3687 (0.3813) data time 0.0011 (0.0040) model time 0.3676 (0.3774) loss 2.3189 (2.9011) grad_norm 1.7964 (2.1613) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][210/625] eta 0:02:38 lr 0.000425 wd 0.0500 time 0.3709 (0.3809) data time 0.0011 (0.0038) model time 0.3698 (0.3770) loss 2.8025 (2.9005) grad_norm 2.9502 (2.1619) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][220/625] eta 0:02:34 lr 0.000425 wd 0.0500 time 0.3772 (0.3808) data time 0.0012 (0.0037) model time 0.3760 (0.3770) loss 2.2050 (2.8992) grad_norm 2.0553 (2.1624) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][230/625] eta 0:02:30 lr 0.000425 wd 0.0500 time 0.3719 (0.3803) data time 0.0011 (0.0036) model time 0.3708 (0.3766) loss 3.4326 (2.9027) grad_norm 1.6371 (2.1460) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][240/625] eta 0:02:26 lr 0.000425 wd 0.0500 time 0.3672 (0.3799) data time 0.0009 (0.0035) model time 0.3662 (0.3762) loss 3.2638 (2.9051) grad_norm 2.3588 (2.1390) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][250/625] eta 0:02:23 lr 0.000425 wd 0.0500 time 0.5521 (0.3819) data time 0.0011 (0.0034) model time 0.5510 (0.3789) loss 2.1964 (2.9007) grad_norm 9.5454 (2.1721) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][260/625] eta 0:02:19 lr 0.000424 wd 0.0500 time 0.3719 (0.3815) data time 0.0009 (0.0033) model time 0.3710 (0.3785) loss 3.2946 (2.9030) grad_norm 3.0954 (2.1914) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 09:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][270/625] eta 0:02:15 lr 0.000424 wd 0.0500 time 0.3669 (0.3811) data time 0.0011 (0.0032) model time 0.3658 (0.3780) loss 2.5834 (2.9039) grad_norm 1.6729 (inf) loss_scale 256.0000 (509.1661) mem 11225MB [2024-09-01 09:47:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][280/625] eta 0:02:11 lr 0.000424 wd 0.0500 time 0.3803 (0.3808) data time 0.0012 (0.0032) model time 0.3792 (0.3777) loss 2.8967 (2.9070) grad_norm 1.9756 (inf) loss_scale 256.0000 (500.1566) mem 11225MB [2024-09-01 09:47:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][290/625] eta 0:02:07 lr 0.000424 wd 0.0500 time 0.3804 (0.3812) data time 0.0010 (0.0031) model time 0.3795 (0.3783) loss 3.5677 (2.9103) grad_norm 1.5761 (inf) loss_scale 256.0000 (491.7663) mem 11225MB [2024-09-01 09:47:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][300/625] eta 0:02:03 lr 0.000424 wd 0.0500 time 0.3726 (0.3809) data time 0.0012 (0.0030) model time 0.3714 (0.3780) loss 3.2334 (2.9114) grad_norm 2.5834 (inf) loss_scale 256.0000 (483.9336) mem 11225MB [2024-09-01 09:47:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][310/625] eta 0:01:59 lr 0.000424 wd 0.0500 time 0.3695 (0.3807) data time 0.0012 (0.0030) model time 0.3683 (0.3778) loss 3.0948 (2.9076) grad_norm 1.6670 (inf) loss_scale 256.0000 (476.6045) mem 11225MB [2024-09-01 09:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][320/625] eta 0:01:56 lr 0.000424 wd 0.0500 time 0.3680 (0.3804) data time 0.0009 (0.0029) model time 0.3671 (0.3775) loss 2.0959 (2.9107) grad_norm 1.4903 (inf) loss_scale 256.0000 (469.7321) mem 11225MB [2024-09-01 09:47:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][330/625] eta 0:01:52 lr 0.000423 wd 0.0500 time 0.3707 (0.3801) data time 0.0011 (0.0029) model time 0.3696 (0.3772) loss 2.9697 (2.9040) grad_norm 1.6799 (inf) loss_scale 256.0000 (463.2749) mem 11225MB [2024-09-01 09:47:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][340/625] eta 0:01:48 lr 0.000423 wd 0.0500 time 0.3669 (0.3798) data time 0.0013 (0.0028) model time 0.3656 (0.3769) loss 3.2590 (2.9059) grad_norm 1.6643 (inf) loss_scale 256.0000 (457.1965) mem 11225MB [2024-09-01 09:48:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][350/625] eta 0:01:44 lr 0.000423 wd 0.0500 time 0.3737 (0.3795) data time 0.0009 (0.0028) model time 0.3728 (0.3767) loss 3.0073 (2.9109) grad_norm 1.6840 (inf) loss_scale 256.0000 (451.4644) mem 11225MB [2024-09-01 09:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][360/625] eta 0:01:40 lr 0.000423 wd 0.0500 time 0.3613 (0.3792) data time 0.0009 (0.0027) model time 0.3603 (0.3763) loss 2.4649 (2.9046) grad_norm 1.6802 (inf) loss_scale 256.0000 (446.0499) mem 11225MB [2024-09-01 09:48:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][370/625] eta 0:01:37 lr 0.000423 wd 0.0500 time 0.5800 (0.3806) data time 0.0008 (0.0027) model time 0.5792 (0.3780) loss 2.5746 (2.9048) grad_norm 1.8468 (inf) loss_scale 256.0000 (440.9272) mem 11225MB [2024-09-01 09:48:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][380/625] eta 0:01:33 lr 0.000423 wd 0.0500 time 0.3818 (0.3805) data time 0.0011 (0.0026) model time 0.3807 (0.3779) loss 3.1573 (2.9051) grad_norm 1.8819 (inf) loss_scale 256.0000 (436.0735) mem 11225MB [2024-09-01 09:48:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][390/625] eta 0:01:29 lr 0.000422 wd 0.0500 time 0.3589 (0.3802) data time 0.0009 (0.0026) model time 0.3580 (0.3776) loss 2.6487 (2.9029) grad_norm 1.5887 (inf) loss_scale 256.0000 (431.4680) mem 11225MB [2024-09-01 09:48:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][400/625] eta 0:01:25 lr 0.000422 wd 0.0500 time 0.3724 (0.3800) data time 0.0008 (0.0026) model time 0.3716 (0.3775) loss 3.1458 (2.8988) grad_norm 1.6692 (inf) loss_scale 256.0000 (427.0923) mem 11225MB [2024-09-01 09:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][410/625] eta 0:01:21 lr 0.000422 wd 0.0500 time 0.3723 (0.3798) data time 0.0012 (0.0025) model time 0.3711 (0.3773) loss 3.1348 (2.8961) grad_norm 1.8305 (inf) loss_scale 256.0000 (422.9294) mem 11225MB [2024-09-01 09:48:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][420/625] eta 0:01:17 lr 0.000422 wd 0.0500 time 0.3692 (0.3796) data time 0.0010 (0.0025) model time 0.3682 (0.3770) loss 3.6152 (2.8944) grad_norm 2.3565 (inf) loss_scale 256.0000 (418.9644) mem 11225MB [2024-09-01 09:48:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][430/625] eta 0:01:13 lr 0.000422 wd 0.0500 time 0.3717 (0.3794) data time 0.0015 (0.0025) model time 0.3702 (0.3768) loss 2.8034 (2.8961) grad_norm 4.4752 (inf) loss_scale 256.0000 (415.1833) mem 11225MB [2024-09-01 09:48:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][440/625] eta 0:01:10 lr 0.000422 wd 0.0500 time 0.3654 (0.3792) data time 0.0009 (0.0025) model time 0.3645 (0.3767) loss 3.6249 (2.8998) grad_norm 2.8901 (inf) loss_scale 256.0000 (411.5737) mem 11225MB [2024-09-01 09:48:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][450/625] eta 0:01:06 lr 0.000422 wd 0.0500 time 0.3668 (0.3790) data time 0.0012 (0.0024) model time 0.3656 (0.3765) loss 2.6685 (2.8997) grad_norm 1.8354 (inf) loss_scale 256.0000 (408.1242) mem 11225MB [2024-09-01 09:48:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][460/625] eta 0:01:02 lr 0.000421 wd 0.0500 time 0.3739 (0.3789) data time 0.0008 (0.0024) model time 0.3731 (0.3764) loss 3.4595 (2.9003) grad_norm 1.9435 (inf) loss_scale 256.0000 (404.8243) mem 11225MB [2024-09-01 09:48:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][470/625] eta 0:00:58 lr 0.000421 wd 0.0500 time 0.3739 (0.3788) data time 0.0009 (0.0024) model time 0.3730 (0.3763) loss 2.5744 (2.8948) grad_norm 2.1030 (inf) loss_scale 256.0000 (401.6645) mem 11225MB [2024-09-01 09:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][480/625] eta 0:00:54 lr 0.000421 wd 0.0500 time 0.3689 (0.3787) data time 0.0012 (0.0023) model time 0.3677 (0.3762) loss 2.5035 (2.8917) grad_norm 1.8796 (inf) loss_scale 256.0000 (398.6362) mem 11225MB [2024-09-01 09:48:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][490/625] eta 0:00:51 lr 0.000421 wd 0.0500 time 0.3616 (0.3785) data time 0.0011 (0.0023) model time 0.3605 (0.3760) loss 3.3162 (2.8889) grad_norm 2.5725 (inf) loss_scale 256.0000 (395.7312) mem 11225MB [2024-09-01 09:48:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][500/625] eta 0:00:47 lr 0.000421 wd 0.0500 time 0.3681 (0.3788) data time 0.0012 (0.0023) model time 0.3670 (0.3764) loss 3.4091 (2.8890) grad_norm 1.7524 (inf) loss_scale 256.0000 (392.9421) mem 11225MB [2024-09-01 09:49:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][510/625] eta 0:00:43 lr 0.000421 wd 0.0500 time 0.3741 (0.3786) data time 0.0009 (0.0023) model time 0.3732 (0.3763) loss 3.0862 (2.8921) grad_norm 1.6500 (inf) loss_scale 256.0000 (390.2622) mem 11225MB [2024-09-01 09:49:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][520/625] eta 0:00:39 lr 0.000421 wd 0.0500 time 0.3739 (0.3785) data time 0.0012 (0.0023) model time 0.3727 (0.3761) loss 3.0746 (2.8990) grad_norm 1.6807 (inf) loss_scale 256.0000 (387.6852) mem 11225MB [2024-09-01 09:49:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][530/625] eta 0:00:35 lr 0.000420 wd 0.0500 time 0.3783 (0.3784) data time 0.0011 (0.0022) model time 0.3772 (0.3760) loss 3.1680 (2.8960) grad_norm 2.8969 (inf) loss_scale 256.0000 (385.2053) mem 11225MB [2024-09-01 09:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][540/625] eta 0:00:32 lr 0.000420 wd 0.0500 time 0.3726 (0.3782) data time 0.0009 (0.0022) model time 0.3716 (0.3759) loss 3.8804 (2.9007) grad_norm 1.4219 (inf) loss_scale 256.0000 (382.8170) mem 11225MB [2024-09-01 09:49:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][550/625] eta 0:00:28 lr 0.000420 wd 0.0500 time 0.3736 (0.3781) data time 0.0008 (0.0022) model time 0.3728 (0.3757) loss 3.4612 (2.9014) grad_norm 2.1650 (inf) loss_scale 256.0000 (380.5154) mem 11225MB [2024-09-01 09:49:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][560/625] eta 0:00:24 lr 0.000420 wd 0.0500 time 0.3590 (0.3780) data time 0.0009 (0.0022) model time 0.3581 (0.3756) loss 2.1030 (2.8977) grad_norm 1.6285 (inf) loss_scale 256.0000 (378.2959) mem 11225MB [2024-09-01 09:49:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][570/625] eta 0:00:20 lr 0.000420 wd 0.0500 time 0.3641 (0.3778) data time 0.0011 (0.0022) model time 0.3630 (0.3755) loss 1.7922 (2.8987) grad_norm 1.6515 (inf) loss_scale 256.0000 (376.1541) mem 11225MB [2024-09-01 09:49:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][580/625] eta 0:00:17 lr 0.000420 wd 0.0500 time 0.3703 (0.3788) data time 0.0009 (0.0021) model time 0.3694 (0.3766) loss 1.9390 (2.8929) grad_norm 1.7130 (inf) loss_scale 256.0000 (374.0861) mem 11225MB [2024-09-01 09:49:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][590/625] eta 0:00:13 lr 0.000420 wd 0.0500 time 0.3791 (0.3787) data time 0.0008 (0.0021) model time 0.3783 (0.3765) loss 3.4604 (2.8937) grad_norm 1.4081 (inf) loss_scale 256.0000 (372.0880) mem 11225MB [2024-09-01 09:49:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][600/625] eta 0:00:09 lr 0.000419 wd 0.0500 time 0.3731 (0.3786) data time 0.0010 (0.0021) model time 0.3721 (0.3764) loss 1.7562 (2.8931) grad_norm 1.5080 (inf) loss_scale 256.0000 (370.1564) mem 11225MB [2024-09-01 09:49:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][610/625] eta 0:00:05 lr 0.000419 wd 0.0500 time 0.3583 (0.3785) data time 0.0006 (0.0021) model time 0.3577 (0.3763) loss 2.7434 (2.8932) grad_norm 1.5135 (inf) loss_scale 256.0000 (368.2881) mem 11225MB [2024-09-01 09:49:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [216/300][620/625] eta 0:00:01 lr 0.000419 wd 0.0500 time 0.3632 (0.3782) data time 0.0007 (0.0021) model time 0.3625 (0.3760) loss 2.3187 (2.8934) grad_norm 1.9948 (inf) loss_scale 256.0000 (366.4799) mem 11225MB [2024-09-01 09:49:45 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 216 training takes 0:03:56 [2024-09-01 09:49:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:49:46 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:49:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.522 (0.522) Loss 0.5391 (0.5391) Acc@1 88.428 (88.428) Acc@5 98.096 (98.096) Mem 11225MB [2024-09-01 09:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.142) Loss 0.8521 (0.6388) Acc@1 79.980 (85.662) Acc@5 95.703 (97.567) Mem 11225MB [2024-09-01 09:49:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9434 (0.7634) Acc@1 77.295 (82.636) Acc@5 94.678 (96.266) Mem 11225MB [2024-09-01 09:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.358 Acc@5 96.273 [2024-09-01 09:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.4% [2024-09-01 09:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.36% [2024-09-01 09:49:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 09:49:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 09:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.504 (0.504) Loss 0.4924 (0.4924) Acc@1 89.551 (89.551) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:49:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 0.7900 (0.5978) Acc@1 81.152 (86.594) Acc@5 96.240 (97.825) Mem 11225MB [2024-09-01 09:49:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.8838 (0.7083) Acc@1 77.930 (83.777) Acc@5 95.557 (96.691) Mem 11225MB [2024-09-01 09:49:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.503 Acc@5 96.725 [2024-09-01 09:49:53 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][0/625] eta 0:12:38 lr 0.000419 wd 0.0500 time 1.2141 (1.2141) data time 0.5035 (0.5035) model time 0.0000 (0.0000) loss 3.1898 (3.1898) grad_norm 2.1569 (2.1569) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:49:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][10/625] eta 0:04:46 lr 0.000419 wd 0.0500 time 0.3677 (0.4660) data time 0.0009 (0.0469) model time 0.0000 (0.0000) loss 2.0673 (2.6172) grad_norm 1.8050 (1.9042) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][20/625] eta 0:04:14 lr 0.000419 wd 0.0500 time 0.3727 (0.4212) data time 0.0009 (0.0251) model time 0.0000 (0.0000) loss 3.4162 (2.7709) grad_norm 1.7521 (1.8541) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][30/625] eta 0:04:00 lr 0.000419 wd 0.0500 time 0.3642 (0.4041) data time 0.0009 (0.0174) model time 0.0000 (0.0000) loss 3.5846 (2.8266) grad_norm 1.9347 (2.0537) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][40/625] eta 0:03:51 lr 0.000419 wd 0.0500 time 0.3775 (0.3958) data time 0.0008 (0.0135) model time 0.0000 (0.0000) loss 1.7238 (2.7844) grad_norm 2.3302 (2.1118) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][50/625] eta 0:03:44 lr 0.000418 wd 0.0500 time 0.3662 (0.3908) data time 0.0011 (0.0111) model time 0.0000 (0.0000) loss 3.2104 (2.8174) grad_norm 3.7973 (2.2587) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][60/625] eta 0:03:38 lr 0.000418 wd 0.0500 time 0.3662 (0.3874) data time 0.0011 (0.0094) model time 0.3651 (0.3687) loss 2.8407 (2.7789) grad_norm 1.7106 (2.2155) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][70/625] eta 0:03:33 lr 0.000418 wd 0.0500 time 0.3714 (0.3847) data time 0.0011 (0.0083) model time 0.3703 (0.3680) loss 3.4113 (2.7955) grad_norm 2.3649 (2.2058) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][80/625] eta 0:03:28 lr 0.000418 wd 0.0500 time 0.3759 (0.3832) data time 0.0012 (0.0074) model time 0.3746 (0.3692) loss 3.1901 (2.8187) grad_norm 2.2502 (2.2053) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][90/625] eta 0:03:24 lr 0.000418 wd 0.0500 time 0.3694 (0.3818) data time 0.0009 (0.0067) model time 0.3685 (0.3692) loss 2.8736 (2.8101) grad_norm 1.8488 (2.1736) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][100/625] eta 0:03:19 lr 0.000418 wd 0.0500 time 0.3765 (0.3809) data time 0.0009 (0.0062) model time 0.3756 (0.3697) loss 2.0215 (2.7939) grad_norm 1.3215 (2.1372) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][110/625] eta 0:03:15 lr 0.000418 wd 0.0500 time 0.3708 (0.3801) data time 0.0011 (0.0057) model time 0.3697 (0.3699) loss 3.1488 (2.7927) grad_norm 1.7640 (2.1010) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][120/625] eta 0:03:11 lr 0.000417 wd 0.0500 time 0.3651 (0.3798) data time 0.0011 (0.0053) model time 0.3640 (0.3706) loss 3.1382 (2.8034) grad_norm 1.6307 (2.0837) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][130/625] eta 0:03:08 lr 0.000417 wd 0.0500 time 0.3673 (0.3806) data time 0.0010 (0.0050) model time 0.3662 (0.3729) loss 2.3519 (2.8044) grad_norm 1.9570 (2.0632) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][140/625] eta 0:03:04 lr 0.000417 wd 0.0500 time 0.3818 (0.3799) data time 0.0010 (0.0048) model time 0.3808 (0.3726) loss 1.9317 (2.8010) grad_norm 1.8842 (2.0513) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][150/625] eta 0:03:00 lr 0.000417 wd 0.0500 time 0.3667 (0.3802) data time 0.0009 (0.0045) model time 0.3658 (0.3735) loss 2.4574 (2.8022) grad_norm 1.3348 (2.0449) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][160/625] eta 0:02:57 lr 0.000417 wd 0.0500 time 0.3661 (0.3822) data time 0.0011 (0.0043) model time 0.3650 (0.3770) loss 2.8918 (2.8068) grad_norm 2.0878 (2.0574) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:50:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][170/625] eta 0:02:53 lr 0.000417 wd 0.0500 time 0.3680 (0.3816) data time 0.0009 (0.0041) model time 0.3672 (0.3765) loss 2.0693 (2.8126) grad_norm 1.8286 (2.0472) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][180/625] eta 0:02:49 lr 0.000417 wd 0.0500 time 0.3798 (0.3810) data time 0.0012 (0.0040) model time 0.3786 (0.3759) loss 2.9167 (2.7982) grad_norm 1.8345 (2.0719) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][190/625] eta 0:02:45 lr 0.000416 wd 0.0500 time 0.3716 (0.3803) data time 0.0013 (0.0039) model time 0.3703 (0.3753) loss 2.8601 (2.8010) grad_norm 1.7279 (2.1043) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][200/625] eta 0:02:41 lr 0.000416 wd 0.0500 time 0.3773 (0.3799) data time 0.0012 (0.0037) model time 0.3761 (0.3749) loss 2.4157 (2.8077) grad_norm 2.3562 (2.1146) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][210/625] eta 0:02:37 lr 0.000416 wd 0.0500 time 0.3695 (0.3796) data time 0.0013 (0.0036) model time 0.3682 (0.3748) loss 2.5634 (2.7930) grad_norm 2.7202 (2.1306) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][220/625] eta 0:02:33 lr 0.000416 wd 0.0500 time 0.3691 (0.3793) data time 0.0011 (0.0035) model time 0.3680 (0.3746) loss 2.5463 (2.7980) grad_norm 2.5749 (2.1318) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][230/625] eta 0:02:29 lr 0.000416 wd 0.0500 time 0.3755 (0.3789) data time 0.0011 (0.0034) model time 0.3744 (0.3743) loss 2.8389 (2.8044) grad_norm 2.7568 (2.1331) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][240/625] eta 0:02:25 lr 0.000416 wd 0.0500 time 0.3740 (0.3787) data time 0.0009 (0.0033) model time 0.3731 (0.3743) loss 3.4506 (2.8087) grad_norm 1.6735 (2.1311) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][250/625] eta 0:02:21 lr 0.000416 wd 0.0500 time 0.3681 (0.3783) data time 0.0009 (0.0032) model time 0.3671 (0.3739) loss 3.3169 (2.8202) grad_norm 1.3714 (2.1258) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][260/625] eta 0:02:17 lr 0.000415 wd 0.0500 time 0.3700 (0.3780) data time 0.0012 (0.0032) model time 0.3688 (0.3737) loss 2.9658 (2.8359) grad_norm 1.8416 (2.1125) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][270/625] eta 0:02:14 lr 0.000415 wd 0.0500 time 0.3681 (0.3778) data time 0.0010 (0.0031) model time 0.3672 (0.3735) loss 2.0141 (2.8417) grad_norm 1.6563 (2.1011) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][280/625] eta 0:02:10 lr 0.000415 wd 0.0500 time 0.3766 (0.3776) data time 0.0013 (0.0030) model time 0.3754 (0.3734) loss 3.0340 (2.8346) grad_norm 1.4636 (2.0881) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][290/625] eta 0:02:06 lr 0.000415 wd 0.0500 time 0.3667 (0.3775) data time 0.0012 (0.0030) model time 0.3655 (0.3734) loss 2.9711 (2.8287) grad_norm 3.0620 (2.0902) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][300/625] eta 0:02:02 lr 0.000415 wd 0.0500 time 0.3697 (0.3772) data time 0.0011 (0.0029) model time 0.3686 (0.3732) loss 3.3188 (2.8330) grad_norm 1.6403 (2.0894) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][310/625] eta 0:01:58 lr 0.000415 wd 0.0500 time 0.3709 (0.3770) data time 0.0011 (0.0029) model time 0.3697 (0.3731) loss 3.1376 (2.8319) grad_norm 1.8137 (2.0851) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][320/625] eta 0:01:54 lr 0.000415 wd 0.0500 time 0.3798 (0.3769) data time 0.0009 (0.0028) model time 0.3788 (0.3730) loss 2.9498 (2.8408) grad_norm 1.6508 (2.0718) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:51:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][330/625] eta 0:01:51 lr 0.000414 wd 0.0500 time 0.3686 (0.3767) data time 0.0011 (0.0028) model time 0.3674 (0.3729) loss 3.0358 (2.8437) grad_norm 2.2250 (2.0717) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][340/625] eta 0:01:47 lr 0.000414 wd 0.0500 time 0.3714 (0.3781) data time 0.0009 (0.0027) model time 0.3705 (0.3746) loss 2.9933 (2.8488) grad_norm 2.3392 (2.0677) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][350/625] eta 0:01:44 lr 0.000414 wd 0.0500 time 0.3723 (0.3792) data time 0.0011 (0.0027) model time 0.3711 (0.3760) loss 3.1211 (2.8471) grad_norm 2.3065 (2.0739) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][360/625] eta 0:01:40 lr 0.000414 wd 0.0500 time 0.3727 (0.3790) data time 0.0009 (0.0027) model time 0.3718 (0.3758) loss 3.3874 (2.8515) grad_norm 1.4559 (2.0885) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][370/625] eta 0:01:36 lr 0.000414 wd 0.0500 time 0.3689 (0.3788) data time 0.0009 (0.0026) model time 0.3680 (0.3756) loss 3.6405 (2.8570) grad_norm 1.6110 (2.0844) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][380/625] eta 0:01:32 lr 0.000414 wd 0.0500 time 0.3742 (0.3786) data time 0.0008 (0.0026) model time 0.3734 (0.3754) loss 2.7274 (2.8559) grad_norm 1.5634 (2.0821) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][390/625] eta 0:01:28 lr 0.000414 wd 0.0500 time 0.3678 (0.3783) data time 0.0009 (0.0025) model time 0.3669 (0.3751) loss 2.7724 (2.8577) grad_norm 2.1856 (2.0881) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][400/625] eta 0:01:25 lr 0.000413 wd 0.0500 time 0.3662 (0.3781) data time 0.0011 (0.0025) model time 0.3651 (0.3750) loss 3.0357 (2.8575) grad_norm 1.8012 (2.0901) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][410/625] eta 0:01:21 lr 0.000413 wd 0.0500 time 0.3711 (0.3779) data time 0.0012 (0.0025) model time 0.3699 (0.3749) loss 2.5150 (2.8597) grad_norm 2.1288 (2.0858) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][420/625] eta 0:01:17 lr 0.000413 wd 0.0500 time 0.3749 (0.3778) data time 0.0008 (0.0024) model time 0.3741 (0.3747) loss 2.6414 (2.8577) grad_norm 1.9634 (2.0981) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][430/625] eta 0:01:13 lr 0.000413 wd 0.0500 time 0.3738 (0.3776) data time 0.0012 (0.0024) model time 0.3726 (0.3746) loss 3.6710 (2.8589) grad_norm 1.9327 (2.0894) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][440/625] eta 0:01:09 lr 0.000413 wd 0.0500 time 0.3716 (0.3775) data time 0.0009 (0.0024) model time 0.3707 (0.3745) loss 3.8918 (2.8612) grad_norm 2.0453 (2.0902) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][450/625] eta 0:01:06 lr 0.000413 wd 0.0500 time 0.3652 (0.3773) data time 0.0009 (0.0024) model time 0.3643 (0.3743) loss 2.9100 (2.8626) grad_norm 2.8163 (2.0910) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][460/625] eta 0:01:02 lr 0.000413 wd 0.0500 time 0.5758 (0.3785) data time 0.0011 (0.0023) model time 0.5747 (0.3757) loss 3.0715 (2.8657) grad_norm 1.9598 (2.0947) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][470/625] eta 0:00:58 lr 0.000412 wd 0.0500 time 0.3822 (0.3788) data time 0.0009 (0.0023) model time 0.3813 (0.3761) loss 2.2867 (2.8624) grad_norm 1.3031 (2.0905) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][480/625] eta 0:00:54 lr 0.000412 wd 0.0500 time 0.3772 (0.3787) data time 0.0009 (0.0023) model time 0.3763 (0.3760) loss 3.0933 (2.8659) grad_norm 2.1289 (2.0843) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][490/625] eta 0:00:51 lr 0.000412 wd 0.0500 time 0.3717 (0.3785) data time 0.0014 (0.0023) model time 0.3703 (0.3758) loss 2.9320 (2.8666) grad_norm 1.9103 (2.0853) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][500/625] eta 0:00:47 lr 0.000412 wd 0.0500 time 0.3692 (0.3784) data time 0.0011 (0.0022) model time 0.3681 (0.3757) loss 2.5137 (2.8617) grad_norm 1.6300 (2.0867) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][510/625] eta 0:00:43 lr 0.000412 wd 0.0500 time 0.3710 (0.3782) data time 0.0011 (0.0022) model time 0.3699 (0.3755) loss 3.2063 (2.8617) grad_norm 2.2379 (2.0819) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][520/625] eta 0:00:39 lr 0.000412 wd 0.0500 time 0.3682 (0.3784) data time 0.0010 (0.0022) model time 0.3671 (0.3758) loss 3.0655 (2.8638) grad_norm 1.7704 (2.0897) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][530/625] eta 0:00:35 lr 0.000412 wd 0.0500 time 0.3911 (0.3783) data time 0.0009 (0.0022) model time 0.3902 (0.3757) loss 3.4076 (2.8637) grad_norm 1.3520 (2.0889) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][540/625] eta 0:00:32 lr 0.000411 wd 0.0500 time 0.3711 (0.3783) data time 0.0009 (0.0022) model time 0.3702 (0.3757) loss 2.7418 (2.8618) grad_norm 2.5460 (2.0871) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][550/625] eta 0:00:28 lr 0.000411 wd 0.0500 time 0.3670 (0.3782) data time 0.0009 (0.0022) model time 0.3661 (0.3756) loss 2.5837 (2.8593) grad_norm 3.0464 (2.1029) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][560/625] eta 0:00:24 lr 0.000411 wd 0.0500 time 0.3738 (0.3780) data time 0.0008 (0.0021) model time 0.3730 (0.3755) loss 3.0730 (2.8616) grad_norm 2.1411 (2.1070) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][570/625] eta 0:00:20 lr 0.000411 wd 0.0500 time 0.3758 (0.3779) data time 0.0011 (0.0021) model time 0.3747 (0.3754) loss 3.5911 (2.8668) grad_norm 1.2891 (2.1153) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][580/625] eta 0:00:17 lr 0.000411 wd 0.0500 time 0.3637 (0.3778) data time 0.0010 (0.0021) model time 0.3627 (0.3753) loss 3.0276 (2.8706) grad_norm 2.1322 (2.1112) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][590/625] eta 0:00:13 lr 0.000411 wd 0.0500 time 0.3755 (0.3777) data time 0.0011 (0.0021) model time 0.3744 (0.3752) loss 2.6392 (2.8680) grad_norm 1.9337 (2.1091) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][600/625] eta 0:00:09 lr 0.000411 wd 0.0500 time 0.3718 (0.3776) data time 0.0011 (0.0021) model time 0.3706 (0.3751) loss 3.2536 (2.8684) grad_norm 2.3788 (2.1107) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][610/625] eta 0:00:05 lr 0.000410 wd 0.0500 time 0.3558 (0.3774) data time 0.0009 (0.0021) model time 0.3549 (0.3750) loss 3.2051 (2.8667) grad_norm 2.2571 (2.1120) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [217/300][620/625] eta 0:00:01 lr 0.000410 wd 0.0500 time 0.3601 (0.3771) data time 0.0006 (0.0020) model time 0.3595 (0.3747) loss 2.0152 (2.8632) grad_norm 2.1589 (2.1116) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 217 training takes 0:03:55 [2024-09-01 09:53:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:53:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:53:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.496 (0.496) Loss 0.5420 (0.5420) Acc@1 89.209 (89.209) Acc@5 98.291 (98.291) Mem 11225MB [2024-09-01 09:53:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.142) Loss 0.8608 (0.6523) Acc@1 79.492 (85.858) Acc@5 95.752 (97.483) Mem 11225MB [2024-09-01 09:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9561 (0.7734) Acc@1 77.148 (82.799) Acc@5 94.385 (96.164) Mem 11225MB [2024-09-01 09:53:53 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.500 Acc@5 96.169 [2024-09-01 09:53:53 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 09:53:53 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.50% [2024-09-01 09:53:53 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 09:53:54 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 09:53:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.4924 (0.4924) Acc@1 89.551 (89.551) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 09:53:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.142) Loss 0.7881 (0.5971) Acc@1 81.006 (86.586) Acc@5 96.240 (97.803) Mem 11225MB [2024-09-01 09:53:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.8838 (0.7076) Acc@1 78.027 (83.796) Acc@5 95.508 (96.696) Mem 11225MB [2024-09-01 09:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.521 Acc@5 96.727 [2024-09-01 09:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 09:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.52% [2024-09-01 09:53:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:53:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][0/625] eta 0:08:27 lr 0.000410 wd 0.0500 time 0.8123 (0.8123) data time 0.4581 (0.4581) model time 0.0000 (0.0000) loss 3.2202 (3.2202) grad_norm 1.8796 (1.8796) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][10/625] eta 0:04:12 lr 0.000410 wd 0.0500 time 0.3726 (0.4105) data time 0.0010 (0.0427) model time 0.0000 (0.0000) loss 2.8834 (2.8761) grad_norm 1.8161 (2.2760) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][20/625] eta 0:04:02 lr 0.000410 wd 0.0500 time 0.5668 (0.4001) data time 0.0011 (0.0230) model time 0.0000 (0.0000) loss 2.6178 (2.9275) grad_norm 1.9400 (2.2480) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][30/625] eta 0:04:04 lr 0.000410 wd 0.0500 time 0.3715 (0.4109) data time 0.0008 (0.0159) model time 0.0000 (0.0000) loss 1.7330 (2.8624) grad_norm 1.7000 (2.2466) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][40/625] eta 0:03:55 lr 0.000410 wd 0.0500 time 0.3770 (0.4018) data time 0.0011 (0.0123) model time 0.0000 (0.0000) loss 2.9755 (2.7795) grad_norm 1.9539 (2.1233) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][50/625] eta 0:03:47 lr 0.000410 wd 0.0500 time 0.3674 (0.3955) data time 0.0012 (0.0102) model time 0.0000 (0.0000) loss 2.7492 (2.8094) grad_norm 2.4264 (2.1004) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][60/625] eta 0:03:41 lr 0.000409 wd 0.0500 time 0.3783 (0.3915) data time 0.0011 (0.0087) model time 0.3772 (0.3698) loss 3.1569 (2.8431) grad_norm 1.5948 (2.0515) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][70/625] eta 0:03:35 lr 0.000409 wd 0.0500 time 0.3875 (0.3890) data time 0.0010 (0.0076) model time 0.3865 (0.3712) loss 3.4077 (2.8343) grad_norm 1.6393 (2.0420) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][80/625] eta 0:03:30 lr 0.000409 wd 0.0500 time 0.3710 (0.3869) data time 0.0011 (0.0068) model time 0.3699 (0.3710) loss 3.2747 (2.8502) grad_norm 2.5064 (2.0601) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][90/625] eta 0:03:26 lr 0.000409 wd 0.0500 time 0.3747 (0.3853) data time 0.0008 (0.0062) model time 0.3739 (0.3712) loss 3.5532 (2.8246) grad_norm 1.7786 (2.0638) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][100/625] eta 0:03:21 lr 0.000409 wd 0.0500 time 0.3629 (0.3839) data time 0.0011 (0.0057) model time 0.3618 (0.3710) loss 2.9626 (2.8270) grad_norm 2.8336 (2.0901) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][110/625] eta 0:03:17 lr 0.000409 wd 0.0500 time 0.3925 (0.3832) data time 0.0008 (0.0053) model time 0.3917 (0.3716) loss 2.8653 (2.8464) grad_norm 1.5012 (2.0770) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][120/625] eta 0:03:13 lr 0.000409 wd 0.0500 time 0.3677 (0.3823) data time 0.0008 (0.0050) model time 0.3669 (0.3716) loss 2.9415 (2.8441) grad_norm 1.6714 (2.0530) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][130/625] eta 0:03:08 lr 0.000408 wd 0.0500 time 0.3698 (0.3817) data time 0.0012 (0.0047) model time 0.3686 (0.3717) loss 3.5015 (2.8573) grad_norm 1.2550 (2.0341) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][140/625] eta 0:03:04 lr 0.000408 wd 0.0500 time 0.3667 (0.3811) data time 0.0012 (0.0044) model time 0.3655 (0.3717) loss 2.9427 (2.8722) grad_norm 3.0827 (2.0334) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][150/625] eta 0:03:01 lr 0.000408 wd 0.0500 time 0.3758 (0.3817) data time 0.0011 (0.0042) model time 0.3747 (0.3734) loss 2.7064 (2.8632) grad_norm 2.3479 (2.0679) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:54:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][160/625] eta 0:02:57 lr 0.000408 wd 0.0500 time 0.3662 (0.3810) data time 0.0011 (0.0040) model time 0.3651 (0.3730) loss 2.3760 (2.8561) grad_norm 2.0583 (2.0615) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][170/625] eta 0:02:53 lr 0.000408 wd 0.0500 time 0.3713 (0.3805) data time 0.0008 (0.0039) model time 0.3706 (0.3729) loss 2.9769 (2.8663) grad_norm 2.3861 (2.0695) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][180/625] eta 0:02:49 lr 0.000408 wd 0.0500 time 0.3732 (0.3800) data time 0.0011 (0.0037) model time 0.3721 (0.3727) loss 3.2681 (2.8814) grad_norm 1.8145 (2.0542) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][190/625] eta 0:02:45 lr 0.000408 wd 0.0500 time 0.3686 (0.3795) data time 0.0009 (0.0036) model time 0.3678 (0.3725) loss 3.4868 (2.8823) grad_norm 2.5145 (2.0656) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][200/625] eta 0:02:41 lr 0.000407 wd 0.0500 time 0.3674 (0.3791) data time 0.0012 (0.0035) model time 0.3662 (0.3723) loss 3.5915 (2.8613) grad_norm 1.8884 (2.0953) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][210/625] eta 0:02:37 lr 0.000407 wd 0.0500 time 0.3837 (0.3787) data time 0.0009 (0.0033) model time 0.3828 (0.3721) loss 2.9882 (2.8624) grad_norm 2.0734 (2.1004) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][220/625] eta 0:02:33 lr 0.000407 wd 0.0500 time 0.3738 (0.3783) data time 0.0008 (0.0033) model time 0.3730 (0.3719) loss 3.1707 (2.8584) grad_norm 13.2000 (2.1963) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][230/625] eta 0:02:29 lr 0.000407 wd 0.0500 time 0.3783 (0.3779) data time 0.0011 (0.0032) model time 0.3772 (0.3717) loss 3.5721 (2.8604) grad_norm 1.6418 (2.2217) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][240/625] eta 0:02:26 lr 0.000407 wd 0.0500 time 0.3724 (0.3802) data time 0.0011 (0.0031) model time 0.3713 (0.3750) loss 3.3500 (2.8662) grad_norm 2.4529 (2.2547) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][250/625] eta 0:02:22 lr 0.000407 wd 0.0500 time 0.3795 (0.3799) data time 0.0009 (0.0030) model time 0.3786 (0.3747) loss 3.4001 (2.8650) grad_norm 1.7509 (2.2504) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][260/625] eta 0:02:18 lr 0.000407 wd 0.0500 time 0.3711 (0.3796) data time 0.0009 (0.0029) model time 0.3702 (0.3746) loss 3.3364 (2.8617) grad_norm 2.1320 (2.2497) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][270/625] eta 0:02:14 lr 0.000406 wd 0.0500 time 0.3828 (0.3793) data time 0.0011 (0.0029) model time 0.3817 (0.3744) loss 3.0814 (2.8675) grad_norm 1.8606 (2.2552) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][280/625] eta 0:02:10 lr 0.000406 wd 0.0500 time 0.3686 (0.3790) data time 0.0009 (0.0028) model time 0.3677 (0.3741) loss 3.0686 (2.8686) grad_norm 1.6203 (2.2428) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][290/625] eta 0:02:06 lr 0.000406 wd 0.0500 time 0.3668 (0.3787) data time 0.0008 (0.0028) model time 0.3660 (0.3740) loss 2.9629 (2.8670) grad_norm 1.5138 (2.2311) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][300/625] eta 0:02:03 lr 0.000406 wd 0.0500 time 0.3705 (0.3785) data time 0.0010 (0.0027) model time 0.3695 (0.3739) loss 2.3627 (2.8615) grad_norm 1.7970 (2.2122) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][310/625] eta 0:01:59 lr 0.000406 wd 0.0500 time 0.3648 (0.3784) data time 0.0009 (0.0027) model time 0.3639 (0.3738) loss 2.9338 (2.8662) grad_norm 1.9170 (2.1978) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:55:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][320/625] eta 0:01:55 lr 0.000406 wd 0.0500 time 0.3711 (0.3781) data time 0.0009 (0.0026) model time 0.3702 (0.3737) loss 3.4071 (2.8669) grad_norm 2.0864 (2.1931) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][330/625] eta 0:01:51 lr 0.000406 wd 0.0500 time 0.3706 (0.3785) data time 0.0009 (0.0026) model time 0.3697 (0.3743) loss 3.0733 (2.8612) grad_norm 1.5611 (2.2221) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][340/625] eta 0:01:47 lr 0.000405 wd 0.0500 time 0.3788 (0.3783) data time 0.0012 (0.0025) model time 0.3776 (0.3741) loss 2.9909 (2.8676) grad_norm 4.1261 (2.2389) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][350/625] eta 0:01:43 lr 0.000405 wd 0.0500 time 0.3708 (0.3780) data time 0.0011 (0.0025) model time 0.3697 (0.3739) loss 2.8357 (2.8707) grad_norm 1.9998 (2.2458) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][360/625] eta 0:01:40 lr 0.000405 wd 0.0500 time 0.3761 (0.3779) data time 0.0012 (0.0024) model time 0.3749 (0.3738) loss 3.2909 (2.8699) grad_norm 1.4562 (2.2379) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][370/625] eta 0:01:36 lr 0.000405 wd 0.0500 time 0.3701 (0.3777) data time 0.0011 (0.0024) model time 0.3690 (0.3737) loss 2.7245 (2.8749) grad_norm 1.3377 (2.2356) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][380/625] eta 0:01:32 lr 0.000405 wd 0.0500 time 0.3720 (0.3775) data time 0.0010 (0.0024) model time 0.3710 (0.3736) loss 3.0824 (2.8807) grad_norm 2.1859 (2.2318) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][390/625] eta 0:01:28 lr 0.000405 wd 0.0500 time 0.3747 (0.3774) data time 0.0009 (0.0024) model time 0.3738 (0.3735) loss 2.1432 (2.8769) grad_norm 2.3745 (2.2295) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][400/625] eta 0:01:24 lr 0.000405 wd 0.0500 time 0.3792 (0.3773) data time 0.0008 (0.0023) model time 0.3783 (0.3734) loss 3.4102 (2.8765) grad_norm 1.8788 (2.2234) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][410/625] eta 0:01:21 lr 0.000404 wd 0.0500 time 0.3741 (0.3772) data time 0.0011 (0.0023) model time 0.3730 (0.3734) loss 3.1268 (2.8764) grad_norm 1.8182 (2.2145) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][420/625] eta 0:01:17 lr 0.000404 wd 0.0500 time 0.5818 (0.3789) data time 0.0011 (0.0023) model time 0.5807 (0.3754) loss 2.8569 (2.8800) grad_norm 2.2331 (2.2148) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][430/625] eta 0:01:13 lr 0.000404 wd 0.0500 time 0.3690 (0.3787) data time 0.0010 (0.0022) model time 0.3680 (0.3753) loss 3.3728 (2.8783) grad_norm 2.4297 (2.2102) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][440/625] eta 0:01:10 lr 0.000404 wd 0.0500 time 0.3721 (0.3786) data time 0.0009 (0.0022) model time 0.3713 (0.3752) loss 2.0954 (2.8757) grad_norm 2.1764 (2.2021) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][450/625] eta 0:01:06 lr 0.000404 wd 0.0500 time 0.3591 (0.3789) data time 0.0011 (0.0022) model time 0.3580 (0.3756) loss 2.8743 (2.8729) grad_norm 1.6372 (2.2011) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][460/625] eta 0:01:02 lr 0.000404 wd 0.0500 time 0.3774 (0.3788) data time 0.0012 (0.0022) model time 0.3762 (0.3755) loss 2.7852 (2.8705) grad_norm 1.8060 (2.1915) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][470/625] eta 0:00:58 lr 0.000404 wd 0.0500 time 0.3704 (0.3786) data time 0.0010 (0.0022) model time 0.3694 (0.3753) loss 2.2981 (2.8686) grad_norm 1.3968 (2.1875) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][480/625] eta 0:00:54 lr 0.000404 wd 0.0500 time 0.3692 (0.3784) data time 0.0008 (0.0021) model time 0.3684 (0.3752) loss 3.3777 (2.8701) grad_norm 1.7374 (2.1814) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][490/625] eta 0:00:51 lr 0.000403 wd 0.0500 time 0.3719 (0.3783) data time 0.0011 (0.0021) model time 0.3708 (0.3751) loss 2.7270 (2.8676) grad_norm 2.4184 (2.1740) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][500/625] eta 0:00:47 lr 0.000403 wd 0.0500 time 0.3737 (0.3781) data time 0.0011 (0.0021) model time 0.3726 (0.3749) loss 2.7247 (2.8651) grad_norm 8.1148 (2.1819) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][510/625] eta 0:00:43 lr 0.000403 wd 0.0500 time 0.3725 (0.3779) data time 0.0011 (0.0021) model time 0.3714 (0.3748) loss 3.0469 (2.8687) grad_norm 1.9216 (2.1753) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][520/625] eta 0:00:39 lr 0.000403 wd 0.0500 time 0.3642 (0.3778) data time 0.0011 (0.0021) model time 0.3631 (0.3747) loss 3.1021 (2.8692) grad_norm 1.6263 (2.1820) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][530/625] eta 0:00:35 lr 0.000403 wd 0.0500 time 0.3708 (0.3777) data time 0.0010 (0.0020) model time 0.3698 (0.3746) loss 2.0017 (2.8610) grad_norm 2.1148 (2.1795) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][540/625] eta 0:00:32 lr 0.000403 wd 0.0500 time 0.3686 (0.3787) data time 0.0016 (0.0020) model time 0.3669 (0.3758) loss 3.0976 (2.8594) grad_norm 2.5736 (2.1761) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][550/625] eta 0:00:28 lr 0.000403 wd 0.0500 time 0.3636 (0.3786) data time 0.0012 (0.0020) model time 0.3624 (0.3757) loss 3.4018 (2.8619) grad_norm 1.6267 (2.1670) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][560/625] eta 0:00:24 lr 0.000402 wd 0.0500 time 0.3643 (0.3784) data time 0.0012 (0.0020) model time 0.3631 (0.3755) loss 3.1870 (2.8620) grad_norm 1.5653 (2.1614) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][570/625] eta 0:00:20 lr 0.000402 wd 0.0500 time 0.3771 (0.3783) data time 0.0009 (0.0020) model time 0.3762 (0.3754) loss 3.3112 (2.8668) grad_norm 2.3709 (2.1578) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][580/625] eta 0:00:17 lr 0.000402 wd 0.0500 time 0.3767 (0.3781) data time 0.0011 (0.0020) model time 0.3756 (0.3753) loss 2.9763 (2.8636) grad_norm 2.6902 (2.1656) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][590/625] eta 0:00:13 lr 0.000402 wd 0.0500 time 0.3729 (0.3780) data time 0.0011 (0.0020) model time 0.3718 (0.3752) loss 2.5930 (2.8607) grad_norm 1.8347 (2.1702) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][600/625] eta 0:00:09 lr 0.000402 wd 0.0500 time 0.3753 (0.3779) data time 0.0009 (0.0019) model time 0.3745 (0.3751) loss 3.8824 (2.8644) grad_norm 2.6570 (2.1679) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][610/625] eta 0:00:05 lr 0.000402 wd 0.0500 time 0.3629 (0.3778) data time 0.0006 (0.0019) model time 0.3623 (0.3750) loss 2.7793 (2.8609) grad_norm 2.0469 (2.1649) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [218/300][620/625] eta 0:00:01 lr 0.000402 wd 0.0500 time 0.3637 (0.3776) data time 0.0008 (0.0019) model time 0.3628 (0.3749) loss 2.5097 (2.8634) grad_norm 2.3019 (2.1613) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:57:54 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 218 training takes 0:03:55 [2024-09-01 09:57:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 09:57:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 09:57:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.517 (0.517) Loss 0.5293 (0.5293) Acc@1 88.916 (88.916) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 09:57:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.8369 (0.6450) Acc@1 79.346 (85.818) Acc@5 96.436 (97.590) Mem 11225MB [2024-09-01 09:57:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9556 (0.7649) Acc@1 76.855 (82.785) Acc@5 94.385 (96.240) Mem 11225MB [2024-09-01 09:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.464 Acc@5 96.267 [2024-09-01 09:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 09:57:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.808 (0.808) Loss 0.4919 (0.4919) Acc@1 89.697 (89.697) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 09:58:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.172) Loss 0.7866 (0.5964) Acc@1 81.104 (86.688) Acc@5 96.240 (97.803) Mem 11225MB [2024-09-01 09:58:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.135) Loss 0.8828 (0.7070) Acc@1 77.881 (83.873) Acc@5 95.410 (96.687) Mem 11225MB [2024-09-01 09:58:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.583 Acc@5 96.725 [2024-09-01 09:58:01 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 09:58:01 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.58% [2024-09-01 09:58:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 09:58:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 09:58:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][0/625] eta 0:08:07 lr 0.000401 wd 0.0500 time 0.7797 (0.7797) data time 0.4466 (0.4466) model time 0.0000 (0.0000) loss 2.7686 (2.7686) grad_norm 2.0423 (2.0423) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][10/625] eta 0:04:11 lr 0.000401 wd 0.0500 time 0.3873 (0.4095) data time 0.0009 (0.0417) model time 0.0000 (0.0000) loss 3.1077 (2.7753) grad_norm 2.0577 (3.4404) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][20/625] eta 0:03:56 lr 0.000401 wd 0.0500 time 0.3702 (0.3912) data time 0.0009 (0.0225) model time 0.0000 (0.0000) loss 3.3923 (2.8564) grad_norm 3.1658 (2.8583) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][30/625] eta 0:03:49 lr 0.000401 wd 0.0500 time 0.3650 (0.3849) data time 0.0012 (0.0156) model time 0.0000 (0.0000) loss 3.0033 (2.8593) grad_norm 2.1677 (2.5799) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][40/625] eta 0:03:43 lr 0.000401 wd 0.0500 time 0.3727 (0.3825) data time 0.0009 (0.0121) model time 0.0000 (0.0000) loss 1.8284 (2.8257) grad_norm 2.9020 (2.5186) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][50/625] eta 0:03:38 lr 0.000401 wd 0.0500 time 0.3654 (0.3807) data time 0.0010 (0.0100) model time 0.0000 (0.0000) loss 3.2044 (2.8539) grad_norm 1.6778 (2.4832) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][60/625] eta 0:03:34 lr 0.000401 wd 0.0500 time 0.3689 (0.3797) data time 0.0011 (0.0085) model time 0.3678 (0.3733) loss 3.2895 (2.8901) grad_norm 2.3207 (2.3949) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][70/625] eta 0:03:35 lr 0.000400 wd 0.0500 time 0.5755 (0.3876) data time 0.0011 (0.0075) model time 0.5744 (0.4039) loss 3.5357 (2.8795) grad_norm 1.6142 (2.3451) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][80/625] eta 0:03:30 lr 0.000400 wd 0.0500 time 0.3725 (0.3856) data time 0.0009 (0.0067) model time 0.3715 (0.3928) loss 1.7265 (2.8402) grad_norm 1.5537 (2.3029) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][90/625] eta 0:03:25 lr 0.000400 wd 0.0500 time 0.3793 (0.3844) data time 0.0012 (0.0061) model time 0.3781 (0.3879) loss 3.2411 (2.8440) grad_norm 1.7624 (2.2526) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][100/625] eta 0:03:21 lr 0.000400 wd 0.0500 time 0.3824 (0.3831) data time 0.0011 (0.0056) model time 0.3814 (0.3844) loss 3.3523 (2.8485) grad_norm 2.5669 (2.2396) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][110/625] eta 0:03:17 lr 0.000400 wd 0.0500 time 0.6057 (0.3840) data time 0.0009 (0.0052) model time 0.6048 (0.3857) loss 3.0104 (2.8599) grad_norm 1.7860 (2.2403) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][120/625] eta 0:03:13 lr 0.000400 wd 0.0500 time 0.3722 (0.3826) data time 0.0012 (0.0049) model time 0.3710 (0.3829) loss 3.2816 (2.8898) grad_norm 1.6620 (2.2169) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][130/625] eta 0:03:08 lr 0.000400 wd 0.0500 time 0.3652 (0.3817) data time 0.0009 (0.0046) model time 0.3643 (0.3811) loss 3.5606 (2.8927) grad_norm 1.8679 (2.2022) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:58:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][140/625] eta 0:03:04 lr 0.000400 wd 0.0500 time 0.3668 (0.3809) data time 0.0011 (0.0044) model time 0.3657 (0.3798) loss 3.3540 (2.9022) grad_norm 2.1693 (2.2426) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][150/625] eta 0:03:00 lr 0.000399 wd 0.0500 time 0.3669 (0.3800) data time 0.0012 (0.0041) model time 0.3657 (0.3784) loss 3.0112 (2.8955) grad_norm 2.2040 (2.2450) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][160/625] eta 0:02:56 lr 0.000399 wd 0.0500 time 0.3709 (0.3793) data time 0.0008 (0.0040) model time 0.3700 (0.3775) loss 3.6645 (2.8850) grad_norm 1.9819 (2.2433) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][170/625] eta 0:02:52 lr 0.000399 wd 0.0500 time 0.3733 (0.3788) data time 0.0011 (0.0038) model time 0.3722 (0.3767) loss 3.2887 (2.8845) grad_norm 1.8857 (2.2182) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][180/625] eta 0:02:48 lr 0.000399 wd 0.0500 time 0.3729 (0.3784) data time 0.0011 (0.0037) model time 0.3718 (0.3763) loss 2.8878 (2.8756) grad_norm 1.5398 (2.1983) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][190/625] eta 0:02:44 lr 0.000399 wd 0.0500 time 0.3717 (0.3781) data time 0.0009 (0.0035) model time 0.3708 (0.3760) loss 2.9966 (2.8753) grad_norm 2.5514 (2.2275) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][200/625] eta 0:02:40 lr 0.000399 wd 0.0500 time 0.3762 (0.3779) data time 0.0011 (0.0034) model time 0.3751 (0.3757) loss 2.7680 (2.8659) grad_norm 2.3481 (2.2520) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][210/625] eta 0:02:36 lr 0.000399 wd 0.0500 time 0.3725 (0.3775) data time 0.0009 (0.0033) model time 0.3716 (0.3753) loss 3.2601 (2.8729) grad_norm 2.2710 (2.2581) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][220/625] eta 0:02:32 lr 0.000398 wd 0.0500 time 0.3728 (0.3773) data time 0.0012 (0.0032) model time 0.3717 (0.3751) loss 3.2353 (2.8661) grad_norm 2.0363 (2.2516) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][230/625] eta 0:02:28 lr 0.000398 wd 0.0500 time 0.3686 (0.3771) data time 0.0010 (0.0031) model time 0.3676 (0.3749) loss 3.0241 (2.8711) grad_norm 1.9240 (2.2569) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][240/625] eta 0:02:25 lr 0.000398 wd 0.0500 time 0.3717 (0.3769) data time 0.0009 (0.0030) model time 0.3708 (0.3746) loss 2.8577 (2.8745) grad_norm 1.2963 (2.2525) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][250/625] eta 0:02:21 lr 0.000398 wd 0.0500 time 0.3721 (0.3767) data time 0.0009 (0.0030) model time 0.3712 (0.3745) loss 1.8720 (2.8712) grad_norm 1.6055 (2.2380) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][260/625] eta 0:02:17 lr 0.000398 wd 0.0500 time 0.3719 (0.3765) data time 0.0012 (0.0029) model time 0.3707 (0.3743) loss 2.3983 (2.8744) grad_norm 2.3575 (2.2314) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][270/625] eta 0:02:13 lr 0.000398 wd 0.0500 time 0.3687 (0.3762) data time 0.0009 (0.0028) model time 0.3678 (0.3740) loss 3.4471 (2.8661) grad_norm 2.1683 (2.2211) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][280/625] eta 0:02:10 lr 0.000398 wd 0.0500 time 0.3667 (0.3776) data time 0.0009 (0.0028) model time 0.3658 (0.3757) loss 2.7775 (2.8692) grad_norm 1.6284 (2.2226) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][290/625] eta 0:02:06 lr 0.000397 wd 0.0500 time 0.3693 (0.3781) data time 0.0008 (0.0027) model time 0.3684 (0.3764) loss 3.1075 (2.8654) grad_norm 2.0308 (2.2129) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 09:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][300/625] eta 0:02:02 lr 0.000397 wd 0.0500 time 0.3714 (0.3779) data time 0.0008 (0.0027) model time 0.3706 (0.3761) loss 3.2314 (2.8718) grad_norm 9.9346 (2.2313) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][310/625] eta 0:01:58 lr 0.000397 wd 0.0500 time 0.3770 (0.3777) data time 0.0012 (0.0026) model time 0.3757 (0.3759) loss 3.1783 (2.8745) grad_norm 2.6654 (2.3281) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][320/625] eta 0:01:55 lr 0.000397 wd 0.0500 time 0.5758 (0.3781) data time 0.0013 (0.0026) model time 0.5745 (0.3764) loss 2.6758 (2.8701) grad_norm 2.0083 (2.3248) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][330/625] eta 0:01:51 lr 0.000397 wd 0.0500 time 0.3687 (0.3778) data time 0.0009 (0.0025) model time 0.3678 (0.3761) loss 2.5480 (2.8694) grad_norm 1.6892 (2.3156) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][340/625] eta 0:01:47 lr 0.000397 wd 0.0500 time 0.3728 (0.3776) data time 0.0011 (0.0025) model time 0.3717 (0.3759) loss 3.1837 (2.8696) grad_norm 1.4240 (2.3072) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][350/625] eta 0:01:43 lr 0.000397 wd 0.0500 time 0.3739 (0.3774) data time 0.0011 (0.0025) model time 0.3728 (0.3757) loss 2.7189 (2.8715) grad_norm 2.9038 (2.2982) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][360/625] eta 0:01:39 lr 0.000396 wd 0.0500 time 0.3718 (0.3773) data time 0.0009 (0.0024) model time 0.3709 (0.3755) loss 3.1403 (2.8781) grad_norm 1.6839 (2.2893) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][370/625] eta 0:01:36 lr 0.000396 wd 0.0500 time 0.3676 (0.3771) data time 0.0014 (0.0024) model time 0.3663 (0.3753) loss 3.0705 (2.8715) grad_norm 2.1559 (2.2839) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][380/625] eta 0:01:32 lr 0.000396 wd 0.0500 time 0.3747 (0.3769) data time 0.0009 (0.0024) model time 0.3738 (0.3751) loss 3.1749 (2.8740) grad_norm 1.8885 (2.2778) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][390/625] eta 0:01:28 lr 0.000396 wd 0.0500 time 0.3668 (0.3767) data time 0.0009 (0.0023) model time 0.3659 (0.3749) loss 3.4845 (2.8791) grad_norm 2.1559 (2.2822) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:00:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][400/625] eta 0:01:24 lr 0.000396 wd 0.0500 time 0.3707 (0.3766) data time 0.0009 (0.0023) model time 0.3698 (0.3748) loss 3.3095 (2.8828) grad_norm 1.7914 (2.2709) loss_scale 512.0000 (261.1072) mem 11225MB [2024-09-01 10:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][410/625] eta 0:01:20 lr 0.000396 wd 0.0500 time 0.3740 (0.3765) data time 0.0010 (0.0023) model time 0.3730 (0.3747) loss 2.5501 (2.8837) grad_norm 1.1629 (2.2574) loss_scale 512.0000 (267.2117) mem 11225MB [2024-09-01 10:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][420/625] eta 0:01:17 lr 0.000396 wd 0.0500 time 0.3717 (0.3764) data time 0.0008 (0.0023) model time 0.3709 (0.3746) loss 3.0697 (2.8845) grad_norm 2.5994 (2.2534) loss_scale 512.0000 (273.0261) mem 11225MB [2024-09-01 10:00:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][430/625] eta 0:01:13 lr 0.000395 wd 0.0500 time 0.3722 (0.3762) data time 0.0011 (0.0022) model time 0.3711 (0.3744) loss 3.0952 (2.8867) grad_norm 1.7087 (2.2470) loss_scale 512.0000 (278.5708) mem 11225MB [2024-09-01 10:00:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][440/625] eta 0:01:09 lr 0.000395 wd 0.0500 time 0.3737 (0.3762) data time 0.0008 (0.0022) model time 0.3728 (0.3744) loss 3.6633 (2.8890) grad_norm 2.6925 (2.2448) loss_scale 512.0000 (283.8639) mem 11225MB [2024-09-01 10:00:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][450/625] eta 0:01:05 lr 0.000395 wd 0.0500 time 0.3756 (0.3761) data time 0.0008 (0.0022) model time 0.3748 (0.3743) loss 3.1967 (2.8909) grad_norm 2.4883 (2.2390) loss_scale 512.0000 (288.9224) mem 11225MB [2024-09-01 10:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][460/625] eta 0:01:02 lr 0.000395 wd 0.0500 time 0.5583 (0.3765) data time 0.0011 (0.0022) model time 0.5573 (0.3748) loss 2.8905 (2.8917) grad_norm 1.7623 (2.2425) loss_scale 512.0000 (293.7614) mem 11225MB [2024-09-01 10:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][470/625] eta 0:00:58 lr 0.000395 wd 0.0500 time 0.3748 (0.3772) data time 0.0010 (0.0021) model time 0.3738 (0.3756) loss 2.9940 (2.8946) grad_norm 1.6225 (2.2405) loss_scale 512.0000 (298.3949) mem 11225MB [2024-09-01 10:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][480/625] eta 0:00:54 lr 0.000395 wd 0.0500 time 0.3693 (0.3770) data time 0.0011 (0.0021) model time 0.3682 (0.3754) loss 3.5074 (2.8941) grad_norm 2.5498 (2.2468) loss_scale 512.0000 (302.8358) mem 11225MB [2024-09-01 10:01:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][490/625] eta 0:00:50 lr 0.000395 wd 0.0500 time 0.3645 (0.3769) data time 0.0008 (0.0021) model time 0.3637 (0.3753) loss 3.1209 (2.8940) grad_norm 1.5412 (2.2421) loss_scale 512.0000 (307.0957) mem 11225MB [2024-09-01 10:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][500/625] eta 0:00:47 lr 0.000394 wd 0.0500 time 0.3601 (0.3767) data time 0.0010 (0.0021) model time 0.3591 (0.3751) loss 2.5027 (2.8934) grad_norm 1.8276 (2.2346) loss_scale 512.0000 (311.1856) mem 11225MB [2024-09-01 10:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][510/625] eta 0:00:43 lr 0.000394 wd 0.0500 time 0.3757 (0.3770) data time 0.0009 (0.0021) model time 0.3748 (0.3754) loss 3.4056 (2.8898) grad_norm 1.3064 (2.2303) loss_scale 512.0000 (315.1155) mem 11225MB [2024-09-01 10:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][520/625] eta 0:00:39 lr 0.000394 wd 0.0500 time 0.3705 (0.3769) data time 0.0014 (0.0021) model time 0.3691 (0.3753) loss 3.0202 (2.8871) grad_norm 1.7364 (2.2239) loss_scale 512.0000 (318.8944) mem 11225MB [2024-09-01 10:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][530/625] eta 0:00:35 lr 0.000394 wd 0.0500 time 0.3696 (0.3768) data time 0.0010 (0.0020) model time 0.3687 (0.3751) loss 2.8656 (2.8864) grad_norm 1.3331 (2.2187) loss_scale 512.0000 (322.5311) mem 11225MB [2024-09-01 10:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][540/625] eta 0:00:32 lr 0.000394 wd 0.0500 time 0.3751 (0.3767) data time 0.0011 (0.0020) model time 0.3740 (0.3751) loss 1.9173 (2.8809) grad_norm 1.8992 (2.2104) loss_scale 512.0000 (326.0333) mem 11225MB [2024-09-01 10:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][550/625] eta 0:00:28 lr 0.000394 wd 0.0500 time 0.3772 (0.3767) data time 0.0011 (0.0020) model time 0.3760 (0.3751) loss 2.8257 (2.8814) grad_norm 1.5710 (2.2110) loss_scale 512.0000 (329.4083) mem 11225MB [2024-09-01 10:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][560/625] eta 0:00:24 lr 0.000394 wd 0.0500 time 0.3717 (0.3766) data time 0.0011 (0.0020) model time 0.3706 (0.3750) loss 2.7261 (2.8816) grad_norm 3.6865 (2.2098) loss_scale 512.0000 (332.6631) mem 11225MB [2024-09-01 10:01:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][570/625] eta 0:00:20 lr 0.000394 wd 0.0500 time 0.3698 (0.3765) data time 0.0012 (0.0020) model time 0.3686 (0.3749) loss 3.1672 (2.8829) grad_norm 3.6852 (2.2089) loss_scale 512.0000 (335.8039) mem 11225MB [2024-09-01 10:01:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][580/625] eta 0:00:16 lr 0.000393 wd 0.0500 time 0.5913 (0.3768) data time 0.0011 (0.0020) model time 0.5902 (0.3752) loss 3.1844 (2.8838) grad_norm 2.7495 (2.2080) loss_scale 512.0000 (338.8365) mem 11225MB [2024-09-01 10:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][590/625] eta 0:00:13 lr 0.000393 wd 0.0500 time 0.3656 (0.3770) data time 0.0011 (0.0020) model time 0.3645 (0.3755) loss 3.1370 (2.8860) grad_norm 2.2771 (2.2192) loss_scale 512.0000 (341.7665) mem 11225MB [2024-09-01 10:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][600/625] eta 0:00:09 lr 0.000393 wd 0.0500 time 0.3722 (0.3769) data time 0.0011 (0.0019) model time 0.3711 (0.3754) loss 3.5589 (2.8882) grad_norm 3.0506 (2.2174) loss_scale 512.0000 (344.5990) mem 11225MB [2024-09-01 10:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][610/625] eta 0:00:05 lr 0.000393 wd 0.0500 time 0.3594 (0.3768) data time 0.0006 (0.0019) model time 0.3589 (0.3753) loss 3.6382 (2.8910) grad_norm 2.0386 (2.2138) loss_scale 512.0000 (347.3388) mem 11225MB [2024-09-01 10:01:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [219/300][620/625] eta 0:00:01 lr 0.000393 wd 0.0500 time 0.3615 (0.3765) data time 0.0008 (0.0019) model time 0.3607 (0.3750) loss 3.3244 (2.8911) grad_norm 2.6842 (2.2164) loss_scale 512.0000 (349.9903) mem 11225MB [2024-09-01 10:01:58 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 219 training takes 0:03:55 [2024-09-01 10:01:58 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:01:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:02:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.518 (0.518) Loss 0.5464 (0.5464) Acc@1 88.477 (88.477) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 0.8848 (0.6554) Acc@1 78.760 (85.702) Acc@5 95.410 (97.510) Mem 11225MB [2024-09-01 10:02:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 0.9727 (0.7733) Acc@1 76.953 (82.778) Acc@5 94.482 (96.215) Mem 11225MB [2024-09-01 10:02:02 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.506 Acc@5 96.263 [2024-09-01 10:02:02 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 10:02:02 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.51% [2024-09-01 10:02:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 10:02:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 10:02:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.523 (0.523) Loss 0.4919 (0.4919) Acc@1 89.795 (89.795) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 10:02:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.7876 (0.5965) Acc@1 81.250 (86.737) Acc@5 96.289 (97.798) Mem 11225MB [2024-09-01 10:02:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.8818 (0.7064) Acc@1 78.027 (83.910) Acc@5 95.410 (96.684) Mem 11225MB [2024-09-01 10:02:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.621 Acc@5 96.719 [2024-09-01 10:02:06 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 10:02:06 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.62% [2024-09-01 10:02:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:02:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:02:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][0/625] eta 0:08:35 lr 0.000393 wd 0.0500 time 0.8251 (0.8251) data time 0.4797 (0.4797) model time 0.0000 (0.0000) loss 2.3983 (2.3983) grad_norm 1.5896 (1.5896) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][10/625] eta 0:04:14 lr 0.000393 wd 0.0500 time 0.3673 (0.4138) data time 0.0008 (0.0447) model time 0.0000 (0.0000) loss 3.3507 (2.8502) grad_norm 1.9597 (1.9671) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][20/625] eta 0:03:58 lr 0.000392 wd 0.0500 time 0.3698 (0.3936) data time 0.0009 (0.0240) model time 0.0000 (0.0000) loss 2.1482 (2.8756) grad_norm 2.1219 (1.8897) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][30/625] eta 0:03:49 lr 0.000392 wd 0.0500 time 0.3672 (0.3863) data time 0.0012 (0.0166) model time 0.0000 (0.0000) loss 2.9234 (2.8548) grad_norm 1.8537 (1.9751) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][40/625] eta 0:03:43 lr 0.000392 wd 0.0500 time 0.3877 (0.3828) data time 0.0011 (0.0129) model time 0.0000 (0.0000) loss 3.1350 (2.8383) grad_norm 1.6000 (1.9521) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][50/625] eta 0:03:38 lr 0.000392 wd 0.0500 time 0.3685 (0.3807) data time 0.0009 (0.0106) model time 0.0000 (0.0000) loss 2.7194 (2.8615) grad_norm 2.1384 (2.0013) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][60/625] eta 0:03:34 lr 0.000392 wd 0.0500 time 0.3752 (0.3797) data time 0.0012 (0.0091) model time 0.3739 (0.3735) loss 3.1133 (2.8631) grad_norm 2.2383 (2.1442) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][70/625] eta 0:03:31 lr 0.000392 wd 0.0500 time 0.3371 (0.3819) data time 0.0012 (0.0080) model time 0.3359 (0.3836) loss 3.4052 (2.8889) grad_norm 2.3891 (2.1559) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][80/625] eta 0:03:27 lr 0.000392 wd 0.0500 time 0.3671 (0.3804) data time 0.0011 (0.0071) model time 0.3660 (0.3786) loss 2.8050 (2.8752) grad_norm 2.0427 (2.1382) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][90/625] eta 0:03:23 lr 0.000391 wd 0.0500 time 0.3780 (0.3796) data time 0.0009 (0.0065) model time 0.3772 (0.3771) loss 1.6838 (2.8756) grad_norm 2.2353 (2.1905) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][100/625] eta 0:03:21 lr 0.000391 wd 0.0500 time 0.3797 (0.3830) data time 0.0009 (0.0059) model time 0.3788 (0.3842) loss 2.2004 (2.8839) grad_norm 2.3296 (2.1669) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][110/625] eta 0:03:16 lr 0.000391 wd 0.0500 time 0.3675 (0.3819) data time 0.0010 (0.0055) model time 0.3664 (0.3818) loss 2.9600 (2.8888) grad_norm 1.9478 (2.1514) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][120/625] eta 0:03:12 lr 0.000391 wd 0.0500 time 0.3628 (0.3810) data time 0.0011 (0.0051) model time 0.3617 (0.3800) loss 2.9045 (2.8785) grad_norm 1.4073 (2.1186) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:02:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][130/625] eta 0:03:08 lr 0.000391 wd 0.0500 time 0.3759 (0.3803) data time 0.0009 (0.0048) model time 0.3750 (0.3789) loss 2.5343 (2.8688) grad_norm 2.0984 (2.0930) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][140/625] eta 0:03:04 lr 0.000391 wd 0.0500 time 0.3771 (0.3799) data time 0.0012 (0.0046) model time 0.3759 (0.3782) loss 1.8911 (2.8730) grad_norm 2.4127 (2.0825) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][150/625] eta 0:03:00 lr 0.000391 wd 0.0500 time 0.3652 (0.3794) data time 0.0011 (0.0044) model time 0.3641 (0.3776) loss 3.0308 (2.8814) grad_norm 1.7567 (2.0739) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][160/625] eta 0:02:56 lr 0.000391 wd 0.0500 time 0.3722 (0.3788) data time 0.0009 (0.0042) model time 0.3714 (0.3767) loss 3.0362 (2.8989) grad_norm 1.6062 (2.0792) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][170/625] eta 0:02:52 lr 0.000390 wd 0.0500 time 0.3664 (0.3782) data time 0.0012 (0.0040) model time 0.3652 (0.3760) loss 2.9441 (2.8909) grad_norm 1.5646 (2.0783) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][180/625] eta 0:02:48 lr 0.000390 wd 0.0500 time 0.3703 (0.3779) data time 0.0011 (0.0038) model time 0.3692 (0.3756) loss 2.9244 (2.8826) grad_norm 1.4047 (2.0716) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][190/625] eta 0:02:44 lr 0.000390 wd 0.0500 time 0.3714 (0.3775) data time 0.0011 (0.0037) model time 0.3704 (0.3752) loss 3.0189 (2.8949) grad_norm 1.6641 (2.0588) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][200/625] eta 0:02:40 lr 0.000390 wd 0.0500 time 0.3717 (0.3771) data time 0.0009 (0.0036) model time 0.3709 (0.3747) loss 1.9399 (2.8880) grad_norm 2.0656 (2.0585) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][210/625] eta 0:02:36 lr 0.000390 wd 0.0500 time 0.3671 (0.3768) data time 0.0009 (0.0035) model time 0.3662 (0.3744) loss 3.2109 (2.8909) grad_norm 2.3236 (2.0827) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][220/625] eta 0:02:32 lr 0.000390 wd 0.0500 time 0.3740 (0.3766) data time 0.0009 (0.0033) model time 0.3732 (0.3742) loss 3.0458 (2.8896) grad_norm 13.2072 (2.1289) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][230/625] eta 0:02:28 lr 0.000390 wd 0.0500 time 0.3671 (0.3764) data time 0.0011 (0.0033) model time 0.3660 (0.3740) loss 2.8522 (2.8678) grad_norm 2.6010 (2.1524) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][240/625] eta 0:02:24 lr 0.000389 wd 0.0500 time 0.3676 (0.3762) data time 0.0009 (0.0032) model time 0.3668 (0.3739) loss 2.9663 (2.8619) grad_norm 1.9546 (2.1721) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][250/625] eta 0:02:20 lr 0.000389 wd 0.0500 time 0.3693 (0.3760) data time 0.0011 (0.0031) model time 0.3681 (0.3736) loss 2.9276 (2.8632) grad_norm 1.1620 (2.1545) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][260/625] eta 0:02:17 lr 0.000389 wd 0.0500 time 0.3714 (0.3759) data time 0.0011 (0.0030) model time 0.3702 (0.3735) loss 2.7295 (2.8617) grad_norm 1.6762 (2.1412) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][270/625] eta 0:02:13 lr 0.000389 wd 0.0500 time 0.3659 (0.3757) data time 0.0011 (0.0029) model time 0.3648 (0.3733) loss 3.2492 (2.8556) grad_norm 2.8700 (2.2209) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][280/625] eta 0:02:09 lr 0.000389 wd 0.0500 time 0.3763 (0.3762) data time 0.0012 (0.0029) model time 0.3751 (0.3740) loss 2.9351 (2.8509) grad_norm 1.8421 (2.2293) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:03:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][290/625] eta 0:02:05 lr 0.000389 wd 0.0500 time 0.3640 (0.3761) data time 0.0009 (0.0028) model time 0.3631 (0.3740) loss 3.0473 (2.8467) grad_norm 2.0385 (2.2361) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][300/625] eta 0:02:02 lr 0.000389 wd 0.0500 time 0.3775 (0.3760) data time 0.0011 (0.0028) model time 0.3764 (0.3739) loss 3.1848 (2.8501) grad_norm 2.0822 (2.2343) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][310/625] eta 0:01:58 lr 0.000388 wd 0.0500 time 0.3671 (0.3771) data time 0.0009 (0.0027) model time 0.3662 (0.3752) loss 3.2056 (2.8510) grad_norm 1.5279 (2.2213) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][320/625] eta 0:01:54 lr 0.000388 wd 0.0500 time 0.3718 (0.3770) data time 0.0008 (0.0027) model time 0.3709 (0.3751) loss 2.7025 (2.8501) grad_norm 1.9226 (2.2076) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][330/625] eta 0:01:51 lr 0.000388 wd 0.0500 time 0.3750 (0.3769) data time 0.0008 (0.0026) model time 0.3742 (0.3750) loss 3.6206 (2.8569) grad_norm 2.2557 (2.1970) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][340/625] eta 0:01:47 lr 0.000388 wd 0.0500 time 0.3770 (0.3768) data time 0.0011 (0.0026) model time 0.3759 (0.3749) loss 3.6416 (2.8562) grad_norm 1.7628 (2.1791) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][350/625] eta 0:01:43 lr 0.000388 wd 0.0500 time 0.3677 (0.3767) data time 0.0011 (0.0026) model time 0.3666 (0.3748) loss 2.9016 (2.8497) grad_norm 1.5465 (2.1672) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][360/625] eta 0:01:39 lr 0.000388 wd 0.0500 time 0.3733 (0.3766) data time 0.0009 (0.0025) model time 0.3724 (0.3747) loss 2.0411 (2.8482) grad_norm 1.6553 (2.1718) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][370/625] eta 0:01:35 lr 0.000388 wd 0.0500 time 0.3708 (0.3764) data time 0.0009 (0.0025) model time 0.3699 (0.3745) loss 3.5813 (2.8482) grad_norm 3.6439 (2.1904) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][380/625] eta 0:01:32 lr 0.000387 wd 0.0500 time 0.3705 (0.3763) data time 0.0014 (0.0025) model time 0.3691 (0.3744) loss 2.9236 (2.8467) grad_norm 3.5946 (2.1899) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][390/625] eta 0:01:28 lr 0.000387 wd 0.0500 time 0.3698 (0.3761) data time 0.0013 (0.0024) model time 0.3686 (0.3743) loss 2.7877 (2.8493) grad_norm 2.7035 (2.1972) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][400/625] eta 0:01:24 lr 0.000387 wd 0.0500 time 0.3710 (0.3759) data time 0.0011 (0.0024) model time 0.3699 (0.3741) loss 3.0186 (2.8421) grad_norm 1.6021 (2.1859) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][410/625] eta 0:01:20 lr 0.000387 wd 0.0500 time 0.3783 (0.3758) data time 0.0013 (0.0024) model time 0.3770 (0.3739) loss 2.8751 (2.8401) grad_norm 1.6784 (2.1811) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][420/625] eta 0:01:17 lr 0.000387 wd 0.0500 time 0.3606 (0.3757) data time 0.0011 (0.0023) model time 0.3595 (0.3738) loss 3.0199 (2.8399) grad_norm 1.7314 (2.1748) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][430/625] eta 0:01:13 lr 0.000387 wd 0.0500 time 0.3751 (0.3756) data time 0.0008 (0.0023) model time 0.3743 (0.3737) loss 2.0141 (2.8388) grad_norm 1.7035 (2.1706) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][440/625] eta 0:01:09 lr 0.000387 wd 0.0500 time 0.3708 (0.3755) data time 0.0011 (0.0023) model time 0.3698 (0.3736) loss 3.2293 (2.8456) grad_norm 1.3337 (2.1649) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][450/625] eta 0:01:05 lr 0.000387 wd 0.0500 time 0.3636 (0.3754) data time 0.0014 (0.0023) model time 0.3622 (0.3736) loss 3.3452 (2.8493) grad_norm 3.3477 (2.1664) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][460/625] eta 0:01:01 lr 0.000386 wd 0.0500 time 0.3685 (0.3754) data time 0.0009 (0.0022) model time 0.3675 (0.3735) loss 2.3174 (2.8475) grad_norm 2.7896 (2.1690) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][470/625] eta 0:00:58 lr 0.000386 wd 0.0500 time 0.3701 (0.3758) data time 0.0012 (0.0022) model time 0.3690 (0.3740) loss 2.3431 (2.8473) grad_norm 3.1099 (2.1744) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][480/625] eta 0:00:54 lr 0.000386 wd 0.0500 time 0.3737 (0.3757) data time 0.0011 (0.0022) model time 0.3726 (0.3739) loss 2.9563 (2.8502) grad_norm 1.8610 (2.1776) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][490/625] eta 0:00:50 lr 0.000386 wd 0.0500 time 0.3696 (0.3764) data time 0.0011 (0.0022) model time 0.3686 (0.3747) loss 3.4545 (2.8543) grad_norm 4.0799 (2.1965) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][500/625] eta 0:00:47 lr 0.000386 wd 0.0500 time 0.3828 (0.3763) data time 0.0011 (0.0022) model time 0.3816 (0.3746) loss 1.9469 (2.8564) grad_norm 2.1453 (2.2066) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][510/625] eta 0:00:43 lr 0.000386 wd 0.0500 time 0.3666 (0.3762) data time 0.0010 (0.0021) model time 0.3656 (0.3745) loss 3.4229 (2.8591) grad_norm 2.0441 (2.2079) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][520/625] eta 0:00:39 lr 0.000386 wd 0.0500 time 0.3737 (0.3762) data time 0.0009 (0.0021) model time 0.3728 (0.3745) loss 1.9371 (2.8557) grad_norm 1.7831 (2.2032) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][530/625] eta 0:00:35 lr 0.000385 wd 0.0500 time 0.3703 (0.3761) data time 0.0009 (0.0021) model time 0.3695 (0.3744) loss 3.2337 (2.8598) grad_norm 2.0452 (2.1966) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][540/625] eta 0:00:31 lr 0.000385 wd 0.0500 time 0.3716 (0.3760) data time 0.0012 (0.0021) model time 0.3705 (0.3743) loss 2.8307 (2.8544) grad_norm 2.0338 (2.2013) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][550/625] eta 0:00:28 lr 0.000385 wd 0.0500 time 0.3699 (0.3760) data time 0.0009 (0.0021) model time 0.3690 (0.3743) loss 1.8844 (2.8558) grad_norm 1.8967 (2.1966) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][560/625] eta 0:00:24 lr 0.000385 wd 0.0500 time 0.3708 (0.3759) data time 0.0009 (0.0021) model time 0.3699 (0.3742) loss 2.4303 (2.8531) grad_norm 2.0023 (2.1926) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][570/625] eta 0:00:20 lr 0.000385 wd 0.0500 time 0.3693 (0.3758) data time 0.0013 (0.0021) model time 0.3680 (0.3741) loss 2.1986 (2.8531) grad_norm 3.0148 (2.2020) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][580/625] eta 0:00:16 lr 0.000385 wd 0.0500 time 0.6189 (0.3761) data time 0.0009 (0.0020) model time 0.6181 (0.3745) loss 3.0076 (2.8544) grad_norm 1.5196 (2.2002) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][590/625] eta 0:00:13 lr 0.000385 wd 0.0500 time 0.3724 (0.3760) data time 0.0011 (0.0020) model time 0.3713 (0.3743) loss 3.2131 (2.8602) grad_norm 1.6039 (2.1942) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][600/625] eta 0:00:09 lr 0.000384 wd 0.0500 time 0.3726 (0.3759) data time 0.0013 (0.0020) model time 0.3713 (0.3743) loss 2.1473 (2.8570) grad_norm 1.5974 (2.1923) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:05:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][610/625] eta 0:00:05 lr 0.000384 wd 0.0500 time 0.3588 (0.3762) data time 0.0008 (0.0020) model time 0.3580 (0.3745) loss 2.8301 (2.8586) grad_norm 1.5398 (2.1848) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [220/300][620/625] eta 0:00:01 lr 0.000384 wd 0.0500 time 0.3612 (0.3759) data time 0.0006 (0.0020) model time 0.3606 (0.3743) loss 2.7599 (2.8614) grad_norm 2.6632 (2.1794) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:02 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 220 training takes 0:03:54 [2024-09-01 10:06:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:06:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:06:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.515 (0.515) Loss 0.5229 (0.5229) Acc@1 89.014 (89.014) Acc@5 98.438 (98.438) Mem 11225MB [2024-09-01 10:06:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.148) Loss 0.8540 (0.6445) Acc@1 79.297 (85.738) Acc@5 95.752 (97.612) Mem 11225MB [2024-09-01 10:06:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.122) Loss 0.9478 (0.7680) Acc@1 77.734 (82.768) Acc@5 94.336 (96.317) Mem 11225MB [2024-09-01 10:06:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.504 Acc@5 96.295 [2024-09-01 10:06:06 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 10:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.730 (0.730) Loss 0.4919 (0.4919) Acc@1 89.746 (89.746) Acc@5 98.633 (98.633) Mem 11225MB [2024-09-01 10:06:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.169) Loss 0.7871 (0.5960) Acc@1 81.152 (86.785) Acc@5 96.240 (97.834) Mem 11225MB [2024-09-01 10:06:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.133) Loss 0.8843 (0.7061) Acc@1 77.979 (83.924) Acc@5 95.459 (96.729) Mem 11225MB [2024-09-01 10:06:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.637 Acc@5 96.765 [2024-09-01 10:06:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 10:06:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.64% [2024-09-01 10:06:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:06:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][0/625] eta 0:08:00 lr 0.000384 wd 0.0500 time 0.7687 (0.7687) data time 0.4346 (0.4346) model time 0.0000 (0.0000) loss 3.6717 (3.6717) grad_norm 2.4046 (2.4046) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][10/625] eta 0:04:11 lr 0.000384 wd 0.0500 time 0.3745 (0.4090) data time 0.0010 (0.0406) model time 0.0000 (0.0000) loss 2.8699 (3.0589) grad_norm 1.7496 (2.1455) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][20/625] eta 0:03:57 lr 0.000384 wd 0.0500 time 0.3688 (0.3924) data time 0.0008 (0.0218) model time 0.0000 (0.0000) loss 3.3512 (3.0467) grad_norm 2.2376 (2.0505) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][30/625] eta 0:03:49 lr 0.000384 wd 0.0500 time 0.3640 (0.3852) data time 0.0011 (0.0152) model time 0.0000 (0.0000) loss 2.9896 (3.0154) grad_norm 1.9837 (2.0058) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][40/625] eta 0:03:43 lr 0.000384 wd 0.0500 time 0.3720 (0.3818) data time 0.0011 (0.0118) model time 0.0000 (0.0000) loss 2.1328 (2.9704) grad_norm 2.5968 (2.2696) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][50/625] eta 0:03:38 lr 0.000383 wd 0.0500 time 0.3692 (0.3798) data time 0.0012 (0.0097) model time 0.0000 (0.0000) loss 2.0565 (2.9589) grad_norm 2.1454 (2.2074) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][60/625] eta 0:03:34 lr 0.000383 wd 0.0500 time 0.3651 (0.3788) data time 0.0010 (0.0083) model time 0.3642 (0.3725) loss 3.0219 (2.9167) grad_norm 2.2869 (2.1636) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][70/625] eta 0:03:29 lr 0.000383 wd 0.0500 time 0.3729 (0.3779) data time 0.0008 (0.0073) model time 0.3721 (0.3719) loss 2.6656 (2.9263) grad_norm 1.3570 (2.1234) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][80/625] eta 0:03:25 lr 0.000383 wd 0.0500 time 0.3797 (0.3775) data time 0.0012 (0.0065) model time 0.3785 (0.3724) loss 2.4837 (2.9378) grad_norm 1.5072 (2.1041) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][90/625] eta 0:03:21 lr 0.000383 wd 0.0500 time 0.3756 (0.3771) data time 0.0011 (0.0059) model time 0.3746 (0.3724) loss 2.7682 (2.9338) grad_norm 1.3253 (2.0999) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][100/625] eta 0:03:17 lr 0.000383 wd 0.0500 time 0.3741 (0.3767) data time 0.0011 (0.0055) model time 0.3730 (0.3724) loss 2.2627 (2.9132) grad_norm 1.4310 (2.0497) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][110/625] eta 0:03:14 lr 0.000383 wd 0.0500 time 0.3734 (0.3773) data time 0.0009 (0.0051) model time 0.3726 (0.3741) loss 3.3380 (2.8747) grad_norm 1.6963 (2.0359) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][120/625] eta 0:03:10 lr 0.000382 wd 0.0500 time 0.3803 (0.3770) data time 0.0011 (0.0048) model time 0.3792 (0.3738) loss 3.3874 (2.8688) grad_norm 1.5787 (2.0578) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][130/625] eta 0:03:06 lr 0.000382 wd 0.0500 time 0.3662 (0.3765) data time 0.0009 (0.0045) model time 0.3653 (0.3732) loss 2.9321 (2.8773) grad_norm 2.3361 (2.0780) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][140/625] eta 0:03:02 lr 0.000382 wd 0.0500 time 0.3702 (0.3762) data time 0.0009 (0.0043) model time 0.3693 (0.3729) loss 2.4362 (2.8891) grad_norm 3.9611 (2.0955) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][150/625] eta 0:02:58 lr 0.000382 wd 0.0500 time 0.3685 (0.3757) data time 0.0011 (0.0041) model time 0.3675 (0.3725) loss 3.0231 (2.8869) grad_norm 2.3847 (2.1042) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][160/625] eta 0:02:54 lr 0.000382 wd 0.0500 time 0.3901 (0.3758) data time 0.0011 (0.0039) model time 0.3890 (0.3728) loss 3.2432 (2.8843) grad_norm 2.3367 (2.1192) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][170/625] eta 0:02:52 lr 0.000382 wd 0.0500 time 0.3688 (0.3781) data time 0.0012 (0.0037) model time 0.3675 (0.3761) loss 1.8937 (2.8758) grad_norm 2.5681 (2.1167) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][180/625] eta 0:02:48 lr 0.000382 wd 0.0500 time 0.3711 (0.3778) data time 0.0013 (0.0036) model time 0.3698 (0.3759) loss 2.9783 (2.8780) grad_norm 2.7209 (2.1431) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][190/625] eta 0:02:44 lr 0.000381 wd 0.0500 time 0.3821 (0.3775) data time 0.0008 (0.0035) model time 0.3813 (0.3755) loss 2.1771 (2.8727) grad_norm 1.9534 (2.1365) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][200/625] eta 0:02:40 lr 0.000381 wd 0.0500 time 0.3775 (0.3771) data time 0.0010 (0.0033) model time 0.3765 (0.3750) loss 1.9038 (2.8658) grad_norm 2.1661 (2.1490) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][210/625] eta 0:02:36 lr 0.000381 wd 0.0500 time 0.3717 (0.3768) data time 0.0008 (0.0032) model time 0.3709 (0.3747) loss 3.1799 (2.8702) grad_norm 2.5643 (2.1602) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][220/625] eta 0:02:32 lr 0.000381 wd 0.0500 time 0.3704 (0.3765) data time 0.0009 (0.0031) model time 0.3695 (0.3743) loss 3.1321 (2.8627) grad_norm 1.7630 (2.1980) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][230/625] eta 0:02:28 lr 0.000381 wd 0.0500 time 0.3741 (0.3764) data time 0.0012 (0.0031) model time 0.3730 (0.3742) loss 3.2452 (2.8670) grad_norm 1.4289 (2.1769) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][240/625] eta 0:02:24 lr 0.000381 wd 0.0500 time 0.3654 (0.3761) data time 0.0009 (0.0030) model time 0.3645 (0.3740) loss 2.2830 (2.8531) grad_norm 1.5601 (2.1742) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][250/625] eta 0:02:20 lr 0.000381 wd 0.0500 time 0.3724 (0.3759) data time 0.0009 (0.0029) model time 0.3714 (0.3737) loss 3.6637 (2.8621) grad_norm 2.6603 (2.1730) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][260/625] eta 0:02:17 lr 0.000381 wd 0.0500 time 0.3692 (0.3757) data time 0.0011 (0.0029) model time 0.3681 (0.3735) loss 3.1454 (2.8633) grad_norm 2.6469 (2.1709) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][270/625] eta 0:02:13 lr 0.000380 wd 0.0500 time 0.3723 (0.3755) data time 0.0011 (0.0028) model time 0.3712 (0.3733) loss 3.1185 (2.8699) grad_norm 4.1109 (2.1688) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:07:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][280/625] eta 0:02:09 lr 0.000380 wd 0.0500 time 0.3767 (0.3755) data time 0.0011 (0.0027) model time 0.3756 (0.3733) loss 2.9861 (2.8740) grad_norm 1.8154 (2.1659) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][290/625] eta 0:02:06 lr 0.000380 wd 0.0500 time 0.3706 (0.3766) data time 0.0008 (0.0027) model time 0.3697 (0.3747) loss 2.8181 (2.8788) grad_norm 2.7361 (2.1612) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][300/625] eta 0:02:02 lr 0.000380 wd 0.0500 time 0.3712 (0.3764) data time 0.0009 (0.0026) model time 0.3702 (0.3745) loss 3.5924 (2.8828) grad_norm 1.7205 (2.1730) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][310/625] eta 0:01:58 lr 0.000380 wd 0.0500 time 0.3688 (0.3762) data time 0.0012 (0.0026) model time 0.3676 (0.3744) loss 3.0218 (2.8883) grad_norm 1.7031 (2.1742) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][320/625] eta 0:01:54 lr 0.000380 wd 0.0500 time 0.3691 (0.3761) data time 0.0009 (0.0025) model time 0.3682 (0.3742) loss 1.9755 (2.8855) grad_norm 1.7373 (2.1690) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][330/625] eta 0:01:50 lr 0.000380 wd 0.0500 time 0.3687 (0.3759) data time 0.0011 (0.0025) model time 0.3677 (0.3740) loss 2.9474 (2.8876) grad_norm 4.4573 (2.1864) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][340/625] eta 0:01:47 lr 0.000379 wd 0.0500 time 0.3777 (0.3758) data time 0.0011 (0.0025) model time 0.3766 (0.3739) loss 3.2981 (2.8885) grad_norm 3.9554 (2.2280) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][350/625] eta 0:01:43 lr 0.000379 wd 0.0500 time 0.3763 (0.3756) data time 0.0011 (0.0024) model time 0.3752 (0.3737) loss 3.1118 (2.8870) grad_norm 1.7109 (2.2260) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][360/625] eta 0:01:39 lr 0.000379 wd 0.0500 time 0.3711 (0.3755) data time 0.0010 (0.0024) model time 0.3701 (0.3736) loss 2.7478 (2.8809) grad_norm 1.5767 (2.2238) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][370/625] eta 0:01:35 lr 0.000379 wd 0.0500 time 0.3691 (0.3754) data time 0.0011 (0.0024) model time 0.3680 (0.3735) loss 2.2955 (2.8737) grad_norm 2.8319 (2.2363) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][380/625] eta 0:01:31 lr 0.000379 wd 0.0500 time 0.3708 (0.3753) data time 0.0009 (0.0023) model time 0.3699 (0.3734) loss 3.2822 (2.8727) grad_norm 2.7341 (2.2392) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][390/625] eta 0:01:28 lr 0.000379 wd 0.0500 time 0.3666 (0.3751) data time 0.0009 (0.0023) model time 0.3657 (0.3732) loss 3.1598 (2.8776) grad_norm 2.7696 (2.2370) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][400/625] eta 0:01:24 lr 0.000379 wd 0.0500 time 0.3707 (0.3751) data time 0.0010 (0.0023) model time 0.3697 (0.3732) loss 3.0307 (2.8808) grad_norm 1.5484 (2.2627) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][410/625] eta 0:01:20 lr 0.000378 wd 0.0500 time 0.3672 (0.3755) data time 0.0011 (0.0023) model time 0.3660 (0.3737) loss 2.9630 (2.8767) grad_norm 1.8235 (2.2551) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][420/625] eta 0:01:16 lr 0.000378 wd 0.0500 time 0.3755 (0.3754) data time 0.0012 (0.0022) model time 0.3743 (0.3736) loss 3.2578 (2.8779) grad_norm 1.7941 (2.2409) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][430/625] eta 0:01:13 lr 0.000378 wd 0.0500 time 0.3672 (0.3753) data time 0.0008 (0.0022) model time 0.3663 (0.3735) loss 2.1287 (2.8748) grad_norm 2.5185 (2.2376) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:08:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][440/625] eta 0:01:09 lr 0.000378 wd 0.0500 time 0.3672 (0.3752) data time 0.0013 (0.0022) model time 0.3659 (0.3734) loss 3.1324 (2.8754) grad_norm 1.6888 (2.2342) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][450/625] eta 0:01:05 lr 0.000378 wd 0.0500 time 0.3696 (0.3752) data time 0.0013 (0.0022) model time 0.3682 (0.3734) loss 2.3994 (2.8710) grad_norm 1.9577 (2.2274) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][460/625] eta 0:01:01 lr 0.000378 wd 0.0500 time 0.3675 (0.3750) data time 0.0009 (0.0021) model time 0.3667 (0.3733) loss 2.6278 (2.8725) grad_norm 3.4840 (2.2330) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][470/625] eta 0:00:58 lr 0.000378 wd 0.0500 time 0.3696 (0.3750) data time 0.0010 (0.0021) model time 0.3686 (0.3732) loss 2.9095 (2.8727) grad_norm 1.8216 (2.2348) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][480/625] eta 0:00:54 lr 0.000378 wd 0.0500 time 0.3669 (0.3749) data time 0.0014 (0.0021) model time 0.3655 (0.3731) loss 3.2844 (2.8753) grad_norm 2.1171 (2.2278) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][490/625] eta 0:00:50 lr 0.000377 wd 0.0500 time 0.3684 (0.3748) data time 0.0012 (0.0021) model time 0.3673 (0.3730) loss 3.1563 (2.8783) grad_norm 1.7257 (2.2235) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][500/625] eta 0:00:46 lr 0.000377 wd 0.0500 time 0.3741 (0.3752) data time 0.0009 (0.0021) model time 0.3732 (0.3735) loss 2.4432 (2.8738) grad_norm 1.2318 (2.2158) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][510/625] eta 0:00:43 lr 0.000377 wd 0.0500 time 0.3739 (0.3751) data time 0.0009 (0.0021) model time 0.3731 (0.3734) loss 3.0322 (2.8743) grad_norm 1.9836 (2.2072) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][520/625] eta 0:00:39 lr 0.000377 wd 0.0500 time 0.3735 (0.3750) data time 0.0011 (0.0020) model time 0.3724 (0.3733) loss 3.1173 (2.8745) grad_norm 1.6322 (2.2012) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][530/625] eta 0:00:35 lr 0.000377 wd 0.0500 time 0.3683 (0.3750) data time 0.0013 (0.0020) model time 0.3670 (0.3733) loss 2.9635 (2.8743) grad_norm 6.1641 (2.2095) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][540/625] eta 0:00:31 lr 0.000377 wd 0.0500 time 0.3733 (0.3748) data time 0.0011 (0.0020) model time 0.3722 (0.3731) loss 3.0694 (2.8761) grad_norm 2.0166 (2.2112) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][550/625] eta 0:00:28 lr 0.000377 wd 0.0500 time 0.3697 (0.3748) data time 0.0009 (0.0020) model time 0.3688 (0.3731) loss 2.5263 (2.8753) grad_norm 1.6100 (2.2173) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][560/625] eta 0:00:24 lr 0.000376 wd 0.0500 time 0.3826 (0.3748) data time 0.0009 (0.0020) model time 0.3817 (0.3730) loss 3.5358 (2.8718) grad_norm 2.0844 (2.2206) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][570/625] eta 0:00:20 lr 0.000376 wd 0.0500 time 0.3883 (0.3748) data time 0.0011 (0.0020) model time 0.3872 (0.3731) loss 3.1580 (2.8735) grad_norm 4.2601 (2.2196) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][580/625] eta 0:00:16 lr 0.000376 wd 0.0500 time 0.3663 (0.3747) data time 0.0009 (0.0020) model time 0.3655 (0.3730) loss 3.8222 (2.8763) grad_norm 1.6782 (2.2233) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][590/625] eta 0:00:13 lr 0.000376 wd 0.0500 time 0.3739 (0.3746) data time 0.0012 (0.0019) model time 0.3727 (0.3729) loss 2.6203 (2.8756) grad_norm 2.8782 (2.2234) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][600/625] eta 0:00:09 lr 0.000376 wd 0.0500 time 0.3763 (0.3746) data time 0.0012 (0.0019) model time 0.3752 (0.3729) loss 2.9318 (2.8741) grad_norm 2.4536 (2.2282) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:09:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][610/625] eta 0:00:05 lr 0.000376 wd 0.0500 time 0.3676 (0.3745) data time 0.0008 (0.0019) model time 0.3668 (0.3728) loss 2.9697 (2.8771) grad_norm 1.8675 (2.2328) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [221/300][620/625] eta 0:00:01 lr 0.000376 wd 0.0500 time 0.3611 (0.3746) data time 0.0008 (0.0019) model time 0.3603 (0.3730) loss 3.0513 (2.8829) grad_norm 1.8775 (2.2365) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:04 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 221 training takes 0:03:54 [2024-09-01 10:10:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:10:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:10:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.531 (0.531) Loss 0.5435 (0.5435) Acc@1 88.232 (88.232) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 10:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.145) Loss 0.8730 (0.6412) Acc@1 79.492 (85.929) Acc@5 95.557 (97.559) Mem 11225MB [2024-09-01 10:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 0.9517 (0.7624) Acc@1 77.734 (82.838) Acc@5 94.727 (96.280) Mem 11225MB [2024-09-01 10:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.562 Acc@5 96.303 [2024-09-01 10:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.6% [2024-09-01 10:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.56% [2024-09-01 10:10:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 10:10:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 10:10:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.521 (0.521) Loss 0.4919 (0.4919) Acc@1 89.648 (89.648) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 10:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.7866 (0.5955) Acc@1 81.445 (86.772) Acc@5 96.240 (97.843) Mem 11225MB [2024-09-01 10:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.8818 (0.7056) Acc@1 78.076 (83.917) Acc@5 95.215 (96.726) Mem 11225MB [2024-09-01 10:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.645 Acc@5 96.757 [2024-09-01 10:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 10:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.65% [2024-09-01 10:10:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:10:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][0/625] eta 0:08:51 lr 0.000376 wd 0.0500 time 0.8502 (0.8502) data time 0.5123 (0.5123) model time 0.0000 (0.0000) loss 3.8308 (3.8308) grad_norm 1.7837 (1.7837) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][10/625] eta 0:04:17 lr 0.000375 wd 0.0500 time 0.3703 (0.4186) data time 0.0009 (0.0477) model time 0.0000 (0.0000) loss 3.4762 (3.1235) grad_norm 1.6873 (1.9256) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][20/625] eta 0:04:01 lr 0.000375 wd 0.0500 time 0.3693 (0.3985) data time 0.0008 (0.0255) model time 0.0000 (0.0000) loss 2.2296 (2.9489) grad_norm 2.3771 (1.9120) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][30/625] eta 0:03:52 lr 0.000375 wd 0.0500 time 0.3698 (0.3902) data time 0.0011 (0.0177) model time 0.0000 (0.0000) loss 2.8588 (2.9364) grad_norm 1.6526 (2.0229) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][40/625] eta 0:03:44 lr 0.000375 wd 0.0500 time 0.3691 (0.3846) data time 0.0011 (0.0137) model time 0.0000 (0.0000) loss 2.5405 (2.9576) grad_norm 1.6127 (1.9623) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][50/625] eta 0:03:39 lr 0.000375 wd 0.0500 time 0.3678 (0.3820) data time 0.0009 (0.0113) model time 0.0000 (0.0000) loss 3.3939 (2.9629) grad_norm 1.8684 (1.9191) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][60/625] eta 0:03:37 lr 0.000375 wd 0.0500 time 0.5805 (0.3857) data time 0.0010 (0.0096) model time 0.5795 (0.4032) loss 2.2656 (2.9612) grad_norm 1.9610 (1.8895) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][70/625] eta 0:03:33 lr 0.000375 wd 0.0500 time 0.3762 (0.3855) data time 0.0009 (0.0084) model time 0.3753 (0.3932) loss 3.3209 (2.9791) grad_norm 1.8981 (1.9024) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][80/625] eta 0:03:29 lr 0.000374 wd 0.0500 time 0.3715 (0.3836) data time 0.0011 (0.0075) model time 0.3704 (0.3850) loss 2.9142 (2.9569) grad_norm 1.6382 (1.9239) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][90/625] eta 0:03:24 lr 0.000374 wd 0.0500 time 0.3803 (0.3821) data time 0.0011 (0.0068) model time 0.3792 (0.3811) loss 3.0048 (2.9827) grad_norm 6.8463 (1.9595) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][100/625] eta 0:03:20 lr 0.000374 wd 0.0500 time 0.3684 (0.3811) data time 0.0011 (0.0063) model time 0.3672 (0.3790) loss 2.4544 (2.9663) grad_norm 1.6721 (1.9749) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][110/625] eta 0:03:15 lr 0.000374 wd 0.0500 time 0.3775 (0.3801) data time 0.0011 (0.0058) model time 0.3764 (0.3773) loss 2.8639 (2.9502) grad_norm 3.6476 (2.0377) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][120/625] eta 0:03:11 lr 0.000374 wd 0.0500 time 0.3678 (0.3794) data time 0.0009 (0.0054) model time 0.3668 (0.3763) loss 3.5583 (2.9324) grad_norm 1.9209 (2.0790) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][130/625] eta 0:03:07 lr 0.000374 wd 0.0500 time 0.3699 (0.3787) data time 0.0013 (0.0051) model time 0.3686 (0.3754) loss 3.5663 (2.9230) grad_norm 2.2814 (2.0712) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][140/625] eta 0:03:03 lr 0.000374 wd 0.0500 time 0.3685 (0.3782) data time 0.0010 (0.0049) model time 0.3675 (0.3748) loss 3.5035 (2.9155) grad_norm 1.6886 (2.0526) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][150/625] eta 0:02:59 lr 0.000373 wd 0.0500 time 0.3682 (0.3778) data time 0.0009 (0.0046) model time 0.3673 (0.3744) loss 3.3293 (2.9216) grad_norm 1.7472 (2.1453) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][160/625] eta 0:02:55 lr 0.000373 wd 0.0500 time 0.3685 (0.3774) data time 0.0011 (0.0044) model time 0.3674 (0.3740) loss 2.8555 (2.9106) grad_norm 1.5290 (2.1370) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][170/625] eta 0:02:51 lr 0.000373 wd 0.0500 time 0.3711 (0.3772) data time 0.0011 (0.0042) model time 0.3699 (0.3740) loss 3.0748 (2.9195) grad_norm 2.0099 (2.1241) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][180/625] eta 0:02:47 lr 0.000373 wd 0.0500 time 0.3777 (0.3769) data time 0.0012 (0.0040) model time 0.3765 (0.3737) loss 3.0789 (2.9235) grad_norm 1.8597 (2.1385) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][190/625] eta 0:02:43 lr 0.000373 wd 0.0500 time 0.3755 (0.3766) data time 0.0009 (0.0039) model time 0.3746 (0.3734) loss 3.2354 (2.9201) grad_norm 2.9818 (2.1586) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][200/625] eta 0:02:39 lr 0.000373 wd 0.0500 time 0.3707 (0.3763) data time 0.0011 (0.0038) model time 0.3696 (0.3732) loss 3.1640 (2.9083) grad_norm 1.7804 (2.1522) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][210/625] eta 0:02:36 lr 0.000373 wd 0.0500 time 0.3761 (0.3761) data time 0.0008 (0.0036) model time 0.3753 (0.3730) loss 3.2616 (2.9052) grad_norm 2.1220 (2.1561) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][220/625] eta 0:02:32 lr 0.000373 wd 0.0500 time 0.3756 (0.3759) data time 0.0011 (0.0035) model time 0.3745 (0.3729) loss 3.4067 (2.9053) grad_norm 2.0472 (2.1520) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][230/625] eta 0:02:28 lr 0.000372 wd 0.0500 time 0.3675 (0.3757) data time 0.0012 (0.0034) model time 0.3664 (0.3727) loss 3.1190 (2.9146) grad_norm 2.8343 (2.2714) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][240/625] eta 0:02:24 lr 0.000372 wd 0.0500 time 0.3761 (0.3756) data time 0.0009 (0.0033) model time 0.3753 (0.3727) loss 2.7949 (2.9167) grad_norm 1.9488 (2.2595) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][250/625] eta 0:02:21 lr 0.000372 wd 0.0500 time 0.3801 (0.3770) data time 0.0009 (0.0032) model time 0.3792 (0.3745) loss 3.0181 (2.9045) grad_norm 1.7434 (2.2448) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][260/625] eta 0:02:17 lr 0.000372 wd 0.0500 time 0.3736 (0.3769) data time 0.0012 (0.0032) model time 0.3724 (0.3745) loss 2.8484 (2.9077) grad_norm 1.9959 (2.2581) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:11:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][270/625] eta 0:02:13 lr 0.000372 wd 0.0500 time 0.3698 (0.3768) data time 0.0009 (0.0031) model time 0.3689 (0.3743) loss 2.6654 (2.9087) grad_norm 4.1707 (2.2674) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][280/625] eta 0:02:10 lr 0.000372 wd 0.0500 time 0.3663 (0.3772) data time 0.0008 (0.0030) model time 0.3655 (0.3750) loss 3.6303 (2.9196) grad_norm 3.2263 (2.2634) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][290/625] eta 0:02:06 lr 0.000372 wd 0.0500 time 0.3624 (0.3769) data time 0.0011 (0.0030) model time 0.3613 (0.3746) loss 3.4110 (2.9050) grad_norm 1.9331 (2.2756) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][300/625] eta 0:02:02 lr 0.000371 wd 0.0500 time 0.3651 (0.3767) data time 0.0012 (0.0029) model time 0.3639 (0.3744) loss 3.3507 (2.9051) grad_norm 3.4305 (2.2703) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][310/625] eta 0:01:58 lr 0.000371 wd 0.0500 time 0.3746 (0.3765) data time 0.0009 (0.0029) model time 0.3737 (0.3742) loss 2.9494 (2.9061) grad_norm 1.6464 (2.2574) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][320/625] eta 0:01:54 lr 0.000371 wd 0.0500 time 0.3627 (0.3763) data time 0.0010 (0.0028) model time 0.3617 (0.3740) loss 2.9497 (2.9110) grad_norm 1.6195 (2.2447) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][330/625] eta 0:01:50 lr 0.000371 wd 0.0500 time 0.3734 (0.3762) data time 0.0011 (0.0028) model time 0.3722 (0.3739) loss 3.0027 (2.9152) grad_norm 1.8837 (2.2627) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][340/625] eta 0:01:47 lr 0.000371 wd 0.0500 time 0.3723 (0.3761) data time 0.0008 (0.0027) model time 0.3715 (0.3738) loss 3.4916 (2.9143) grad_norm 1.8873 (2.2503) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][350/625] eta 0:01:43 lr 0.000371 wd 0.0500 time 0.3675 (0.3759) data time 0.0016 (0.0027) model time 0.3660 (0.3737) loss 2.5938 (2.9071) grad_norm 1.8827 (2.2326) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][360/625] eta 0:01:39 lr 0.000371 wd 0.0500 time 0.3593 (0.3757) data time 0.0011 (0.0026) model time 0.3582 (0.3735) loss 2.1768 (2.9034) grad_norm 1.6177 (2.2223) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][370/625] eta 0:01:35 lr 0.000370 wd 0.0500 time 0.3742 (0.3762) data time 0.0009 (0.0026) model time 0.3734 (0.3740) loss 2.2528 (2.9019) grad_norm 2.2863 (2.2128) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][380/625] eta 0:01:32 lr 0.000370 wd 0.0500 time 0.3705 (0.3761) data time 0.0009 (0.0026) model time 0.3696 (0.3739) loss 2.3453 (2.8942) grad_norm 2.3558 (2.2065) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][390/625] eta 0:01:28 lr 0.000370 wd 0.0500 time 0.3820 (0.3759) data time 0.0013 (0.0025) model time 0.3807 (0.3738) loss 2.3867 (2.8922) grad_norm 1.7160 (2.2034) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][400/625] eta 0:01:24 lr 0.000370 wd 0.0500 time 0.3653 (0.3758) data time 0.0012 (0.0025) model time 0.3641 (0.3737) loss 2.4520 (2.8884) grad_norm 1.8603 (2.1974) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][410/625] eta 0:01:20 lr 0.000370 wd 0.0500 time 0.3645 (0.3757) data time 0.0011 (0.0025) model time 0.3634 (0.3735) loss 2.4844 (2.8868) grad_norm 2.1638 (2.1907) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][420/625] eta 0:01:16 lr 0.000370 wd 0.0500 time 0.3684 (0.3756) data time 0.0011 (0.0024) model time 0.3673 (0.3735) loss 2.9564 (2.8852) grad_norm 2.5286 (2.1832) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][430/625] eta 0:01:13 lr 0.000370 wd 0.0500 time 0.3864 (0.3755) data time 0.0009 (0.0024) model time 0.3855 (0.3734) loss 1.9000 (2.8829) grad_norm 1.5115 (2.1852) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:12:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][440/625] eta 0:01:09 lr 0.000370 wd 0.0500 time 0.3740 (0.3754) data time 0.0011 (0.0024) model time 0.3729 (0.3734) loss 3.2629 (2.8776) grad_norm 1.7324 (2.1792) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:13:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][450/625] eta 0:01:05 lr 0.000369 wd 0.0500 time 0.3660 (0.3753) data time 0.0010 (0.0023) model time 0.3650 (0.3732) loss 2.9993 (2.8800) grad_norm 2.5156 (2.2008) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:13:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][460/625] eta 0:01:01 lr 0.000369 wd 0.0500 time 0.3668 (0.3753) data time 0.0011 (0.0023) model time 0.3657 (0.3732) loss 3.3434 (2.8807) grad_norm 1.7744 (2.1923) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][470/625] eta 0:00:58 lr 0.000369 wd 0.0500 time 0.3834 (0.3756) data time 0.0008 (0.0023) model time 0.3825 (0.3736) loss 2.4982 (2.8764) grad_norm 2.1305 (2.1868) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:13:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][480/625] eta 0:00:54 lr 0.000369 wd 0.0500 time 0.3687 (0.3755) data time 0.0011 (0.0023) model time 0.3676 (0.3736) loss 2.9427 (2.8813) grad_norm 1.6485 (2.1815) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:13:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][490/625] eta 0:00:50 lr 0.000369 wd 0.0500 time 0.3648 (0.3755) data time 0.0008 (0.0022) model time 0.3640 (0.3735) loss 2.4865 (2.8820) grad_norm 2.0913 (2.1816) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:13:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][500/625] eta 0:00:46 lr 0.000369 wd 0.0500 time 0.3759 (0.3754) data time 0.0009 (0.0022) model time 0.3751 (0.3734) loss 2.5245 (2.8802) grad_norm 2.1218 (2.1838) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:13:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][510/625] eta 0:00:43 lr 0.000369 wd 0.0500 time 0.3677 (0.3753) data time 0.0009 (0.0022) model time 0.3668 (0.3733) loss 3.6254 (2.8807) grad_norm 2.6211 (2.1959) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:13:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][520/625] eta 0:00:39 lr 0.000368 wd 0.0500 time 0.3664 (0.3752) data time 0.0012 (0.0022) model time 0.3652 (0.3733) loss 2.3108 (2.8783) grad_norm 2.9963 (2.1986) loss_scale 1024.0000 (514.9482) mem 11225MB [2024-09-01 10:13:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][530/625] eta 0:00:35 lr 0.000368 wd 0.0500 time 0.3635 (0.3752) data time 0.0011 (0.0022) model time 0.3624 (0.3733) loss 3.2926 (2.8797) grad_norm 2.4646 (2.1960) loss_scale 1024.0000 (524.5348) mem 11225MB [2024-09-01 10:13:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][540/625] eta 0:00:31 lr 0.000368 wd 0.0500 time 0.3845 (0.3752) data time 0.0010 (0.0022) model time 0.3835 (0.3733) loss 1.6727 (2.8706) grad_norm 1.4671 (2.1948) loss_scale 1024.0000 (533.7671) mem 11225MB [2024-09-01 10:13:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][550/625] eta 0:00:28 lr 0.000368 wd 0.0500 time 0.3813 (0.3751) data time 0.0014 (0.0021) model time 0.3798 (0.3732) loss 3.0847 (2.8716) grad_norm 1.8773 (2.1893) loss_scale 1024.0000 (542.6642) mem 11225MB [2024-09-01 10:13:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][560/625] eta 0:00:24 lr 0.000368 wd 0.0500 time 0.3720 (0.3752) data time 0.0012 (0.0021) model time 0.3708 (0.3733) loss 2.1125 (2.8727) grad_norm 2.7265 (2.1884) loss_scale 1024.0000 (551.2442) mem 11225MB [2024-09-01 10:13:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][570/625] eta 0:00:20 lr 0.000368 wd 0.0500 time 0.3700 (0.3751) data time 0.0016 (0.0021) model time 0.3684 (0.3732) loss 2.0718 (2.8688) grad_norm 1.7078 (2.1884) loss_scale 1024.0000 (559.5236) mem 11225MB [2024-09-01 10:13:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][580/625] eta 0:00:16 lr 0.000368 wd 0.0500 time 0.6043 (0.3759) data time 0.0011 (0.0021) model time 0.6032 (0.3741) loss 3.1934 (2.8684) grad_norm 1.7889 (2.1812) loss_scale 1024.0000 (567.5181) mem 11225MB [2024-09-01 10:13:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][590/625] eta 0:00:13 lr 0.000368 wd 0.0500 time 0.3789 (0.3757) data time 0.0010 (0.0021) model time 0.3779 (0.3739) loss 2.7033 (2.8680) grad_norm 2.2089 (2.1796) loss_scale 1024.0000 (575.2420) mem 11225MB [2024-09-01 10:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][600/625] eta 0:00:09 lr 0.000367 wd 0.0500 time 0.3778 (0.3756) data time 0.0011 (0.0021) model time 0.3767 (0.3738) loss 2.7867 (2.8669) grad_norm 1.8656 (2.1771) loss_scale 1024.0000 (582.7088) mem 11225MB [2024-09-01 10:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][610/625] eta 0:00:05 lr 0.000367 wd 0.0500 time 0.3651 (0.3756) data time 0.0008 (0.0021) model time 0.3643 (0.3738) loss 2.8512 (2.8722) grad_norm 2.4057 (2.1769) loss_scale 1024.0000 (589.9313) mem 11225MB [2024-09-01 10:14:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [222/300][620/625] eta 0:00:01 lr 0.000367 wd 0.0500 time 0.3614 (0.3753) data time 0.0008 (0.0021) model time 0.3606 (0.3735) loss 2.4709 (2.8679) grad_norm 2.0442 (2.1747) loss_scale 1024.0000 (596.9211) mem 11225MB [2024-09-01 10:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 222 training takes 0:03:54 [2024-09-01 10:14:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:14:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:14:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.529 (0.529) Loss 0.5508 (0.5508) Acc@1 88.330 (88.330) Acc@5 98.291 (98.291) Mem 11225MB [2024-09-01 10:14:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.8799 (0.6564) Acc@1 78.906 (85.542) Acc@5 95.752 (97.510) Mem 11225MB [2024-09-01 10:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 1.0146 (0.7760) Acc@1 76.074 (82.761) Acc@5 94.482 (96.277) Mem 11225MB [2024-09-01 10:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.528 Acc@5 96.311 [2024-09-01 10:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.5% [2024-09-01 10:14:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 1.048 (1.048) Loss 0.4922 (0.4922) Acc@1 89.600 (89.600) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 10:14:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.195) Loss 0.7881 (0.5955) Acc@1 81.396 (86.768) Acc@5 96.240 (97.852) Mem 11225MB [2024-09-01 10:14:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.146) Loss 0.8813 (0.7054) Acc@1 78.125 (83.896) Acc@5 95.215 (96.740) Mem 11225MB [2024-09-01 10:14:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.627 Acc@5 96.771 [2024-09-01 10:14:16 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 10:14:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][0/625] eta 0:12:18 lr 0.000367 wd 0.0500 time 1.1817 (1.1817) data time 0.6428 (0.6428) model time 0.0000 (0.0000) loss 3.0237 (3.0237) grad_norm 1.9533 (1.9533) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][10/625] eta 0:04:33 lr 0.000367 wd 0.0500 time 0.3696 (0.4441) data time 0.0010 (0.0597) model time 0.0000 (0.0000) loss 2.6665 (2.7040) grad_norm 2.8690 (2.1776) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][20/625] eta 0:04:08 lr 0.000367 wd 0.0500 time 0.3681 (0.4109) data time 0.0011 (0.0319) model time 0.0000 (0.0000) loss 2.7469 (2.7792) grad_norm 2.4108 (2.5367) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][30/625] eta 0:03:57 lr 0.000367 wd 0.0500 time 0.3851 (0.3988) data time 0.0009 (0.0220) model time 0.0000 (0.0000) loss 3.2851 (2.8288) grad_norm 1.7039 (2.4770) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][40/625] eta 0:03:49 lr 0.000367 wd 0.0500 time 0.3716 (0.3925) data time 0.0011 (0.0169) model time 0.0000 (0.0000) loss 2.7999 (2.8528) grad_norm 2.5714 (2.4869) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][50/625] eta 0:03:43 lr 0.000366 wd 0.0500 time 0.3713 (0.3886) data time 0.0008 (0.0138) model time 0.0000 (0.0000) loss 2.4543 (2.8714) grad_norm 18.0553 (2.7438) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][60/625] eta 0:03:37 lr 0.000366 wd 0.0500 time 0.3740 (0.3857) data time 0.0009 (0.0117) model time 0.3731 (0.3697) loss 2.4257 (2.8771) grad_norm 1.9756 (2.7347) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][70/625] eta 0:03:32 lr 0.000366 wd 0.0500 time 0.3615 (0.3838) data time 0.0011 (0.0102) model time 0.3604 (0.3705) loss 3.4920 (2.8783) grad_norm 2.3014 (2.6693) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][80/625] eta 0:03:28 lr 0.000366 wd 0.0500 time 0.3763 (0.3824) data time 0.0011 (0.0091) model time 0.3752 (0.3709) loss 2.9784 (2.8701) grad_norm 2.3321 (2.6028) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][90/625] eta 0:03:23 lr 0.000366 wd 0.0500 time 0.3729 (0.3812) data time 0.0008 (0.0082) model time 0.3720 (0.3706) loss 3.6814 (2.8889) grad_norm 1.5191 (2.5122) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][100/625] eta 0:03:19 lr 0.000366 wd 0.0500 time 0.3976 (0.3808) data time 0.0011 (0.0075) model time 0.3964 (0.3717) loss 2.5906 (2.8811) grad_norm 1.8873 (2.4624) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:14:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][110/625] eta 0:03:15 lr 0.000366 wd 0.0500 time 0.3737 (0.3801) data time 0.0011 (0.0070) model time 0.3725 (0.3718) loss 2.1953 (2.8717) grad_norm 1.7272 (2.4155) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][120/625] eta 0:03:11 lr 0.000365 wd 0.0500 time 0.3697 (0.3798) data time 0.0011 (0.0065) model time 0.3686 (0.3723) loss 2.9434 (2.8709) grad_norm 2.6185 (2.4084) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][130/625] eta 0:03:07 lr 0.000365 wd 0.0500 time 0.3680 (0.3792) data time 0.0009 (0.0061) model time 0.3671 (0.3720) loss 3.1913 (2.8674) grad_norm 1.9098 (2.4070) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][140/625] eta 0:03:04 lr 0.000365 wd 0.0500 time 0.3862 (0.3803) data time 0.0011 (0.0057) model time 0.3851 (0.3744) loss 3.0172 (2.8773) grad_norm 1.8647 (2.4118) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][150/625] eta 0:03:01 lr 0.000365 wd 0.0500 time 0.5881 (0.3813) data time 0.0010 (0.0054) model time 0.5870 (0.3764) loss 2.6145 (2.8659) grad_norm 1.3773 (2.3904) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][160/625] eta 0:02:57 lr 0.000365 wd 0.0500 time 0.3878 (0.3823) data time 0.0011 (0.0052) model time 0.3867 (0.3781) loss 3.2953 (2.8680) grad_norm 1.6415 (2.3644) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][170/625] eta 0:02:53 lr 0.000365 wd 0.0500 time 0.3653 (0.3816) data time 0.0010 (0.0050) model time 0.3644 (0.3774) loss 2.0540 (2.8578) grad_norm 1.7815 (2.3490) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][180/625] eta 0:02:49 lr 0.000365 wd 0.0500 time 0.3669 (0.3810) data time 0.0009 (0.0047) model time 0.3660 (0.3769) loss 2.4924 (2.8473) grad_norm 1.6112 (2.3500) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][190/625] eta 0:02:45 lr 0.000364 wd 0.0500 time 0.3834 (0.3807) data time 0.0011 (0.0046) model time 0.3824 (0.3767) loss 2.2478 (2.8548) grad_norm 1.6372 (2.3289) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][200/625] eta 0:02:41 lr 0.000364 wd 0.0500 time 0.3719 (0.3802) data time 0.0010 (0.0044) model time 0.3709 (0.3762) loss 3.4322 (2.8658) grad_norm 1.5672 (2.3144) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][210/625] eta 0:02:37 lr 0.000364 wd 0.0500 time 0.3730 (0.3797) data time 0.0011 (0.0043) model time 0.3719 (0.3757) loss 2.8269 (2.8632) grad_norm 2.6937 (2.2896) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][220/625] eta 0:02:33 lr 0.000364 wd 0.0500 time 0.3738 (0.3793) data time 0.0011 (0.0041) model time 0.3726 (0.3753) loss 3.0195 (2.8605) grad_norm 1.9306 (2.2833) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][230/625] eta 0:02:29 lr 0.000364 wd 0.0500 time 0.3784 (0.3790) data time 0.0009 (0.0040) model time 0.3775 (0.3752) loss 2.9051 (2.8569) grad_norm 1.3829 (2.2946) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][240/625] eta 0:02:25 lr 0.000364 wd 0.0500 time 0.3707 (0.3789) data time 0.0011 (0.0039) model time 0.3696 (0.3751) loss 2.6606 (2.8598) grad_norm 2.2120 (2.2901) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][250/625] eta 0:02:22 lr 0.000364 wd 0.0500 time 0.3729 (0.3787) data time 0.0011 (0.0038) model time 0.3718 (0.3750) loss 2.8425 (2.8443) grad_norm 1.7664 (2.2778) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][260/625] eta 0:02:18 lr 0.000364 wd 0.0500 time 0.3724 (0.3785) data time 0.0009 (0.0037) model time 0.3715 (0.3749) loss 3.2215 (2.8515) grad_norm 1.8233 (2.2627) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:15:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][270/625] eta 0:02:14 lr 0.000363 wd 0.0500 time 0.3752 (0.3782) data time 0.0009 (0.0036) model time 0.3743 (0.3746) loss 2.2333 (2.8386) grad_norm 1.6427 (2.2551) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][280/625] eta 0:02:10 lr 0.000363 wd 0.0500 time 0.3709 (0.3780) data time 0.0009 (0.0035) model time 0.3700 (0.3745) loss 3.6052 (2.8387) grad_norm 1.5554 (2.2387) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][290/625] eta 0:02:06 lr 0.000363 wd 0.0500 time 0.3769 (0.3778) data time 0.0011 (0.0034) model time 0.3758 (0.3743) loss 2.9715 (2.8371) grad_norm 1.5403 (2.2324) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][300/625] eta 0:02:02 lr 0.000363 wd 0.0500 time 0.3721 (0.3776) data time 0.0012 (0.0033) model time 0.3708 (0.3741) loss 2.5948 (2.8414) grad_norm 2.1210 (2.2322) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][310/625] eta 0:01:58 lr 0.000363 wd 0.0500 time 0.3757 (0.3774) data time 0.0008 (0.0033) model time 0.3748 (0.3741) loss 2.2927 (2.8407) grad_norm 2.1087 (2.2232) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][320/625] eta 0:01:55 lr 0.000363 wd 0.0500 time 0.3670 (0.3779) data time 0.0010 (0.0032) model time 0.3660 (0.3747) loss 3.2125 (2.8462) grad_norm 1.9708 (2.2145) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][330/625] eta 0:01:51 lr 0.000363 wd 0.0500 time 0.3748 (0.3777) data time 0.0012 (0.0031) model time 0.3736 (0.3745) loss 2.9987 (2.8471) grad_norm 2.0426 (2.2333) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][340/625] eta 0:01:48 lr 0.000362 wd 0.0500 time 0.3719 (0.3794) data time 0.0009 (0.0031) model time 0.3710 (0.3766) loss 3.2085 (2.8420) grad_norm 1.9216 (2.2271) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][350/625] eta 0:01:44 lr 0.000362 wd 0.0500 time 0.3731 (0.3792) data time 0.0012 (0.0030) model time 0.3719 (0.3764) loss 3.1217 (2.8426) grad_norm 1.9155 (2.2196) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][360/625] eta 0:01:40 lr 0.000362 wd 0.0500 time 0.3739 (0.3789) data time 0.0011 (0.0030) model time 0.3727 (0.3762) loss 2.9968 (2.8491) grad_norm 1.9789 (2.2224) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][370/625] eta 0:01:36 lr 0.000362 wd 0.0500 time 0.3653 (0.3787) data time 0.0011 (0.0029) model time 0.3642 (0.3760) loss 2.9085 (2.8524) grad_norm 1.4318 (2.2205) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][380/625] eta 0:01:32 lr 0.000362 wd 0.0500 time 0.3713 (0.3786) data time 0.0008 (0.0029) model time 0.3705 (0.3758) loss 3.1931 (2.8546) grad_norm 3.4204 (2.2189) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][390/625] eta 0:01:28 lr 0.000362 wd 0.0500 time 0.3746 (0.3784) data time 0.0010 (0.0028) model time 0.3737 (0.3756) loss 3.0827 (2.8540) grad_norm 1.6065 (2.2098) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][400/625] eta 0:01:25 lr 0.000362 wd 0.0500 time 0.3734 (0.3782) data time 0.0008 (0.0028) model time 0.3726 (0.3755) loss 3.1178 (2.8567) grad_norm 3.6275 (2.2061) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][410/625] eta 0:01:21 lr 0.000362 wd 0.0500 time 0.3702 (0.3781) data time 0.0012 (0.0028) model time 0.3690 (0.3754) loss 3.0876 (2.8544) grad_norm 1.9202 (2.2214) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][420/625] eta 0:01:17 lr 0.000361 wd 0.0500 time 0.3752 (0.3780) data time 0.0012 (0.0027) model time 0.3739 (0.3753) loss 2.3100 (2.8574) grad_norm 2.0326 (2.2213) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][430/625] eta 0:01:13 lr 0.000361 wd 0.0500 time 0.3695 (0.3779) data time 0.0012 (0.0027) model time 0.3683 (0.3752) loss 3.3080 (2.8587) grad_norm 2.7464 (2.2157) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][440/625] eta 0:01:09 lr 0.000361 wd 0.0500 time 0.3675 (0.3782) data time 0.0011 (0.0027) model time 0.3664 (0.3756) loss 3.0416 (2.8588) grad_norm 2.1221 (2.2042) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][450/625] eta 0:01:06 lr 0.000361 wd 0.0500 time 0.3746 (0.3780) data time 0.0011 (0.0026) model time 0.3734 (0.3754) loss 2.8688 (2.8612) grad_norm 1.5735 (2.2035) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][460/625] eta 0:01:02 lr 0.000361 wd 0.0500 time 0.3666 (0.3788) data time 0.0009 (0.0026) model time 0.3657 (0.3763) loss 3.1453 (2.8606) grad_norm 2.5553 (2.2002) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][470/625] eta 0:00:58 lr 0.000361 wd 0.0500 time 0.3718 (0.3786) data time 0.0011 (0.0026) model time 0.3707 (0.3762) loss 3.3804 (2.8594) grad_norm 1.8008 (2.1973) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][480/625] eta 0:00:54 lr 0.000361 wd 0.0500 time 0.3702 (0.3785) data time 0.0009 (0.0025) model time 0.3693 (0.3761) loss 3.3834 (2.8635) grad_norm 1.9415 (2.1914) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][490/625] eta 0:00:51 lr 0.000360 wd 0.0500 time 0.3703 (0.3783) data time 0.0008 (0.0025) model time 0.3694 (0.3759) loss 3.5272 (2.8614) grad_norm 1.8535 (2.1866) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][500/625] eta 0:00:47 lr 0.000360 wd 0.0500 time 0.3772 (0.3782) data time 0.0008 (0.0025) model time 0.3763 (0.3758) loss 3.6782 (2.8663) grad_norm 1.8047 (2.1858) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][510/625] eta 0:00:43 lr 0.000360 wd 0.0500 time 0.3717 (0.3781) data time 0.0011 (0.0025) model time 0.3706 (0.3757) loss 3.1790 (2.8671) grad_norm 2.9099 (2.1948) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][520/625] eta 0:00:39 lr 0.000360 wd 0.0500 time 0.3605 (0.3779) data time 0.0009 (0.0024) model time 0.3596 (0.3755) loss 2.0514 (2.8632) grad_norm 1.7397 (2.1931) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][530/625] eta 0:00:35 lr 0.000360 wd 0.0500 time 0.3763 (0.3778) data time 0.0009 (0.0024) model time 0.3755 (0.3754) loss 2.3220 (2.8620) grad_norm 1.9446 (2.1899) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][540/625] eta 0:00:32 lr 0.000360 wd 0.0500 time 0.3727 (0.3776) data time 0.0011 (0.0024) model time 0.3716 (0.3753) loss 3.4672 (2.8591) grad_norm 1.5583 (2.1827) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][550/625] eta 0:00:28 lr 0.000360 wd 0.0500 time 0.3724 (0.3775) data time 0.0011 (0.0024) model time 0.3713 (0.3751) loss 3.2132 (2.8593) grad_norm 1.9331 (2.1856) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][560/625] eta 0:00:24 lr 0.000360 wd 0.0500 time 0.3677 (0.3774) data time 0.0009 (0.0023) model time 0.3669 (0.3750) loss 3.3012 (2.8569) grad_norm 1.5434 (2.1832) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][570/625] eta 0:00:20 lr 0.000359 wd 0.0500 time 0.3703 (0.3772) data time 0.0010 (0.0023) model time 0.3694 (0.3749) loss 2.4642 (2.8546) grad_norm 2.0698 (2.1807) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][580/625] eta 0:00:16 lr 0.000359 wd 0.0500 time 0.3707 (0.3771) data time 0.0008 (0.0023) model time 0.3698 (0.3748) loss 2.8628 (2.8570) grad_norm 2.0160 (2.1857) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:17:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][590/625] eta 0:00:13 lr 0.000359 wd 0.0500 time 0.3665 (0.3770) data time 0.0014 (0.0023) model time 0.3651 (0.3747) loss 2.9199 (2.8544) grad_norm 2.9912 (inf) loss_scale 512.0000 (1018.8020) mem 11225MB [2024-09-01 10:18:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][600/625] eta 0:00:09 lr 0.000359 wd 0.0500 time 0.3658 (0.3769) data time 0.0012 (0.0023) model time 0.3646 (0.3746) loss 3.3114 (2.8533) grad_norm 2.5011 (inf) loss_scale 512.0000 (1010.3694) mem 11225MB [2024-09-01 10:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][610/625] eta 0:00:05 lr 0.000359 wd 0.0500 time 0.3637 (0.3767) data time 0.0006 (0.0023) model time 0.3631 (0.3744) loss 3.2313 (2.8522) grad_norm 2.0006 (inf) loss_scale 512.0000 (1002.2128) mem 11225MB [2024-09-01 10:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [223/300][620/625] eta 0:00:01 lr 0.000359 wd 0.0500 time 0.3642 (0.3764) data time 0.0006 (0.0022) model time 0.3635 (0.3742) loss 2.0163 (2.8510) grad_norm 2.4025 (inf) loss_scale 512.0000 (994.3188) mem 11225MB [2024-09-01 10:18:11 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 223 training takes 0:03:55 [2024-09-01 10:18:11 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:18:12 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.475 (0.475) Loss 0.5283 (0.5283) Acc@1 88.428 (88.428) Acc@5 98.438 (98.438) Mem 11225MB [2024-09-01 10:18:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.145) Loss 0.8687 (0.6360) Acc@1 79.736 (86.137) Acc@5 95.703 (97.612) Mem 11225MB [2024-09-01 10:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.096 (0.122) Loss 0.9463 (0.7576) Acc@1 76.758 (82.952) Acc@5 94.580 (96.250) Mem 11225MB [2024-09-01 10:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.616 Acc@5 96.293 [2024-09-01 10:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.6% [2024-09-01 10:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.62% [2024-09-01 10:18:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 10:18:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 10:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.510 (0.510) Loss 0.4919 (0.4919) Acc@1 89.746 (89.746) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 10:18:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 0.7886 (0.5954) Acc@1 81.396 (86.808) Acc@5 96.240 (97.860) Mem 11225MB [2024-09-01 10:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 0.8804 (0.7052) Acc@1 78.320 (83.954) Acc@5 95.215 (96.724) Mem 11225MB [2024-09-01 10:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.673 Acc@5 96.759 [2024-09-01 10:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 10:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.67% [2024-09-01 10:18:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:18:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][0/625] eta 0:08:04 lr 0.000359 wd 0.0500 time 0.7749 (0.7749) data time 0.4419 (0.4419) model time 0.0000 (0.0000) loss 3.0768 (3.0768) grad_norm 1.8275 (1.8275) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][10/625] eta 0:04:08 lr 0.000359 wd 0.0500 time 0.3651 (0.4048) data time 0.0011 (0.0414) model time 0.0000 (0.0000) loss 2.8234 (2.7551) grad_norm 2.4358 (2.6877) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][20/625] eta 0:03:55 lr 0.000358 wd 0.0500 time 0.3684 (0.3884) data time 0.0012 (0.0223) model time 0.0000 (0.0000) loss 3.0692 (2.8643) grad_norm 3.0120 (2.6801) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][30/625] eta 0:03:59 lr 0.000358 wd 0.0500 time 0.3779 (0.4033) data time 0.0012 (0.0155) model time 0.0000 (0.0000) loss 2.9033 (2.8989) grad_norm 2.0373 (2.4362) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][40/625] eta 0:03:51 lr 0.000358 wd 0.0500 time 0.3641 (0.3950) data time 0.0011 (0.0120) model time 0.0000 (0.0000) loss 2.4700 (2.9001) grad_norm 2.0603 (2.4293) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][50/625] eta 0:03:44 lr 0.000358 wd 0.0500 time 0.3726 (0.3905) data time 0.0012 (0.0099) model time 0.0000 (0.0000) loss 2.9678 (2.9027) grad_norm 2.9503 (2.6034) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][60/625] eta 0:03:38 lr 0.000358 wd 0.0500 time 0.3672 (0.3872) data time 0.0011 (0.0085) model time 0.3660 (0.3692) loss 2.3625 (2.8780) grad_norm 2.3135 (2.5579) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][70/625] eta 0:03:33 lr 0.000358 wd 0.0500 time 0.3671 (0.3848) data time 0.0011 (0.0075) model time 0.3660 (0.3691) loss 2.9394 (2.9016) grad_norm 2.4864 (2.5254) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][80/625] eta 0:03:28 lr 0.000358 wd 0.0500 time 0.3636 (0.3833) data time 0.0009 (0.0067) model time 0.3627 (0.3698) loss 2.0720 (2.9062) grad_norm 2.3261 (2.5559) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][90/625] eta 0:03:24 lr 0.000357 wd 0.0500 time 0.3688 (0.3823) data time 0.0011 (0.0061) model time 0.3677 (0.3706) loss 3.0993 (2.9251) grad_norm 3.3025 (2.6371) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:18:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][100/625] eta 0:03:20 lr 0.000357 wd 0.0500 time 0.3700 (0.3811) data time 0.0009 (0.0056) model time 0.3691 (0.3703) loss 3.4669 (2.9350) grad_norm 2.5523 (2.6781) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][110/625] eta 0:03:15 lr 0.000357 wd 0.0500 time 0.3728 (0.3801) data time 0.0011 (0.0052) model time 0.3717 (0.3700) loss 2.4991 (2.9201) grad_norm 2.1415 (2.6662) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][120/625] eta 0:03:11 lr 0.000357 wd 0.0500 time 0.3705 (0.3794) data time 0.0011 (0.0049) model time 0.3693 (0.3701) loss 3.2183 (2.9060) grad_norm 1.7084 (2.6029) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][130/625] eta 0:03:07 lr 0.000357 wd 0.0500 time 0.3625 (0.3788) data time 0.0010 (0.0046) model time 0.3616 (0.3701) loss 2.7307 (2.8906) grad_norm 2.1107 (2.5553) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][140/625] eta 0:03:03 lr 0.000357 wd 0.0500 time 0.3681 (0.3782) data time 0.0009 (0.0044) model time 0.3672 (0.3700) loss 3.0348 (2.9032) grad_norm 1.7777 (2.5181) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][150/625] eta 0:03:00 lr 0.000357 wd 0.0500 time 0.3586 (0.3791) data time 0.0011 (0.0042) model time 0.3575 (0.3720) loss 2.2029 (2.8948) grad_norm 2.2179 (2.4957) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][160/625] eta 0:02:56 lr 0.000357 wd 0.0500 time 0.3746 (0.3786) data time 0.0011 (0.0040) model time 0.3735 (0.3718) loss 2.4940 (2.9007) grad_norm 4.0114 (2.4826) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][170/625] eta 0:02:52 lr 0.000356 wd 0.0500 time 0.3695 (0.3781) data time 0.0011 (0.0038) model time 0.3684 (0.3715) loss 2.7268 (2.8981) grad_norm 1.9468 (2.4570) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][180/625] eta 0:02:48 lr 0.000356 wd 0.0500 time 0.3779 (0.3780) data time 0.0008 (0.0037) model time 0.3770 (0.3719) loss 3.5335 (2.8955) grad_norm 1.9835 (2.4498) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][190/625] eta 0:02:44 lr 0.000356 wd 0.0500 time 0.3759 (0.3779) data time 0.0011 (0.0036) model time 0.3749 (0.3720) loss 2.7902 (2.9021) grad_norm 1.9224 (2.4426) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][200/625] eta 0:02:40 lr 0.000356 wd 0.0500 time 0.3672 (0.3776) data time 0.0013 (0.0034) model time 0.3659 (0.3720) loss 2.7114 (2.8962) grad_norm 2.7064 (2.4363) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][210/625] eta 0:02:36 lr 0.000356 wd 0.0500 time 0.3739 (0.3773) data time 0.0012 (0.0033) model time 0.3727 (0.3719) loss 2.9387 (2.9062) grad_norm 2.0717 (2.4105) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][220/625] eta 0:02:32 lr 0.000356 wd 0.0500 time 0.3724 (0.3769) data time 0.0011 (0.0032) model time 0.3713 (0.3716) loss 3.4943 (2.9142) grad_norm 2.0041 (2.4204) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][230/625] eta 0:02:28 lr 0.000356 wd 0.0500 time 0.3701 (0.3766) data time 0.0014 (0.0032) model time 0.3687 (0.3714) loss 2.9244 (2.9240) grad_norm 2.3178 (2.4080) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][240/625] eta 0:02:25 lr 0.000355 wd 0.0500 time 0.3579 (0.3781) data time 0.0012 (0.0031) model time 0.3567 (0.3736) loss 2.5196 (2.9250) grad_norm 2.1070 (2.3936) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][250/625] eta 0:02:21 lr 0.000355 wd 0.0500 time 0.3687 (0.3779) data time 0.0012 (0.0030) model time 0.3675 (0.3734) loss 3.1840 (2.9204) grad_norm 2.4852 (2.4064) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:19:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][260/625] eta 0:02:17 lr 0.000355 wd 0.0500 time 0.3780 (0.3778) data time 0.0009 (0.0029) model time 0.3771 (0.3735) loss 2.9667 (2.9189) grad_norm 1.6907 (2.3947) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][270/625] eta 0:02:14 lr 0.000355 wd 0.0500 time 0.3763 (0.3784) data time 0.0010 (0.0029) model time 0.3753 (0.3744) loss 3.2657 (2.9228) grad_norm 2.0881 (2.3963) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][280/625] eta 0:02:10 lr 0.000355 wd 0.0500 time 0.3632 (0.3784) data time 0.0012 (0.0028) model time 0.3620 (0.3744) loss 2.2836 (2.9220) grad_norm 1.7302 (2.3994) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][290/625] eta 0:02:06 lr 0.000355 wd 0.0500 time 0.3689 (0.3781) data time 0.0012 (0.0028) model time 0.3677 (0.3742) loss 2.8356 (2.9151) grad_norm 1.6288 (2.3923) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][300/625] eta 0:02:02 lr 0.000355 wd 0.0500 time 0.3722 (0.3779) data time 0.0010 (0.0027) model time 0.3713 (0.3741) loss 2.4488 (2.9089) grad_norm 2.3630 (2.3818) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][310/625] eta 0:01:58 lr 0.000355 wd 0.0500 time 0.3674 (0.3776) data time 0.0009 (0.0027) model time 0.3665 (0.3739) loss 3.2866 (2.9029) grad_norm 1.9427 (2.3840) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][320/625] eta 0:01:55 lr 0.000354 wd 0.0500 time 0.3637 (0.3774) data time 0.0013 (0.0026) model time 0.3624 (0.3737) loss 2.9276 (2.8989) grad_norm 1.6713 (2.3744) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][330/625] eta 0:01:51 lr 0.000354 wd 0.0500 time 0.3652 (0.3772) data time 0.0011 (0.0026) model time 0.3641 (0.3735) loss 2.9056 (2.8997) grad_norm 1.8738 (2.3585) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][340/625] eta 0:01:47 lr 0.000354 wd 0.0500 time 0.3625 (0.3770) data time 0.0009 (0.0025) model time 0.3616 (0.3734) loss 2.7794 (2.8976) grad_norm 2.4783 (2.3453) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][350/625] eta 0:01:43 lr 0.000354 wd 0.0500 time 0.3694 (0.3768) data time 0.0010 (0.0025) model time 0.3684 (0.3732) loss 2.6144 (2.8928) grad_norm 1.6217 (2.3360) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][360/625] eta 0:01:39 lr 0.000354 wd 0.0500 time 0.3697 (0.3767) data time 0.0008 (0.0025) model time 0.3689 (0.3732) loss 1.8309 (2.8903) grad_norm 1.8523 (2.3250) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][370/625] eta 0:01:36 lr 0.000354 wd 0.0500 time 0.3630 (0.3765) data time 0.0012 (0.0024) model time 0.3618 (0.3730) loss 3.5800 (2.8956) grad_norm 2.5406 (2.3339) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][380/625] eta 0:01:32 lr 0.000354 wd 0.0500 time 0.3731 (0.3764) data time 0.0013 (0.0024) model time 0.3719 (0.3729) loss 3.1630 (2.8911) grad_norm 2.0468 (2.3513) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][390/625] eta 0:01:28 lr 0.000353 wd 0.0500 time 0.3742 (0.3762) data time 0.0012 (0.0024) model time 0.3731 (0.3728) loss 2.8012 (2.8939) grad_norm 1.9596 (2.3621) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][400/625] eta 0:01:24 lr 0.000353 wd 0.0500 time 0.3589 (0.3761) data time 0.0011 (0.0023) model time 0.3577 (0.3727) loss 2.5147 (2.8896) grad_norm 1.7386 (2.3660) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][410/625] eta 0:01:20 lr 0.000353 wd 0.0500 time 0.3739 (0.3759) data time 0.0012 (0.0023) model time 0.3727 (0.3726) loss 2.9025 (2.8884) grad_norm 1.4814 (2.3645) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:20:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][420/625] eta 0:01:17 lr 0.000353 wd 0.0500 time 0.3735 (0.3773) data time 0.0011 (0.0023) model time 0.3724 (0.3742) loss 3.2413 (2.8916) grad_norm 1.4293 (2.3646) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][430/625] eta 0:01:13 lr 0.000353 wd 0.0500 time 0.3756 (0.3771) data time 0.0011 (0.0022) model time 0.3745 (0.3741) loss 3.2557 (2.8885) grad_norm 1.8600 (2.3590) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][440/625] eta 0:01:09 lr 0.000353 wd 0.0500 time 0.3612 (0.3770) data time 0.0009 (0.0022) model time 0.3602 (0.3740) loss 2.6790 (2.8862) grad_norm 1.9063 (2.3534) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][450/625] eta 0:01:05 lr 0.000353 wd 0.0500 time 0.3709 (0.3769) data time 0.0009 (0.0022) model time 0.3701 (0.3739) loss 2.8147 (2.8796) grad_norm 1.3066 (2.3481) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][460/625] eta 0:01:02 lr 0.000353 wd 0.0500 time 0.3663 (0.3767) data time 0.0013 (0.0022) model time 0.3650 (0.3738) loss 3.0139 (2.8801) grad_norm 2.6911 (2.3468) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][470/625] eta 0:00:58 lr 0.000352 wd 0.0500 time 0.3711 (0.3766) data time 0.0011 (0.0022) model time 0.3700 (0.3737) loss 2.0259 (2.8759) grad_norm 1.6696 (2.3336) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][480/625] eta 0:00:54 lr 0.000352 wd 0.0500 time 0.3694 (0.3770) data time 0.0011 (0.0021) model time 0.3683 (0.3741) loss 2.7204 (2.8738) grad_norm 2.1474 (2.3289) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][490/625] eta 0:00:50 lr 0.000352 wd 0.0500 time 0.3657 (0.3769) data time 0.0013 (0.0021) model time 0.3644 (0.3741) loss 3.0799 (2.8760) grad_norm 2.5108 (2.3258) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][500/625] eta 0:00:47 lr 0.000352 wd 0.0500 time 0.3736 (0.3768) data time 0.0011 (0.0021) model time 0.3725 (0.3740) loss 3.1009 (2.8762) grad_norm 2.6098 (2.3268) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][510/625] eta 0:00:43 lr 0.000352 wd 0.0500 time 0.3671 (0.3767) data time 0.0011 (0.0021) model time 0.3660 (0.3739) loss 2.8449 (2.8768) grad_norm 2.1267 (2.3227) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][520/625] eta 0:00:39 lr 0.000352 wd 0.0500 time 0.3724 (0.3766) data time 0.0011 (0.0021) model time 0.3714 (0.3738) loss 3.1749 (2.8745) grad_norm 1.8933 (2.3187) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][530/625] eta 0:00:35 lr 0.000352 wd 0.0500 time 0.3870 (0.3765) data time 0.0014 (0.0021) model time 0.3856 (0.3738) loss 2.8175 (2.8769) grad_norm 1.8012 (2.3271) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][540/625] eta 0:00:32 lr 0.000351 wd 0.0500 time 0.3673 (0.3772) data time 0.0008 (0.0020) model time 0.3665 (0.3746) loss 3.2875 (2.8800) grad_norm 3.8230 (2.3267) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][550/625] eta 0:00:28 lr 0.000351 wd 0.0500 time 0.3737 (0.3771) data time 0.0009 (0.0020) model time 0.3729 (0.3745) loss 2.7187 (2.8790) grad_norm 2.9384 (2.3603) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][560/625] eta 0:00:24 lr 0.000351 wd 0.0500 time 0.3730 (0.3769) data time 0.0009 (0.0020) model time 0.3721 (0.3744) loss 1.8573 (2.8784) grad_norm 1.3877 (2.3528) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][570/625] eta 0:00:20 lr 0.000351 wd 0.0500 time 0.3717 (0.3768) data time 0.0012 (0.0020) model time 0.3706 (0.3743) loss 2.6270 (2.8787) grad_norm 1.7921 (2.3533) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:21:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][580/625] eta 0:00:16 lr 0.000351 wd 0.0500 time 0.3645 (0.3767) data time 0.0009 (0.0020) model time 0.3636 (0.3742) loss 2.7488 (2.8811) grad_norm 1.4862 (2.3474) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][590/625] eta 0:00:13 lr 0.000351 wd 0.0500 time 0.4005 (0.3767) data time 0.0009 (0.0020) model time 0.3996 (0.3742) loss 2.6806 (2.8800) grad_norm 2.1387 (2.3405) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][600/625] eta 0:00:09 lr 0.000351 wd 0.0500 time 0.3707 (0.3766) data time 0.0009 (0.0020) model time 0.3698 (0.3741) loss 2.2818 (2.8811) grad_norm 2.3880 (2.3358) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][610/625] eta 0:00:05 lr 0.000351 wd 0.0500 time 0.3700 (0.3766) data time 0.0006 (0.0019) model time 0.3694 (0.3741) loss 2.2895 (2.8789) grad_norm 2.0087 (2.3323) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [224/300][620/625] eta 0:00:01 lr 0.000350 wd 0.0500 time 0.3676 (0.3764) data time 0.0006 (0.0019) model time 0.3671 (0.3739) loss 2.9969 (2.8798) grad_norm 2.1113 (2.3356) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:16 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 224 training takes 0:03:55 [2024-09-01 10:22:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:22:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:22:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.5308 (0.5308) Acc@1 89.014 (89.014) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 0.8750 (0.6427) Acc@1 79.297 (85.938) Acc@5 95.459 (97.514) Mem 11225MB [2024-09-01 10:22:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9604 (0.7612) Acc@1 77.344 (82.857) Acc@5 94.482 (96.236) Mem 11225MB [2024-09-01 10:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.590 Acc@5 96.259 [2024-09-01 10:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.6% [2024-09-01 10:22:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.777 (0.777) Loss 0.4917 (0.4917) Acc@1 89.697 (89.697) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 10:22:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.099 (0.172) Loss 0.7891 (0.5951) Acc@1 81.592 (86.816) Acc@5 96.240 (97.856) Mem 11225MB [2024-09-01 10:22:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.135) Loss 0.8804 (0.7051) Acc@1 78.369 (83.954) Acc@5 95.068 (96.708) Mem 11225MB [2024-09-01 10:22:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.687 Acc@5 96.749 [2024-09-01 10:22:23 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 10:22:23 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.69% [2024-09-01 10:22:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:22:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:22:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][0/625] eta 0:08:22 lr 0.000350 wd 0.0500 time 0.8041 (0.8041) data time 0.4683 (0.4683) model time 0.0000 (0.0000) loss 2.6264 (2.6264) grad_norm 5.0446 (5.0446) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][10/625] eta 0:04:12 lr 0.000350 wd 0.0500 time 0.3745 (0.4104) data time 0.0009 (0.0436) model time 0.0000 (0.0000) loss 1.8549 (2.7012) grad_norm 2.1747 (2.7356) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][20/625] eta 0:03:56 lr 0.000350 wd 0.0500 time 0.3640 (0.3912) data time 0.0009 (0.0234) model time 0.0000 (0.0000) loss 3.1705 (2.7952) grad_norm 3.4208 (2.7098) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][30/625] eta 0:03:48 lr 0.000350 wd 0.0500 time 0.3821 (0.3847) data time 0.0009 (0.0163) model time 0.0000 (0.0000) loss 3.2059 (2.8107) grad_norm 1.8898 (2.5463) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][40/625] eta 0:03:43 lr 0.000350 wd 0.0500 time 0.3722 (0.3814) data time 0.0011 (0.0127) model time 0.0000 (0.0000) loss 3.2379 (2.8049) grad_norm 1.6950 (2.6226) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][50/625] eta 0:03:38 lr 0.000350 wd 0.0500 time 0.3745 (0.3794) data time 0.0011 (0.0104) model time 0.0000 (0.0000) loss 3.0224 (2.8158) grad_norm 2.2839 (2.5248) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][60/625] eta 0:03:33 lr 0.000350 wd 0.0500 time 0.3723 (0.3785) data time 0.0011 (0.0089) model time 0.3712 (0.3727) loss 3.4300 (2.8247) grad_norm 1.5648 (2.4313) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][70/625] eta 0:03:34 lr 0.000349 wd 0.0500 time 0.3723 (0.3858) data time 0.0012 (0.0078) model time 0.3711 (0.4008) loss 3.2492 (2.8165) grad_norm 1.6203 (2.3294) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][80/625] eta 0:03:29 lr 0.000349 wd 0.0500 time 0.3725 (0.3839) data time 0.0009 (0.0070) model time 0.3716 (0.3904) loss 2.8685 (2.8219) grad_norm 2.4081 (2.2771) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:22:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][90/625] eta 0:03:24 lr 0.000349 wd 0.0500 time 0.3768 (0.3826) data time 0.0011 (0.0064) model time 0.3756 (0.3853) loss 2.3139 (2.7987) grad_norm 1.7558 (2.2320) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][100/625] eta 0:03:20 lr 0.000349 wd 0.0500 time 0.3698 (0.3814) data time 0.0009 (0.0059) model time 0.3689 (0.3822) loss 3.3003 (2.8126) grad_norm 13.9224 (2.3362) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][110/625] eta 0:03:15 lr 0.000349 wd 0.0500 time 0.3725 (0.3802) data time 0.0011 (0.0055) model time 0.3713 (0.3797) loss 3.2385 (2.8202) grad_norm 1.5091 (2.2909) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][120/625] eta 0:03:11 lr 0.000349 wd 0.0500 time 0.3694 (0.3794) data time 0.0011 (0.0051) model time 0.3682 (0.3782) loss 2.3916 (2.8038) grad_norm 1.5962 (2.2621) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][130/625] eta 0:03:07 lr 0.000349 wd 0.0500 time 0.3676 (0.3786) data time 0.0013 (0.0048) model time 0.3664 (0.3768) loss 2.9879 (2.8121) grad_norm 1.6490 (2.2374) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][140/625] eta 0:03:03 lr 0.000348 wd 0.0500 time 0.3706 (0.3780) data time 0.0011 (0.0046) model time 0.3694 (0.3759) loss 3.2223 (2.8295) grad_norm 1.6507 (2.2177) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][150/625] eta 0:02:59 lr 0.000348 wd 0.0500 time 0.3704 (0.3775) data time 0.0011 (0.0043) model time 0.3693 (0.3753) loss 3.2080 (2.8282) grad_norm 2.5268 (2.2387) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][160/625] eta 0:02:55 lr 0.000348 wd 0.0500 time 0.3723 (0.3771) data time 0.0009 (0.0041) model time 0.3715 (0.3748) loss 4.1315 (2.8414) grad_norm 2.1178 (2.2471) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][170/625] eta 0:02:51 lr 0.000348 wd 0.0500 time 0.3678 (0.3768) data time 0.0011 (0.0040) model time 0.3666 (0.3745) loss 2.7248 (2.8553) grad_norm 2.3082 (2.3853) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][180/625] eta 0:02:47 lr 0.000348 wd 0.0500 time 0.3724 (0.3766) data time 0.0008 (0.0038) model time 0.3716 (0.3743) loss 2.5550 (2.8544) grad_norm 1.7469 (2.3672) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][190/625] eta 0:02:43 lr 0.000348 wd 0.0500 time 0.3721 (0.3762) data time 0.0012 (0.0037) model time 0.3709 (0.3738) loss 3.0175 (2.8539) grad_norm 2.0737 (2.3799) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][200/625] eta 0:02:39 lr 0.000348 wd 0.0500 time 0.3670 (0.3758) data time 0.0009 (0.0036) model time 0.3660 (0.3734) loss 3.0118 (2.8367) grad_norm 2.2189 (2.3811) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][210/625] eta 0:02:36 lr 0.000348 wd 0.0500 time 0.3642 (0.3766) data time 0.0008 (0.0034) model time 0.3634 (0.3744) loss 3.3921 (2.8322) grad_norm 2.6822 (2.3607) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][220/625] eta 0:02:32 lr 0.000347 wd 0.0500 time 0.3725 (0.3763) data time 0.0009 (0.0034) model time 0.3716 (0.3740) loss 3.1886 (2.8242) grad_norm 1.6560 (2.3302) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][230/625] eta 0:02:28 lr 0.000347 wd 0.0500 time 0.3719 (0.3760) data time 0.0008 (0.0033) model time 0.3711 (0.3737) loss 3.0859 (2.8178) grad_norm 2.2989 (2.3359) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][240/625] eta 0:02:24 lr 0.000347 wd 0.0500 time 0.3732 (0.3757) data time 0.0009 (0.0032) model time 0.3723 (0.3735) loss 3.2458 (2.8083) grad_norm 1.9094 (2.3278) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:23:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][250/625] eta 0:02:20 lr 0.000347 wd 0.0500 time 0.3658 (0.3755) data time 0.0011 (0.0031) model time 0.3647 (0.3732) loss 2.7192 (2.8071) grad_norm 1.9429 (2.3489) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][260/625] eta 0:02:16 lr 0.000347 wd 0.0500 time 0.3669 (0.3752) data time 0.0011 (0.0031) model time 0.3658 (0.3730) loss 3.1502 (2.8097) grad_norm 2.2551 (2.3787) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][270/625] eta 0:02:13 lr 0.000347 wd 0.0500 time 0.3848 (0.3751) data time 0.0013 (0.0030) model time 0.3835 (0.3729) loss 2.1380 (2.8080) grad_norm 2.1361 (2.3707) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][280/625] eta 0:02:10 lr 0.000347 wd 0.0500 time 0.3703 (0.3772) data time 0.0013 (0.0029) model time 0.3690 (0.3755) loss 2.5696 (2.8051) grad_norm 1.9227 (2.3560) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][290/625] eta 0:02:06 lr 0.000346 wd 0.0500 time 0.3757 (0.3771) data time 0.0009 (0.0029) model time 0.3748 (0.3753) loss 3.1194 (2.7961) grad_norm 2.2124 (2.3471) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][300/625] eta 0:02:02 lr 0.000346 wd 0.0500 time 0.3696 (0.3769) data time 0.0011 (0.0028) model time 0.3685 (0.3751) loss 2.8819 (2.7915) grad_norm 1.6075 (2.3330) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][310/625] eta 0:01:58 lr 0.000346 wd 0.0500 time 0.3787 (0.3768) data time 0.0009 (0.0028) model time 0.3778 (0.3750) loss 3.4204 (2.7936) grad_norm 1.5511 (2.3208) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][320/625] eta 0:01:54 lr 0.000346 wd 0.0500 time 0.3680 (0.3767) data time 0.0015 (0.0027) model time 0.3665 (0.3749) loss 2.9580 (2.7896) grad_norm 2.6379 (2.3194) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][330/625] eta 0:01:51 lr 0.000346 wd 0.0500 time 0.3787 (0.3765) data time 0.0015 (0.0027) model time 0.3772 (0.3748) loss 2.9412 (2.7902) grad_norm 2.0753 (2.3104) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][340/625] eta 0:01:47 lr 0.000346 wd 0.0500 time 0.3750 (0.3764) data time 0.0009 (0.0026) model time 0.3741 (0.3747) loss 3.5989 (2.7954) grad_norm 2.8780 (2.3000) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][350/625] eta 0:01:43 lr 0.000346 wd 0.0500 time 0.3731 (0.3763) data time 0.0011 (0.0026) model time 0.3720 (0.3745) loss 2.7403 (2.7968) grad_norm 1.6417 (2.2894) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][360/625] eta 0:01:39 lr 0.000346 wd 0.0500 time 0.3750 (0.3762) data time 0.0009 (0.0026) model time 0.3741 (0.3744) loss 2.2072 (2.8007) grad_norm 1.6588 (2.2850) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][370/625] eta 0:01:35 lr 0.000345 wd 0.0500 time 0.3692 (0.3760) data time 0.0011 (0.0025) model time 0.3681 (0.3742) loss 3.1201 (2.8013) grad_norm 2.7052 (2.2777) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][380/625] eta 0:01:32 lr 0.000345 wd 0.0500 time 0.3719 (0.3759) data time 0.0009 (0.0025) model time 0.3710 (0.3741) loss 2.9246 (2.8037) grad_norm 2.6633 (2.2721) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][390/625] eta 0:01:28 lr 0.000345 wd 0.0500 time 0.3698 (0.3763) data time 0.0011 (0.0024) model time 0.3687 (0.3746) loss 3.1618 (2.8064) grad_norm 2.6907 (2.3033) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][400/625] eta 0:01:24 lr 0.000345 wd 0.0500 time 0.3775 (0.3762) data time 0.0011 (0.0024) model time 0.3765 (0.3744) loss 3.2250 (2.8141) grad_norm 2.0854 (2.3019) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:24:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][410/625] eta 0:01:20 lr 0.000345 wd 0.0500 time 0.3685 (0.3760) data time 0.0013 (0.0024) model time 0.3673 (0.3742) loss 3.0622 (2.8143) grad_norm 1.8519 (2.2966) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][420/625] eta 0:01:17 lr 0.000345 wd 0.0500 time 0.3799 (0.3759) data time 0.0013 (0.0024) model time 0.3785 (0.3742) loss 2.9464 (2.8142) grad_norm 1.9621 (2.2873) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][430/625] eta 0:01:13 lr 0.000345 wd 0.0500 time 0.3769 (0.3758) data time 0.0009 (0.0023) model time 0.3760 (0.3741) loss 1.9150 (2.8111) grad_norm 2.8436 (2.3012) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][440/625] eta 0:01:09 lr 0.000345 wd 0.0500 time 0.3715 (0.3757) data time 0.0012 (0.0023) model time 0.3703 (0.3740) loss 2.9980 (2.8153) grad_norm 1.9466 (2.2943) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][450/625] eta 0:01:05 lr 0.000344 wd 0.0500 time 0.3770 (0.3757) data time 0.0010 (0.0023) model time 0.3760 (0.3740) loss 2.3747 (2.8145) grad_norm 2.4006 (2.2877) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][460/625] eta 0:01:02 lr 0.000344 wd 0.0500 time 0.3728 (0.3760) data time 0.0013 (0.0023) model time 0.3715 (0.3744) loss 2.5267 (2.8122) grad_norm 2.3686 (2.2790) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][470/625] eta 0:00:58 lr 0.000344 wd 0.0500 time 0.3689 (0.3772) data time 0.0011 (0.0022) model time 0.3677 (0.3756) loss 2.8784 (2.8062) grad_norm 1.7690 (2.2686) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][480/625] eta 0:00:54 lr 0.000344 wd 0.0500 time 0.3735 (0.3770) data time 0.0011 (0.0022) model time 0.3724 (0.3755) loss 1.8085 (2.8054) grad_norm 2.0863 (2.2710) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][490/625] eta 0:00:50 lr 0.000344 wd 0.0500 time 0.3729 (0.3769) data time 0.0011 (0.0022) model time 0.3718 (0.3754) loss 3.0193 (2.8049) grad_norm 2.9560 (2.2690) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][500/625] eta 0:00:47 lr 0.000344 wd 0.0500 time 0.3767 (0.3768) data time 0.0009 (0.0022) model time 0.3758 (0.3752) loss 2.7677 (2.8042) grad_norm 2.8786 (2.2684) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][510/625] eta 0:00:43 lr 0.000344 wd 0.0500 time 0.3661 (0.3770) data time 0.0008 (0.0022) model time 0.3652 (0.3755) loss 1.6887 (2.8027) grad_norm 1.6658 (2.2659) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][520/625] eta 0:00:39 lr 0.000343 wd 0.0500 time 0.3623 (0.3769) data time 0.0011 (0.0021) model time 0.3612 (0.3754) loss 2.7513 (2.8016) grad_norm 1.6930 (2.2613) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][530/625] eta 0:00:35 lr 0.000343 wd 0.0500 time 0.3738 (0.3768) data time 0.0012 (0.0021) model time 0.3727 (0.3753) loss 2.5323 (2.7984) grad_norm 8.1192 (2.2654) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][540/625] eta 0:00:32 lr 0.000343 wd 0.0500 time 0.3693 (0.3767) data time 0.0011 (0.0021) model time 0.3681 (0.3752) loss 2.9859 (2.7957) grad_norm 1.8839 (2.2585) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][550/625] eta 0:00:28 lr 0.000343 wd 0.0500 time 0.3766 (0.3766) data time 0.0011 (0.0021) model time 0.3755 (0.3751) loss 3.1417 (2.7939) grad_norm 1.9579 (2.2829) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][560/625] eta 0:00:24 lr 0.000343 wd 0.0500 time 0.3711 (0.3765) data time 0.0009 (0.0021) model time 0.3702 (0.3750) loss 3.2591 (2.7936) grad_norm 1.7703 (2.2788) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][570/625] eta 0:00:20 lr 0.000343 wd 0.0500 time 0.3740 (0.3765) data time 0.0011 (0.0021) model time 0.3729 (0.3750) loss 3.0138 (2.7954) grad_norm 1.5528 (2.2707) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][580/625] eta 0:00:16 lr 0.000343 wd 0.0500 time 0.3595 (0.3767) data time 0.0011 (0.0020) model time 0.3585 (0.3752) loss 2.6859 (2.7941) grad_norm 1.8400 (2.2658) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][590/625] eta 0:00:13 lr 0.000343 wd 0.0500 time 0.3878 (0.3773) data time 0.0009 (0.0020) model time 0.3869 (0.3759) loss 2.8467 (2.7944) grad_norm 1.6726 (2.2584) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][600/625] eta 0:00:09 lr 0.000342 wd 0.0500 time 0.3808 (0.3773) data time 0.0011 (0.0020) model time 0.3798 (0.3759) loss 2.7329 (2.7945) grad_norm 1.5564 (2.2559) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][610/625] eta 0:00:05 lr 0.000342 wd 0.0500 time 0.3663 (0.3773) data time 0.0006 (0.0020) model time 0.3657 (0.3758) loss 2.8022 (2.7964) grad_norm 1.6305 (2.2539) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [225/300][620/625] eta 0:00:01 lr 0.000342 wd 0.0500 time 0.3716 (0.3770) data time 0.0006 (0.0020) model time 0.3711 (0.3756) loss 3.3591 (2.8005) grad_norm 3.4631 (2.2597) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:20 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 225 training takes 0:03:55 [2024-09-01 10:26:20 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:26:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:26:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.5225 (0.5225) Acc@1 88.965 (88.965) Acc@5 98.389 (98.389) Mem 11225MB [2024-09-01 10:26:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.144) Loss 0.8628 (0.6373) Acc@1 79.346 (85.929) Acc@5 95.996 (97.661) Mem 11225MB [2024-09-01 10:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.121) Loss 0.9321 (0.7558) Acc@1 77.295 (82.957) Acc@5 95.068 (96.380) Mem 11225MB [2024-09-01 10:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.616 Acc@5 96.355 [2024-09-01 10:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.6% [2024-09-01 10:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.62% [2024-09-01 10:26:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 10:26:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 10:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.521 (0.521) Loss 0.4915 (0.4915) Acc@1 89.697 (89.697) Acc@5 98.633 (98.633) Mem 11225MB [2024-09-01 10:26:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.144) Loss 0.7915 (0.5948) Acc@1 81.396 (86.808) Acc@5 96.240 (97.865) Mem 11225MB [2024-09-01 10:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.121) Loss 0.8799 (0.7047) Acc@1 78.369 (83.973) Acc@5 95.215 (96.731) Mem 11225MB [2024-09-01 10:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.701 Acc@5 96.775 [2024-09-01 10:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 10:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.70% [2024-09-01 10:26:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:26:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][0/625] eta 0:10:52 lr 0.000342 wd 0.0500 time 1.0437 (1.0437) data time 0.4738 (0.4738) model time 0.0000 (0.0000) loss 3.1280 (3.1280) grad_norm 2.8668 (2.8668) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][10/625] eta 0:04:26 lr 0.000342 wd 0.0500 time 0.3732 (0.4329) data time 0.0008 (0.0440) model time 0.0000 (0.0000) loss 2.1753 (2.6748) grad_norm 2.1725 (2.5454) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][20/625] eta 0:04:05 lr 0.000342 wd 0.0500 time 0.3688 (0.4057) data time 0.0011 (0.0236) model time 0.0000 (0.0000) loss 3.0783 (2.7692) grad_norm 2.2832 (2.6539) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][30/625] eta 0:03:55 lr 0.000342 wd 0.0500 time 0.3729 (0.3955) data time 0.0011 (0.0164) model time 0.0000 (0.0000) loss 2.2584 (2.7348) grad_norm 1.7641 (2.4704) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][40/625] eta 0:03:48 lr 0.000342 wd 0.0500 time 0.3741 (0.3900) data time 0.0015 (0.0127) model time 0.0000 (0.0000) loss 2.1695 (2.7597) grad_norm 2.3495 (2.4146) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][50/625] eta 0:03:41 lr 0.000341 wd 0.0500 time 0.3711 (0.3861) data time 0.0008 (0.0105) model time 0.0000 (0.0000) loss 2.3811 (2.7817) grad_norm 1.8552 (2.4036) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][60/625] eta 0:03:36 lr 0.000341 wd 0.0500 time 0.3644 (0.3836) data time 0.0008 (0.0089) model time 0.3635 (0.3699) loss 2.9399 (2.7987) grad_norm 2.8293 (2.4798) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:26:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][70/625] eta 0:03:32 lr 0.000341 wd 0.0500 time 0.3736 (0.3820) data time 0.0011 (0.0078) model time 0.3725 (0.3706) loss 2.6249 (2.8129) grad_norm 2.4250 (2.5569) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][80/625] eta 0:03:27 lr 0.000341 wd 0.0500 time 0.3823 (0.3809) data time 0.0008 (0.0070) model time 0.3814 (0.3710) loss 3.1869 (2.8252) grad_norm 2.3510 (2.5277) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][90/625] eta 0:03:23 lr 0.000341 wd 0.0500 time 0.3716 (0.3797) data time 0.0008 (0.0064) model time 0.3708 (0.3705) loss 3.3283 (2.8308) grad_norm 1.8917 (2.5209) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][100/625] eta 0:03:21 lr 0.000341 wd 0.0500 time 0.3681 (0.3846) data time 0.0013 (0.0059) model time 0.3668 (0.3820) loss 2.4652 (2.8270) grad_norm 2.3712 (2.5101) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][110/625] eta 0:03:17 lr 0.000341 wd 0.0500 time 0.3682 (0.3834) data time 0.0011 (0.0054) model time 0.3671 (0.3800) loss 2.0080 (2.8292) grad_norm 2.2331 (2.4714) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][120/625] eta 0:03:13 lr 0.000341 wd 0.0500 time 0.3650 (0.3823) data time 0.0013 (0.0051) model time 0.3636 (0.3784) loss 1.8105 (2.8405) grad_norm 2.2493 (2.4280) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][130/625] eta 0:03:08 lr 0.000340 wd 0.0500 time 0.3759 (0.3817) data time 0.0011 (0.0048) model time 0.3748 (0.3776) loss 1.7556 (2.8402) grad_norm 2.3026 (2.4306) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][140/625] eta 0:03:04 lr 0.000340 wd 0.0500 time 0.3746 (0.3809) data time 0.0012 (0.0045) model time 0.3734 (0.3767) loss 3.3077 (2.8489) grad_norm 1.7916 (2.4054) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][150/625] eta 0:03:00 lr 0.000340 wd 0.0500 time 0.3750 (0.3802) data time 0.0009 (0.0043) model time 0.3741 (0.3761) loss 1.7586 (2.8566) grad_norm 2.9157 (2.4645) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][160/625] eta 0:02:56 lr 0.000340 wd 0.0500 time 0.3692 (0.3797) data time 0.0012 (0.0041) model time 0.3680 (0.3755) loss 3.3967 (2.8675) grad_norm 2.6231 (2.5568) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][170/625] eta 0:02:52 lr 0.000340 wd 0.0500 time 0.3843 (0.3793) data time 0.0014 (0.0040) model time 0.3828 (0.3752) loss 2.3293 (2.8637) grad_norm 2.1154 (2.5271) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][180/625] eta 0:02:48 lr 0.000340 wd 0.0500 time 0.3787 (0.3788) data time 0.0008 (0.0038) model time 0.3779 (0.3747) loss 2.0286 (2.8681) grad_norm 1.9100 (2.4967) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][190/625] eta 0:02:45 lr 0.000340 wd 0.0500 time 0.3723 (0.3795) data time 0.0009 (0.0037) model time 0.3715 (0.3759) loss 3.2821 (2.8721) grad_norm 1.7002 (2.4746) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][200/625] eta 0:02:41 lr 0.000339 wd 0.0500 time 0.3730 (0.3791) data time 0.0012 (0.0036) model time 0.3718 (0.3755) loss 3.0355 (2.8747) grad_norm 2.4146 (2.4556) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][210/625] eta 0:02:37 lr 0.000339 wd 0.0500 time 0.3717 (0.3787) data time 0.0008 (0.0035) model time 0.3709 (0.3751) loss 2.2835 (2.8698) grad_norm 1.8134 (2.4425) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][220/625] eta 0:02:33 lr 0.000339 wd 0.0500 time 0.3781 (0.3783) data time 0.0011 (0.0034) model time 0.3769 (0.3747) loss 3.2754 (2.8629) grad_norm 2.5677 (2.4320) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:27:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][230/625] eta 0:02:29 lr 0.000339 wd 0.0500 time 0.3750 (0.3780) data time 0.0011 (0.0033) model time 0.3739 (0.3745) loss 3.0867 (2.8664) grad_norm 3.2237 (2.4076) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][240/625] eta 0:02:25 lr 0.000339 wd 0.0500 time 0.3638 (0.3777) data time 0.0009 (0.0032) model time 0.3629 (0.3742) loss 3.5030 (2.8669) grad_norm 2.4230 (2.3982) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][250/625] eta 0:02:21 lr 0.000339 wd 0.0500 time 0.3646 (0.3774) data time 0.0012 (0.0031) model time 0.3634 (0.3739) loss 3.4880 (2.8667) grad_norm 2.9720 (2.3842) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][260/625] eta 0:02:17 lr 0.000339 wd 0.0500 time 0.3650 (0.3773) data time 0.0009 (0.0030) model time 0.3641 (0.3740) loss 2.7081 (2.8585) grad_norm 2.2225 (2.3695) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][270/625] eta 0:02:13 lr 0.000339 wd 0.0500 time 0.3696 (0.3771) data time 0.0009 (0.0030) model time 0.3688 (0.3738) loss 1.8320 (2.8514) grad_norm 1.7253 (2.3522) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][280/625] eta 0:02:10 lr 0.000338 wd 0.0500 time 0.3665 (0.3769) data time 0.0011 (0.0029) model time 0.3654 (0.3736) loss 2.8647 (2.8433) grad_norm 4.7976 (2.3537) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][290/625] eta 0:02:06 lr 0.000338 wd 0.0500 time 0.3714 (0.3767) data time 0.0011 (0.0028) model time 0.3703 (0.3734) loss 3.3579 (2.8524) grad_norm 2.4712 (2.3531) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][300/625] eta 0:02:02 lr 0.000338 wd 0.0500 time 0.3722 (0.3765) data time 0.0012 (0.0028) model time 0.3710 (0.3733) loss 2.9148 (2.8525) grad_norm 1.8210 (2.3624) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][310/625] eta 0:01:59 lr 0.000338 wd 0.0500 time 0.3736 (0.3783) data time 0.0008 (0.0027) model time 0.3728 (0.3756) loss 2.8442 (2.8557) grad_norm 2.2548 (2.3591) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][320/625] eta 0:01:55 lr 0.000338 wd 0.0500 time 0.3676 (0.3781) data time 0.0012 (0.0027) model time 0.3664 (0.3754) loss 3.2760 (2.8553) grad_norm 2.2862 (2.3575) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][330/625] eta 0:01:51 lr 0.000338 wd 0.0500 time 0.3751 (0.3780) data time 0.0010 (0.0026) model time 0.3741 (0.3753) loss 3.8739 (2.8552) grad_norm 1.9081 (2.3477) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][340/625] eta 0:01:47 lr 0.000338 wd 0.0500 time 0.3875 (0.3778) data time 0.0009 (0.0026) model time 0.3866 (0.3751) loss 2.8174 (2.8526) grad_norm 1.3915 (2.3277) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][350/625] eta 0:01:43 lr 0.000337 wd 0.0500 time 0.3745 (0.3777) data time 0.0011 (0.0026) model time 0.3734 (0.3750) loss 3.2908 (2.8516) grad_norm 3.0048 (2.3187) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][360/625] eta 0:01:40 lr 0.000337 wd 0.0500 time 0.3670 (0.3774) data time 0.0012 (0.0025) model time 0.3658 (0.3748) loss 3.2725 (2.8500) grad_norm 2.2543 (2.3165) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][370/625] eta 0:01:36 lr 0.000337 wd 0.0500 time 0.3778 (0.3773) data time 0.0009 (0.0025) model time 0.3769 (0.3747) loss 3.6267 (2.8518) grad_norm 2.0476 (2.3070) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][380/625] eta 0:01:32 lr 0.000337 wd 0.0500 time 0.3662 (0.3771) data time 0.0009 (0.0024) model time 0.3654 (0.3745) loss 2.8423 (2.8548) grad_norm 2.5237 (2.3011) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:28:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][390/625] eta 0:01:28 lr 0.000337 wd 0.0500 time 0.3633 (0.3769) data time 0.0012 (0.0024) model time 0.3621 (0.3743) loss 3.0726 (2.8595) grad_norm 2.2756 (2.2950) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][400/625] eta 0:01:24 lr 0.000337 wd 0.0500 time 0.3853 (0.3768) data time 0.0009 (0.0024) model time 0.3845 (0.3742) loss 2.9642 (2.8620) grad_norm 1.8492 (2.2937) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][410/625] eta 0:01:20 lr 0.000337 wd 0.0500 time 0.3729 (0.3766) data time 0.0010 (0.0024) model time 0.3719 (0.3741) loss 3.1611 (2.8593) grad_norm 2.3881 (2.2916) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][420/625] eta 0:01:17 lr 0.000337 wd 0.0500 time 0.3739 (0.3764) data time 0.0009 (0.0023) model time 0.3731 (0.3739) loss 2.9650 (2.8562) grad_norm 2.6630 (2.2965) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][430/625] eta 0:01:13 lr 0.000336 wd 0.0500 time 0.3648 (0.3763) data time 0.0011 (0.0023) model time 0.3638 (0.3737) loss 2.4243 (2.8570) grad_norm 1.8776 (2.2931) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][440/625] eta 0:01:09 lr 0.000336 wd 0.0500 time 0.3682 (0.3762) data time 0.0012 (0.0023) model time 0.3670 (0.3736) loss 3.0375 (2.8598) grad_norm 2.0652 (2.2869) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][450/625] eta 0:01:05 lr 0.000336 wd 0.0500 time 0.3758 (0.3761) data time 0.0009 (0.0023) model time 0.3749 (0.3736) loss 3.7072 (2.8587) grad_norm 1.7996 (2.2802) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][460/625] eta 0:01:02 lr 0.000336 wd 0.0500 time 0.3676 (0.3760) data time 0.0010 (0.0022) model time 0.3666 (0.3735) loss 2.8401 (2.8541) grad_norm 2.4337 (2.2722) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][470/625] eta 0:00:58 lr 0.000336 wd 0.0500 time 0.3757 (0.3758) data time 0.0009 (0.0022) model time 0.3748 (0.3734) loss 2.0589 (2.8467) grad_norm 2.3888 (2.2674) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][480/625] eta 0:00:54 lr 0.000336 wd 0.0500 time 0.3678 (0.3757) data time 0.0008 (0.0022) model time 0.3670 (0.3733) loss 3.2341 (2.8489) grad_norm 1.5534 (2.2627) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][490/625] eta 0:00:50 lr 0.000336 wd 0.0500 time 0.3756 (0.3764) data time 0.0010 (0.0022) model time 0.3746 (0.3741) loss 3.2846 (2.8540) grad_norm 3.4500 (2.2632) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][500/625] eta 0:00:47 lr 0.000336 wd 0.0500 time 0.3612 (0.3763) data time 0.0009 (0.0022) model time 0.3602 (0.3740) loss 2.7024 (2.8570) grad_norm 2.5091 (2.2715) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][510/625] eta 0:00:43 lr 0.000335 wd 0.0500 time 0.3705 (0.3762) data time 0.0009 (0.0021) model time 0.3697 (0.3739) loss 2.7170 (2.8545) grad_norm 2.8483 (2.2663) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][520/625] eta 0:00:39 lr 0.000335 wd 0.0500 time 0.3714 (0.3765) data time 0.0010 (0.0021) model time 0.3704 (0.3742) loss 3.3793 (2.8561) grad_norm 2.8751 (2.2596) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][530/625] eta 0:00:35 lr 0.000335 wd 0.0500 time 0.3710 (0.3764) data time 0.0012 (0.0021) model time 0.3699 (0.3741) loss 3.4393 (2.8591) grad_norm 1.8197 (2.2528) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][540/625] eta 0:00:31 lr 0.000335 wd 0.0500 time 0.3740 (0.3763) data time 0.0008 (0.0021) model time 0.3732 (0.3740) loss 3.5055 (2.8534) grad_norm 1.4180 (2.2538) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:29:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][550/625] eta 0:00:28 lr 0.000335 wd 0.0500 time 0.3736 (0.3762) data time 0.0011 (0.0021) model time 0.3725 (0.3740) loss 3.0030 (2.8528) grad_norm 2.8633 (2.2502) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][560/625] eta 0:00:24 lr 0.000335 wd 0.0500 time 0.3684 (0.3761) data time 0.0009 (0.0021) model time 0.3675 (0.3738) loss 3.0174 (2.8502) grad_norm 3.7243 (2.2509) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][570/625] eta 0:00:20 lr 0.000335 wd 0.0500 time 0.3699 (0.3760) data time 0.0009 (0.0020) model time 0.3691 (0.3738) loss 2.2819 (2.8506) grad_norm 2.6958 (2.2529) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][580/625] eta 0:00:16 lr 0.000335 wd 0.0500 time 0.3658 (0.3759) data time 0.0009 (0.0020) model time 0.3649 (0.3737) loss 3.1391 (2.8582) grad_norm 2.1529 (2.2526) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][590/625] eta 0:00:13 lr 0.000334 wd 0.0500 time 0.3666 (0.3758) data time 0.0009 (0.0020) model time 0.3658 (0.3736) loss 1.9099 (2.8526) grad_norm 2.1550 (2.2605) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][600/625] eta 0:00:09 lr 0.000334 wd 0.0500 time 0.3631 (0.3757) data time 0.0012 (0.0020) model time 0.3620 (0.3735) loss 2.7047 (2.8535) grad_norm 1.6316 (2.2590) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][610/625] eta 0:00:05 lr 0.000334 wd 0.0500 time 0.3648 (0.3763) data time 0.0008 (0.0020) model time 0.3640 (0.3742) loss 3.1947 (2.8525) grad_norm 2.0151 (2.2542) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [226/300][620/625] eta 0:00:01 lr 0.000334 wd 0.0500 time 0.3688 (0.3761) data time 0.0006 (0.0020) model time 0.3683 (0.3740) loss 3.4534 (2.8525) grad_norm 2.0205 (2.2500) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:24 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 226 training takes 0:03:55 [2024-09-01 10:30:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:30:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:30:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.491 (0.491) Loss 0.5220 (0.5220) Acc@1 88.477 (88.477) Acc@5 98.438 (98.438) Mem 11225MB [2024-09-01 10:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.139) Loss 0.8560 (0.6297) Acc@1 79.248 (86.044) Acc@5 95.752 (97.692) Mem 11225MB [2024-09-01 10:30:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 0.9517 (0.7528) Acc@1 77.100 (83.033) Acc@5 94.873 (96.394) Mem 11225MB [2024-09-01 10:30:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.798 Acc@5 96.427 [2024-09-01 10:30:28 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.8% [2024-09-01 10:30:28 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.80% [2024-09-01 10:30:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 10:30:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 10:30:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.520 (0.520) Loss 0.4912 (0.4912) Acc@1 89.648 (89.648) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.7915 (0.5945) Acc@1 81.348 (86.790) Acc@5 96.240 (97.865) Mem 11225MB [2024-09-01 10:30:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.8784 (0.7042) Acc@1 78.467 (83.956) Acc@5 95.117 (96.724) Mem 11225MB [2024-09-01 10:30:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.691 Acc@5 96.761 [2024-09-01 10:30:32 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 10:30:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][0/625] eta 0:12:41 lr 0.000334 wd 0.0500 time 1.2188 (1.2188) data time 0.5777 (0.5777) model time 0.0000 (0.0000) loss 2.9515 (2.9515) grad_norm 1.7499 (1.7499) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][10/625] eta 0:04:36 lr 0.000334 wd 0.0500 time 0.3718 (0.4490) data time 0.0008 (0.0536) model time 0.0000 (0.0000) loss 2.1930 (2.6249) grad_norm 2.0566 (2.7160) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][20/625] eta 0:04:15 lr 0.000334 wd 0.0500 time 0.3665 (0.4218) data time 0.0009 (0.0286) model time 0.0000 (0.0000) loss 3.0391 (2.6953) grad_norm 3.1325 (2.5441) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][30/625] eta 0:04:01 lr 0.000334 wd 0.0500 time 0.3762 (0.4059) data time 0.0011 (0.0198) model time 0.0000 (0.0000) loss 3.1546 (2.8057) grad_norm 2.5559 (2.4561) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][40/625] eta 0:03:52 lr 0.000333 wd 0.0500 time 0.3673 (0.3973) data time 0.0011 (0.0152) model time 0.0000 (0.0000) loss 3.3134 (2.7950) grad_norm 2.4691 (2.6065) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][50/625] eta 0:03:45 lr 0.000333 wd 0.0500 time 0.3707 (0.3920) data time 0.0011 (0.0125) model time 0.0000 (0.0000) loss 2.0064 (2.7795) grad_norm 2.8299 (2.6412) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][60/625] eta 0:03:39 lr 0.000333 wd 0.0500 time 0.3719 (0.3885) data time 0.0009 (0.0106) model time 0.3710 (0.3694) loss 3.2665 (2.8341) grad_norm 1.9766 (2.5740) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:30:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][70/625] eta 0:03:34 lr 0.000333 wd 0.0500 time 0.3692 (0.3858) data time 0.0008 (0.0093) model time 0.3683 (0.3686) loss 2.8905 (2.8474) grad_norm 1.9158 (2.5283) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:31:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][80/625] eta 0:03:29 lr 0.000333 wd 0.0500 time 0.3706 (0.3841) data time 0.0013 (0.0083) model time 0.3693 (0.3696) loss 3.1040 (2.8753) grad_norm 1.8001 (2.4694) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:31:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][90/625] eta 0:03:24 lr 0.000333 wd 0.0500 time 0.3686 (0.3828) data time 0.0013 (0.0075) model time 0.3673 (0.3700) loss 2.9205 (2.8587) grad_norm 2.8229 (2.4393) loss_scale 1024.0000 (545.7582) mem 11225MB [2024-09-01 10:31:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][100/625] eta 0:03:20 lr 0.000333 wd 0.0500 time 0.3731 (0.3816) data time 0.0009 (0.0069) model time 0.3723 (0.3698) loss 3.4736 (2.8409) grad_norm 2.0772 (2.4411) loss_scale 1024.0000 (593.1089) mem 11225MB [2024-09-01 10:31:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][110/625] eta 0:03:16 lr 0.000332 wd 0.0500 time 0.3680 (0.3809) data time 0.0009 (0.0063) model time 0.3671 (0.3703) loss 3.0593 (2.8381) grad_norm 2.0301 (2.4888) loss_scale 1024.0000 (631.9279) mem 11225MB [2024-09-01 10:31:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][120/625] eta 0:03:11 lr 0.000332 wd 0.0500 time 0.3668 (0.3801) data time 0.0012 (0.0059) model time 0.3656 (0.3703) loss 3.0332 (2.8270) grad_norm 1.7559 (2.4398) loss_scale 1024.0000 (664.3306) mem 11225MB [2024-09-01 10:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][130/625] eta 0:03:07 lr 0.000332 wd 0.0500 time 0.3715 (0.3794) data time 0.0008 (0.0056) model time 0.3707 (0.3702) loss 3.2994 (2.8219) grad_norm 1.6978 (2.3917) loss_scale 1024.0000 (691.7863) mem 11225MB [2024-09-01 10:31:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][140/625] eta 0:03:04 lr 0.000332 wd 0.0500 time 0.3703 (0.3802) data time 0.0012 (0.0053) model time 0.3692 (0.3723) loss 3.2164 (2.8315) grad_norm 1.9071 (2.3565) loss_scale 1024.0000 (715.3475) mem 11225MB [2024-09-01 10:31:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][150/625] eta 0:03:00 lr 0.000332 wd 0.0500 time 0.3691 (0.3797) data time 0.0009 (0.0050) model time 0.3682 (0.3722) loss 3.0137 (2.8279) grad_norm 2.1753 (2.3380) loss_scale 1024.0000 (735.7881) mem 11225MB [2024-09-01 10:31:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][160/625] eta 0:02:56 lr 0.000332 wd 0.0500 time 0.3647 (0.3789) data time 0.0011 (0.0047) model time 0.3636 (0.3717) loss 3.3146 (2.7993) grad_norm 3.8029 (2.3215) loss_scale 1024.0000 (753.6894) mem 11225MB [2024-09-01 10:31:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][170/625] eta 0:02:53 lr 0.000332 wd 0.0500 time 0.3685 (0.3818) data time 0.0011 (0.0045) model time 0.3674 (0.3762) loss 3.0732 (2.7993) grad_norm 2.6863 (2.3249) loss_scale 1024.0000 (769.4971) mem 11225MB [2024-09-01 10:31:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][180/625] eta 0:02:49 lr 0.000332 wd 0.0500 time 0.3681 (0.3811) data time 0.0011 (0.0044) model time 0.3670 (0.3756) loss 2.9920 (2.8002) grad_norm 2.5057 (2.3210) loss_scale 1024.0000 (783.5580) mem 11225MB [2024-09-01 10:31:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][190/625] eta 0:02:45 lr 0.000331 wd 0.0500 time 0.3791 (0.3806) data time 0.0013 (0.0042) model time 0.3778 (0.3753) loss 2.9200 (2.7976) grad_norm 5.4522 (2.3420) loss_scale 1024.0000 (796.1466) mem 11225MB [2024-09-01 10:31:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][200/625] eta 0:02:41 lr 0.000331 wd 0.0500 time 0.3679 (0.3801) data time 0.0012 (0.0040) model time 0.3667 (0.3749) loss 3.1416 (2.8028) grad_norm 2.0737 (2.3298) loss_scale 1024.0000 (807.4826) mem 11225MB [2024-09-01 10:31:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][210/625] eta 0:02:37 lr 0.000331 wd 0.0500 time 0.3727 (0.3798) data time 0.0012 (0.0039) model time 0.3716 (0.3747) loss 2.9501 (2.8087) grad_norm 1.8162 (2.3513) loss_scale 1024.0000 (817.7441) mem 11225MB [2024-09-01 10:31:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][220/625] eta 0:02:33 lr 0.000331 wd 0.0500 time 0.3767 (0.3794) data time 0.0008 (0.0038) model time 0.3759 (0.3745) loss 3.3600 (2.8187) grad_norm 2.6281 (2.3608) loss_scale 1024.0000 (827.0769) mem 11225MB [2024-09-01 10:32:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][230/625] eta 0:02:29 lr 0.000331 wd 0.0500 time 0.3769 (0.3793) data time 0.0009 (0.0037) model time 0.3761 (0.3745) loss 2.6162 (2.8255) grad_norm 2.7813 (2.3798) loss_scale 1024.0000 (835.6017) mem 11225MB [2024-09-01 10:32:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][240/625] eta 0:02:25 lr 0.000331 wd 0.0500 time 0.3718 (0.3789) data time 0.0010 (0.0036) model time 0.3709 (0.3742) loss 2.1179 (2.8205) grad_norm 2.0308 (2.3747) loss_scale 1024.0000 (843.4191) mem 11225MB [2024-09-01 10:32:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][250/625] eta 0:02:21 lr 0.000331 wd 0.0500 time 0.3662 (0.3785) data time 0.0010 (0.0035) model time 0.3652 (0.3739) loss 2.2191 (2.8046) grad_norm 1.6084 (2.3561) loss_scale 1024.0000 (850.6135) mem 11225MB [2024-09-01 10:32:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][260/625] eta 0:02:18 lr 0.000331 wd 0.0500 time 0.3668 (0.3782) data time 0.0012 (0.0034) model time 0.3656 (0.3736) loss 3.1018 (2.7928) grad_norm 1.4925 (2.3419) loss_scale 1024.0000 (857.2567) mem 11225MB [2024-09-01 10:32:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][270/625] eta 0:02:14 lr 0.000330 wd 0.0500 time 0.3713 (0.3779) data time 0.0008 (0.0033) model time 0.3704 (0.3734) loss 2.3464 (2.7891) grad_norm 1.4982 (2.3249) loss_scale 1024.0000 (863.4096) mem 11225MB [2024-09-01 10:32:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][280/625] eta 0:02:10 lr 0.000330 wd 0.0500 time 0.3672 (0.3776) data time 0.0013 (0.0032) model time 0.3659 (0.3733) loss 3.1466 (2.7993) grad_norm 2.2906 (2.3205) loss_scale 1024.0000 (869.1246) mem 11225MB [2024-09-01 10:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][290/625] eta 0:02:06 lr 0.000330 wd 0.0500 time 0.3742 (0.3788) data time 0.0009 (0.0032) model time 0.3733 (0.3749) loss 3.3962 (2.8094) grad_norm 2.2680 (2.3168) loss_scale 1024.0000 (874.4467) mem 11225MB [2024-09-01 10:32:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][300/625] eta 0:02:03 lr 0.000330 wd 0.0500 time 0.3739 (0.3786) data time 0.0009 (0.0031) model time 0.3730 (0.3747) loss 3.2598 (2.8099) grad_norm 2.0212 (2.3320) loss_scale 1024.0000 (879.4153) mem 11225MB [2024-09-01 10:32:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][310/625] eta 0:01:59 lr 0.000330 wd 0.0500 time 0.3781 (0.3784) data time 0.0009 (0.0030) model time 0.3772 (0.3746) loss 3.1139 (2.8085) grad_norm 2.7178 (2.3945) loss_scale 1024.0000 (884.0643) mem 11225MB [2024-09-01 10:32:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][320/625] eta 0:01:55 lr 0.000330 wd 0.0500 time 0.3726 (0.3782) data time 0.0010 (0.0030) model time 0.3715 (0.3744) loss 2.5354 (2.8109) grad_norm 2.0419 (2.4021) loss_scale 1024.0000 (888.4237) mem 11225MB [2024-09-01 10:32:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][330/625] eta 0:01:51 lr 0.000330 wd 0.0500 time 0.3718 (0.3780) data time 0.0013 (0.0029) model time 0.3704 (0.3743) loss 2.9975 (2.8117) grad_norm 2.0061 (2.3911) loss_scale 1024.0000 (892.5196) mem 11225MB [2024-09-01 10:32:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][340/625] eta 0:01:47 lr 0.000330 wd 0.0500 time 0.3673 (0.3779) data time 0.0011 (0.0029) model time 0.3662 (0.3742) loss 2.9302 (2.8126) grad_norm 1.6098 (2.3811) loss_scale 1024.0000 (896.3754) mem 11225MB [2024-09-01 10:32:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][350/625] eta 0:01:44 lr 0.000329 wd 0.0500 time 0.3699 (0.3783) data time 0.0009 (0.0028) model time 0.3690 (0.3748) loss 2.1271 (2.8102) grad_norm 1.6010 (2.3715) loss_scale 1024.0000 (900.0114) mem 11225MB [2024-09-01 10:32:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][360/625] eta 0:01:40 lr 0.000329 wd 0.0500 time 0.3771 (0.3781) data time 0.0009 (0.0028) model time 0.3763 (0.3746) loss 1.9238 (2.8091) grad_norm 2.1279 (2.3820) loss_scale 1024.0000 (903.4460) mem 11225MB [2024-09-01 10:32:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][370/625] eta 0:01:36 lr 0.000329 wd 0.0500 time 0.3723 (0.3780) data time 0.0011 (0.0027) model time 0.3712 (0.3746) loss 3.0942 (2.8033) grad_norm 1.2929 (2.3694) loss_scale 1024.0000 (906.6954) mem 11225MB [2024-09-01 10:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][380/625] eta 0:01:32 lr 0.000329 wd 0.0500 time 0.3749 (0.3778) data time 0.0011 (0.0027) model time 0.3738 (0.3744) loss 3.5454 (2.7982) grad_norm 2.0896 (2.3619) loss_scale 1024.0000 (909.7743) mem 11225MB [2024-09-01 10:33:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][390/625] eta 0:01:28 lr 0.000329 wd 0.0500 time 0.3676 (0.3776) data time 0.0012 (0.0027) model time 0.3664 (0.3743) loss 3.2599 (2.7979) grad_norm 2.3800 (2.3624) loss_scale 1024.0000 (912.6957) mem 11225MB [2024-09-01 10:33:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][400/625] eta 0:01:24 lr 0.000329 wd 0.0500 time 0.3699 (0.3776) data time 0.0013 (0.0026) model time 0.3687 (0.3743) loss 2.5354 (2.7941) grad_norm 4.9194 (2.3675) loss_scale 1024.0000 (915.4713) mem 11225MB [2024-09-01 10:33:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][410/625] eta 0:01:21 lr 0.000329 wd 0.0500 time 0.3657 (0.3775) data time 0.0011 (0.0026) model time 0.3646 (0.3742) loss 2.0613 (2.7960) grad_norm 1.3915 (2.3631) loss_scale 1024.0000 (918.1119) mem 11225MB [2024-09-01 10:33:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][420/625] eta 0:01:17 lr 0.000328 wd 0.0500 time 0.3699 (0.3774) data time 0.0009 (0.0026) model time 0.3690 (0.3742) loss 2.1078 (2.7898) grad_norm 1.6192 (2.3580) loss_scale 1024.0000 (920.6271) mem 11225MB [2024-09-01 10:33:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][430/625] eta 0:01:13 lr 0.000328 wd 0.0500 time 0.3673 (0.3773) data time 0.0009 (0.0025) model time 0.3664 (0.3741) loss 3.2414 (2.7904) grad_norm 2.1749 (2.3579) loss_scale 1024.0000 (923.0255) mem 11225MB [2024-09-01 10:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][440/625] eta 0:01:09 lr 0.000328 wd 0.0500 time 0.3688 (0.3772) data time 0.0009 (0.0025) model time 0.3678 (0.3741) loss 2.4332 (2.7887) grad_norm 2.3043 (2.3476) loss_scale 1024.0000 (925.3152) mem 11225MB [2024-09-01 10:33:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][450/625] eta 0:01:05 lr 0.000328 wd 0.0500 time 0.3675 (0.3771) data time 0.0009 (0.0025) model time 0.3666 (0.3740) loss 2.9525 (2.7871) grad_norm 1.5069 (2.3460) loss_scale 1024.0000 (927.5033) mem 11225MB [2024-09-01 10:33:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][460/625] eta 0:01:02 lr 0.000328 wd 0.0500 time 0.3762 (0.3771) data time 0.0011 (0.0024) model time 0.3752 (0.3740) loss 3.2523 (2.7895) grad_norm 1.8473 (2.3333) loss_scale 1024.0000 (929.5965) mem 11225MB [2024-09-01 10:33:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][470/625] eta 0:00:58 lr 0.000328 wd 0.0500 time 0.3678 (0.3769) data time 0.0011 (0.0024) model time 0.3668 (0.3739) loss 2.5489 (2.7889) grad_norm 1.9262 (2.3500) loss_scale 1024.0000 (931.6008) mem 11225MB [2024-09-01 10:33:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][480/625] eta 0:00:54 lr 0.000328 wd 0.0500 time 0.3719 (0.3769) data time 0.0012 (0.0024) model time 0.3708 (0.3739) loss 1.8228 (2.7928) grad_norm 1.9765 (2.3460) loss_scale 1024.0000 (933.5218) mem 11225MB [2024-09-01 10:33:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][490/625] eta 0:00:50 lr 0.000328 wd 0.0500 time 0.3896 (0.3769) data time 0.0011 (0.0024) model time 0.3885 (0.3739) loss 2.1861 (2.7872) grad_norm 1.6272 (2.3562) loss_scale 1024.0000 (935.3646) mem 11225MB [2024-09-01 10:33:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][500/625] eta 0:00:47 lr 0.000327 wd 0.0500 time 0.3717 (0.3776) data time 0.0011 (0.0023) model time 0.3706 (0.3748) loss 3.0569 (2.7829) grad_norm 2.3342 (2.3506) loss_scale 1024.0000 (937.1337) mem 11225MB [2024-09-01 10:33:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][510/625] eta 0:00:43 lr 0.000327 wd 0.0500 time 0.3764 (0.3775) data time 0.0012 (0.0023) model time 0.3752 (0.3747) loss 3.0129 (2.7795) grad_norm 2.8940 (2.3579) loss_scale 1024.0000 (938.8337) mem 11225MB [2024-09-01 10:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][520/625] eta 0:00:39 lr 0.000327 wd 0.0500 time 0.3757 (0.3774) data time 0.0011 (0.0023) model time 0.3746 (0.3746) loss 2.9853 (2.7826) grad_norm 1.5357 (2.3595) loss_scale 1024.0000 (940.4683) mem 11225MB [2024-09-01 10:33:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][530/625] eta 0:00:35 lr 0.000327 wd 0.0500 time 0.5542 (0.3777) data time 0.0010 (0.0023) model time 0.5532 (0.3750) loss 3.2909 (2.7861) grad_norm 2.3093 (2.3604) loss_scale 1024.0000 (942.0414) mem 11225MB [2024-09-01 10:33:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][540/625] eta 0:00:32 lr 0.000327 wd 0.0500 time 0.3703 (0.3775) data time 0.0009 (0.0023) model time 0.3694 (0.3748) loss 3.5301 (2.7847) grad_norm 1.7491 (2.3519) loss_scale 1024.0000 (943.5564) mem 11225MB [2024-09-01 10:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][550/625] eta 0:00:28 lr 0.000327 wd 0.0500 time 0.3733 (0.3775) data time 0.0012 (0.0022) model time 0.3721 (0.3748) loss 3.0880 (2.7852) grad_norm 2.3786 (2.3485) loss_scale 1024.0000 (945.0163) mem 11225MB [2024-09-01 10:34:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][560/625] eta 0:00:24 lr 0.000327 wd 0.0500 time 0.3673 (0.3774) data time 0.0011 (0.0022) model time 0.3662 (0.3747) loss 2.5328 (2.7862) grad_norm 2.4848 (2.3446) loss_scale 1024.0000 (946.4242) mem 11225MB [2024-09-01 10:34:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][570/625] eta 0:00:20 lr 0.000327 wd 0.0500 time 0.3738 (0.3773) data time 0.0011 (0.0022) model time 0.3727 (0.3747) loss 3.0333 (2.7901) grad_norm 2.5672 (2.3418) loss_scale 1024.0000 (947.7828) mem 11225MB [2024-09-01 10:34:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][580/625] eta 0:00:16 lr 0.000326 wd 0.0500 time 0.3701 (0.3773) data time 0.0009 (0.0022) model time 0.3693 (0.3746) loss 2.6364 (2.7885) grad_norm 1.7488 (2.3394) loss_scale 1024.0000 (949.0947) mem 11225MB [2024-09-01 10:34:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][590/625] eta 0:00:13 lr 0.000326 wd 0.0500 time 0.3800 (0.3772) data time 0.0012 (0.0022) model time 0.3788 (0.3746) loss 3.2007 (2.7920) grad_norm 3.3210 (2.3442) loss_scale 1024.0000 (950.3621) mem 11225MB [2024-09-01 10:34:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][600/625] eta 0:00:09 lr 0.000326 wd 0.0500 time 0.3745 (0.3772) data time 0.0011 (0.0022) model time 0.3734 (0.3746) loss 3.3231 (2.7969) grad_norm 2.1634 (2.3572) loss_scale 1024.0000 (951.5874) mem 11225MB [2024-09-01 10:34:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][610/625] eta 0:00:05 lr 0.000326 wd 0.0500 time 0.3613 (0.3771) data time 0.0006 (0.0021) model time 0.3608 (0.3745) loss 3.1337 (2.7987) grad_norm 1.5935 (2.3601) loss_scale 1024.0000 (952.7725) mem 11225MB [2024-09-01 10:34:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [227/300][620/625] eta 0:00:01 lr 0.000326 wd 0.0500 time 0.3626 (0.3769) data time 0.0006 (0.0021) model time 0.3620 (0.3743) loss 3.0558 (2.7998) grad_norm 2.1570 (2.3733) loss_scale 1024.0000 (953.9195) mem 11225MB [2024-09-01 10:34:28 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 227 training takes 0:03:55 [2024-09-01 10:34:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:34:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:34:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.514 (0.514) Loss 0.5610 (0.5610) Acc@1 88.086 (88.086) Acc@5 98.438 (98.438) Mem 11225MB [2024-09-01 10:34:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 0.8740 (0.6615) Acc@1 79.102 (86.080) Acc@5 95.703 (97.607) Mem 11225MB [2024-09-01 10:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.9585 (0.7779) Acc@1 77.344 (82.973) Acc@5 94.922 (96.366) Mem 11225MB [2024-09-01 10:34:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.676 Acc@5 96.385 [2024-09-01 10:34:32 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.7% [2024-09-01 10:34:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.808 (0.808) Loss 0.4905 (0.4905) Acc@1 89.746 (89.746) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:34:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.174) Loss 0.7915 (0.5947) Acc@1 81.299 (86.830) Acc@5 96.289 (97.883) Mem 11225MB [2024-09-01 10:34:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.135) Loss 0.8784 (0.7043) Acc@1 78.711 (84.008) Acc@5 95.215 (96.761) Mem 11225MB [2024-09-01 10:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.741 Acc@5 96.797 [2024-09-01 10:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 10:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.74% [2024-09-01 10:34:35 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:34:36 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:34:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][0/625] eta 0:07:32 lr 0.000326 wd 0.0500 time 0.7243 (0.7243) data time 0.3805 (0.3805) model time 0.0000 (0.0000) loss 3.0964 (3.0964) grad_norm 2.7587 (2.7587) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:34:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][10/625] eta 0:04:07 lr 0.000326 wd 0.0500 time 0.3749 (0.4029) data time 0.0011 (0.0357) model time 0.0000 (0.0000) loss 1.5802 (2.8629) grad_norm 2.9854 (2.4770) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:34:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][20/625] eta 0:03:54 lr 0.000326 wd 0.0500 time 0.3663 (0.3881) data time 0.0009 (0.0192) model time 0.0000 (0.0000) loss 3.5128 (2.8600) grad_norm 2.0399 (2.4342) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:34:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][30/625] eta 0:03:47 lr 0.000325 wd 0.0500 time 0.3706 (0.3823) data time 0.0011 (0.0134) model time 0.0000 (0.0000) loss 3.0552 (2.8744) grad_norm 2.1269 (2.2614) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][40/625] eta 0:03:42 lr 0.000325 wd 0.0500 time 0.3726 (0.3800) data time 0.0009 (0.0105) model time 0.0000 (0.0000) loss 2.4702 (2.7700) grad_norm 2.2795 (2.2356) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:34:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][50/625] eta 0:03:37 lr 0.000325 wd 0.0500 time 0.3662 (0.3781) data time 0.0011 (0.0086) model time 0.0000 (0.0000) loss 3.2008 (2.7664) grad_norm 2.3079 (2.1733) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][60/625] eta 0:03:34 lr 0.000325 wd 0.0500 time 0.3804 (0.3799) data time 0.0013 (0.0074) model time 0.3791 (0.3877) loss 2.7454 (2.7521) grad_norm 1.8222 (2.1165) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][70/625] eta 0:03:33 lr 0.000325 wd 0.0500 time 0.3734 (0.3850) data time 0.0011 (0.0066) model time 0.3722 (0.4013) loss 3.1122 (2.7801) grad_norm 1.9828 (2.1197) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][80/625] eta 0:03:29 lr 0.000325 wd 0.0500 time 0.3728 (0.3837) data time 0.0009 (0.0059) model time 0.3719 (0.3919) loss 2.7477 (2.8084) grad_norm 2.3091 (2.1374) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][90/625] eta 0:03:24 lr 0.000325 wd 0.0500 time 0.3662 (0.3823) data time 0.0009 (0.0054) model time 0.3654 (0.3864) loss 3.3033 (2.8105) grad_norm 1.6785 (2.1161) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][100/625] eta 0:03:20 lr 0.000325 wd 0.0500 time 0.3693 (0.3816) data time 0.0009 (0.0050) model time 0.3684 (0.3839) loss 2.2347 (2.8104) grad_norm 1.3425 (2.1103) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][110/625] eta 0:03:16 lr 0.000324 wd 0.0500 time 0.3744 (0.3806) data time 0.0009 (0.0046) model time 0.3736 (0.3815) loss 2.0646 (2.7994) grad_norm 2.3955 (2.1811) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][120/625] eta 0:03:11 lr 0.000324 wd 0.0500 time 0.3765 (0.3798) data time 0.0012 (0.0043) model time 0.3753 (0.3798) loss 3.1268 (2.8169) grad_norm 3.1013 (2.2369) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][130/625] eta 0:03:08 lr 0.000324 wd 0.0500 time 0.3719 (0.3807) data time 0.0012 (0.0041) model time 0.3707 (0.3811) loss 3.1310 (2.8155) grad_norm 2.6734 (2.2477) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][140/625] eta 0:03:04 lr 0.000324 wd 0.0500 time 0.3696 (0.3801) data time 0.0012 (0.0039) model time 0.3683 (0.3800) loss 3.1297 (2.8303) grad_norm 2.0447 (2.2633) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][150/625] eta 0:03:00 lr 0.000324 wd 0.0500 time 0.3671 (0.3794) data time 0.0009 (0.0037) model time 0.3662 (0.3788) loss 2.5178 (2.8169) grad_norm 1.9119 (2.2617) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][160/625] eta 0:02:56 lr 0.000324 wd 0.0500 time 0.3733 (0.3788) data time 0.0013 (0.0036) model time 0.3720 (0.3778) loss 2.7318 (2.8097) grad_norm 1.3739 (2.2512) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][170/625] eta 0:02:52 lr 0.000324 wd 0.0500 time 0.3681 (0.3784) data time 0.0009 (0.0034) model time 0.3672 (0.3773) loss 3.2954 (2.8141) grad_norm 3.3487 (2.2390) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][180/625] eta 0:02:48 lr 0.000324 wd 0.0500 time 0.3836 (0.3784) data time 0.0009 (0.0033) model time 0.3827 (0.3772) loss 1.9778 (2.8060) grad_norm 2.3550 (2.2335) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][190/625] eta 0:02:44 lr 0.000323 wd 0.0500 time 0.3792 (0.3781) data time 0.0010 (0.0032) model time 0.3782 (0.3769) loss 3.3257 (2.8008) grad_norm 2.0446 (2.2371) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][200/625] eta 0:02:40 lr 0.000323 wd 0.0500 time 0.3703 (0.3778) data time 0.0009 (0.0031) model time 0.3694 (0.3765) loss 1.7722 (2.7925) grad_norm 2.0274 (2.2261) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:35:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][210/625] eta 0:02:36 lr 0.000323 wd 0.0500 time 0.3676 (0.3776) data time 0.0011 (0.0030) model time 0.3665 (0.3763) loss 3.3051 (2.8045) grad_norm 1.4000 (2.2383) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][220/625] eta 0:02:32 lr 0.000323 wd 0.0500 time 0.3840 (0.3775) data time 0.0013 (0.0029) model time 0.3828 (0.3760) loss 2.6089 (2.8163) grad_norm 2.3987 (2.2370) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][230/625] eta 0:02:28 lr 0.000323 wd 0.0500 time 0.3578 (0.3772) data time 0.0015 (0.0029) model time 0.3563 (0.3757) loss 2.9214 (2.8228) grad_norm 1.9123 (2.2424) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][240/625] eta 0:02:25 lr 0.000323 wd 0.0500 time 0.5156 (0.3776) data time 0.0011 (0.0028) model time 0.5145 (0.3763) loss 3.2508 (2.8213) grad_norm 1.6593 (2.2342) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][250/625] eta 0:02:22 lr 0.000323 wd 0.0500 time 0.3708 (0.3790) data time 0.0012 (0.0027) model time 0.3696 (0.3780) loss 2.9295 (2.8293) grad_norm 1.6465 (2.2597) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][260/625] eta 0:02:18 lr 0.000323 wd 0.0500 time 0.3671 (0.3786) data time 0.0009 (0.0027) model time 0.3663 (0.3775) loss 3.2551 (2.8309) grad_norm 1.8296 (2.2417) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][270/625] eta 0:02:14 lr 0.000322 wd 0.0500 time 0.3737 (0.3784) data time 0.0013 (0.0026) model time 0.3725 (0.3773) loss 3.3494 (2.8327) grad_norm 1.6007 (2.2297) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][280/625] eta 0:02:10 lr 0.000322 wd 0.0500 time 0.3666 (0.3783) data time 0.0012 (0.0026) model time 0.3654 (0.3771) loss 2.3739 (2.8267) grad_norm 2.2359 (2.2247) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][290/625] eta 0:02:06 lr 0.000322 wd 0.0500 time 0.3720 (0.3781) data time 0.0009 (0.0025) model time 0.3711 (0.3768) loss 3.6727 (2.8359) grad_norm 2.0790 (2.2170) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][300/625] eta 0:02:02 lr 0.000322 wd 0.0500 time 0.3681 (0.3778) data time 0.0011 (0.0025) model time 0.3670 (0.3765) loss 2.9819 (2.8377) grad_norm 6.1618 (2.2228) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][310/625] eta 0:01:58 lr 0.000322 wd 0.0500 time 0.3728 (0.3776) data time 0.0013 (0.0024) model time 0.3715 (0.3763) loss 3.0723 (2.8493) grad_norm 2.0221 (2.2166) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][320/625] eta 0:01:55 lr 0.000322 wd 0.0500 time 0.3713 (0.3775) data time 0.0009 (0.0024) model time 0.3704 (0.3762) loss 2.9710 (2.8483) grad_norm 2.7426 (2.2805) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][330/625] eta 0:01:51 lr 0.000322 wd 0.0500 time 0.3728 (0.3773) data time 0.0015 (0.0024) model time 0.3713 (0.3760) loss 2.6679 (2.8496) grad_norm 2.1263 (2.2771) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][340/625] eta 0:01:47 lr 0.000321 wd 0.0500 time 0.3718 (0.3778) data time 0.0011 (0.0023) model time 0.3707 (0.3765) loss 2.8376 (2.8472) grad_norm 2.5214 (2.2786) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][350/625] eta 0:01:43 lr 0.000321 wd 0.0500 time 0.3674 (0.3776) data time 0.0011 (0.0023) model time 0.3663 (0.3763) loss 3.3660 (2.8487) grad_norm 1.5676 (2.2692) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][360/625] eta 0:01:40 lr 0.000321 wd 0.0500 time 0.5580 (0.3779) data time 0.0012 (0.0023) model time 0.5569 (0.3767) loss 3.2591 (2.8495) grad_norm 1.8002 (2.2729) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:36:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][370/625] eta 0:01:36 lr 0.000321 wd 0.0500 time 0.3695 (0.3784) data time 0.0008 (0.0022) model time 0.3687 (0.3772) loss 3.0091 (2.8543) grad_norm 1.9910 (2.2752) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][380/625] eta 0:01:32 lr 0.000321 wd 0.0500 time 0.3746 (0.3783) data time 0.0012 (0.0022) model time 0.3735 (0.3771) loss 2.2346 (2.8561) grad_norm 1.7575 (2.2743) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][390/625] eta 0:01:28 lr 0.000321 wd 0.0500 time 0.3752 (0.3781) data time 0.0009 (0.0022) model time 0.3742 (0.3769) loss 3.5994 (2.8579) grad_norm 1.7720 (2.2668) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][400/625] eta 0:01:25 lr 0.000321 wd 0.0500 time 0.3675 (0.3779) data time 0.0011 (0.0022) model time 0.3664 (0.3766) loss 3.1872 (2.8517) grad_norm 1.9348 (2.2640) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][410/625] eta 0:01:21 lr 0.000321 wd 0.0500 time 0.3777 (0.3778) data time 0.0010 (0.0021) model time 0.3768 (0.3765) loss 2.9933 (2.8530) grad_norm 1.8593 (2.2585) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][420/625] eta 0:01:17 lr 0.000320 wd 0.0500 time 0.3698 (0.3777) data time 0.0013 (0.0021) model time 0.3685 (0.3764) loss 3.0513 (2.8598) grad_norm 2.3446 (2.2520) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][430/625] eta 0:01:13 lr 0.000320 wd 0.0500 time 0.3719 (0.3775) data time 0.0010 (0.0021) model time 0.3709 (0.3763) loss 2.0097 (2.8577) grad_norm 1.8675 (2.2447) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][440/625] eta 0:01:09 lr 0.000320 wd 0.0500 time 0.3722 (0.3775) data time 0.0009 (0.0021) model time 0.3713 (0.3762) loss 3.2571 (2.8543) grad_norm 2.0882 (2.2412) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][450/625] eta 0:01:06 lr 0.000320 wd 0.0500 time 0.3745 (0.3773) data time 0.0009 (0.0020) model time 0.3736 (0.3760) loss 1.8296 (2.8478) grad_norm 1.5902 (2.2297) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][460/625] eta 0:01:02 lr 0.000320 wd 0.0500 time 0.3695 (0.3772) data time 0.0013 (0.0020) model time 0.3682 (0.3759) loss 3.0941 (2.8477) grad_norm 3.6561 (2.2281) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][470/625] eta 0:00:58 lr 0.000320 wd 0.0500 time 0.3671 (0.3771) data time 0.0012 (0.0020) model time 0.3658 (0.3757) loss 2.6389 (2.8493) grad_norm 4.0198 (2.2397) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][480/625] eta 0:00:54 lr 0.000320 wd 0.0500 time 0.3724 (0.3769) data time 0.0009 (0.0020) model time 0.3715 (0.3756) loss 2.4014 (2.8489) grad_norm 2.2149 (2.2448) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][490/625] eta 0:00:50 lr 0.000320 wd 0.0500 time 0.3776 (0.3768) data time 0.0016 (0.0020) model time 0.3761 (0.3755) loss 2.8088 (2.8460) grad_norm 3.0425 (2.2463) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][500/625] eta 0:00:47 lr 0.000319 wd 0.0500 time 0.3627 (0.3767) data time 0.0009 (0.0020) model time 0.3618 (0.3753) loss 2.9694 (2.8468) grad_norm 1.5420 (2.2413) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][510/625] eta 0:00:43 lr 0.000319 wd 0.0500 time 0.3641 (0.3765) data time 0.0011 (0.0020) model time 0.3630 (0.3751) loss 3.1013 (2.8451) grad_norm 2.2481 (2.2340) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][520/625] eta 0:00:39 lr 0.000319 wd 0.0500 time 0.3671 (0.3768) data time 0.0012 (0.0019) model time 0.3659 (0.3755) loss 2.9707 (2.8452) grad_norm 1.9612 (2.2274) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:37:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][530/625] eta 0:00:35 lr 0.000319 wd 0.0500 time 0.3766 (0.3767) data time 0.0011 (0.0019) model time 0.3755 (0.3754) loss 2.8135 (2.8446) grad_norm 4.2007 (2.2299) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][540/625] eta 0:00:32 lr 0.000319 wd 0.0500 time 0.3786 (0.3766) data time 0.0011 (0.0019) model time 0.3776 (0.3753) loss 2.9889 (2.8421) grad_norm 1.5708 (2.2322) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][550/625] eta 0:00:28 lr 0.000319 wd 0.0500 time 0.3786 (0.3766) data time 0.0009 (0.0019) model time 0.3777 (0.3752) loss 2.6676 (2.8421) grad_norm 1.2899 (2.2297) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][560/625] eta 0:00:24 lr 0.000319 wd 0.0500 time 0.3710 (0.3765) data time 0.0009 (0.0019) model time 0.3701 (0.3751) loss 2.1806 (2.8413) grad_norm 1.7764 (2.2261) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][570/625] eta 0:00:20 lr 0.000319 wd 0.0500 time 0.3724 (0.3767) data time 0.0009 (0.0019) model time 0.3715 (0.3754) loss 2.7922 (2.8435) grad_norm 1.9408 (2.2303) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][580/625] eta 0:00:16 lr 0.000318 wd 0.0500 time 0.3692 (0.3770) data time 0.0009 (0.0019) model time 0.3683 (0.3757) loss 2.4352 (2.8432) grad_norm 1.9523 (2.2254) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][590/625] eta 0:00:13 lr 0.000318 wd 0.0500 time 0.3604 (0.3769) data time 0.0013 (0.0019) model time 0.3591 (0.3756) loss 2.9689 (2.8438) grad_norm 2.8880 (2.2361) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][600/625] eta 0:00:09 lr 0.000318 wd 0.0500 time 0.3732 (0.3768) data time 0.0012 (0.0018) model time 0.3720 (0.3755) loss 3.3172 (2.8453) grad_norm 2.7526 (2.2380) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][610/625] eta 0:00:05 lr 0.000318 wd 0.0500 time 0.3670 (0.3767) data time 0.0008 (0.0018) model time 0.3662 (0.3754) loss 2.9981 (2.8467) grad_norm 1.5399 (2.2357) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [228/300][620/625] eta 0:00:01 lr 0.000318 wd 0.0500 time 0.3598 (0.3765) data time 0.0008 (0.0018) model time 0.3590 (0.3751) loss 2.8263 (2.8467) grad_norm 1.8503 (2.2433) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:32 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 228 training takes 0:03:55 [2024-09-01 10:38:32 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:38:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:38:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.488 (0.488) Loss 0.5146 (0.5146) Acc@1 89.111 (89.111) Acc@5 98.486 (98.486) Mem 11225MB [2024-09-01 10:38:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.139) Loss 0.8774 (0.6292) Acc@1 77.734 (86.035) Acc@5 95.898 (97.630) Mem 11225MB [2024-09-01 10:38:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 0.9517 (0.7520) Acc@1 77.881 (83.029) Acc@5 94.873 (96.391) Mem 11225MB [2024-09-01 10:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.694 Acc@5 96.379 [2024-09-01 10:38:36 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.7% [2024-09-01 10:38:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.782 (0.782) Loss 0.4910 (0.4910) Acc@1 89.746 (89.746) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:38:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.173) Loss 0.7925 (0.5945) Acc@1 81.104 (86.794) Acc@5 96.289 (97.860) Mem 11225MB [2024-09-01 10:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.136) Loss 0.8774 (0.7045) Acc@1 78.906 (84.005) Acc@5 95.166 (96.754) Mem 11225MB [2024-09-01 10:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.733 Acc@5 96.785 [2024-09-01 10:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 10:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][0/625] eta 0:12:50 lr 0.000318 wd 0.0500 time 1.2335 (1.2335) data time 0.5855 (0.5855) model time 0.0000 (0.0000) loss 2.6460 (2.6460) grad_norm 2.2024 (2.2024) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][10/625] eta 0:04:38 lr 0.000318 wd 0.0500 time 0.3870 (0.4521) data time 0.0010 (0.0544) model time 0.0000 (0.0000) loss 2.9990 (2.8666) grad_norm 1.6609 (2.1543) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][20/625] eta 0:04:10 lr 0.000318 wd 0.0500 time 0.3672 (0.4133) data time 0.0012 (0.0290) model time 0.0000 (0.0000) loss 3.0918 (2.7218) grad_norm 1.5973 (2.0685) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][30/625] eta 0:03:57 lr 0.000317 wd 0.0500 time 0.3884 (0.3995) data time 0.0009 (0.0200) model time 0.0000 (0.0000) loss 2.5795 (2.8200) grad_norm 1.4971 (1.9680) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][40/625] eta 0:03:49 lr 0.000317 wd 0.0500 time 0.3644 (0.3924) data time 0.0010 (0.0155) model time 0.0000 (0.0000) loss 3.1387 (2.8140) grad_norm 2.0620 (2.0230) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][50/625] eta 0:03:43 lr 0.000317 wd 0.0500 time 0.3752 (0.3890) data time 0.0011 (0.0127) model time 0.0000 (0.0000) loss 3.0682 (2.8316) grad_norm 2.3253 (2.1143) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][60/625] eta 0:03:38 lr 0.000317 wd 0.0500 time 0.3664 (0.3861) data time 0.0011 (0.0108) model time 0.3653 (0.3700) loss 3.0274 (2.8555) grad_norm 1.9637 (2.1365) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][70/625] eta 0:03:32 lr 0.000317 wd 0.0500 time 0.3726 (0.3836) data time 0.0010 (0.0094) model time 0.3716 (0.3686) loss 3.2753 (2.8960) grad_norm 1.6934 (2.1012) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][80/625] eta 0:03:28 lr 0.000317 wd 0.0500 time 0.3705 (0.3819) data time 0.0008 (0.0084) model time 0.3696 (0.3688) loss 3.1273 (2.8877) grad_norm 2.4677 (2.1149) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][90/625] eta 0:03:23 lr 0.000317 wd 0.0500 time 0.3623 (0.3807) data time 0.0010 (0.0076) model time 0.3614 (0.3691) loss 3.6779 (2.9169) grad_norm 1.5887 (2.1403) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][100/625] eta 0:03:19 lr 0.000317 wd 0.0500 time 0.3637 (0.3797) data time 0.0012 (0.0069) model time 0.3625 (0.3691) loss 3.2816 (2.9158) grad_norm 2.2625 (2.1222) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][110/625] eta 0:03:15 lr 0.000316 wd 0.0500 time 0.3712 (0.3790) data time 0.0011 (0.0064) model time 0.3701 (0.3693) loss 3.0060 (2.9008) grad_norm 1.8396 (2.1222) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][120/625] eta 0:03:11 lr 0.000316 wd 0.0500 time 0.3723 (0.3784) data time 0.0009 (0.0060) model time 0.3714 (0.3696) loss 2.7637 (2.9085) grad_norm 1.9971 (2.1063) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][130/625] eta 0:03:07 lr 0.000316 wd 0.0500 time 0.3674 (0.3778) data time 0.0009 (0.0056) model time 0.3665 (0.3696) loss 3.2568 (2.9049) grad_norm 1.6479 (2.0976) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][140/625] eta 0:03:03 lr 0.000316 wd 0.0500 time 0.3704 (0.3774) data time 0.0009 (0.0053) model time 0.3695 (0.3697) loss 1.7857 (2.8938) grad_norm 2.7950 (2.0918) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][150/625] eta 0:02:59 lr 0.000316 wd 0.0500 time 0.3759 (0.3784) data time 0.0009 (0.0050) model time 0.3749 (0.3719) loss 3.1052 (2.8888) grad_norm 2.7274 (2.1108) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][160/625] eta 0:02:56 lr 0.000316 wd 0.0500 time 0.3760 (0.3796) data time 0.0012 (0.0048) model time 0.3748 (0.3740) loss 3.1483 (2.8934) grad_norm 1.6193 (2.1257) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][170/625] eta 0:02:52 lr 0.000316 wd 0.0500 time 0.3741 (0.3793) data time 0.0012 (0.0046) model time 0.3729 (0.3740) loss 2.2359 (2.8800) grad_norm 2.0199 (2.1529) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][180/625] eta 0:02:48 lr 0.000316 wd 0.0500 time 0.3710 (0.3790) data time 0.0009 (0.0044) model time 0.3701 (0.3738) loss 3.1233 (2.8796) grad_norm 2.0050 (2.1477) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][190/625] eta 0:02:44 lr 0.000315 wd 0.0500 time 0.3693 (0.3787) data time 0.0009 (0.0042) model time 0.3684 (0.3737) loss 3.3463 (2.8838) grad_norm 2.7632 (2.1356) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][200/625] eta 0:02:41 lr 0.000315 wd 0.0500 time 0.3744 (0.3793) data time 0.0009 (0.0041) model time 0.3736 (0.3749) loss 2.9921 (2.8784) grad_norm 2.9496 (2.1291) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:39:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][210/625] eta 0:02:37 lr 0.000315 wd 0.0500 time 0.3715 (0.3790) data time 0.0012 (0.0039) model time 0.3703 (0.3747) loss 3.0852 (2.8894) grad_norm 2.1874 (2.1326) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][220/625] eta 0:02:33 lr 0.000315 wd 0.0500 time 0.3665 (0.3786) data time 0.0009 (0.0038) model time 0.3656 (0.3744) loss 2.1585 (2.8900) grad_norm 2.1283 (2.1387) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][230/625] eta 0:02:29 lr 0.000315 wd 0.0500 time 0.3727 (0.3784) data time 0.0011 (0.0037) model time 0.3716 (0.3741) loss 2.4632 (2.8880) grad_norm 2.3815 (2.1493) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][240/625] eta 0:02:25 lr 0.000315 wd 0.0500 time 0.3727 (0.3782) data time 0.0011 (0.0036) model time 0.3716 (0.3740) loss 2.9791 (2.8751) grad_norm 2.2715 (2.1611) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][250/625] eta 0:02:21 lr 0.000315 wd 0.0500 time 0.3736 (0.3780) data time 0.0012 (0.0036) model time 0.3724 (0.3739) loss 2.2909 (2.8687) grad_norm 1.7290 (2.1543) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][260/625] eta 0:02:17 lr 0.000315 wd 0.0500 time 0.3734 (0.3777) data time 0.0009 (0.0035) model time 0.3725 (0.3738) loss 1.8278 (2.8676) grad_norm 2.5733 (2.1529) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][270/625] eta 0:02:14 lr 0.000314 wd 0.0500 time 0.3737 (0.3776) data time 0.0012 (0.0034) model time 0.3725 (0.3737) loss 3.2819 (2.8612) grad_norm 2.2351 (2.1475) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][280/625] eta 0:02:10 lr 0.000314 wd 0.0500 time 0.3795 (0.3774) data time 0.0012 (0.0033) model time 0.3783 (0.3736) loss 2.9619 (2.8540) grad_norm 1.3876 (2.1371) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][290/625] eta 0:02:06 lr 0.000314 wd 0.0500 time 0.3705 (0.3772) data time 0.0011 (0.0032) model time 0.3694 (0.3734) loss 1.9687 (2.8458) grad_norm 5.8390 (2.1524) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][300/625] eta 0:02:02 lr 0.000314 wd 0.0500 time 0.3790 (0.3772) data time 0.0009 (0.0032) model time 0.3780 (0.3735) loss 3.1726 (2.8264) grad_norm 2.4448 (2.2413) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][310/625] eta 0:01:58 lr 0.000314 wd 0.0500 time 0.3704 (0.3770) data time 0.0009 (0.0031) model time 0.3696 (0.3734) loss 2.8952 (2.8225) grad_norm 1.5218 (2.2545) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][320/625] eta 0:01:54 lr 0.000314 wd 0.0500 time 0.3818 (0.3768) data time 0.0011 (0.0030) model time 0.3807 (0.3733) loss 1.8500 (2.8176) grad_norm 1.3495 (2.2488) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][330/625] eta 0:01:51 lr 0.000314 wd 0.0500 time 0.3699 (0.3766) data time 0.0011 (0.0030) model time 0.3688 (0.3732) loss 3.1620 (2.8206) grad_norm 2.2009 (2.2475) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][340/625] eta 0:01:47 lr 0.000314 wd 0.0500 time 0.3731 (0.3776) data time 0.0009 (0.0029) model time 0.3722 (0.3744) loss 2.1519 (2.8204) grad_norm 2.2535 (2.2376) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][350/625] eta 0:01:43 lr 0.000313 wd 0.0500 time 0.3784 (0.3775) data time 0.0015 (0.0029) model time 0.3769 (0.3743) loss 2.7931 (2.8202) grad_norm 1.7819 (2.2274) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][360/625] eta 0:01:39 lr 0.000313 wd 0.0500 time 0.3713 (0.3773) data time 0.0012 (0.0028) model time 0.3701 (0.3741) loss 2.9557 (2.8156) grad_norm 3.0480 (2.2353) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:40:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][370/625] eta 0:01:36 lr 0.000313 wd 0.0500 time 0.3732 (0.3771) data time 0.0012 (0.0028) model time 0.3720 (0.3740) loss 1.7507 (2.8080) grad_norm 2.2651 (2.2444) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:41:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][380/625] eta 0:01:32 lr 0.000313 wd 0.0500 time 0.3719 (0.3774) data time 0.0011 (0.0028) model time 0.3708 (0.3744) loss 3.4788 (2.8071) grad_norm 1.9514 (2.2409) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:41:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][390/625] eta 0:01:28 lr 0.000313 wd 0.0500 time 0.3712 (0.3773) data time 0.0012 (0.0027) model time 0.3700 (0.3743) loss 3.3177 (2.8142) grad_norm 3.2146 (2.2509) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][400/625] eta 0:01:24 lr 0.000313 wd 0.0500 time 0.3727 (0.3771) data time 0.0011 (0.0027) model time 0.3715 (0.3742) loss 2.7318 (2.8168) grad_norm 2.1383 (2.2603) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:41:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][410/625] eta 0:01:21 lr 0.000313 wd 0.0500 time 0.3686 (0.3770) data time 0.0012 (0.0026) model time 0.3674 (0.3741) loss 2.4104 (2.8152) grad_norm 1.5900 (2.2502) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:41:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][420/625] eta 0:01:17 lr 0.000313 wd 0.0500 time 0.3737 (0.3769) data time 0.0009 (0.0026) model time 0.3728 (0.3740) loss 1.8563 (2.8106) grad_norm 2.0390 (2.2478) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:41:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][430/625] eta 0:01:13 lr 0.000312 wd 0.0500 time 0.3670 (0.3768) data time 0.0008 (0.0026) model time 0.3661 (0.3739) loss 2.1170 (2.8093) grad_norm 1.7509 (2.2423) loss_scale 1024.0000 (1024.0000) mem 11225MB [2024-09-01 10:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][440/625] eta 0:01:09 lr 0.000312 wd 0.0500 time 0.3731 (0.3767) data time 0.0011 (0.0026) model time 0.3720 (0.3738) loss 2.4937 (2.8110) grad_norm 1.4762 (inf) loss_scale 512.0000 (1017.0340) mem 11225MB [2024-09-01 10:41:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][450/625] eta 0:01:05 lr 0.000312 wd 0.0500 time 0.5606 (0.3770) data time 0.0008 (0.0025) model time 0.5598 (0.3743) loss 2.3406 (2.8121) grad_norm 1.4261 (inf) loss_scale 512.0000 (1005.8359) mem 11225MB [2024-09-01 10:41:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][460/625] eta 0:01:02 lr 0.000312 wd 0.0500 time 0.3905 (0.3775) data time 0.0009 (0.0025) model time 0.3896 (0.3748) loss 3.4088 (2.8126) grad_norm 2.5491 (inf) loss_scale 512.0000 (995.1236) mem 11225MB [2024-09-01 10:41:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][470/625] eta 0:00:58 lr 0.000312 wd 0.0500 time 0.3711 (0.3774) data time 0.0011 (0.0025) model time 0.3700 (0.3748) loss 2.6565 (2.8109) grad_norm 3.3568 (inf) loss_scale 512.0000 (984.8662) mem 11225MB [2024-09-01 10:41:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][480/625] eta 0:00:54 lr 0.000312 wd 0.0500 time 0.3735 (0.3773) data time 0.0009 (0.0024) model time 0.3726 (0.3747) loss 3.4476 (2.8144) grad_norm 2.0193 (inf) loss_scale 512.0000 (975.0353) mem 11225MB [2024-09-01 10:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][490/625] eta 0:00:50 lr 0.000312 wd 0.0500 time 0.3752 (0.3773) data time 0.0011 (0.0024) model time 0.3741 (0.3747) loss 2.9347 (2.8118) grad_norm 2.6661 (inf) loss_scale 512.0000 (965.6049) mem 11225MB [2024-09-01 10:41:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][500/625] eta 0:00:47 lr 0.000312 wd 0.0500 time 0.3713 (0.3776) data time 0.0009 (0.0024) model time 0.3705 (0.3750) loss 2.4185 (2.8103) grad_norm 2.3171 (inf) loss_scale 512.0000 (956.5509) mem 11225MB [2024-09-01 10:41:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][510/625] eta 0:00:43 lr 0.000311 wd 0.0500 time 0.3724 (0.3775) data time 0.0011 (0.0024) model time 0.3713 (0.3750) loss 3.1665 (2.8135) grad_norm 3.0858 (inf) loss_scale 512.0000 (947.8513) mem 11225MB [2024-09-01 10:41:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][520/625] eta 0:00:39 lr 0.000311 wd 0.0500 time 0.3786 (0.3774) data time 0.0009 (0.0023) model time 0.3777 (0.3749) loss 3.0091 (2.8154) grad_norm 2.3176 (inf) loss_scale 512.0000 (939.4856) mem 11225MB [2024-09-01 10:41:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][530/625] eta 0:00:35 lr 0.000311 wd 0.0500 time 0.3770 (0.3773) data time 0.0014 (0.0023) model time 0.3756 (0.3748) loss 2.6064 (2.8117) grad_norm 1.6622 (inf) loss_scale 512.0000 (931.4350) mem 11225MB [2024-09-01 10:42:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][540/625] eta 0:00:32 lr 0.000311 wd 0.0500 time 0.3714 (0.3772) data time 0.0011 (0.0023) model time 0.3703 (0.3747) loss 1.7533 (2.8128) grad_norm 2.2994 (inf) loss_scale 512.0000 (923.6821) mem 11225MB [2024-09-01 10:42:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][550/625] eta 0:00:28 lr 0.000311 wd 0.0500 time 0.3814 (0.3770) data time 0.0009 (0.0023) model time 0.3805 (0.3746) loss 3.4036 (2.8135) grad_norm 1.7240 (inf) loss_scale 512.0000 (916.2105) mem 11225MB [2024-09-01 10:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][560/625] eta 0:00:24 lr 0.000311 wd 0.0500 time 0.3740 (0.3769) data time 0.0010 (0.0023) model time 0.3730 (0.3745) loss 2.2597 (2.8130) grad_norm 2.5535 (inf) loss_scale 512.0000 (909.0053) mem 11225MB [2024-09-01 10:42:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][570/625] eta 0:00:20 lr 0.000311 wd 0.0500 time 0.3722 (0.3769) data time 0.0061 (0.0023) model time 0.3661 (0.3744) loss 2.2385 (2.8115) grad_norm 2.4523 (inf) loss_scale 512.0000 (902.0525) mem 11225MB [2024-09-01 10:42:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][580/625] eta 0:00:16 lr 0.000311 wd 0.0500 time 0.3733 (0.3767) data time 0.0012 (0.0022) model time 0.3721 (0.3743) loss 3.4125 (2.8105) grad_norm 1.6235 (inf) loss_scale 512.0000 (895.3391) mem 11225MB [2024-09-01 10:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][590/625] eta 0:00:13 lr 0.000310 wd 0.0500 time 0.3715 (0.3766) data time 0.0011 (0.0022) model time 0.3704 (0.3742) loss 3.2649 (2.8098) grad_norm 1.7602 (inf) loss_scale 512.0000 (888.8528) mem 11225MB [2024-09-01 10:42:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][600/625] eta 0:00:09 lr 0.000310 wd 0.0500 time 0.3701 (0.3766) data time 0.0011 (0.0022) model time 0.3690 (0.3742) loss 2.5712 (2.8109) grad_norm 2.3197 (inf) loss_scale 512.0000 (882.5824) mem 11225MB [2024-09-01 10:42:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][610/625] eta 0:00:05 lr 0.000310 wd 0.0500 time 0.3572 (0.3765) data time 0.0006 (0.0022) model time 0.3566 (0.3742) loss 3.1199 (2.8111) grad_norm 2.8752 (inf) loss_scale 512.0000 (876.5172) mem 11225MB [2024-09-01 10:42:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [229/300][620/625] eta 0:00:01 lr 0.000310 wd 0.0500 time 0.3524 (0.3763) data time 0.0006 (0.0022) model time 0.3518 (0.3739) loss 2.9901 (2.8094) grad_norm 1.8446 (inf) loss_scale 512.0000 (870.6473) mem 11225MB [2024-09-01 10:42:34 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 229 training takes 0:03:55 [2024-09-01 10:42:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:42:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:42:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.522 (0.522) Loss 0.5254 (0.5254) Acc@1 88.574 (88.574) Acc@5 98.242 (98.242) Mem 11225MB [2024-09-01 10:42:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 0.8335 (0.6263) Acc@1 80.908 (86.213) Acc@5 95.703 (97.603) Mem 11225MB [2024-09-01 10:42:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9492 (0.7467) Acc@1 77.686 (83.208) Acc@5 94.971 (96.410) Mem 11225MB [2024-09-01 10:42:38 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.921 Acc@5 96.443 [2024-09-01 10:42:38 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 10:42:38 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.92% [2024-09-01 10:42:38 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 10:42:39 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 10:42:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 0.4912 (0.4912) Acc@1 89.795 (89.795) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:42:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.140) Loss 0.7944 (0.5942) Acc@1 80.859 (86.843) Acc@5 96.289 (97.865) Mem 11225MB [2024-09-01 10:42:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 0.8755 (0.7041) Acc@1 79.053 (84.063) Acc@5 95.215 (96.761) Mem 11225MB [2024-09-01 10:42:42 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.769 Acc@5 96.789 [2024-09-01 10:42:42 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 10:42:42 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.77% [2024-09-01 10:42:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:42:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:42:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][0/625] eta 0:08:15 lr 0.000310 wd 0.0500 time 0.7929 (0.7929) data time 0.4445 (0.4445) model time 0.0000 (0.0000) loss 2.5251 (2.5251) grad_norm 2.6575 (2.6575) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:42:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][10/625] eta 0:04:12 lr 0.000310 wd 0.0500 time 0.3698 (0.4105) data time 0.0012 (0.0415) model time 0.0000 (0.0000) loss 2.5655 (2.8025) grad_norm 2.1579 (2.3377) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:42:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][20/625] eta 0:04:01 lr 0.000310 wd 0.0500 time 0.5003 (0.3988) data time 0.0009 (0.0223) model time 0.0000 (0.0000) loss 2.3288 (2.7668) grad_norm 1.9646 (2.5018) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:42:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][30/625] eta 0:03:56 lr 0.000310 wd 0.0500 time 0.3760 (0.3969) data time 0.0009 (0.0155) model time 0.0000 (0.0000) loss 2.7575 (2.7991) grad_norm 2.2980 (2.4416) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][40/625] eta 0:03:48 lr 0.000309 wd 0.0500 time 0.3715 (0.3906) data time 0.0013 (0.0120) model time 0.0000 (0.0000) loss 3.0034 (2.7183) grad_norm 1.9149 (2.5046) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][50/625] eta 0:03:42 lr 0.000309 wd 0.0500 time 0.3947 (0.3875) data time 0.0013 (0.0099) model time 0.0000 (0.0000) loss 3.1324 (2.7560) grad_norm 2.2887 (2.4898) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][60/625] eta 0:03:37 lr 0.000309 wd 0.0500 time 0.3780 (0.3849) data time 0.0012 (0.0085) model time 0.3769 (0.3703) loss 1.7771 (2.7404) grad_norm 1.5041 (2.4607) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][70/625] eta 0:03:32 lr 0.000309 wd 0.0500 time 0.3770 (0.3830) data time 0.0010 (0.0075) model time 0.3759 (0.3704) loss 3.2679 (2.7911) grad_norm 2.1530 (2.4331) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][80/625] eta 0:03:28 lr 0.000309 wd 0.0500 time 0.3762 (0.3817) data time 0.0011 (0.0067) model time 0.3752 (0.3707) loss 3.4804 (2.7872) grad_norm 1.6621 (2.3904) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][90/625] eta 0:03:23 lr 0.000309 wd 0.0500 time 0.3732 (0.3805) data time 0.0008 (0.0061) model time 0.3723 (0.3703) loss 3.0272 (2.8102) grad_norm 1.7396 (2.3576) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][100/625] eta 0:03:19 lr 0.000309 wd 0.0500 time 0.3704 (0.3795) data time 0.0011 (0.0056) model time 0.3693 (0.3701) loss 3.0688 (2.8096) grad_norm 2.2113 (2.3158) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][110/625] eta 0:03:15 lr 0.000309 wd 0.0500 time 0.3671 (0.3789) data time 0.0010 (0.0052) model time 0.3660 (0.3703) loss 3.7120 (2.7910) grad_norm 1.6213 (2.2840) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][120/625] eta 0:03:10 lr 0.000308 wd 0.0500 time 0.3656 (0.3780) data time 0.0008 (0.0049) model time 0.3648 (0.3698) loss 3.4193 (2.8040) grad_norm 1.4200 (2.2612) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][130/625] eta 0:03:06 lr 0.000308 wd 0.0500 time 0.3704 (0.3774) data time 0.0009 (0.0046) model time 0.3695 (0.3697) loss 3.0850 (2.8180) grad_norm 1.7391 (2.2666) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][140/625] eta 0:03:02 lr 0.000308 wd 0.0500 time 0.3673 (0.3770) data time 0.0013 (0.0044) model time 0.3660 (0.3699) loss 2.9786 (2.8205) grad_norm 2.1471 (2.2701) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][150/625] eta 0:02:58 lr 0.000308 wd 0.0500 time 0.3690 (0.3766) data time 0.0009 (0.0041) model time 0.3681 (0.3698) loss 3.1867 (2.8273) grad_norm 2.7064 (2.2801) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][160/625] eta 0:02:54 lr 0.000308 wd 0.0500 time 0.3672 (0.3763) data time 0.0008 (0.0040) model time 0.3663 (0.3699) loss 3.1911 (2.8281) grad_norm 2.5401 (2.2830) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][170/625] eta 0:02:51 lr 0.000308 wd 0.0500 time 0.3726 (0.3760) data time 0.0012 (0.0038) model time 0.3715 (0.3699) loss 2.1814 (2.8283) grad_norm 2.7655 (2.3099) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][180/625] eta 0:02:47 lr 0.000308 wd 0.0500 time 0.3756 (0.3758) data time 0.0011 (0.0037) model time 0.3745 (0.3700) loss 2.8705 (2.8288) grad_norm 2.3680 (2.3482) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][190/625] eta 0:02:43 lr 0.000308 wd 0.0500 time 0.3718 (0.3757) data time 0.0009 (0.0035) model time 0.3709 (0.3701) loss 2.6519 (2.8252) grad_norm 1.8405 (2.3363) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:43:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][200/625] eta 0:02:39 lr 0.000307 wd 0.0500 time 0.3735 (0.3755) data time 0.0009 (0.0034) model time 0.3726 (0.3703) loss 3.3253 (2.8289) grad_norm 1.4673 (2.3258) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][210/625] eta 0:02:36 lr 0.000307 wd 0.0500 time 0.3772 (0.3764) data time 0.0008 (0.0033) model time 0.3763 (0.3716) loss 3.0487 (2.8299) grad_norm 5.7801 (2.3381) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][220/625] eta 0:02:32 lr 0.000307 wd 0.0500 time 0.3734 (0.3761) data time 0.0012 (0.0032) model time 0.3723 (0.3715) loss 2.9732 (2.8270) grad_norm 10.7475 (2.3650) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][230/625] eta 0:02:28 lr 0.000307 wd 0.0500 time 0.5922 (0.3769) data time 0.0009 (0.0031) model time 0.5913 (0.3727) loss 2.7960 (2.8268) grad_norm 1.8103 (2.3507) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][240/625] eta 0:02:25 lr 0.000307 wd 0.0500 time 0.3678 (0.3767) data time 0.0009 (0.0031) model time 0.3669 (0.3726) loss 2.3464 (2.8249) grad_norm 4.6227 (2.3486) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][250/625] eta 0:02:21 lr 0.000307 wd 0.0500 time 0.3733 (0.3765) data time 0.0009 (0.0030) model time 0.3724 (0.3724) loss 3.4836 (2.8339) grad_norm 2.1275 (2.3413) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][260/625] eta 0:02:17 lr 0.000307 wd 0.0500 time 0.3734 (0.3762) data time 0.0012 (0.0029) model time 0.3722 (0.3722) loss 2.5380 (2.8329) grad_norm 3.4826 (2.3351) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][270/625] eta 0:02:13 lr 0.000307 wd 0.0500 time 0.3657 (0.3760) data time 0.0011 (0.0029) model time 0.3646 (0.3721) loss 3.1215 (2.8434) grad_norm 2.4122 (2.3468) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][280/625] eta 0:02:09 lr 0.000306 wd 0.0500 time 0.3680 (0.3758) data time 0.0011 (0.0028) model time 0.3669 (0.3720) loss 2.8327 (2.8300) grad_norm 2.1084 (2.3358) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][290/625] eta 0:02:05 lr 0.000306 wd 0.0500 time 0.3707 (0.3757) data time 0.0008 (0.0027) model time 0.3699 (0.3720) loss 2.6590 (2.8288) grad_norm 2.7915 (2.3285) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][300/625] eta 0:02:02 lr 0.000306 wd 0.0500 time 0.3713 (0.3755) data time 0.0008 (0.0027) model time 0.3705 (0.3719) loss 1.8906 (2.8175) grad_norm 2.2429 (2.3256) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][310/625] eta 0:01:58 lr 0.000306 wd 0.0500 time 0.3713 (0.3754) data time 0.0009 (0.0026) model time 0.3705 (0.3719) loss 3.1202 (2.8163) grad_norm 2.8863 (2.3194) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][320/625] eta 0:01:54 lr 0.000306 wd 0.0500 time 0.3720 (0.3753) data time 0.0009 (0.0026) model time 0.3711 (0.3718) loss 1.9047 (2.8083) grad_norm 2.8536 (2.3120) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][330/625] eta 0:01:50 lr 0.000306 wd 0.0500 time 0.3703 (0.3752) data time 0.0009 (0.0026) model time 0.3694 (0.3718) loss 2.7895 (2.8134) grad_norm 3.3807 (2.3212) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][340/625] eta 0:01:46 lr 0.000306 wd 0.0500 time 0.3728 (0.3751) data time 0.0011 (0.0025) model time 0.3717 (0.3718) loss 2.5328 (2.8141) grad_norm 1.9902 (2.3295) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][350/625] eta 0:01:43 lr 0.000306 wd 0.0500 time 0.3724 (0.3751) data time 0.0009 (0.0025) model time 0.3715 (0.3718) loss 3.2959 (2.8209) grad_norm 1.9902 (2.3352) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:44:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][360/625] eta 0:01:39 lr 0.000305 wd 0.0500 time 0.3671 (0.3749) data time 0.0011 (0.0024) model time 0.3660 (0.3717) loss 2.7362 (2.8272) grad_norm 1.8642 (2.3403) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][370/625] eta 0:01:35 lr 0.000305 wd 0.0500 time 0.3723 (0.3748) data time 0.0014 (0.0024) model time 0.3710 (0.3716) loss 3.1985 (2.8153) grad_norm 2.8051 (2.3460) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][380/625] eta 0:01:31 lr 0.000305 wd 0.0500 time 0.3739 (0.3748) data time 0.0010 (0.0024) model time 0.3729 (0.3716) loss 3.3698 (2.8122) grad_norm 2.8304 (2.3378) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][390/625] eta 0:01:28 lr 0.000305 wd 0.0500 time 0.6079 (0.3753) data time 0.0009 (0.0023) model time 0.6070 (0.3723) loss 2.9668 (2.8123) grad_norm 1.9597 (2.3398) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][400/625] eta 0:01:24 lr 0.000305 wd 0.0500 time 0.3765 (0.3752) data time 0.0009 (0.0023) model time 0.3756 (0.3722) loss 2.2430 (2.8067) grad_norm 1.7068 (2.3397) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][410/625] eta 0:01:20 lr 0.000305 wd 0.0500 time 0.3695 (0.3751) data time 0.0012 (0.0023) model time 0.3684 (0.3721) loss 3.3875 (2.8074) grad_norm 1.9612 (2.3414) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][420/625] eta 0:01:17 lr 0.000305 wd 0.0500 time 0.3696 (0.3760) data time 0.0010 (0.0023) model time 0.3687 (0.3732) loss 3.3594 (2.8116) grad_norm 2.9209 (2.3507) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][430/625] eta 0:01:13 lr 0.000305 wd 0.0500 time 0.3661 (0.3758) data time 0.0013 (0.0022) model time 0.3648 (0.3731) loss 3.1894 (2.8106) grad_norm 1.8304 (2.3452) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][440/625] eta 0:01:09 lr 0.000304 wd 0.0500 time 0.3667 (0.3757) data time 0.0011 (0.0022) model time 0.3655 (0.3729) loss 3.2058 (2.8085) grad_norm 2.0252 (2.3383) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][450/625] eta 0:01:05 lr 0.000304 wd 0.0500 time 0.3732 (0.3756) data time 0.0008 (0.0022) model time 0.3724 (0.3729) loss 3.3887 (2.8042) grad_norm 1.5876 (2.3363) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][460/625] eta 0:01:01 lr 0.000304 wd 0.0500 time 0.3745 (0.3756) data time 0.0009 (0.0022) model time 0.3737 (0.3729) loss 3.3218 (2.8040) grad_norm 1.9058 (2.3394) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][470/625] eta 0:00:58 lr 0.000304 wd 0.0500 time 0.3704 (0.3755) data time 0.0012 (0.0022) model time 0.3692 (0.3729) loss 2.8759 (2.8082) grad_norm 1.9769 (2.3271) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][480/625] eta 0:00:54 lr 0.000304 wd 0.0500 time 0.3678 (0.3755) data time 0.0009 (0.0021) model time 0.3669 (0.3728) loss 3.2665 (2.8108) grad_norm 2.0687 (2.3199) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][490/625] eta 0:00:50 lr 0.000304 wd 0.0500 time 0.3651 (0.3754) data time 0.0011 (0.0021) model time 0.3639 (0.3728) loss 3.2219 (2.8113) grad_norm 1.4415 (2.3286) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][500/625] eta 0:00:46 lr 0.000304 wd 0.0500 time 0.3658 (0.3753) data time 0.0011 (0.0021) model time 0.3647 (0.3727) loss 2.8694 (2.8148) grad_norm 1.9209 (2.3250) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][510/625] eta 0:00:43 lr 0.000304 wd 0.0500 time 0.3340 (0.3756) data time 0.0009 (0.0021) model time 0.3331 (0.3731) loss 1.9554 (2.8119) grad_norm 1.9299 (2.3249) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:45:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][520/625] eta 0:00:39 lr 0.000303 wd 0.0500 time 0.3701 (0.3755) data time 0.0011 (0.0021) model time 0.3690 (0.3730) loss 3.3309 (2.8130) grad_norm 2.1571 (2.3215) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][530/625] eta 0:00:35 lr 0.000303 wd 0.0500 time 0.3635 (0.3754) data time 0.0011 (0.0021) model time 0.3624 (0.3729) loss 1.9817 (2.8135) grad_norm 2.0331 (2.3220) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][540/625] eta 0:00:31 lr 0.000303 wd 0.0500 time 0.3640 (0.3761) data time 0.0011 (0.0020) model time 0.3629 (0.3737) loss 3.7997 (2.8163) grad_norm 2.0242 (2.3219) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][550/625] eta 0:00:28 lr 0.000303 wd 0.0500 time 0.3710 (0.3760) data time 0.0009 (0.0020) model time 0.3701 (0.3736) loss 2.9847 (2.8194) grad_norm 1.6693 (2.3155) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][560/625] eta 0:00:24 lr 0.000303 wd 0.0500 time 0.3815 (0.3759) data time 0.0012 (0.0020) model time 0.3804 (0.3736) loss 3.2002 (2.8163) grad_norm 1.5579 (2.3216) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][570/625] eta 0:00:20 lr 0.000303 wd 0.0500 time 0.3671 (0.3758) data time 0.0014 (0.0020) model time 0.3657 (0.3735) loss 2.0727 (2.8117) grad_norm 1.9518 (2.3126) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][580/625] eta 0:00:16 lr 0.000303 wd 0.0500 time 0.3670 (0.3758) data time 0.0009 (0.0020) model time 0.3660 (0.3735) loss 3.2657 (2.8140) grad_norm 3.4606 (2.3196) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][590/625] eta 0:00:13 lr 0.000303 wd 0.0500 time 0.3672 (0.3757) data time 0.0009 (0.0020) model time 0.3663 (0.3734) loss 2.7466 (2.8155) grad_norm 2.7402 (2.3200) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][600/625] eta 0:00:09 lr 0.000302 wd 0.0500 time 0.3708 (0.3756) data time 0.0011 (0.0019) model time 0.3697 (0.3733) loss 2.9179 (2.8193) grad_norm 1.8010 (2.3193) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][610/625] eta 0:00:05 lr 0.000302 wd 0.0500 time 0.3631 (0.3756) data time 0.0008 (0.0019) model time 0.3623 (0.3733) loss 2.8560 (2.8188) grad_norm 2.2652 (2.3174) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [230/300][620/625] eta 0:00:01 lr 0.000302 wd 0.0500 time 0.3574 (0.3753) data time 0.0006 (0.0019) model time 0.3568 (0.3730) loss 3.6184 (2.8175) grad_norm 1.5992 (2.3100) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:38 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 230 training takes 0:03:54 [2024-09-01 10:46:38 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:46:39 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:46:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.512 (0.512) Loss 0.5176 (0.5176) Acc@1 88.574 (88.574) Acc@5 98.389 (98.389) Mem 11225MB [2024-09-01 10:46:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.142) Loss 0.8638 (0.6321) Acc@1 79.297 (86.022) Acc@5 95.605 (97.576) Mem 11225MB [2024-09-01 10:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9468 (0.7472) Acc@1 77.002 (83.073) Acc@5 95.068 (96.408) Mem 11225MB [2024-09-01 10:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.835 Acc@5 96.443 [2024-09-01 10:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.8% [2024-09-01 10:46:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.868 (0.868) Loss 0.4907 (0.4907) Acc@1 89.746 (89.746) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:46:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.177) Loss 0.7959 (0.5942) Acc@1 80.957 (86.910) Acc@5 96.240 (97.856) Mem 11225MB [2024-09-01 10:46:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8755 (0.7039) Acc@1 79.150 (84.080) Acc@5 95.264 (96.756) Mem 11225MB [2024-09-01 10:46:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.795 Acc@5 96.787 [2024-09-01 10:46:45 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 10:46:45 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.80% [2024-09-01 10:46:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:46:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:46:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][0/625] eta 0:07:38 lr 0.000302 wd 0.0500 time 0.7337 (0.7337) data time 0.3993 (0.3993) model time 0.0000 (0.0000) loss 3.1162 (3.1162) grad_norm 1.8703 (1.8703) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][10/625] eta 0:04:08 lr 0.000302 wd 0.0500 time 0.3681 (0.4044) data time 0.0009 (0.0374) model time 0.0000 (0.0000) loss 3.3602 (3.0870) grad_norm 2.5011 (2.7239) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][20/625] eta 0:03:54 lr 0.000302 wd 0.0500 time 0.3679 (0.3869) data time 0.0008 (0.0201) model time 0.0000 (0.0000) loss 2.6858 (2.8843) grad_norm 2.3254 (2.5415) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:46:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][30/625] eta 0:03:47 lr 0.000302 wd 0.0500 time 0.3742 (0.3824) data time 0.0009 (0.0140) model time 0.0000 (0.0000) loss 1.9995 (2.8238) grad_norm 1.6197 (2.4978) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][40/625] eta 0:03:44 lr 0.000302 wd 0.0500 time 0.3703 (0.3832) data time 0.0012 (0.0109) model time 0.0000 (0.0000) loss 2.9516 (2.8187) grad_norm 2.4468 (2.3969) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][50/625] eta 0:03:39 lr 0.000302 wd 0.0500 time 0.3751 (0.3811) data time 0.0009 (0.0090) model time 0.0000 (0.0000) loss 3.6091 (2.8131) grad_norm 1.7411 (2.4057) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][60/625] eta 0:03:34 lr 0.000301 wd 0.0500 time 0.3796 (0.3800) data time 0.0012 (0.0078) model time 0.3783 (0.3725) loss 2.7847 (2.8089) grad_norm 2.3424 (2.4256) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][70/625] eta 0:03:33 lr 0.000301 wd 0.0500 time 0.3710 (0.3840) data time 0.0014 (0.0069) model time 0.3697 (0.3899) loss 1.7803 (2.8072) grad_norm 2.4094 (2.4256) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][80/625] eta 0:03:28 lr 0.000301 wd 0.0500 time 0.3740 (0.3826) data time 0.0010 (0.0061) model time 0.3730 (0.3839) loss 3.4348 (2.8096) grad_norm 1.6089 (2.3748) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][90/625] eta 0:03:24 lr 0.000301 wd 0.0500 time 0.3729 (0.3814) data time 0.0009 (0.0056) model time 0.3720 (0.3805) loss 2.9098 (2.8178) grad_norm 1.8413 (2.3739) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][100/625] eta 0:03:19 lr 0.000301 wd 0.0500 time 0.3760 (0.3805) data time 0.0011 (0.0052) model time 0.3749 (0.3787) loss 2.4663 (2.8224) grad_norm 1.8688 (2.3406) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][110/625] eta 0:03:15 lr 0.000301 wd 0.0500 time 0.3688 (0.3797) data time 0.0009 (0.0048) model time 0.3679 (0.3773) loss 1.8234 (2.8345) grad_norm 2.7173 (2.3044) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][120/625] eta 0:03:11 lr 0.000301 wd 0.0500 time 0.3792 (0.3794) data time 0.0009 (0.0046) model time 0.3784 (0.3767) loss 2.0406 (2.8176) grad_norm 1.9002 (2.3675) loss_scale 512.0000 (512.0000) mem 11225MB [2024-09-01 10:47:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][130/625] eta 0:03:07 lr 0.000301 wd 0.0500 time 0.3557 (0.3790) data time 0.0008 (0.0043) model time 0.3548 (0.3762) loss 1.9608 (2.8018) grad_norm inf (inf) loss_scale 256.0000 (510.0458) mem 11225MB [2024-09-01 10:47:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][140/625] eta 0:03:03 lr 0.000300 wd 0.0500 time 0.3738 (0.3784) data time 0.0010 (0.0041) model time 0.3728 (0.3755) loss 3.6517 (2.8124) grad_norm 4.0330 (inf) loss_scale 256.0000 (492.0284) mem 11225MB [2024-09-01 10:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][150/625] eta 0:02:59 lr 0.000300 wd 0.0500 time 0.3656 (0.3779) data time 0.0012 (0.0039) model time 0.3644 (0.3750) loss 2.9659 (2.8201) grad_norm 2.0774 (inf) loss_scale 256.0000 (476.3974) mem 11225MB [2024-09-01 10:47:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][160/625] eta 0:02:55 lr 0.000300 wd 0.0500 time 0.3668 (0.3776) data time 0.0009 (0.0037) model time 0.3660 (0.3747) loss 2.4828 (2.8136) grad_norm 2.0416 (inf) loss_scale 256.0000 (462.7081) mem 11225MB [2024-09-01 10:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][170/625] eta 0:02:51 lr 0.000300 wd 0.0500 time 0.3699 (0.3772) data time 0.0012 (0.0036) model time 0.3687 (0.3742) loss 2.3324 (2.8207) grad_norm 2.9420 (inf) loss_scale 256.0000 (450.6199) mem 11225MB [2024-09-01 10:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][180/625] eta 0:02:47 lr 0.000300 wd 0.0500 time 0.3686 (0.3769) data time 0.0014 (0.0035) model time 0.3673 (0.3740) loss 3.3813 (2.8275) grad_norm 2.7725 (inf) loss_scale 256.0000 (439.8674) mem 11225MB [2024-09-01 10:47:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][190/625] eta 0:02:43 lr 0.000300 wd 0.0500 time 0.3845 (0.3767) data time 0.0009 (0.0033) model time 0.3836 (0.3738) loss 3.2668 (2.8345) grad_norm 3.1471 (inf) loss_scale 256.0000 (430.2408) mem 11225MB [2024-09-01 10:48:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][200/625] eta 0:02:40 lr 0.000300 wd 0.0500 time 0.3719 (0.3765) data time 0.0009 (0.0032) model time 0.3710 (0.3737) loss 3.2440 (2.8293) grad_norm 4.0475 (inf) loss_scale 256.0000 (421.5721) mem 11225MB [2024-09-01 10:48:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][210/625] eta 0:02:36 lr 0.000300 wd 0.0500 time 0.3705 (0.3762) data time 0.0012 (0.0031) model time 0.3694 (0.3733) loss 3.1106 (2.8318) grad_norm 1.7185 (inf) loss_scale 256.0000 (413.7251) mem 11225MB [2024-09-01 10:48:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][220/625] eta 0:02:32 lr 0.000299 wd 0.0500 time 0.3326 (0.3765) data time 0.0012 (0.0031) model time 0.3314 (0.3739) loss 2.8894 (2.8358) grad_norm 2.3092 (inf) loss_scale 256.0000 (406.5882) mem 11225MB [2024-09-01 10:48:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][230/625] eta 0:02:28 lr 0.000299 wd 0.0500 time 0.3678 (0.3764) data time 0.0009 (0.0030) model time 0.3669 (0.3738) loss 3.3575 (2.8443) grad_norm 2.8444 (inf) loss_scale 256.0000 (400.0693) mem 11225MB [2024-09-01 10:48:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][240/625] eta 0:02:24 lr 0.000299 wd 0.0500 time 0.3714 (0.3763) data time 0.0010 (0.0029) model time 0.3704 (0.3737) loss 3.7764 (2.8455) grad_norm 1.7579 (inf) loss_scale 256.0000 (394.0913) mem 11225MB [2024-09-01 10:48:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][250/625] eta 0:02:21 lr 0.000299 wd 0.0500 time 0.3812 (0.3761) data time 0.0013 (0.0028) model time 0.3798 (0.3736) loss 2.6807 (2.8422) grad_norm 1.8084 (inf) loss_scale 256.0000 (388.5896) mem 11225MB [2024-09-01 10:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][260/625] eta 0:02:17 lr 0.000299 wd 0.0500 time 0.3679 (0.3759) data time 0.0009 (0.0028) model time 0.3670 (0.3734) loss 1.7134 (2.8350) grad_norm 2.0744 (inf) loss_scale 256.0000 (383.5096) mem 11225MB [2024-09-01 10:48:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][270/625] eta 0:02:13 lr 0.000299 wd 0.0500 time 0.3681 (0.3757) data time 0.0011 (0.0027) model time 0.3670 (0.3732) loss 2.6430 (2.8321) grad_norm 2.1016 (inf) loss_scale 256.0000 (378.8044) mem 11225MB [2024-09-01 10:48:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][280/625] eta 0:02:09 lr 0.000299 wd 0.0500 time 0.3754 (0.3765) data time 0.0012 (0.0027) model time 0.3742 (0.3743) loss 3.2639 (2.8196) grad_norm 4.1136 (inf) loss_scale 256.0000 (374.4342) mem 11225MB [2024-09-01 10:48:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][290/625] eta 0:02:06 lr 0.000299 wd 0.0500 time 0.3586 (0.3763) data time 0.0010 (0.0026) model time 0.3575 (0.3741) loss 3.0946 (2.8201) grad_norm 2.3940 (inf) loss_scale 256.0000 (370.3643) mem 11225MB [2024-09-01 10:48:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][300/625] eta 0:02:02 lr 0.000298 wd 0.0500 time 0.3657 (0.3761) data time 0.0009 (0.0026) model time 0.3648 (0.3738) loss 2.7001 (2.8165) grad_norm 4.3348 (inf) loss_scale 256.0000 (366.5648) mem 11225MB [2024-09-01 10:48:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][310/625] eta 0:01:58 lr 0.000298 wd 0.0500 time 0.3732 (0.3759) data time 0.0011 (0.0025) model time 0.3721 (0.3736) loss 3.1815 (2.8129) grad_norm 2.3533 (inf) loss_scale 256.0000 (363.0096) mem 11225MB [2024-09-01 10:48:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][320/625] eta 0:01:54 lr 0.000298 wd 0.0500 time 0.3755 (0.3758) data time 0.0009 (0.0025) model time 0.3746 (0.3735) loss 1.9887 (2.8114) grad_norm 1.7865 (inf) loss_scale 256.0000 (359.6760) mem 11225MB [2024-09-01 10:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][330/625] eta 0:01:50 lr 0.000298 wd 0.0500 time 0.3696 (0.3757) data time 0.0010 (0.0024) model time 0.3686 (0.3734) loss 2.2781 (2.8099) grad_norm 2.5166 (inf) loss_scale 256.0000 (356.5438) mem 11225MB [2024-09-01 10:48:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][340/625] eta 0:01:47 lr 0.000298 wd 0.0500 time 0.3450 (0.3762) data time 0.0012 (0.0024) model time 0.3438 (0.3741) loss 2.6373 (2.8035) grad_norm 3.9927 (inf) loss_scale 256.0000 (353.5953) mem 11225MB [2024-09-01 10:48:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][350/625] eta 0:01:43 lr 0.000298 wd 0.0500 time 0.3693 (0.3761) data time 0.0011 (0.0024) model time 0.3681 (0.3739) loss 2.9787 (2.8080) grad_norm 2.7818 (inf) loss_scale 256.0000 (350.8148) mem 11225MB [2024-09-01 10:49:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][360/625] eta 0:01:39 lr 0.000298 wd 0.0500 time 0.3722 (0.3760) data time 0.0008 (0.0023) model time 0.3713 (0.3739) loss 1.8022 (2.8053) grad_norm 2.3566 (inf) loss_scale 256.0000 (348.1884) mem 11225MB [2024-09-01 10:49:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][370/625] eta 0:01:35 lr 0.000298 wd 0.0500 time 0.3745 (0.3758) data time 0.0008 (0.0023) model time 0.3736 (0.3738) loss 3.4434 (2.8090) grad_norm 8.7876 (inf) loss_scale 256.0000 (345.7035) mem 11225MB [2024-09-01 10:49:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][380/625] eta 0:01:32 lr 0.000297 wd 0.0500 time 0.3684 (0.3757) data time 0.0011 (0.0023) model time 0.3673 (0.3736) loss 2.9025 (2.8066) grad_norm 1.3655 (inf) loss_scale 256.0000 (343.3491) mem 11225MB [2024-09-01 10:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][390/625] eta 0:01:28 lr 0.000297 wd 0.0500 time 0.3754 (0.3757) data time 0.0011 (0.0023) model time 0.3744 (0.3736) loss 3.0261 (2.8103) grad_norm 1.5930 (inf) loss_scale 256.0000 (341.1151) mem 11225MB [2024-09-01 10:49:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][400/625] eta 0:01:24 lr 0.000297 wd 0.0500 time 0.3680 (0.3756) data time 0.0010 (0.0022) model time 0.3670 (0.3735) loss 2.0676 (2.8106) grad_norm 1.6633 (inf) loss_scale 256.0000 (338.9925) mem 11225MB [2024-09-01 10:49:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][410/625] eta 0:01:20 lr 0.000297 wd 0.0500 time 0.3723 (0.3755) data time 0.0011 (0.0022) model time 0.3711 (0.3735) loss 1.5862 (2.8052) grad_norm 2.4830 (inf) loss_scale 256.0000 (336.9732) mem 11225MB [2024-09-01 10:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][420/625] eta 0:01:16 lr 0.000297 wd 0.0500 time 0.3678 (0.3755) data time 0.0012 (0.0022) model time 0.3666 (0.3734) loss 3.1765 (2.8069) grad_norm 2.0207 (inf) loss_scale 256.0000 (335.0499) mem 11225MB [2024-09-01 10:49:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][430/625] eta 0:01:13 lr 0.000297 wd 0.0500 time 0.3791 (0.3754) data time 0.0008 (0.0022) model time 0.3782 (0.3734) loss 2.0455 (2.8099) grad_norm 1.8637 (inf) loss_scale 256.0000 (333.2158) mem 11225MB [2024-09-01 10:49:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][440/625] eta 0:01:09 lr 0.000297 wd 0.0500 time 0.3949 (0.3753) data time 0.0012 (0.0021) model time 0.3938 (0.3733) loss 2.6181 (2.8097) grad_norm 1.6139 (inf) loss_scale 256.0000 (331.4649) mem 11225MB [2024-09-01 10:49:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][450/625] eta 0:01:05 lr 0.000297 wd 0.0500 time 0.3689 (0.3752) data time 0.0011 (0.0021) model time 0.3678 (0.3732) loss 3.1017 (2.8055) grad_norm 1.8703 (inf) loss_scale 256.0000 (329.7916) mem 11225MB [2024-09-01 10:49:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][460/625] eta 0:01:02 lr 0.000296 wd 0.0500 time 0.5879 (0.3761) data time 0.0011 (0.0021) model time 0.5867 (0.3742) loss 2.1941 (2.8049) grad_norm 2.0695 (inf) loss_scale 256.0000 (328.1909) mem 11225MB [2024-09-01 10:49:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][470/625] eta 0:00:58 lr 0.000296 wd 0.0500 time 0.3664 (0.3760) data time 0.0009 (0.0021) model time 0.3655 (0.3741) loss 3.1581 (2.8070) grad_norm 2.3813 (inf) loss_scale 256.0000 (326.6582) mem 11225MB [2024-09-01 10:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][480/625] eta 0:00:54 lr 0.000296 wd 0.0500 time 0.3601 (0.3759) data time 0.0009 (0.0021) model time 0.3592 (0.3741) loss 2.3177 (2.8061) grad_norm 1.7530 (inf) loss_scale 256.0000 (325.1892) mem 11225MB [2024-09-01 10:49:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][490/625] eta 0:00:50 lr 0.000296 wd 0.0500 time 0.3709 (0.3758) data time 0.0008 (0.0020) model time 0.3700 (0.3739) loss 1.7328 (2.8092) grad_norm 2.1701 (inf) loss_scale 256.0000 (323.7800) mem 11225MB [2024-09-01 10:49:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][500/625] eta 0:00:46 lr 0.000296 wd 0.0500 time 0.3727 (0.3757) data time 0.0011 (0.0020) model time 0.3716 (0.3739) loss 2.5708 (2.8130) grad_norm 2.8900 (inf) loss_scale 256.0000 (322.4271) mem 11225MB [2024-09-01 10:49:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][510/625] eta 0:00:43 lr 0.000296 wd 0.0500 time 0.3655 (0.3756) data time 0.0012 (0.0020) model time 0.3643 (0.3738) loss 2.7522 (2.8132) grad_norm 1.9692 (inf) loss_scale 256.0000 (321.1272) mem 11225MB [2024-09-01 10:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][520/625] eta 0:00:39 lr 0.000296 wd 0.0500 time 0.3715 (0.3756) data time 0.0012 (0.0020) model time 0.3703 (0.3737) loss 3.0757 (2.8113) grad_norm 1.7606 (inf) loss_scale 256.0000 (319.8772) mem 11225MB [2024-09-01 10:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][530/625] eta 0:00:35 lr 0.000296 wd 0.0500 time 0.3701 (0.3755) data time 0.0008 (0.0020) model time 0.3692 (0.3737) loss 2.8332 (2.8071) grad_norm 5.5899 (inf) loss_scale 256.0000 (318.6742) mem 11225MB [2024-09-01 10:50:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][540/625] eta 0:00:31 lr 0.000295 wd 0.0500 time 0.3775 (0.3754) data time 0.0012 (0.0020) model time 0.3763 (0.3736) loss 3.4454 (2.8087) grad_norm 2.3582 (inf) loss_scale 256.0000 (317.5157) mem 11225MB [2024-09-01 10:50:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][550/625] eta 0:00:28 lr 0.000295 wd 0.0500 time 0.3809 (0.3758) data time 0.0009 (0.0020) model time 0.3800 (0.3740) loss 2.9672 (2.8131) grad_norm 1.8006 (inf) loss_scale 256.0000 (316.3993) mem 11225MB [2024-09-01 10:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][560/625] eta 0:00:24 lr 0.000295 wd 0.0500 time 0.3846 (0.3758) data time 0.0013 (0.0019) model time 0.3833 (0.3740) loss 2.1125 (2.8161) grad_norm 1.9275 (inf) loss_scale 256.0000 (315.3226) mem 11225MB [2024-09-01 10:50:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][570/625] eta 0:00:20 lr 0.000295 wd 0.0500 time 0.3747 (0.3757) data time 0.0009 (0.0019) model time 0.3739 (0.3740) loss 3.4461 (2.8143) grad_norm 1.7519 (inf) loss_scale 256.0000 (314.2837) mem 11225MB [2024-09-01 10:50:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][580/625] eta 0:00:16 lr 0.000295 wd 0.0500 time 0.3701 (0.3763) data time 0.0012 (0.0019) model time 0.3689 (0.3746) loss 3.1214 (2.8156) grad_norm 2.1297 (inf) loss_scale 256.0000 (313.2806) mem 11225MB [2024-09-01 10:50:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][590/625] eta 0:00:13 lr 0.000295 wd 0.0500 time 0.3707 (0.3763) data time 0.0011 (0.0019) model time 0.3696 (0.3746) loss 3.0276 (2.8106) grad_norm 4.0352 (inf) loss_scale 256.0000 (312.3113) mem 11225MB [2024-09-01 10:50:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][600/625] eta 0:00:09 lr 0.000295 wd 0.0500 time 0.3668 (0.3762) data time 0.0011 (0.0019) model time 0.3657 (0.3745) loss 3.3792 (2.8109) grad_norm 2.3791 (inf) loss_scale 256.0000 (311.3744) mem 11225MB [2024-09-01 10:50:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][610/625] eta 0:00:05 lr 0.000295 wd 0.0500 time 0.3600 (0.3761) data time 0.0008 (0.0019) model time 0.3592 (0.3744) loss 2.3845 (2.8117) grad_norm 2.0750 (inf) loss_scale 256.0000 (310.4681) mem 11225MB [2024-09-01 10:50:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [231/300][620/625] eta 0:00:01 lr 0.000294 wd 0.0500 time 0.3552 (0.3758) data time 0.0008 (0.0019) model time 0.3543 (0.3741) loss 2.8160 (2.8109) grad_norm 2.1408 (inf) loss_scale 256.0000 (309.5910) mem 11225MB [2024-09-01 10:50:41 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 231 training takes 0:03:54 [2024-09-01 10:50:41 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:50:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 0.5322 (0.5322) Acc@1 88.867 (88.867) Acc@5 98.291 (98.291) Mem 11225MB [2024-09-01 10:50:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 0.8315 (0.6355) Acc@1 80.469 (86.284) Acc@5 95.605 (97.576) Mem 11225MB [2024-09-01 10:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.096 (0.120) Loss 0.9390 (0.7528) Acc@1 77.930 (83.157) Acc@5 94.873 (96.410) Mem 11225MB [2024-09-01 10:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.835 Acc@5 96.421 [2024-09-01 10:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.8% [2024-09-01 10:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.780 (0.780) Loss 0.4907 (0.4907) Acc@1 89.795 (89.795) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.170) Loss 0.7944 (0.5936) Acc@1 80.908 (86.941) Acc@5 96.289 (97.860) Mem 11225MB [2024-09-01 10:50:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.134) Loss 0.8770 (0.7033) Acc@1 78.809 (84.084) Acc@5 95.459 (96.768) Mem 11225MB [2024-09-01 10:50:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.799 Acc@5 96.801 [2024-09-01 10:50:49 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 10:50:49 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.80% [2024-09-01 10:50:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:50:50 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:50:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][0/625] eta 0:07:55 lr 0.000294 wd 0.0500 time 0.7615 (0.7615) data time 0.4295 (0.4295) model time 0.0000 (0.0000) loss 2.7429 (2.7429) grad_norm 2.2727 (2.2727) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:50:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][10/625] eta 0:04:12 lr 0.000294 wd 0.0500 time 0.3764 (0.4109) data time 0.0011 (0.0401) model time 0.0000 (0.0000) loss 2.8722 (3.0273) grad_norm 1.9855 (2.0658) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:50:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][20/625] eta 0:03:57 lr 0.000294 wd 0.0500 time 0.3674 (0.3924) data time 0.0013 (0.0216) model time 0.0000 (0.0000) loss 2.7038 (2.8888) grad_norm 2.3632 (2.2694) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][30/625] eta 0:03:49 lr 0.000294 wd 0.0500 time 0.3655 (0.3859) data time 0.0011 (0.0151) model time 0.0000 (0.0000) loss 3.2429 (2.8654) grad_norm 2.1339 (2.2532) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][40/625] eta 0:03:43 lr 0.000294 wd 0.0500 time 0.3730 (0.3820) data time 0.0009 (0.0117) model time 0.0000 (0.0000) loss 1.8186 (2.8528) grad_norm 2.0976 (2.1566) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][50/625] eta 0:03:38 lr 0.000294 wd 0.0500 time 0.3655 (0.3795) data time 0.0008 (0.0096) model time 0.0000 (0.0000) loss 2.7981 (2.8533) grad_norm 1.5845 (2.1249) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][60/625] eta 0:03:33 lr 0.000294 wd 0.0500 time 0.3691 (0.3780) data time 0.0009 (0.0082) model time 0.3682 (0.3688) loss 2.2758 (2.8361) grad_norm 2.5226 (2.0916) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][70/625] eta 0:03:30 lr 0.000294 wd 0.0500 time 0.3741 (0.3800) data time 0.0012 (0.0072) model time 0.3728 (0.3800) loss 3.0111 (2.8165) grad_norm 1.7618 (2.0591) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][80/625] eta 0:03:26 lr 0.000293 wd 0.0500 time 0.3674 (0.3791) data time 0.0013 (0.0065) model time 0.3661 (0.3772) loss 3.2541 (2.7974) grad_norm 1.7979 (2.0700) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][90/625] eta 0:03:22 lr 0.000293 wd 0.0500 time 0.3745 (0.3783) data time 0.0012 (0.0059) model time 0.3733 (0.3755) loss 2.8105 (2.7866) grad_norm 2.6348 (2.1130) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][100/625] eta 0:03:20 lr 0.000293 wd 0.0500 time 0.3728 (0.3814) data time 0.0009 (0.0055) model time 0.3719 (0.3821) loss 2.8795 (2.8066) grad_norm 1.9158 (2.1290) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][110/625] eta 0:03:15 lr 0.000293 wd 0.0500 time 0.3681 (0.3803) data time 0.0013 (0.0051) model time 0.3668 (0.3797) loss 3.0505 (2.7849) grad_norm 1.9720 (2.2005) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][120/625] eta 0:03:11 lr 0.000293 wd 0.0500 time 0.3721 (0.3795) data time 0.0012 (0.0048) model time 0.3710 (0.3782) loss 2.8106 (2.7996) grad_norm 2.4397 (2.1927) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][130/625] eta 0:03:07 lr 0.000293 wd 0.0500 time 0.3653 (0.3788) data time 0.0011 (0.0045) model time 0.3641 (0.3771) loss 2.7376 (2.7910) grad_norm 1.6283 (2.1820) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][140/625] eta 0:03:03 lr 0.000293 wd 0.0500 time 0.3726 (0.3782) data time 0.0009 (0.0043) model time 0.3717 (0.3762) loss 2.0480 (2.8011) grad_norm 2.2005 (2.1842) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][150/625] eta 0:02:59 lr 0.000293 wd 0.0500 time 0.3736 (0.3777) data time 0.0012 (0.0041) model time 0.3723 (0.3756) loss 2.0085 (2.7950) grad_norm 2.6644 (2.2135) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][160/625] eta 0:02:55 lr 0.000292 wd 0.0500 time 0.3609 (0.3773) data time 0.0011 (0.0039) model time 0.3598 (0.3751) loss 3.2548 (2.7942) grad_norm 2.0372 (2.2070) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][170/625] eta 0:02:51 lr 0.000292 wd 0.0500 time 0.3666 (0.3769) data time 0.0012 (0.0037) model time 0.3655 (0.3746) loss 1.8651 (2.8029) grad_norm 2.0995 (2.2129) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:51:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][180/625] eta 0:02:47 lr 0.000292 wd 0.0500 time 0.3722 (0.3767) data time 0.0012 (0.0036) model time 0.3710 (0.3744) loss 2.8621 (2.8000) grad_norm 2.0224 (2.2030) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][190/625] eta 0:02:44 lr 0.000292 wd 0.0500 time 0.3715 (0.3775) data time 0.0015 (0.0035) model time 0.3700 (0.3755) loss 2.9213 (2.7837) grad_norm 1.8838 (2.1885) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][200/625] eta 0:02:40 lr 0.000292 wd 0.0500 time 0.3728 (0.3772) data time 0.0009 (0.0033) model time 0.3719 (0.3752) loss 2.9196 (2.7918) grad_norm 5.3855 (2.1999) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][210/625] eta 0:02:36 lr 0.000292 wd 0.0500 time 0.3733 (0.3770) data time 0.0009 (0.0032) model time 0.3724 (0.3749) loss 2.2054 (2.7952) grad_norm 2.0459 (2.2162) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][220/625] eta 0:02:32 lr 0.000292 wd 0.0500 time 0.3677 (0.3766) data time 0.0011 (0.0032) model time 0.3666 (0.3745) loss 2.9588 (2.7932) grad_norm 2.4032 (2.3069) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][230/625] eta 0:02:28 lr 0.000292 wd 0.0500 time 0.3757 (0.3764) data time 0.0009 (0.0031) model time 0.3748 (0.3743) loss 3.2698 (2.8012) grad_norm 1.9559 (2.2979) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][240/625] eta 0:02:24 lr 0.000291 wd 0.0500 time 0.3777 (0.3762) data time 0.0008 (0.0030) model time 0.3768 (0.3741) loss 1.6267 (2.8016) grad_norm 1.7558 (2.2895) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][250/625] eta 0:02:20 lr 0.000291 wd 0.0500 time 0.3668 (0.3759) data time 0.0011 (0.0029) model time 0.3657 (0.3738) loss 2.5316 (2.7995) grad_norm 2.7968 (2.3108) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][260/625] eta 0:02:17 lr 0.000291 wd 0.0500 time 0.3747 (0.3759) data time 0.0011 (0.0028) model time 0.3736 (0.3739) loss 3.2910 (2.7954) grad_norm 2.7797 (2.3402) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][270/625] eta 0:02:13 lr 0.000291 wd 0.0500 time 0.3659 (0.3757) data time 0.0008 (0.0028) model time 0.3651 (0.3737) loss 2.0634 (2.7954) grad_norm 1.6937 (2.3421) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][280/625] eta 0:02:09 lr 0.000291 wd 0.0500 time 0.3736 (0.3757) data time 0.0012 (0.0027) model time 0.3724 (0.3736) loss 2.7421 (2.7863) grad_norm 2.7717 (2.3435) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][290/625] eta 0:02:05 lr 0.000291 wd 0.0500 time 0.3558 (0.3755) data time 0.0012 (0.0027) model time 0.3546 (0.3735) loss 3.1276 (2.7893) grad_norm 1.5716 (2.3473) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][300/625] eta 0:02:02 lr 0.000291 wd 0.0500 time 0.3718 (0.3754) data time 0.0009 (0.0026) model time 0.3709 (0.3734) loss 3.6642 (2.7916) grad_norm 2.0435 (2.3568) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][310/625] eta 0:01:58 lr 0.000291 wd 0.0500 time 0.3707 (0.3760) data time 0.0011 (0.0026) model time 0.3696 (0.3741) loss 3.2242 (2.7968) grad_norm 3.2072 (2.3503) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][320/625] eta 0:01:54 lr 0.000291 wd 0.0500 time 0.3716 (0.3758) data time 0.0010 (0.0025) model time 0.3706 (0.3740) loss 2.4146 (2.7922) grad_norm 1.9159 (2.3382) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][330/625] eta 0:01:50 lr 0.000290 wd 0.0500 time 0.3738 (0.3757) data time 0.0011 (0.0025) model time 0.3727 (0.3738) loss 2.9618 (2.7910) grad_norm 1.8348 (2.3587) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:52:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][340/625] eta 0:01:46 lr 0.000290 wd 0.0500 time 0.3574 (0.3754) data time 0.0010 (0.0025) model time 0.3564 (0.3735) loss 3.0103 (2.7888) grad_norm 2.4126 (2.4002) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][350/625] eta 0:01:43 lr 0.000290 wd 0.0500 time 0.3713 (0.3753) data time 0.0011 (0.0024) model time 0.3702 (0.3733) loss 3.0056 (2.7881) grad_norm 2.6046 (2.3931) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][360/625] eta 0:01:39 lr 0.000290 wd 0.0500 time 0.3709 (0.3750) data time 0.0011 (0.0024) model time 0.3699 (0.3731) loss 2.4202 (2.7849) grad_norm 2.4999 (2.3935) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][370/625] eta 0:01:35 lr 0.000290 wd 0.0500 time 0.3752 (0.3749) data time 0.0013 (0.0024) model time 0.3739 (0.3730) loss 3.1814 (2.7855) grad_norm 1.7586 (2.3836) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][380/625] eta 0:01:31 lr 0.000290 wd 0.0500 time 0.3665 (0.3748) data time 0.0009 (0.0023) model time 0.3656 (0.3729) loss 2.6911 (2.7844) grad_norm 8.2384 (2.3928) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][390/625] eta 0:01:28 lr 0.000290 wd 0.0500 time 0.3792 (0.3748) data time 0.0011 (0.0023) model time 0.3781 (0.3729) loss 3.2033 (2.7839) grad_norm 2.0702 (2.4093) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][400/625] eta 0:01:24 lr 0.000290 wd 0.0500 time 0.3693 (0.3752) data time 0.0011 (0.0023) model time 0.3682 (0.3734) loss 2.9100 (2.7828) grad_norm 1.8371 (2.3992) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][410/625] eta 0:01:20 lr 0.000289 wd 0.0500 time 0.3684 (0.3752) data time 0.0012 (0.0022) model time 0.3672 (0.3734) loss 2.7085 (2.7850) grad_norm 2.4256 (2.3948) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][420/625] eta 0:01:16 lr 0.000289 wd 0.0500 time 0.3679 (0.3750) data time 0.0009 (0.0022) model time 0.3670 (0.3732) loss 2.8923 (2.7886) grad_norm 1.7907 (2.3952) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][430/625] eta 0:01:13 lr 0.000289 wd 0.0500 time 0.3899 (0.3750) data time 0.0009 (0.0022) model time 0.3890 (0.3732) loss 3.2856 (2.7960) grad_norm 1.6531 (2.3968) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][440/625] eta 0:01:09 lr 0.000289 wd 0.0500 time 0.3815 (0.3750) data time 0.0011 (0.0022) model time 0.3804 (0.3732) loss 3.0307 (2.7953) grad_norm 2.2018 (2.3925) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][450/625] eta 0:01:05 lr 0.000289 wd 0.0500 time 0.3684 (0.3749) data time 0.0009 (0.0021) model time 0.3676 (0.3731) loss 3.0825 (2.7902) grad_norm 1.7529 (2.3849) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][460/625] eta 0:01:01 lr 0.000289 wd 0.0500 time 0.3689 (0.3747) data time 0.0010 (0.0021) model time 0.3679 (0.3730) loss 2.7273 (2.7892) grad_norm 1.8417 (2.3811) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][470/625] eta 0:00:58 lr 0.000289 wd 0.0500 time 0.3870 (0.3747) data time 0.0009 (0.0021) model time 0.3862 (0.3729) loss 3.3664 (2.7957) grad_norm 5.0677 (2.3755) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][480/625] eta 0:00:54 lr 0.000289 wd 0.0500 time 0.3708 (0.3747) data time 0.0011 (0.0021) model time 0.3697 (0.3729) loss 2.4292 (2.7995) grad_norm 1.6493 (2.3703) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][490/625] eta 0:00:50 lr 0.000288 wd 0.0500 time 0.3693 (0.3754) data time 0.0009 (0.0021) model time 0.3684 (0.3738) loss 2.7479 (2.8047) grad_norm 2.1119 (2.3716) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:53:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][500/625] eta 0:00:46 lr 0.000288 wd 0.0500 time 0.3686 (0.3753) data time 0.0011 (0.0020) model time 0.3675 (0.3737) loss 2.7070 (2.8110) grad_norm 1.9396 (2.3648) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][510/625] eta 0:00:43 lr 0.000288 wd 0.0500 time 0.3709 (0.3753) data time 0.0012 (0.0020) model time 0.3697 (0.3736) loss 2.6601 (2.8057) grad_norm 1.4478 (2.3533) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][520/625] eta 0:00:39 lr 0.000288 wd 0.0500 time 0.3755 (0.3752) data time 0.0012 (0.0020) model time 0.3744 (0.3735) loss 2.7669 (2.7982) grad_norm 1.7744 (2.3462) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][530/625] eta 0:00:35 lr 0.000288 wd 0.0500 time 0.3655 (0.3751) data time 0.0011 (0.0020) model time 0.3644 (0.3734) loss 3.0441 (2.7990) grad_norm 2.7540 (2.3469) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][540/625] eta 0:00:31 lr 0.000288 wd 0.0500 time 0.3686 (0.3750) data time 0.0012 (0.0020) model time 0.3673 (0.3733) loss 1.8982 (2.7940) grad_norm 2.2956 (2.3521) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][550/625] eta 0:00:28 lr 0.000288 wd 0.0500 time 0.3705 (0.3749) data time 0.0012 (0.0020) model time 0.3693 (0.3732) loss 2.6905 (2.7962) grad_norm 2.2706 (2.3568) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][560/625] eta 0:00:24 lr 0.000288 wd 0.0500 time 0.3679 (0.3748) data time 0.0013 (0.0020) model time 0.3667 (0.3731) loss 3.5150 (2.7989) grad_norm 2.7231 (2.3613) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][570/625] eta 0:00:20 lr 0.000287 wd 0.0500 time 0.3688 (0.3747) data time 0.0009 (0.0019) model time 0.3679 (0.3730) loss 2.6883 (2.8013) grad_norm 3.8221 (2.3644) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][580/625] eta 0:00:16 lr 0.000287 wd 0.0500 time 0.3767 (0.3750) data time 0.0011 (0.0019) model time 0.3756 (0.3733) loss 2.6577 (2.8029) grad_norm 2.0171 (2.3721) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][590/625] eta 0:00:13 lr 0.000287 wd 0.0500 time 0.3637 (0.3749) data time 0.0010 (0.0019) model time 0.3627 (0.3733) loss 2.7744 (2.8016) grad_norm 1.6171 (2.3692) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][600/625] eta 0:00:09 lr 0.000287 wd 0.0500 time 0.3700 (0.3749) data time 0.0014 (0.0019) model time 0.3686 (0.3732) loss 2.9669 (2.8055) grad_norm 1.8003 (2.3601) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][610/625] eta 0:00:05 lr 0.000287 wd 0.0500 time 0.3681 (0.3755) data time 0.0006 (0.0019) model time 0.3675 (0.3739) loss 2.9325 (2.8058) grad_norm 2.0074 (2.3598) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [232/300][620/625] eta 0:00:01 lr 0.000287 wd 0.0500 time 0.3655 (0.3753) data time 0.0008 (0.0019) model time 0.3647 (0.3737) loss 2.0235 (2.8015) grad_norm 2.0730 (2.3661) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:44 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 232 training takes 0:03:54 [2024-09-01 10:54:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:54:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:54:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.517 (0.517) Loss 0.5273 (0.5273) Acc@1 89.307 (89.307) Acc@5 98.633 (98.633) Mem 11225MB [2024-09-01 10:54:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.144) Loss 0.8691 (0.6441) Acc@1 79.199 (86.102) Acc@5 95.654 (97.510) Mem 11225MB [2024-09-01 10:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 0.9795 (0.7643) Acc@1 77.197 (83.096) Acc@5 94.141 (96.282) Mem 11225MB [2024-09-01 10:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.857 Acc@5 96.309 [2024-09-01 10:54:48 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 10:54:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.864 (0.864) Loss 0.4910 (0.4910) Acc@1 89.844 (89.844) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.176) Loss 0.7944 (0.5934) Acc@1 81.006 (86.963) Acc@5 96.191 (97.812) Mem 11225MB [2024-09-01 10:54:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8779 (0.7031) Acc@1 78.857 (84.129) Acc@5 95.459 (96.733) Mem 11225MB [2024-09-01 10:54:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.839 Acc@5 96.777 [2024-09-01 10:54:51 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 10:54:51 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.84% [2024-09-01 10:54:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 10:54:53 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 10:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][0/625] eta 0:07:54 lr 0.000287 wd 0.0500 time 0.7595 (0.7595) data time 0.4218 (0.4218) model time 0.0000 (0.0000) loss 2.6611 (2.6611) grad_norm 2.8253 (2.8253) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][10/625] eta 0:04:10 lr 0.000287 wd 0.0500 time 0.3716 (0.4070) data time 0.0011 (0.0394) model time 0.0000 (0.0000) loss 1.6795 (2.6871) grad_norm 1.7618 (2.3347) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][20/625] eta 0:03:55 lr 0.000287 wd 0.0500 time 0.3880 (0.3896) data time 0.0011 (0.0212) model time 0.0000 (0.0000) loss 2.6202 (2.8377) grad_norm 1.8292 (2.4044) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][30/625] eta 0:03:47 lr 0.000286 wd 0.0500 time 0.3733 (0.3831) data time 0.0011 (0.0148) model time 0.0000 (0.0000) loss 3.4443 (2.8035) grad_norm 3.6436 (2.5880) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][40/625] eta 0:03:45 lr 0.000286 wd 0.0500 time 0.3711 (0.3857) data time 0.0008 (0.0115) model time 0.0000 (0.0000) loss 2.3404 (2.7403) grad_norm 2.4284 (2.5805) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][50/625] eta 0:03:40 lr 0.000286 wd 0.0500 time 0.3713 (0.3832) data time 0.0009 (0.0095) model time 0.0000 (0.0000) loss 3.3404 (2.8173) grad_norm 3.5466 (2.5549) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][60/625] eta 0:03:35 lr 0.000286 wd 0.0500 time 0.3707 (0.3809) data time 0.0011 (0.0081) model time 0.3696 (0.3680) loss 3.4799 (2.8142) grad_norm 2.7926 (2.5001) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][70/625] eta 0:03:30 lr 0.000286 wd 0.0500 time 0.3609 (0.3797) data time 0.0011 (0.0071) model time 0.3598 (0.3697) loss 3.3221 (2.8045) grad_norm 2.2470 (2.4540) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][80/625] eta 0:03:26 lr 0.000286 wd 0.0500 time 0.3673 (0.3787) data time 0.0009 (0.0064) model time 0.3665 (0.3700) loss 2.9779 (2.7982) grad_norm 1.9759 (2.4336) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][90/625] eta 0:03:22 lr 0.000286 wd 0.0500 time 0.3688 (0.3779) data time 0.0011 (0.0058) model time 0.3678 (0.3700) loss 3.4094 (2.8064) grad_norm 2.5129 (2.4279) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][100/625] eta 0:03:18 lr 0.000286 wd 0.0500 time 0.3743 (0.3772) data time 0.0009 (0.0054) model time 0.3734 (0.3699) loss 3.1541 (2.8190) grad_norm 2.0774 (2.5998) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][110/625] eta 0:03:13 lr 0.000285 wd 0.0500 time 0.3742 (0.3766) data time 0.0010 (0.0050) model time 0.3732 (0.3699) loss 3.0733 (2.8018) grad_norm 2.3262 (2.5617) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][120/625] eta 0:03:09 lr 0.000285 wd 0.0500 time 0.3729 (0.3761) data time 0.0009 (0.0047) model time 0.3720 (0.3698) loss 2.5331 (2.7843) grad_norm 1.9184 (2.5200) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][130/625] eta 0:03:05 lr 0.000285 wd 0.0500 time 0.3752 (0.3757) data time 0.0011 (0.0044) model time 0.3741 (0.3698) loss 3.0707 (2.7743) grad_norm 2.8609 (2.4847) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][140/625] eta 0:03:02 lr 0.000285 wd 0.0500 time 0.3655 (0.3753) data time 0.0009 (0.0042) model time 0.3646 (0.3697) loss 2.4527 (2.7688) grad_norm 2.5109 (2.4537) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][150/625] eta 0:02:58 lr 0.000285 wd 0.0500 time 0.3755 (0.3749) data time 0.0009 (0.0040) model time 0.3745 (0.3696) loss 3.3177 (2.7614) grad_norm 1.8658 (2.4208) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][160/625] eta 0:02:54 lr 0.000285 wd 0.0500 time 0.3735 (0.3747) data time 0.0009 (0.0038) model time 0.3726 (0.3696) loss 3.3611 (2.7666) grad_norm 2.5249 (2.3920) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:55:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][170/625] eta 0:02:51 lr 0.000285 wd 0.0500 time 0.3723 (0.3767) data time 0.0009 (0.0036) model time 0.3714 (0.3728) loss 2.8116 (2.7524) grad_norm 1.9483 (2.3669) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][180/625] eta 0:02:47 lr 0.000285 wd 0.0500 time 0.3680 (0.3765) data time 0.0010 (0.0035) model time 0.3670 (0.3727) loss 3.0545 (2.7566) grad_norm 1.8120 (2.3603) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][190/625] eta 0:02:43 lr 0.000285 wd 0.0500 time 0.3907 (0.3764) data time 0.0008 (0.0034) model time 0.3898 (0.3728) loss 2.9292 (2.7626) grad_norm 1.9241 (2.4000) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][200/625] eta 0:02:39 lr 0.000284 wd 0.0500 time 0.3712 (0.3763) data time 0.0009 (0.0033) model time 0.3703 (0.3728) loss 3.1885 (2.7738) grad_norm 2.5696 (2.3858) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][210/625] eta 0:02:36 lr 0.000284 wd 0.0500 time 0.3673 (0.3761) data time 0.0011 (0.0032) model time 0.3662 (0.3726) loss 2.3395 (2.7686) grad_norm 1.4761 (2.3774) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][220/625] eta 0:02:32 lr 0.000284 wd 0.0500 time 0.3928 (0.3759) data time 0.0008 (0.0031) model time 0.3920 (0.3726) loss 2.4635 (2.7773) grad_norm 2.1420 (2.3994) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][230/625] eta 0:02:28 lr 0.000284 wd 0.0500 time 0.3708 (0.3759) data time 0.0010 (0.0030) model time 0.3699 (0.3727) loss 3.0971 (2.7737) grad_norm 2.0254 (2.3888) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][240/625] eta 0:02:24 lr 0.000284 wd 0.0500 time 0.3686 (0.3759) data time 0.0011 (0.0029) model time 0.3674 (0.3727) loss 2.2510 (2.7745) grad_norm 1.8936 (2.3694) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][250/625] eta 0:02:21 lr 0.000284 wd 0.0500 time 0.3649 (0.3765) data time 0.0010 (0.0029) model time 0.3639 (0.3737) loss 3.0960 (2.7786) grad_norm 2.0949 (2.4681) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][260/625] eta 0:02:17 lr 0.000284 wd 0.0500 time 0.3795 (0.3765) data time 0.0009 (0.0028) model time 0.3786 (0.3737) loss 3.1232 (2.7893) grad_norm 2.2353 (2.4600) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][270/625] eta 0:02:13 lr 0.000284 wd 0.0500 time 0.3778 (0.3763) data time 0.0011 (0.0027) model time 0.3767 (0.3735) loss 2.3753 (2.7861) grad_norm 1.4879 (2.4607) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][280/625] eta 0:02:09 lr 0.000283 wd 0.0500 time 0.3968 (0.3763) data time 0.0008 (0.0027) model time 0.3960 (0.3736) loss 1.7418 (2.7815) grad_norm 3.0183 (2.4524) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][290/625] eta 0:02:06 lr 0.000283 wd 0.0500 time 0.3701 (0.3776) data time 0.0013 (0.0026) model time 0.3689 (0.3752) loss 2.4588 (2.7749) grad_norm 1.7502 (2.4466) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][300/625] eta 0:02:02 lr 0.000283 wd 0.0500 time 0.3729 (0.3774) data time 0.0013 (0.0026) model time 0.3715 (0.3750) loss 3.0115 (2.7807) grad_norm 1.8241 (2.4421) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][310/625] eta 0:01:58 lr 0.000283 wd 0.0500 time 0.3681 (0.3773) data time 0.0008 (0.0025) model time 0.3673 (0.3749) loss 2.9170 (2.7837) grad_norm 2.0373 (2.4369) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][320/625] eta 0:01:55 lr 0.000283 wd 0.0500 time 0.3776 (0.3771) data time 0.0009 (0.0025) model time 0.3768 (0.3747) loss 1.7867 (2.7805) grad_norm 4.2880 (2.4339) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:56:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][330/625] eta 0:01:51 lr 0.000283 wd 0.0500 time 0.3707 (0.3769) data time 0.0011 (0.0025) model time 0.3696 (0.3746) loss 2.9432 (2.7800) grad_norm 2.9388 (2.4244) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][340/625] eta 0:01:47 lr 0.000283 wd 0.0500 time 0.3703 (0.3768) data time 0.0009 (0.0024) model time 0.3695 (0.3745) loss 3.6357 (2.7812) grad_norm 3.0662 (2.4240) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][350/625] eta 0:01:43 lr 0.000283 wd 0.0500 time 0.3650 (0.3766) data time 0.0011 (0.0024) model time 0.3639 (0.3744) loss 2.6768 (2.7847) grad_norm 3.3174 (2.4265) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][360/625] eta 0:01:39 lr 0.000282 wd 0.0500 time 0.3756 (0.3765) data time 0.0009 (0.0024) model time 0.3747 (0.3742) loss 3.3977 (2.7882) grad_norm 3.0037 (2.4165) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][370/625] eta 0:01:35 lr 0.000282 wd 0.0500 time 0.3788 (0.3764) data time 0.0008 (0.0023) model time 0.3779 (0.3741) loss 2.2546 (2.7856) grad_norm 1.6931 (2.4080) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][380/625] eta 0:01:32 lr 0.000282 wd 0.0500 time 0.3655 (0.3763) data time 0.0011 (0.0023) model time 0.3645 (0.3740) loss 2.9550 (2.7859) grad_norm 1.9962 (2.4178) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][390/625] eta 0:01:28 lr 0.000282 wd 0.0500 time 0.3921 (0.3762) data time 0.0010 (0.0023) model time 0.3912 (0.3740) loss 2.1930 (2.7878) grad_norm 2.1421 (2.4212) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][400/625] eta 0:01:24 lr 0.000282 wd 0.0500 time 0.3749 (0.3761) data time 0.0011 (0.0022) model time 0.3738 (0.3739) loss 2.9690 (2.7873) grad_norm 2.5512 (2.4275) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][410/625] eta 0:01:20 lr 0.000282 wd 0.0500 time 0.3801 (0.3760) data time 0.0009 (0.0022) model time 0.3792 (0.3738) loss 3.2191 (2.7896) grad_norm 1.6034 (2.4471) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][420/625] eta 0:01:17 lr 0.000282 wd 0.0500 time 0.3683 (0.3759) data time 0.0009 (0.0022) model time 0.3674 (0.3737) loss 2.9720 (2.7888) grad_norm 2.1560 (2.4395) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][430/625] eta 0:01:13 lr 0.000282 wd 0.0500 time 0.3758 (0.3762) data time 0.0011 (0.0022) model time 0.3746 (0.3741) loss 3.0967 (2.7928) grad_norm 1.9502 (2.4310) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][440/625] eta 0:01:09 lr 0.000281 wd 0.0500 time 0.3719 (0.3762) data time 0.0011 (0.0021) model time 0.3707 (0.3741) loss 2.9939 (2.7908) grad_norm 1.8535 (2.4172) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][450/625] eta 0:01:05 lr 0.000281 wd 0.0500 time 0.3725 (0.3762) data time 0.0010 (0.0021) model time 0.3715 (0.3741) loss 3.2145 (2.7884) grad_norm 1.5690 (2.4097) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][460/625] eta 0:01:02 lr 0.000281 wd 0.0500 time 0.3687 (0.3761) data time 0.0013 (0.0021) model time 0.3674 (0.3740) loss 3.0281 (2.7869) grad_norm 1.8379 (2.4011) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][470/625] eta 0:00:58 lr 0.000281 wd 0.0500 time 0.3689 (0.3760) data time 0.0011 (0.0021) model time 0.3679 (0.3739) loss 2.0280 (2.7840) grad_norm 2.1935 (2.4052) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][480/625] eta 0:00:54 lr 0.000281 wd 0.0500 time 0.3689 (0.3759) data time 0.0011 (0.0021) model time 0.3679 (0.3739) loss 2.3640 (2.7794) grad_norm 2.9999 (2.4073) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:57:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][490/625] eta 0:00:50 lr 0.000281 wd 0.0500 time 0.3669 (0.3758) data time 0.0011 (0.0020) model time 0.3658 (0.3738) loss 2.9220 (2.7806) grad_norm 4.5915 (2.4098) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][500/625] eta 0:00:47 lr 0.000281 wd 0.0500 time 0.3726 (0.3766) data time 0.0011 (0.0020) model time 0.3714 (0.3746) loss 2.7657 (2.7821) grad_norm 1.7892 (2.4077) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][510/625] eta 0:00:43 lr 0.000281 wd 0.0500 time 0.3747 (0.3765) data time 0.0009 (0.0020) model time 0.3739 (0.3746) loss 2.5743 (2.7811) grad_norm 1.3797 (2.4049) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][520/625] eta 0:00:39 lr 0.000281 wd 0.0500 time 0.3830 (0.3765) data time 0.0009 (0.0020) model time 0.3822 (0.3746) loss 3.3519 (2.7833) grad_norm 2.0639 (2.4010) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][530/625] eta 0:00:35 lr 0.000280 wd 0.0500 time 0.3682 (0.3764) data time 0.0011 (0.0020) model time 0.3671 (0.3745) loss 2.6733 (2.7835) grad_norm 1.7788 (2.4041) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][540/625] eta 0:00:31 lr 0.000280 wd 0.0500 time 0.3698 (0.3762) data time 0.0011 (0.0020) model time 0.3687 (0.3743) loss 2.8817 (2.7858) grad_norm 2.2470 (2.4019) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][550/625] eta 0:00:28 lr 0.000280 wd 0.0500 time 0.3720 (0.3765) data time 0.0009 (0.0019) model time 0.3711 (0.3747) loss 1.9221 (2.7792) grad_norm 2.2493 (2.3974) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][560/625] eta 0:00:24 lr 0.000280 wd 0.0500 time 0.3717 (0.3764) data time 0.0012 (0.0019) model time 0.3705 (0.3746) loss 2.8513 (2.7773) grad_norm 1.9010 (2.4266) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][570/625] eta 0:00:20 lr 0.000280 wd 0.0500 time 0.3650 (0.3763) data time 0.0012 (0.0019) model time 0.3638 (0.3745) loss 2.0251 (2.7793) grad_norm 2.0639 (2.4208) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][580/625] eta 0:00:16 lr 0.000280 wd 0.0500 time 0.3735 (0.3763) data time 0.0008 (0.0019) model time 0.3727 (0.3744) loss 3.5551 (2.7805) grad_norm 2.4562 (2.4187) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][590/625] eta 0:00:13 lr 0.000280 wd 0.0500 time 0.3669 (0.3762) data time 0.0011 (0.0019) model time 0.3658 (0.3743) loss 2.6515 (2.7817) grad_norm 2.0255 (2.4354) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][600/625] eta 0:00:09 lr 0.000280 wd 0.0500 time 0.3703 (0.3761) data time 0.0008 (0.0019) model time 0.3695 (0.3743) loss 2.3548 (2.7771) grad_norm 1.9095 (2.4334) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][610/625] eta 0:00:05 lr 0.000279 wd 0.0500 time 0.3556 (0.3760) data time 0.0006 (0.0019) model time 0.3550 (0.3742) loss 2.7669 (2.7792) grad_norm 3.7373 (2.4304) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [233/300][620/625] eta 0:00:01 lr 0.000279 wd 0.0500 time 0.3553 (0.3757) data time 0.0008 (0.0019) model time 0.3545 (0.3739) loss 3.2080 (2.7811) grad_norm 2.1772 (2.4271) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:58:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 233 training takes 0:03:54 [2024-09-01 10:58:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 10:58:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 10:58:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.534 (0.534) Loss 0.4978 (0.4978) Acc@1 88.770 (88.770) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:58:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.145) Loss 0.8481 (0.6164) Acc@1 79.883 (86.164) Acc@5 95.410 (97.678) Mem 11225MB [2024-09-01 10:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 0.9292 (0.7390) Acc@1 77.930 (83.203) Acc@5 94.531 (96.331) Mem 11225MB [2024-09-01 10:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.921 Acc@5 96.339 [2024-09-01 10:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 10:58:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.800 (0.800) Loss 0.4907 (0.4907) Acc@1 89.795 (89.795) Acc@5 98.535 (98.535) Mem 11225MB [2024-09-01 10:58:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.172) Loss 0.7954 (0.5931) Acc@1 80.908 (86.972) Acc@5 96.143 (97.829) Mem 11225MB [2024-09-01 10:58:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.134) Loss 0.8794 (0.7030) Acc@1 78.955 (84.145) Acc@5 95.361 (96.733) Mem 11225MB [2024-09-01 10:58:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.835 Acc@5 96.771 [2024-09-01 10:58:55 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 10:58:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][0/625] eta 0:13:06 lr 0.000279 wd 0.0500 time 1.2580 (1.2580) data time 0.6067 (0.6067) model time 0.0000 (0.0000) loss 2.2034 (2.2034) grad_norm 1.6291 (1.6291) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][10/625] eta 0:04:38 lr 0.000279 wd 0.0500 time 0.3605 (0.4530) data time 0.0009 (0.0563) model time 0.0000 (0.0000) loss 3.5150 (2.6317) grad_norm 3.0555 (2.4337) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][20/625] eta 0:04:11 lr 0.000279 wd 0.0500 time 0.3696 (0.4157) data time 0.0012 (0.0300) model time 0.0000 (0.0000) loss 2.7706 (2.7013) grad_norm 3.9357 (3.0554) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][30/625] eta 0:03:59 lr 0.000279 wd 0.0500 time 0.3764 (0.4026) data time 0.0009 (0.0207) model time 0.0000 (0.0000) loss 2.1966 (2.6009) grad_norm 2.2765 (2.8450) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][40/625] eta 0:03:51 lr 0.000279 wd 0.0500 time 0.3769 (0.3959) data time 0.0008 (0.0159) model time 0.0000 (0.0000) loss 3.0061 (2.6677) grad_norm 2.2574 (2.6973) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][50/625] eta 0:03:45 lr 0.000279 wd 0.0500 time 0.3673 (0.3916) data time 0.0012 (0.0130) model time 0.0000 (0.0000) loss 3.1143 (2.7051) grad_norm 1.6884 (2.6294) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][60/625] eta 0:03:41 lr 0.000279 wd 0.0500 time 0.3693 (0.3923) data time 0.0009 (0.0111) model time 0.3684 (0.3948) loss 3.4945 (2.7233) grad_norm 1.7884 (2.5542) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][70/625] eta 0:03:37 lr 0.000278 wd 0.0500 time 0.3719 (0.3925) data time 0.0013 (0.0098) model time 0.3706 (0.3932) loss 3.1688 (2.7317) grad_norm 2.6387 (2.4933) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][80/625] eta 0:03:32 lr 0.000278 wd 0.0500 time 0.3638 (0.3898) data time 0.0009 (0.0088) model time 0.3629 (0.3851) loss 3.5588 (2.7586) grad_norm 2.4269 (2.5606) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][90/625] eta 0:03:27 lr 0.000278 wd 0.0500 time 0.3742 (0.3875) data time 0.0009 (0.0079) model time 0.3733 (0.3808) loss 1.8197 (2.7508) grad_norm 3.2094 (2.5973) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][100/625] eta 0:03:22 lr 0.000278 wd 0.0500 time 0.3668 (0.3858) data time 0.0009 (0.0073) model time 0.3660 (0.3784) loss 3.4079 (2.7598) grad_norm 2.1037 (2.5915) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][110/625] eta 0:03:18 lr 0.000278 wd 0.0500 time 0.3648 (0.3861) data time 0.0009 (0.0067) model time 0.3639 (0.3800) loss 2.9019 (2.7601) grad_norm 2.4964 (2.6042) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][120/625] eta 0:03:14 lr 0.000278 wd 0.0500 time 0.3714 (0.3849) data time 0.0011 (0.0063) model time 0.3703 (0.3786) loss 3.3769 (2.7632) grad_norm 3.6838 (2.8131) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][130/625] eta 0:03:09 lr 0.000278 wd 0.0500 time 0.3659 (0.3838) data time 0.0012 (0.0059) model time 0.3647 (0.3776) loss 2.8826 (2.7753) grad_norm 1.8080 (2.8222) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][140/625] eta 0:03:05 lr 0.000278 wd 0.0500 time 0.3738 (0.3833) data time 0.0012 (0.0055) model time 0.3726 (0.3773) loss 2.5953 (2.7750) grad_norm 2.2288 (2.7777) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][150/625] eta 0:03:01 lr 0.000277 wd 0.0500 time 0.3720 (0.3825) data time 0.0011 (0.0053) model time 0.3708 (0.3766) loss 3.2585 (2.7936) grad_norm 2.1173 (2.7557) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 10:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][160/625] eta 0:02:57 lr 0.000277 wd 0.0500 time 0.3719 (0.3817) data time 0.0009 (0.0050) model time 0.3710 (0.3759) loss 1.8418 (2.7705) grad_norm 1.6849 (2.7155) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][170/625] eta 0:02:53 lr 0.000277 wd 0.0500 time 0.3706 (0.3811) data time 0.0011 (0.0048) model time 0.3695 (0.3753) loss 3.1060 (2.7546) grad_norm 4.4762 (2.7111) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][180/625] eta 0:02:49 lr 0.000277 wd 0.0500 time 0.3726 (0.3806) data time 0.0009 (0.0046) model time 0.3717 (0.3750) loss 3.2901 (2.7704) grad_norm 2.4874 (2.6755) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][190/625] eta 0:02:45 lr 0.000277 wd 0.0500 time 0.3691 (0.3801) data time 0.0009 (0.0044) model time 0.3683 (0.3746) loss 3.4009 (2.7677) grad_norm 2.1220 (2.6552) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][200/625] eta 0:02:41 lr 0.000277 wd 0.0500 time 0.3719 (0.3796) data time 0.0009 (0.0042) model time 0.3711 (0.3743) loss 2.9892 (2.7620) grad_norm 1.8203 (2.6216) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][210/625] eta 0:02:37 lr 0.000277 wd 0.0500 time 0.3766 (0.3792) data time 0.0009 (0.0041) model time 0.3758 (0.3740) loss 3.4316 (2.7658) grad_norm 1.8655 (2.6100) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][220/625] eta 0:02:33 lr 0.000277 wd 0.0500 time 0.3713 (0.3788) data time 0.0011 (0.0040) model time 0.3702 (0.3737) loss 3.1222 (2.7714) grad_norm 2.4652 (2.6335) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][230/625] eta 0:02:29 lr 0.000277 wd 0.0500 time 0.3690 (0.3784) data time 0.0011 (0.0038) model time 0.3679 (0.3734) loss 2.6550 (2.7833) grad_norm 2.7695 (2.6403) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][240/625] eta 0:02:25 lr 0.000276 wd 0.0500 time 0.3777 (0.3782) data time 0.0011 (0.0037) model time 0.3766 (0.3734) loss 3.1155 (2.7806) grad_norm 2.2769 (2.6376) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][250/625] eta 0:02:22 lr 0.000276 wd 0.0500 time 0.3691 (0.3806) data time 0.0009 (0.0037) model time 0.3682 (0.3765) loss 1.8068 (2.7768) grad_norm 1.8857 (2.6462) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:00:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][260/625] eta 0:02:18 lr 0.000276 wd 0.0500 time 0.3772 (0.3804) data time 0.0008 (0.0036) model time 0.3763 (0.3764) loss 2.4412 (2.7774) grad_norm 1.5872 (2.6210) loss_scale 512.0000 (261.8851) mem 11225MB [2024-09-01 11:00:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][270/625] eta 0:02:14 lr 0.000276 wd 0.0500 time 0.3724 (0.3801) data time 0.0011 (0.0035) model time 0.3712 (0.3761) loss 1.9219 (2.7735) grad_norm 2.0715 (2.6034) loss_scale 512.0000 (271.1144) mem 11225MB [2024-09-01 11:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][280/625] eta 0:02:11 lr 0.000276 wd 0.0500 time 0.3676 (0.3798) data time 0.0009 (0.0034) model time 0.3666 (0.3759) loss 1.8058 (2.7761) grad_norm 2.4227 (2.5916) loss_scale 512.0000 (279.6868) mem 11225MB [2024-09-01 11:00:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][290/625] eta 0:02:07 lr 0.000276 wd 0.0500 time 0.3680 (0.3802) data time 0.0011 (0.0033) model time 0.3669 (0.3765) loss 3.0816 (2.7801) grad_norm 2.1888 (nan) loss_scale 256.0000 (282.3918) mem 11225MB [2024-09-01 11:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][300/625] eta 0:02:03 lr 0.000276 wd 0.0500 time 0.3672 (0.3799) data time 0.0009 (0.0032) model time 0.3663 (0.3762) loss 3.3310 (2.7771) grad_norm 1.9586 (nan) loss_scale 256.0000 (281.5150) mem 11225MB [2024-09-01 11:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][310/625] eta 0:01:59 lr 0.000276 wd 0.0500 time 0.3661 (0.3795) data time 0.0010 (0.0032) model time 0.3650 (0.3759) loss 2.8540 (2.7753) grad_norm 2.2769 (nan) loss_scale 256.0000 (280.6945) mem 11225MB [2024-09-01 11:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][320/625] eta 0:01:55 lr 0.000275 wd 0.0500 time 0.3669 (0.3793) data time 0.0011 (0.0031) model time 0.3658 (0.3757) loss 2.1135 (2.7765) grad_norm 2.0867 (nan) loss_scale 256.0000 (279.9252) mem 11225MB [2024-09-01 11:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][330/625] eta 0:01:51 lr 0.000275 wd 0.0500 time 0.3674 (0.3790) data time 0.0011 (0.0031) model time 0.3663 (0.3755) loss 2.3477 (2.7751) grad_norm 1.9226 (nan) loss_scale 256.0000 (279.2024) mem 11225MB [2024-09-01 11:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][340/625] eta 0:01:47 lr 0.000275 wd 0.0500 time 0.3678 (0.3789) data time 0.0011 (0.0030) model time 0.3667 (0.3754) loss 3.0824 (2.7784) grad_norm 1.5869 (nan) loss_scale 256.0000 (278.5220) mem 11225MB [2024-09-01 11:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][350/625] eta 0:01:44 lr 0.000275 wd 0.0500 time 0.3737 (0.3787) data time 0.0010 (0.0029) model time 0.3727 (0.3752) loss 2.1631 (2.7784) grad_norm 1.6166 (nan) loss_scale 256.0000 (277.8803) mem 11225MB [2024-09-01 11:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][360/625] eta 0:01:40 lr 0.000275 wd 0.0500 time 0.3720 (0.3785) data time 0.0008 (0.0029) model time 0.3712 (0.3751) loss 3.4871 (2.7722) grad_norm 1.7465 (nan) loss_scale 256.0000 (277.2742) mem 11225MB [2024-09-01 11:01:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][370/625] eta 0:01:36 lr 0.000275 wd 0.0500 time 0.3723 (0.3801) data time 0.0011 (0.0028) model time 0.3712 (0.3770) loss 3.0297 (2.7719) grad_norm 1.3367 (nan) loss_scale 256.0000 (276.7008) mem 11225MB [2024-09-01 11:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][380/625] eta 0:01:33 lr 0.000275 wd 0.0500 time 0.3757 (0.3799) data time 0.0008 (0.0028) model time 0.3748 (0.3769) loss 3.0718 (2.7706) grad_norm 3.7245 (nan) loss_scale 256.0000 (276.1575) mem 11225MB [2024-09-01 11:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][390/625] eta 0:01:29 lr 0.000275 wd 0.0500 time 0.3748 (0.3797) data time 0.0011 (0.0028) model time 0.3737 (0.3767) loss 3.0668 (2.7776) grad_norm 2.4088 (nan) loss_scale 256.0000 (275.6419) mem 11225MB [2024-09-01 11:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][400/625] eta 0:01:25 lr 0.000274 wd 0.0500 time 0.3711 (0.3796) data time 0.0011 (0.0027) model time 0.3699 (0.3767) loss 2.8876 (2.7828) grad_norm 1.4915 (nan) loss_scale 256.0000 (275.1521) mem 11225MB [2024-09-01 11:01:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][410/625] eta 0:01:21 lr 0.000274 wd 0.0500 time 0.3747 (0.3800) data time 0.0012 (0.0027) model time 0.3735 (0.3771) loss 3.2356 (2.7772) grad_norm 2.0042 (nan) loss_scale 256.0000 (274.6861) mem 11225MB [2024-09-01 11:01:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][420/625] eta 0:01:17 lr 0.000274 wd 0.0500 time 0.3744 (0.3799) data time 0.0011 (0.0026) model time 0.3734 (0.3770) loss 2.7563 (2.7776) grad_norm 1.3470 (nan) loss_scale 256.0000 (274.2423) mem 11225MB [2024-09-01 11:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][430/625] eta 0:01:14 lr 0.000274 wd 0.0500 time 0.3707 (0.3797) data time 0.0010 (0.0026) model time 0.3698 (0.3769) loss 1.8257 (2.7746) grad_norm 2.7836 (nan) loss_scale 256.0000 (273.8190) mem 11225MB [2024-09-01 11:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][440/625] eta 0:01:10 lr 0.000274 wd 0.0500 time 0.3703 (0.3795) data time 0.0009 (0.0026) model time 0.3695 (0.3767) loss 2.7286 (2.7759) grad_norm 3.5877 (nan) loss_scale 256.0000 (273.4150) mem 11225MB [2024-09-01 11:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][450/625] eta 0:01:06 lr 0.000274 wd 0.0500 time 0.3666 (0.3792) data time 0.0010 (0.0026) model time 0.3656 (0.3764) loss 2.6430 (2.7782) grad_norm 1.9562 (nan) loss_scale 256.0000 (273.0288) mem 11225MB [2024-09-01 11:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][460/625] eta 0:01:02 lr 0.000274 wd 0.0500 time 0.3676 (0.3790) data time 0.0011 (0.0025) model time 0.3666 (0.3762) loss 3.0552 (2.7731) grad_norm 1.9331 (nan) loss_scale 256.0000 (272.6594) mem 11225MB [2024-09-01 11:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][470/625] eta 0:00:58 lr 0.000274 wd 0.0500 time 0.3873 (0.3789) data time 0.0008 (0.0025) model time 0.3865 (0.3761) loss 3.0759 (2.7718) grad_norm 2.0446 (nan) loss_scale 256.0000 (272.3057) mem 11225MB [2024-09-01 11:01:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][480/625] eta 0:00:54 lr 0.000274 wd 0.0500 time 0.3692 (0.3786) data time 0.0012 (0.0025) model time 0.3680 (0.3759) loss 2.8746 (2.7718) grad_norm 1.8097 (nan) loss_scale 256.0000 (271.9667) mem 11225MB [2024-09-01 11:02:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][490/625] eta 0:00:51 lr 0.000273 wd 0.0500 time 0.3724 (0.3785) data time 0.0011 (0.0024) model time 0.3713 (0.3757) loss 3.0748 (2.7746) grad_norm 2.0427 (nan) loss_scale 256.0000 (271.6415) mem 11225MB [2024-09-01 11:02:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][500/625] eta 0:00:47 lr 0.000273 wd 0.0500 time 0.3724 (0.3783) data time 0.0011 (0.0024) model time 0.3713 (0.3756) loss 2.1450 (2.7760) grad_norm 2.1496 (nan) loss_scale 256.0000 (271.3293) mem 11225MB [2024-09-01 11:02:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][510/625] eta 0:00:43 lr 0.000273 wd 0.0500 time 0.3753 (0.3782) data time 0.0013 (0.0024) model time 0.3740 (0.3755) loss 3.0142 (2.7764) grad_norm 5.5153 (nan) loss_scale 256.0000 (271.0294) mem 11225MB [2024-09-01 11:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][520/625] eta 0:00:39 lr 0.000273 wd 0.0500 time 0.3662 (0.3781) data time 0.0010 (0.0024) model time 0.3652 (0.3754) loss 1.8333 (2.7755) grad_norm 1.7919 (nan) loss_scale 256.0000 (270.7409) mem 11225MB [2024-09-01 11:02:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][530/625] eta 0:00:35 lr 0.000273 wd 0.0500 time 0.3739 (0.3780) data time 0.0012 (0.0024) model time 0.3727 (0.3753) loss 3.2155 (2.7748) grad_norm 1.8766 (nan) loss_scale 256.0000 (270.4633) mem 11225MB [2024-09-01 11:02:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][540/625] eta 0:00:32 lr 0.000273 wd 0.0500 time 0.3722 (0.3779) data time 0.0009 (0.0023) model time 0.3714 (0.3752) loss 3.2862 (2.7745) grad_norm 2.6902 (nan) loss_scale 256.0000 (270.1959) mem 11225MB [2024-09-01 11:02:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][550/625] eta 0:00:28 lr 0.000273 wd 0.0500 time 0.3684 (0.3778) data time 0.0011 (0.0023) model time 0.3673 (0.3751) loss 2.9701 (2.7765) grad_norm 1.8463 (nan) loss_scale 256.0000 (269.9383) mem 11225MB [2024-09-01 11:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][560/625] eta 0:00:24 lr 0.000273 wd 0.0500 time 0.3690 (0.3777) data time 0.0009 (0.0023) model time 0.3681 (0.3751) loss 3.2879 (2.7791) grad_norm 2.1259 (nan) loss_scale 256.0000 (269.6898) mem 11225MB [2024-09-01 11:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][570/625] eta 0:00:20 lr 0.000272 wd 0.0500 time 0.5945 (0.3780) data time 0.0012 (0.0023) model time 0.5933 (0.3755) loss 3.1087 (2.7842) grad_norm 1.8411 (nan) loss_scale 256.0000 (269.4501) mem 11225MB [2024-09-01 11:02:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][580/625] eta 0:00:17 lr 0.000272 wd 0.0500 time 0.3758 (0.3786) data time 0.0010 (0.0023) model time 0.3749 (0.3762) loss 1.9457 (2.7855) grad_norm 1.8318 (nan) loss_scale 256.0000 (269.2186) mem 11225MB [2024-09-01 11:02:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][590/625] eta 0:00:13 lr 0.000272 wd 0.0500 time 0.3688 (0.3785) data time 0.0008 (0.0022) model time 0.3679 (0.3760) loss 1.9649 (2.7852) grad_norm 2.4465 (nan) loss_scale 256.0000 (268.9949) mem 11225MB [2024-09-01 11:02:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][600/625] eta 0:00:09 lr 0.000272 wd 0.0500 time 0.3733 (0.3784) data time 0.0009 (0.0022) model time 0.3725 (0.3760) loss 3.0291 (2.7860) grad_norm 2.0708 (nan) loss_scale 256.0000 (268.7787) mem 11225MB [2024-09-01 11:02:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][610/625] eta 0:00:05 lr 0.000272 wd 0.0500 time 0.3569 (0.3783) data time 0.0008 (0.0022) model time 0.3561 (0.3758) loss 3.0469 (2.7871) grad_norm 2.0459 (nan) loss_scale 256.0000 (268.5696) mem 11225MB [2024-09-01 11:02:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [234/300][620/625] eta 0:00:01 lr 0.000272 wd 0.0500 time 0.3698 (0.3784) data time 0.0008 (0.0022) model time 0.3690 (0.3760) loss 2.9786 (2.7877) grad_norm 2.3039 (nan) loss_scale 256.0000 (268.3671) mem 11225MB [2024-09-01 11:02:51 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 234 training takes 0:03:56 [2024-09-01 11:02:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:02:52 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:02:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.524 (0.524) Loss 0.5239 (0.5239) Acc@1 88.379 (88.379) Acc@5 98.438 (98.438) Mem 11225MB [2024-09-01 11:02:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.143) Loss 0.8408 (0.6244) Acc@1 80.029 (86.204) Acc@5 95.557 (97.607) Mem 11225MB [2024-09-01 11:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.099 (0.120) Loss 0.9102 (0.7440) Acc@1 77.246 (83.166) Acc@5 95.410 (96.361) Mem 11225MB [2024-09-01 11:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.851 Acc@5 96.363 [2024-09-01 11:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 11:02:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.795 (0.795) Loss 0.4900 (0.4900) Acc@1 89.697 (89.697) Acc@5 98.633 (98.633) Mem 11225MB [2024-09-01 11:02:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.174) Loss 0.7969 (0.5930) Acc@1 80.957 (86.998) Acc@5 96.094 (97.843) Mem 11225MB [2024-09-01 11:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.8789 (0.7027) Acc@1 78.955 (84.124) Acc@5 95.361 (96.733) Mem 11225MB [2024-09-01 11:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.817 Acc@5 96.765 [2024-09-01 11:02:58 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 11:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][0/625] eta 0:12:38 lr 0.000272 wd 0.0500 time 1.2132 (1.2132) data time 0.6002 (0.6002) model time 0.0000 (0.0000) loss 1.8849 (1.8849) grad_norm 2.0454 (2.0454) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][10/625] eta 0:04:35 lr 0.000272 wd 0.0500 time 0.3716 (0.4481) data time 0.0011 (0.0557) model time 0.0000 (0.0000) loss 2.1898 (2.5922) grad_norm 2.0985 (2.3415) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][20/625] eta 0:04:08 lr 0.000272 wd 0.0500 time 0.3688 (0.4108) data time 0.0009 (0.0297) model time 0.0000 (0.0000) loss 3.2395 (2.7832) grad_norm 2.7387 (2.3410) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][30/625] eta 0:03:56 lr 0.000271 wd 0.0500 time 0.3726 (0.3980) data time 0.0011 (0.0205) model time 0.0000 (0.0000) loss 2.9179 (2.7875) grad_norm 2.1423 (2.4408) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][40/625] eta 0:03:48 lr 0.000271 wd 0.0500 time 0.3721 (0.3910) data time 0.0009 (0.0158) model time 0.0000 (0.0000) loss 2.6457 (2.8076) grad_norm 1.6186 (2.3140) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][50/625] eta 0:03:42 lr 0.000271 wd 0.0500 time 0.3709 (0.3875) data time 0.0012 (0.0129) model time 0.0000 (0.0000) loss 3.3080 (2.8245) grad_norm 2.2622 (2.2983) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][60/625] eta 0:03:37 lr 0.000271 wd 0.0500 time 0.3713 (0.3852) data time 0.0009 (0.0110) model time 0.3704 (0.3722) loss 2.7937 (2.8244) grad_norm 1.9599 (2.2923) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][70/625] eta 0:03:33 lr 0.000271 wd 0.0500 time 0.3808 (0.3840) data time 0.0010 (0.0096) model time 0.3797 (0.3737) loss 3.7383 (2.8390) grad_norm 1.9631 (2.3512) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][80/625] eta 0:03:28 lr 0.000271 wd 0.0500 time 0.3743 (0.3827) data time 0.0011 (0.0086) model time 0.3731 (0.3733) loss 3.0345 (2.8444) grad_norm 3.1903 (2.3326) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][90/625] eta 0:03:24 lr 0.000271 wd 0.0500 time 0.3715 (0.3817) data time 0.0011 (0.0078) model time 0.3703 (0.3731) loss 2.3393 (2.8645) grad_norm 2.7019 (2.3579) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][100/625] eta 0:03:20 lr 0.000271 wd 0.0500 time 0.3708 (0.3824) data time 0.0009 (0.0071) model time 0.3698 (0.3760) loss 2.5993 (2.8616) grad_norm 2.0073 (2.3494) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][110/625] eta 0:03:16 lr 0.000271 wd 0.0500 time 0.3664 (0.3815) data time 0.0012 (0.0066) model time 0.3652 (0.3753) loss 2.6582 (2.8800) grad_norm 2.1251 (2.5461) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][120/625] eta 0:03:12 lr 0.000270 wd 0.0500 time 0.3691 (0.3809) data time 0.0012 (0.0061) model time 0.3680 (0.3749) loss 3.1101 (2.8829) grad_norm 2.1125 (2.5639) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][130/625] eta 0:03:08 lr 0.000270 wd 0.0500 time 0.3692 (0.3805) data time 0.0008 (0.0058) model time 0.3684 (0.3748) loss 3.3184 (2.8899) grad_norm 2.7133 (2.5391) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][140/625] eta 0:03:04 lr 0.000270 wd 0.0500 time 0.3706 (0.3798) data time 0.0012 (0.0054) model time 0.3694 (0.3743) loss 2.9984 (2.8808) grad_norm 2.1344 (2.5092) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:03:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][150/625] eta 0:03:01 lr 0.000270 wd 0.0500 time 0.3701 (0.3818) data time 0.0009 (0.0051) model time 0.3692 (0.3777) loss 2.8465 (2.8719) grad_norm 2.4560 (2.4889) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][160/625] eta 0:02:57 lr 0.000270 wd 0.0500 time 0.3723 (0.3824) data time 0.0009 (0.0049) model time 0.3714 (0.3789) loss 2.8031 (2.8675) grad_norm 3.2215 (2.5697) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][170/625] eta 0:02:53 lr 0.000270 wd 0.0500 time 0.3820 (0.3818) data time 0.0009 (0.0047) model time 0.3811 (0.3781) loss 3.3514 (2.8614) grad_norm 2.4372 (2.5674) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][180/625] eta 0:02:49 lr 0.000270 wd 0.0500 time 0.3701 (0.3812) data time 0.0014 (0.0045) model time 0.3688 (0.3775) loss 3.2831 (2.8613) grad_norm 1.9552 (2.5423) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][190/625] eta 0:02:45 lr 0.000270 wd 0.0500 time 0.3669 (0.3807) data time 0.0009 (0.0043) model time 0.3659 (0.3770) loss 1.9350 (2.8456) grad_norm 1.8496 (2.5096) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][200/625] eta 0:02:41 lr 0.000269 wd 0.0500 time 0.3722 (0.3803) data time 0.0009 (0.0041) model time 0.3713 (0.3766) loss 3.2349 (2.8430) grad_norm 2.3054 (2.5012) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][210/625] eta 0:02:37 lr 0.000269 wd 0.0500 time 0.3733 (0.3798) data time 0.0011 (0.0040) model time 0.3722 (0.3761) loss 2.9506 (2.8328) grad_norm 2.0546 (2.4852) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][220/625] eta 0:02:33 lr 0.000269 wd 0.0500 time 0.3723 (0.3793) data time 0.0010 (0.0039) model time 0.3713 (0.3757) loss 2.8337 (2.8268) grad_norm 1.9001 (2.4782) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][230/625] eta 0:02:29 lr 0.000269 wd 0.0500 time 0.3763 (0.3790) data time 0.0009 (0.0038) model time 0.3754 (0.3754) loss 3.6410 (2.8377) grad_norm 2.0311 (2.4690) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][240/625] eta 0:02:25 lr 0.000269 wd 0.0500 time 0.3741 (0.3787) data time 0.0009 (0.0037) model time 0.3731 (0.3752) loss 2.2410 (2.8396) grad_norm 2.2084 (2.4616) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][250/625] eta 0:02:21 lr 0.000269 wd 0.0500 time 0.3697 (0.3784) data time 0.0012 (0.0036) model time 0.3685 (0.3749) loss 2.8460 (2.8328) grad_norm 1.9018 (2.4673) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][260/625] eta 0:02:18 lr 0.000269 wd 0.0500 time 0.3711 (0.3783) data time 0.0011 (0.0035) model time 0.3700 (0.3748) loss 3.1033 (2.8390) grad_norm 3.2706 (2.4782) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][270/625] eta 0:02:14 lr 0.000269 wd 0.0500 time 0.3727 (0.3780) data time 0.0009 (0.0034) model time 0.3719 (0.3746) loss 2.9673 (2.8381) grad_norm 1.6458 (2.4593) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][280/625] eta 0:02:10 lr 0.000269 wd 0.0500 time 0.3764 (0.3778) data time 0.0014 (0.0033) model time 0.3751 (0.3745) loss 3.2521 (2.8308) grad_norm 1.8602 (2.4509) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][290/625] eta 0:02:06 lr 0.000268 wd 0.0500 time 0.3933 (0.3777) data time 0.0011 (0.0032) model time 0.3921 (0.3744) loss 2.7644 (2.8238) grad_norm 2.2063 (2.4741) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][300/625] eta 0:02:02 lr 0.000268 wd 0.0500 time 0.3656 (0.3775) data time 0.0009 (0.0032) model time 0.3647 (0.3743) loss 3.1122 (2.8193) grad_norm 1.5131 (2.4633) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][310/625] eta 0:01:59 lr 0.000268 wd 0.0500 time 0.3733 (0.3779) data time 0.0011 (0.0031) model time 0.3722 (0.3749) loss 2.5591 (2.8147) grad_norm 1.4772 (2.4519) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][320/625] eta 0:01:55 lr 0.000268 wd 0.0500 time 0.3724 (0.3777) data time 0.0011 (0.0031) model time 0.3713 (0.3746) loss 3.1698 (2.8128) grad_norm 4.3450 (2.4590) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][330/625] eta 0:01:51 lr 0.000268 wd 0.0500 time 0.3708 (0.3774) data time 0.0011 (0.0030) model time 0.3697 (0.3744) loss 2.3212 (2.8181) grad_norm 2.3394 (2.4535) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][340/625] eta 0:01:47 lr 0.000268 wd 0.0500 time 0.3674 (0.3788) data time 0.0009 (0.0029) model time 0.3665 (0.3761) loss 2.3992 (2.8177) grad_norm 2.3141 (2.4436) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][350/625] eta 0:01:44 lr 0.000268 wd 0.0500 time 0.3714 (0.3786) data time 0.0008 (0.0029) model time 0.3706 (0.3759) loss 2.9700 (2.8238) grad_norm 1.9670 (2.4288) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][360/625] eta 0:01:40 lr 0.000268 wd 0.0500 time 0.3693 (0.3783) data time 0.0009 (0.0028) model time 0.3684 (0.3756) loss 2.8901 (2.8204) grad_norm 2.5012 (2.4171) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][370/625] eta 0:01:36 lr 0.000267 wd 0.0500 time 0.3707 (0.3781) data time 0.0010 (0.0028) model time 0.3697 (0.3754) loss 3.3547 (2.8246) grad_norm 1.4320 (2.3990) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][380/625] eta 0:01:32 lr 0.000267 wd 0.0500 time 0.3704 (0.3779) data time 0.0010 (0.0028) model time 0.3695 (0.3752) loss 3.1761 (2.8237) grad_norm 3.1237 (2.3962) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][390/625] eta 0:01:28 lr 0.000267 wd 0.0500 time 0.3706 (0.3778) data time 0.0009 (0.0027) model time 0.3697 (0.3751) loss 2.2564 (2.8136) grad_norm 2.8080 (2.3973) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][400/625] eta 0:01:24 lr 0.000267 wd 0.0500 time 0.3703 (0.3776) data time 0.0011 (0.0027) model time 0.3691 (0.3750) loss 2.7056 (2.8103) grad_norm 1.8876 (2.4192) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][410/625] eta 0:01:21 lr 0.000267 wd 0.0500 time 0.3758 (0.3774) data time 0.0009 (0.0026) model time 0.3749 (0.3748) loss 2.5240 (2.8109) grad_norm 2.4734 (2.4209) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][420/625] eta 0:01:17 lr 0.000267 wd 0.0500 time 0.3720 (0.3773) data time 0.0009 (0.0026) model time 0.3711 (0.3747) loss 3.1874 (2.8088) grad_norm 2.2423 (2.4157) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][430/625] eta 0:01:13 lr 0.000267 wd 0.0500 time 0.3681 (0.3771) data time 0.0013 (0.0026) model time 0.3669 (0.3746) loss 2.7926 (2.8062) grad_norm 2.0787 (2.4153) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][440/625] eta 0:01:09 lr 0.000267 wd 0.0500 time 0.3749 (0.3770) data time 0.0009 (0.0025) model time 0.3740 (0.3744) loss 3.2415 (2.8037) grad_norm 2.1894 (2.4049) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][450/625] eta 0:01:05 lr 0.000267 wd 0.0500 time 0.3760 (0.3769) data time 0.0019 (0.0025) model time 0.3741 (0.3743) loss 3.2751 (2.8020) grad_norm 1.4547 (2.4009) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][460/625] eta 0:01:02 lr 0.000266 wd 0.0500 time 0.3687 (0.3777) data time 0.0012 (0.0025) model time 0.3676 (0.3753) loss 3.4026 (2.8047) grad_norm 3.0787 (2.4355) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][470/625] eta 0:00:58 lr 0.000266 wd 0.0500 time 0.3658 (0.3776) data time 0.0010 (0.0025) model time 0.3648 (0.3752) loss 1.6268 (2.7983) grad_norm 1.6460 (2.4310) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][480/625] eta 0:00:54 lr 0.000266 wd 0.0500 time 0.3698 (0.3775) data time 0.0011 (0.0024) model time 0.3687 (0.3751) loss 3.1334 (2.7921) grad_norm 1.9369 (2.4578) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][490/625] eta 0:00:50 lr 0.000266 wd 0.0500 time 0.3654 (0.3777) data time 0.0010 (0.0024) model time 0.3644 (0.3754) loss 3.2672 (2.7960) grad_norm 2.2178 (2.4608) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][500/625] eta 0:00:47 lr 0.000266 wd 0.0500 time 0.3695 (0.3775) data time 0.0009 (0.0024) model time 0.3686 (0.3752) loss 2.7276 (2.7956) grad_norm 2.8755 (2.4747) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][510/625] eta 0:00:43 lr 0.000266 wd 0.0500 time 0.3681 (0.3774) data time 0.0009 (0.0024) model time 0.3672 (0.3751) loss 1.8141 (2.7977) grad_norm 2.4431 (2.4717) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][520/625] eta 0:00:39 lr 0.000266 wd 0.0500 time 0.3632 (0.3773) data time 0.0009 (0.0023) model time 0.3623 (0.3750) loss 2.4923 (2.7982) grad_norm 1.8194 (2.4618) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][530/625] eta 0:00:35 lr 0.000266 wd 0.0500 time 0.3702 (0.3772) data time 0.0014 (0.0023) model time 0.3688 (0.3749) loss 2.7771 (2.7950) grad_norm 2.7078 (2.4559) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][540/625] eta 0:00:32 lr 0.000265 wd 0.0500 time 0.3730 (0.3772) data time 0.0011 (0.0023) model time 0.3719 (0.3749) loss 2.4217 (2.7943) grad_norm 1.7430 (2.4514) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][550/625] eta 0:00:28 lr 0.000265 wd 0.0500 time 0.3746 (0.3771) data time 0.0009 (0.0023) model time 0.3737 (0.3748) loss 3.2541 (2.7981) grad_norm 2.1771 (2.4435) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][560/625] eta 0:00:24 lr 0.000265 wd 0.0500 time 0.3701 (0.3770) data time 0.0011 (0.0023) model time 0.3689 (0.3747) loss 2.7259 (2.7934) grad_norm 2.5893 (2.4418) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][570/625] eta 0:00:20 lr 0.000265 wd 0.0500 time 0.3808 (0.3769) data time 0.0009 (0.0022) model time 0.3800 (0.3747) loss 2.0677 (2.7887) grad_norm 1.9376 (2.4634) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][580/625] eta 0:00:16 lr 0.000265 wd 0.0500 time 0.3691 (0.3767) data time 0.0009 (0.0022) model time 0.3682 (0.3745) loss 1.7417 (2.7878) grad_norm 2.6811 (2.4731) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][590/625] eta 0:00:13 lr 0.000265 wd 0.0500 time 0.3702 (0.3767) data time 0.0011 (0.0022) model time 0.3691 (0.3744) loss 2.8152 (2.7871) grad_norm 2.3879 (2.4987) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][600/625] eta 0:00:09 lr 0.000265 wd 0.0500 time 0.3703 (0.3766) data time 0.0011 (0.0022) model time 0.3692 (0.3744) loss 2.3956 (2.7868) grad_norm 1.6685 (2.4959) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][610/625] eta 0:00:05 lr 0.000265 wd 0.0500 time 0.3564 (0.3768) data time 0.0006 (0.0022) model time 0.3558 (0.3746) loss 2.1037 (2.7852) grad_norm 1.9085 (2.5071) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [235/300][620/625] eta 0:00:01 lr 0.000265 wd 0.0500 time 0.3544 (0.3765) data time 0.0006 (0.0022) model time 0.3539 (0.3743) loss 2.5944 (2.7890) grad_norm 2.0728 (2.5034) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 235 training takes 0:03:55 [2024-09-01 11:06:53 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:06:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.519 (0.519) Loss 0.5298 (0.5298) Acc@1 89.160 (89.160) Acc@5 98.291 (98.291) Mem 11225MB [2024-09-01 11:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.142) Loss 0.8691 (0.6364) Acc@1 79.150 (86.230) Acc@5 95.166 (97.607) Mem 11225MB [2024-09-01 11:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.8984 (0.7505) Acc@1 78.125 (83.252) Acc@5 95.020 (96.426) Mem 11225MB [2024-09-01 11:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.923 Acc@5 96.439 [2024-09-01 11:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 11:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 82.92% [2024-09-01 11:06:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 11:06:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 11:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.520 (0.520) Loss 0.4893 (0.4893) Acc@1 89.600 (89.600) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 11:07:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.7974 (0.5928) Acc@1 81.006 (87.003) Acc@5 96.094 (97.852) Mem 11225MB [2024-09-01 11:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.8774 (0.7026) Acc@1 79.053 (84.145) Acc@5 95.410 (96.735) Mem 11225MB [2024-09-01 11:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.847 Acc@5 96.771 [2024-09-01 11:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 11:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.85% [2024-09-01 11:07:01 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 11:07:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 11:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][0/625] eta 0:08:02 lr 0.000264 wd 0.0500 time 0.7719 (0.7719) data time 0.4397 (0.4397) model time 0.0000 (0.0000) loss 2.5230 (2.5230) grad_norm 4.0009 (4.0009) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][10/625] eta 0:04:11 lr 0.000264 wd 0.0500 time 0.3795 (0.4090) data time 0.0009 (0.0411) model time 0.0000 (0.0000) loss 2.9746 (2.8716) grad_norm 1.9988 (2.2652) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][20/625] eta 0:04:02 lr 0.000264 wd 0.0500 time 0.3683 (0.4011) data time 0.0009 (0.0221) model time 0.0000 (0.0000) loss 1.7334 (2.7983) grad_norm 1.6144 (2.3917) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][30/625] eta 0:03:56 lr 0.000264 wd 0.0500 time 0.3797 (0.3974) data time 0.0013 (0.0154) model time 0.0000 (0.0000) loss 2.9382 (2.7890) grad_norm 1.6437 (2.8581) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][40/625] eta 0:03:48 lr 0.000264 wd 0.0500 time 0.3736 (0.3913) data time 0.0012 (0.0119) model time 0.0000 (0.0000) loss 2.4607 (2.8144) grad_norm 1.8950 (2.7929) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][50/625] eta 0:03:42 lr 0.000264 wd 0.0500 time 0.3681 (0.3874) data time 0.0009 (0.0099) model time 0.0000 (0.0000) loss 2.9653 (2.7937) grad_norm 2.3524 (2.6596) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][60/625] eta 0:03:37 lr 0.000264 wd 0.0500 time 0.3764 (0.3846) data time 0.0009 (0.0084) model time 0.3756 (0.3694) loss 2.9814 (2.7521) grad_norm 1.5561 (2.5371) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][70/625] eta 0:03:33 lr 0.000264 wd 0.0500 time 0.3662 (0.3853) data time 0.0009 (0.0074) model time 0.3653 (0.3791) loss 2.4745 (2.7275) grad_norm 2.0857 (2.4641) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][80/625] eta 0:03:28 lr 0.000264 wd 0.0500 time 0.3641 (0.3834) data time 0.0009 (0.0066) model time 0.3632 (0.3756) loss 3.5637 (2.7281) grad_norm 1.5358 (2.4381) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][90/625] eta 0:03:24 lr 0.000263 wd 0.0500 time 0.3660 (0.3822) data time 0.0009 (0.0060) model time 0.3652 (0.3744) loss 2.2565 (2.7395) grad_norm 3.4930 (2.6523) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][100/625] eta 0:03:20 lr 0.000263 wd 0.0500 time 0.3814 (0.3812) data time 0.0009 (0.0056) model time 0.3804 (0.3738) loss 2.7241 (2.7414) grad_norm 1.8467 (2.6211) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][110/625] eta 0:03:15 lr 0.000263 wd 0.0500 time 0.3741 (0.3803) data time 0.0012 (0.0052) model time 0.3730 (0.3731) loss 2.6088 (2.7535) grad_norm 2.3951 (2.6610) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][120/625] eta 0:03:11 lr 0.000263 wd 0.0500 time 0.3715 (0.3796) data time 0.0009 (0.0048) model time 0.3706 (0.3727) loss 3.1757 (2.7779) grad_norm 2.7100 (2.6228) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][130/625] eta 0:03:07 lr 0.000263 wd 0.0500 time 0.3736 (0.3791) data time 0.0009 (0.0046) model time 0.3727 (0.3727) loss 2.2534 (2.7767) grad_norm 2.8705 (2.5928) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:07:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][140/625] eta 0:03:03 lr 0.000263 wd 0.0500 time 0.3757 (0.3787) data time 0.0012 (0.0043) model time 0.3746 (0.3726) loss 2.6796 (2.7695) grad_norm 1.4309 (2.5486) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][150/625] eta 0:02:59 lr 0.000263 wd 0.0500 time 0.3742 (0.3782) data time 0.0011 (0.0041) model time 0.3731 (0.3724) loss 2.9763 (2.7765) grad_norm 2.0261 (2.5056) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][160/625] eta 0:02:55 lr 0.000263 wd 0.0500 time 0.3796 (0.3778) data time 0.0011 (0.0039) model time 0.3785 (0.3721) loss 2.8123 (2.7742) grad_norm 1.5700 (2.4821) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][170/625] eta 0:02:51 lr 0.000262 wd 0.0500 time 0.3679 (0.3772) data time 0.0013 (0.0038) model time 0.3665 (0.3717) loss 2.7882 (2.7654) grad_norm 2.2137 (2.4457) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][180/625] eta 0:02:47 lr 0.000262 wd 0.0500 time 0.3736 (0.3768) data time 0.0009 (0.0036) model time 0.3727 (0.3714) loss 3.4077 (2.7566) grad_norm 2.8300 (2.4414) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][190/625] eta 0:02:43 lr 0.000262 wd 0.0500 time 0.3759 (0.3765) data time 0.0011 (0.0035) model time 0.3749 (0.3714) loss 2.6178 (2.7589) grad_norm 2.6173 (2.4371) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][200/625] eta 0:02:39 lr 0.000262 wd 0.0500 time 0.3766 (0.3763) data time 0.0012 (0.0034) model time 0.3755 (0.3714) loss 2.1011 (2.7708) grad_norm 2.4926 (2.4444) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][210/625] eta 0:02:36 lr 0.000262 wd 0.0500 time 0.3634 (0.3761) data time 0.0009 (0.0033) model time 0.3625 (0.3713) loss 2.3569 (2.7708) grad_norm 2.1038 (2.4709) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][220/625] eta 0:02:32 lr 0.000262 wd 0.0500 time 0.3754 (0.3760) data time 0.0012 (0.0032) model time 0.3742 (0.3714) loss 3.0001 (2.7678) grad_norm 5.7429 (2.4861) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][230/625] eta 0:02:28 lr 0.000262 wd 0.0500 time 0.5924 (0.3768) data time 0.0011 (0.0031) model time 0.5913 (0.3726) loss 3.1234 (2.7675) grad_norm 2.2440 (2.4713) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][240/625] eta 0:02:25 lr 0.000262 wd 0.0500 time 0.3693 (0.3774) data time 0.0011 (0.0030) model time 0.3682 (0.3735) loss 3.1189 (2.7722) grad_norm 1.9843 (2.4580) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][250/625] eta 0:02:21 lr 0.000262 wd 0.0500 time 0.3768 (0.3771) data time 0.0013 (0.0030) model time 0.3755 (0.3733) loss 2.6980 (2.7704) grad_norm 2.6514 (2.4483) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][260/625] eta 0:02:17 lr 0.000261 wd 0.0500 time 0.3885 (0.3770) data time 0.0011 (0.0029) model time 0.3874 (0.3732) loss 2.1856 (2.7615) grad_norm 4.7769 (2.4410) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][270/625] eta 0:02:13 lr 0.000261 wd 0.0500 time 0.3672 (0.3768) data time 0.0008 (0.0029) model time 0.3664 (0.3731) loss 2.4565 (2.7614) grad_norm 1.7093 (2.4413) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][280/625] eta 0:02:10 lr 0.000261 wd 0.0500 time 0.3678 (0.3773) data time 0.0013 (0.0028) model time 0.3665 (0.3738) loss 3.0275 (2.7644) grad_norm 1.6724 (2.4357) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][290/625] eta 0:02:06 lr 0.000261 wd 0.0500 time 0.3766 (0.3772) data time 0.0011 (0.0027) model time 0.3755 (0.3738) loss 2.2858 (2.7648) grad_norm 2.1407 (2.4332) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:08:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][300/625] eta 0:02:02 lr 0.000261 wd 0.0500 time 0.3675 (0.3770) data time 0.0011 (0.0027) model time 0.3663 (0.3736) loss 2.5333 (2.7710) grad_norm 1.8872 (2.4341) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][310/625] eta 0:01:58 lr 0.000261 wd 0.0500 time 0.3736 (0.3768) data time 0.0008 (0.0026) model time 0.3728 (0.3735) loss 1.7438 (2.7728) grad_norm 1.5833 (2.4244) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][320/625] eta 0:01:54 lr 0.000261 wd 0.0500 time 0.3706 (0.3766) data time 0.0011 (0.0026) model time 0.3695 (0.3733) loss 3.1024 (2.7720) grad_norm 1.9862 (2.4194) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][330/625] eta 0:01:51 lr 0.000261 wd 0.0500 time 0.3712 (0.3763) data time 0.0009 (0.0025) model time 0.3703 (0.3731) loss 3.1022 (2.7796) grad_norm 1.7553 (2.4162) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][340/625] eta 0:01:47 lr 0.000260 wd 0.0500 time 0.3674 (0.3761) data time 0.0012 (0.0025) model time 0.3662 (0.3729) loss 2.3688 (2.7760) grad_norm 1.5877 (2.4069) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][350/625] eta 0:01:43 lr 0.000260 wd 0.0500 time 0.3711 (0.3759) data time 0.0011 (0.0025) model time 0.3700 (0.3728) loss 2.6622 (2.7765) grad_norm 1.8184 (2.3955) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][360/625] eta 0:01:39 lr 0.000260 wd 0.0500 time 0.3678 (0.3758) data time 0.0010 (0.0024) model time 0.3669 (0.3727) loss 3.2550 (2.7703) grad_norm 2.1007 (2.3863) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][370/625] eta 0:01:35 lr 0.000260 wd 0.0500 time 0.3769 (0.3757) data time 0.0009 (0.0024) model time 0.3760 (0.3726) loss 2.7131 (2.7698) grad_norm 1.9320 (2.4105) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][380/625] eta 0:01:32 lr 0.000260 wd 0.0500 time 0.3683 (0.3756) data time 0.0012 (0.0024) model time 0.3671 (0.3726) loss 2.9742 (2.7697) grad_norm 2.0598 (2.4143) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][390/625] eta 0:01:28 lr 0.000260 wd 0.0500 time 0.3702 (0.3755) data time 0.0011 (0.0023) model time 0.3691 (0.3724) loss 2.8892 (2.7663) grad_norm 1.6246 (2.4269) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][400/625] eta 0:01:24 lr 0.000260 wd 0.0500 time 0.3692 (0.3753) data time 0.0012 (0.0023) model time 0.3680 (0.3723) loss 2.4293 (2.7709) grad_norm 1.8185 (2.4294) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][410/625] eta 0:01:20 lr 0.000260 wd 0.0500 time 0.3709 (0.3752) data time 0.0013 (0.0023) model time 0.3695 (0.3722) loss 2.9123 (2.7747) grad_norm 4.3107 (2.5651) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][420/625] eta 0:01:17 lr 0.000260 wd 0.0500 time 0.3726 (0.3764) data time 0.0013 (0.0023) model time 0.3714 (0.3737) loss 2.3303 (2.7728) grad_norm 3.5965 (2.6112) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][430/625] eta 0:01:13 lr 0.000259 wd 0.0500 time 0.3708 (0.3763) data time 0.0011 (0.0022) model time 0.3698 (0.3736) loss 2.7498 (2.7759) grad_norm 2.2478 (2.6025) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][440/625] eta 0:01:09 lr 0.000259 wd 0.0500 time 0.3704 (0.3762) data time 0.0012 (0.0022) model time 0.3693 (0.3735) loss 3.0802 (2.7776) grad_norm 2.9894 (2.5936) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][450/625] eta 0:01:05 lr 0.000259 wd 0.0500 time 0.3700 (0.3761) data time 0.0012 (0.0022) model time 0.3688 (0.3735) loss 2.3576 (2.7792) grad_norm 2.0990 (2.5832) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][460/625] eta 0:01:02 lr 0.000259 wd 0.0500 time 0.3686 (0.3763) data time 0.0009 (0.0022) model time 0.3677 (0.3737) loss 1.9460 (2.7808) grad_norm 2.4631 (2.5725) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][470/625] eta 0:00:58 lr 0.000259 wd 0.0500 time 0.3688 (0.3761) data time 0.0009 (0.0022) model time 0.3679 (0.3736) loss 3.5360 (2.7807) grad_norm 3.9661 (2.5688) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:10:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][480/625] eta 0:00:54 lr 0.000259 wd 0.0500 time 0.3699 (0.3760) data time 0.0009 (0.0021) model time 0.3691 (0.3734) loss 2.1583 (2.7769) grad_norm 2.3786 (2.5582) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][490/625] eta 0:00:50 lr 0.000259 wd 0.0500 time 0.3687 (0.3759) data time 0.0011 (0.0021) model time 0.3676 (0.3734) loss 1.9041 (2.7753) grad_norm 2.3199 (2.5784) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][500/625] eta 0:00:46 lr 0.000259 wd 0.0500 time 0.3706 (0.3758) data time 0.0012 (0.0021) model time 0.3694 (0.3732) loss 3.3574 (2.7757) grad_norm 1.3996 (2.5740) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][510/625] eta 0:00:43 lr 0.000259 wd 0.0500 time 0.3671 (0.3756) data time 0.0008 (0.0021) model time 0.3662 (0.3731) loss 3.5552 (2.7733) grad_norm 1.9980 (2.5621) loss_scale 256.0000 (256.0000) mem 11225MB [2024-09-01 11:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][520/625] eta 0:00:39 lr 0.000258 wd 0.0500 time 0.3747 (0.3755) data time 0.0012 (0.0021) model time 0.3735 (0.3730) loss 2.4768 (2.7726) grad_norm 3.0352 (nan) loss_scale 128.0000 (254.7716) mem 11225MB [2024-09-01 11:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][530/625] eta 0:00:35 lr 0.000258 wd 0.0500 time 0.3716 (0.3754) data time 0.0012 (0.0020) model time 0.3705 (0.3729) loss 3.2342 (2.7758) grad_norm 2.1134 (nan) loss_scale 128.0000 (252.3842) mem 11225MB [2024-09-01 11:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][540/625] eta 0:00:31 lr 0.000258 wd 0.0500 time 0.3750 (0.3761) data time 0.0011 (0.0020) model time 0.3739 (0.3737) loss 2.6589 (2.7737) grad_norm 3.2105 (nan) loss_scale 128.0000 (250.0850) mem 11225MB [2024-09-01 11:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][550/625] eta 0:00:28 lr 0.000258 wd 0.0500 time 0.3749 (0.3760) data time 0.0008 (0.0020) model time 0.3740 (0.3736) loss 3.0958 (2.7717) grad_norm 2.6083 (nan) loss_scale 128.0000 (247.8693) mem 11225MB [2024-09-01 11:10:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][560/625] eta 0:00:24 lr 0.000258 wd 0.0500 time 0.3734 (0.3759) data time 0.0009 (0.0020) model time 0.3725 (0.3736) loss 3.4389 (2.7707) grad_norm 1.7238 (nan) loss_scale 128.0000 (245.7326) mem 11225MB [2024-09-01 11:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][570/625] eta 0:00:20 lr 0.000258 wd 0.0500 time 0.3711 (0.3759) data time 0.0011 (0.0020) model time 0.3700 (0.3735) loss 2.9786 (2.7740) grad_norm 2.6077 (nan) loss_scale 128.0000 (243.6708) mem 11225MB [2024-09-01 11:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][580/625] eta 0:00:16 lr 0.000258 wd 0.0500 time 0.4033 (0.3763) data time 0.0011 (0.0020) model time 0.4022 (0.3740) loss 2.8859 (2.7718) grad_norm 3.4670 (nan) loss_scale 128.0000 (241.6799) mem 11225MB [2024-09-01 11:10:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][590/625] eta 0:00:13 lr 0.000258 wd 0.0500 time 0.3696 (0.3762) data time 0.0012 (0.0020) model time 0.3685 (0.3739) loss 1.8736 (2.7725) grad_norm 1.8615 (nan) loss_scale 128.0000 (239.7563) mem 11225MB [2024-09-01 11:10:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][600/625] eta 0:00:09 lr 0.000257 wd 0.0500 time 0.3902 (0.3762) data time 0.0013 (0.0019) model time 0.3890 (0.3740) loss 2.7909 (2.7715) grad_norm 2.8463 (nan) loss_scale 128.0000 (237.8968) mem 11225MB [2024-09-01 11:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][610/625] eta 0:00:05 lr 0.000257 wd 0.0500 time 0.3577 (0.3761) data time 0.0006 (0.0019) model time 0.3571 (0.3738) loss 2.3741 (2.7668) grad_norm 2.1495 (nan) loss_scale 128.0000 (236.0982) mem 11225MB [2024-09-01 11:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [236/300][620/625] eta 0:00:01 lr 0.000257 wd 0.0500 time 0.3539 (0.3758) data time 0.0009 (0.0019) model time 0.3530 (0.3736) loss 2.8755 (2.7646) grad_norm 2.1127 (nan) loss_scale 128.0000 (234.3575) mem 11225MB [2024-09-01 11:10:58 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 236 training takes 0:03:54 [2024-09-01 11:10:58 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:10:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:10:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.526 (0.526) Loss 0.5146 (0.5146) Acc@1 88.818 (88.818) Acc@5 98.633 (98.633) Mem 11225MB [2024-09-01 11:11:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.147) Loss 0.8442 (0.6263) Acc@1 79.932 (86.248) Acc@5 95.557 (97.710) Mem 11225MB [2024-09-01 11:11:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.096 (0.122) Loss 0.9355 (0.7441) Acc@1 77.051 (83.298) Acc@5 94.824 (96.522) Mem 11225MB [2024-09-01 11:11:02 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.997 Acc@5 96.515 [2024-09-01 11:11:02 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.0% [2024-09-01 11:11:02 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.00% [2024-09-01 11:11:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 11:11:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 11:11:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.523 (0.523) Loss 0.4890 (0.4890) Acc@1 89.600 (89.600) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 11:11:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.150) Loss 0.7983 (0.5922) Acc@1 81.104 (87.016) Acc@5 96.045 (97.834) Mem 11225MB [2024-09-01 11:11:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.123) Loss 0.8760 (0.7020) Acc@1 79.199 (84.161) Acc@5 95.312 (96.722) Mem 11225MB [2024-09-01 11:11:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.857 Acc@5 96.759 [2024-09-01 11:11:06 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:11:06 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.86% [2024-09-01 11:11:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 11:11:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 11:11:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][0/625] eta 0:07:54 lr 0.000257 wd 0.0500 time 0.7598 (0.7598) data time 0.4240 (0.4240) model time 0.0000 (0.0000) loss 2.5390 (2.5390) grad_norm 1.8854 (1.8854) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][10/625] eta 0:04:09 lr 0.000257 wd 0.0500 time 0.3716 (0.4051) data time 0.0011 (0.0396) model time 0.0000 (0.0000) loss 2.9911 (2.9893) grad_norm 2.0527 (2.2213) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][20/625] eta 0:03:55 lr 0.000257 wd 0.0500 time 0.3667 (0.3897) data time 0.0013 (0.0213) model time 0.0000 (0.0000) loss 2.7619 (2.8723) grad_norm 2.3441 (2.3047) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][30/625] eta 0:03:48 lr 0.000257 wd 0.0500 time 0.3662 (0.3833) data time 0.0009 (0.0148) model time 0.0000 (0.0000) loss 3.4822 (2.8713) grad_norm 9.8190 (3.5092) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][40/625] eta 0:03:42 lr 0.000257 wd 0.0500 time 0.3732 (0.3805) data time 0.0012 (0.0115) model time 0.0000 (0.0000) loss 2.5679 (2.8813) grad_norm 3.7025 (3.2833) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][50/625] eta 0:03:37 lr 0.000257 wd 0.0500 time 0.3645 (0.3789) data time 0.0012 (0.0095) model time 0.0000 (0.0000) loss 3.1774 (2.8576) grad_norm 1.7460 (3.1832) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][60/625] eta 0:03:33 lr 0.000257 wd 0.0500 time 0.3636 (0.3775) data time 0.0012 (0.0081) model time 0.3624 (0.3696) loss 2.0328 (2.8629) grad_norm 2.1917 (3.0910) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][70/625] eta 0:03:32 lr 0.000256 wd 0.0500 time 0.3847 (0.3820) data time 0.0012 (0.0071) model time 0.3835 (0.3889) loss 2.8456 (2.8458) grad_norm 2.2246 (2.9880) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][80/625] eta 0:03:27 lr 0.000256 wd 0.0500 time 0.3688 (0.3805) data time 0.0013 (0.0064) model time 0.3675 (0.3820) loss 2.3860 (2.8536) grad_norm 2.0213 (2.9231) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][90/625] eta 0:03:23 lr 0.000256 wd 0.0500 time 0.3688 (0.3795) data time 0.0009 (0.0059) model time 0.3679 (0.3791) loss 2.3566 (2.8558) grad_norm 3.1165 (2.9390) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][100/625] eta 0:03:18 lr 0.000256 wd 0.0500 time 0.3756 (0.3787) data time 0.0013 (0.0054) model time 0.3744 (0.3773) loss 2.8379 (2.8469) grad_norm 2.6144 (3.0936) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][110/625] eta 0:03:14 lr 0.000256 wd 0.0500 time 0.3677 (0.3782) data time 0.0012 (0.0050) model time 0.3665 (0.3765) loss 2.3124 (2.8280) grad_norm 2.4461 (3.0267) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][120/625] eta 0:03:10 lr 0.000256 wd 0.0500 time 0.3700 (0.3780) data time 0.0008 (0.0047) model time 0.3691 (0.3761) loss 2.6110 (2.8293) grad_norm 2.4811 (2.9591) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:11:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][130/625] eta 0:03:06 lr 0.000256 wd 0.0500 time 0.3774 (0.3776) data time 0.0011 (0.0044) model time 0.3763 (0.3756) loss 2.2608 (2.8157) grad_norm 1.9449 (2.9343) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][140/625] eta 0:03:02 lr 0.000256 wd 0.0500 time 0.3723 (0.3772) data time 0.0012 (0.0042) model time 0.3711 (0.3751) loss 2.9991 (2.8197) grad_norm 2.4716 (2.8871) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][150/625] eta 0:02:59 lr 0.000255 wd 0.0500 time 0.3753 (0.3772) data time 0.0012 (0.0040) model time 0.3742 (0.3751) loss 2.9527 (2.8182) grad_norm 2.3261 (2.8294) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][160/625] eta 0:02:55 lr 0.000255 wd 0.0500 time 0.3731 (0.3770) data time 0.0008 (0.0038) model time 0.3723 (0.3749) loss 3.6741 (2.8181) grad_norm 2.8928 (2.7798) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][170/625] eta 0:02:51 lr 0.000255 wd 0.0500 time 0.3644 (0.3766) data time 0.0009 (0.0037) model time 0.3635 (0.3745) loss 3.0183 (2.8340) grad_norm 2.5557 (2.7730) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][180/625] eta 0:02:47 lr 0.000255 wd 0.0500 time 0.3702 (0.3761) data time 0.0011 (0.0035) model time 0.3691 (0.3739) loss 2.6985 (2.8298) grad_norm 1.9458 (2.7429) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][190/625] eta 0:02:43 lr 0.000255 wd 0.0500 time 0.3688 (0.3759) data time 0.0011 (0.0034) model time 0.3677 (0.3736) loss 3.4301 (2.8273) grad_norm 2.4619 (2.7298) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][200/625] eta 0:02:39 lr 0.000255 wd 0.0500 time 0.3778 (0.3759) data time 0.0011 (0.0033) model time 0.3767 (0.3737) loss 2.9057 (2.8188) grad_norm 1.7647 (2.7020) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][210/625] eta 0:02:35 lr 0.000255 wd 0.0500 time 0.3657 (0.3756) data time 0.0009 (0.0032) model time 0.3648 (0.3734) loss 3.0277 (2.8227) grad_norm 1.9283 (2.6868) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][220/625] eta 0:02:32 lr 0.000255 wd 0.0500 time 0.3718 (0.3756) data time 0.0014 (0.0031) model time 0.3704 (0.3734) loss 2.6698 (2.8187) grad_norm 1.5623 (2.6676) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][230/625] eta 0:02:28 lr 0.000255 wd 0.0500 time 0.3714 (0.3755) data time 0.0009 (0.0030) model time 0.3705 (0.3734) loss 2.9498 (2.8224) grad_norm 2.3726 (2.6550) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][240/625] eta 0:02:24 lr 0.000254 wd 0.0500 time 0.3772 (0.3755) data time 0.0010 (0.0029) model time 0.3761 (0.3734) loss 3.3335 (2.8173) grad_norm 2.7039 (2.6479) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][250/625] eta 0:02:20 lr 0.000254 wd 0.0500 time 0.3760 (0.3753) data time 0.0011 (0.0029) model time 0.3750 (0.3732) loss 2.9445 (2.8069) grad_norm 2.3089 (2.6398) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][260/625] eta 0:02:17 lr 0.000254 wd 0.0500 time 0.3678 (0.3759) data time 0.0011 (0.0028) model time 0.3667 (0.3740) loss 2.9655 (2.8020) grad_norm 4.3901 (2.6290) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][270/625] eta 0:02:13 lr 0.000254 wd 0.0500 time 0.3747 (0.3757) data time 0.0010 (0.0027) model time 0.3736 (0.3738) loss 2.7550 (2.7917) grad_norm 1.9689 (2.6096) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][280/625] eta 0:02:10 lr 0.000254 wd 0.0500 time 0.5793 (0.3771) data time 0.0011 (0.0027) model time 0.5783 (0.3756) loss 2.9053 (2.7867) grad_norm 2.1130 (2.5839) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][290/625] eta 0:02:06 lr 0.000254 wd 0.0500 time 0.3679 (0.3769) data time 0.0011 (0.0026) model time 0.3668 (0.3752) loss 2.6206 (2.7865) grad_norm 2.9297 (2.5714) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][300/625] eta 0:02:02 lr 0.000254 wd 0.0500 time 0.3673 (0.3767) data time 0.0011 (0.0026) model time 0.3662 (0.3751) loss 3.4568 (2.7908) grad_norm 2.6962 (2.5855) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][310/625] eta 0:01:58 lr 0.000254 wd 0.0500 time 0.3692 (0.3765) data time 0.0011 (0.0025) model time 0.3681 (0.3749) loss 2.4036 (2.7854) grad_norm 3.4226 (2.6183) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][320/625] eta 0:01:54 lr 0.000254 wd 0.0500 time 0.3757 (0.3764) data time 0.0011 (0.0025) model time 0.3745 (0.3747) loss 3.1106 (2.7853) grad_norm 1.8658 (2.6298) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][330/625] eta 0:01:50 lr 0.000253 wd 0.0500 time 0.3697 (0.3762) data time 0.0011 (0.0025) model time 0.3686 (0.3745) loss 2.7446 (2.7870) grad_norm 2.4248 (2.6237) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][340/625] eta 0:01:47 lr 0.000253 wd 0.0500 time 0.3732 (0.3761) data time 0.0013 (0.0024) model time 0.3720 (0.3745) loss 3.1864 (2.7851) grad_norm 2.3347 (2.6255) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][350/625] eta 0:01:43 lr 0.000253 wd 0.0500 time 0.3745 (0.3760) data time 0.0009 (0.0024) model time 0.3736 (0.3743) loss 2.0890 (2.7780) grad_norm 2.3431 (2.6130) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][360/625] eta 0:01:39 lr 0.000253 wd 0.0500 time 0.3689 (0.3758) data time 0.0010 (0.0024) model time 0.3679 (0.3741) loss 3.2502 (2.7807) grad_norm 5.3654 (2.7596) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][370/625] eta 0:01:35 lr 0.000253 wd 0.0500 time 0.6048 (0.3763) data time 0.0011 (0.0023) model time 0.6037 (0.3747) loss 2.3082 (2.7815) grad_norm 1.8577 (2.7417) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][380/625] eta 0:01:32 lr 0.000253 wd 0.0500 time 0.3711 (0.3761) data time 0.0011 (0.0023) model time 0.3700 (0.3744) loss 3.1417 (2.7821) grad_norm 2.3476 (2.7396) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][390/625] eta 0:01:28 lr 0.000253 wd 0.0500 time 0.3588 (0.3760) data time 0.0012 (0.0023) model time 0.3576 (0.3744) loss 2.7995 (2.7837) grad_norm 3.2064 (2.7329) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][400/625] eta 0:01:24 lr 0.000253 wd 0.0500 time 0.3722 (0.3759) data time 0.0012 (0.0022) model time 0.3710 (0.3742) loss 3.0086 (2.7814) grad_norm 1.7889 (2.7215) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][410/625] eta 0:01:20 lr 0.000252 wd 0.0500 time 0.3704 (0.3758) data time 0.0012 (0.0022) model time 0.3692 (0.3742) loss 3.2235 (2.7770) grad_norm 1.8818 (2.7043) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][420/625] eta 0:01:17 lr 0.000252 wd 0.0500 time 0.3682 (0.3759) data time 0.0009 (0.0022) model time 0.3673 (0.3742) loss 2.9814 (2.7733) grad_norm 2.0318 (2.6865) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][430/625] eta 0:01:13 lr 0.000252 wd 0.0500 time 0.3666 (0.3757) data time 0.0012 (0.0022) model time 0.3654 (0.3741) loss 3.0107 (2.7701) grad_norm 2.3493 (2.7991) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][440/625] eta 0:01:09 lr 0.000252 wd 0.0500 time 0.3743 (0.3756) data time 0.0010 (0.0022) model time 0.3734 (0.3740) loss 2.7260 (2.7730) grad_norm 3.1474 (2.7925) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:13:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][450/625] eta 0:01:05 lr 0.000252 wd 0.0500 time 0.3738 (0.3755) data time 0.0009 (0.0021) model time 0.3729 (0.3739) loss 3.0689 (2.7694) grad_norm 3.1184 (2.7918) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][460/625] eta 0:01:02 lr 0.000252 wd 0.0500 time 0.5847 (0.3759) data time 0.0009 (0.0021) model time 0.5839 (0.3743) loss 3.3411 (2.7738) grad_norm 2.4616 (2.7807) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][470/625] eta 0:00:58 lr 0.000252 wd 0.0500 time 0.3773 (0.3768) data time 0.0012 (0.0021) model time 0.3761 (0.3753) loss 2.7988 (2.7756) grad_norm 2.3661 (2.7720) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][480/625] eta 0:00:54 lr 0.000252 wd 0.0500 time 0.3698 (0.3767) data time 0.0012 (0.0021) model time 0.3686 (0.3752) loss 2.9426 (2.7760) grad_norm 2.3798 (2.7714) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][490/625] eta 0:00:50 lr 0.000252 wd 0.0500 time 0.3743 (0.3766) data time 0.0011 (0.0021) model time 0.3732 (0.3751) loss 2.4633 (2.7785) grad_norm 2.3711 (2.7771) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][500/625] eta 0:00:47 lr 0.000251 wd 0.0500 time 0.3720 (0.3765) data time 0.0012 (0.0020) model time 0.3707 (0.3750) loss 2.2450 (2.7796) grad_norm 2.8657 (2.7684) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][510/625] eta 0:00:43 lr 0.000251 wd 0.0500 time 0.3721 (0.3764) data time 0.0009 (0.0020) model time 0.3712 (0.3749) loss 2.7229 (2.7756) grad_norm 2.8870 (2.7601) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][520/625] eta 0:00:39 lr 0.000251 wd 0.0500 time 0.3702 (0.3763) data time 0.0009 (0.0020) model time 0.3693 (0.3748) loss 3.4252 (2.7779) grad_norm 2.0299 (2.7556) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][530/625] eta 0:00:35 lr 0.000251 wd 0.0500 time 0.3675 (0.3762) data time 0.0009 (0.0020) model time 0.3666 (0.3747) loss 3.1743 (2.7784) grad_norm 2.5398 (2.7479) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][540/625] eta 0:00:31 lr 0.000251 wd 0.0500 time 0.3761 (0.3761) data time 0.0011 (0.0020) model time 0.3750 (0.3746) loss 2.0574 (2.7809) grad_norm 1.7480 (2.7393) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][550/625] eta 0:00:28 lr 0.000251 wd 0.0500 time 0.3678 (0.3760) data time 0.0011 (0.0020) model time 0.3666 (0.3746) loss 2.4880 (2.7792) grad_norm 2.3578 (2.7283) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][560/625] eta 0:00:24 lr 0.000251 wd 0.0500 time 0.3699 (0.3760) data time 0.0009 (0.0020) model time 0.3690 (0.3745) loss 2.2213 (2.7776) grad_norm 2.1936 (2.7269) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][570/625] eta 0:00:20 lr 0.000251 wd 0.0500 time 0.3644 (0.3759) data time 0.0009 (0.0019) model time 0.3635 (0.3744) loss 2.1798 (2.7768) grad_norm 2.1254 (2.7167) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][580/625] eta 0:00:16 lr 0.000251 wd 0.0500 time 0.5992 (0.3766) data time 0.0011 (0.0019) model time 0.5981 (0.3751) loss 2.8172 (2.7785) grad_norm 1.9800 (2.7070) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][590/625] eta 0:00:13 lr 0.000250 wd 0.0500 time 0.3772 (0.3768) data time 0.0009 (0.0019) model time 0.3763 (0.3754) loss 2.7035 (2.7768) grad_norm 2.0218 (2.6990) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][600/625] eta 0:00:09 lr 0.000250 wd 0.0500 time 0.3722 (0.3768) data time 0.0013 (0.0019) model time 0.3710 (0.3754) loss 2.4393 (2.7740) grad_norm 2.7376 (2.6902) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:14:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][610/625] eta 0:00:05 lr 0.000250 wd 0.0500 time 0.3610 (0.3767) data time 0.0008 (0.0019) model time 0.3601 (0.3752) loss 2.7589 (2.7726) grad_norm 1.9784 (2.6798) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [237/300][620/625] eta 0:00:01 lr 0.000250 wd 0.0500 time 0.3659 (0.3763) data time 0.0008 (0.0019) model time 0.3650 (0.3749) loss 2.5737 (2.7742) grad_norm 2.9031 (2.6812) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:02 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 237 training takes 0:03:55 [2024-09-01 11:15:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:15:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:15:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.519 (0.519) Loss 0.5137 (0.5137) Acc@1 88.867 (88.867) Acc@5 98.828 (98.828) Mem 11225MB [2024-09-01 11:15:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.149) Loss 0.8657 (0.6287) Acc@1 79.834 (85.977) Acc@5 94.824 (97.532) Mem 11225MB [2024-09-01 11:15:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.123) Loss 0.9277 (0.7412) Acc@1 77.686 (83.194) Acc@5 94.238 (96.375) Mem 11225MB [2024-09-01 11:15:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.881 Acc@5 96.381 [2024-09-01 11:15:06 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 11:15:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.804 (0.804) Loss 0.4885 (0.4885) Acc@1 89.600 (89.600) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 11:15:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.174) Loss 0.7998 (0.5925) Acc@1 81.055 (87.012) Acc@5 96.094 (97.825) Mem 11225MB [2024-09-01 11:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.136) Loss 0.8750 (0.7020) Acc@1 79.297 (84.163) Acc@5 95.459 (96.710) Mem 11225MB [2024-09-01 11:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.863 Acc@5 96.743 [2024-09-01 11:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.86% [2024-09-01 11:15:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 11:15:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 11:15:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][0/625] eta 0:07:56 lr 0.000250 wd 0.0500 time 0.7622 (0.7622) data time 0.4234 (0.4234) model time 0.0000 (0.0000) loss 2.7254 (2.7254) grad_norm 2.0848 (2.0848) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][10/625] eta 0:04:10 lr 0.000250 wd 0.0500 time 0.3790 (0.4069) data time 0.0011 (0.0395) model time 0.0000 (0.0000) loss 2.5232 (2.9074) grad_norm 1.5835 (2.0636) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][20/625] eta 0:03:55 lr 0.000250 wd 0.0500 time 0.3765 (0.3897) data time 0.0011 (0.0213) model time 0.0000 (0.0000) loss 2.6790 (2.9050) grad_norm 1.7648 (2.1062) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][30/625] eta 0:03:48 lr 0.000250 wd 0.0500 time 0.3708 (0.3840) data time 0.0011 (0.0148) model time 0.0000 (0.0000) loss 2.6119 (2.8711) grad_norm 2.4500 (2.1875) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][40/625] eta 0:03:42 lr 0.000250 wd 0.0500 time 0.3759 (0.3811) data time 0.0009 (0.0115) model time 0.0000 (0.0000) loss 1.7628 (2.8504) grad_norm 2.9116 (2.3449) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][50/625] eta 0:03:38 lr 0.000249 wd 0.0500 time 0.3766 (0.3792) data time 0.0009 (0.0095) model time 0.0000 (0.0000) loss 3.3949 (2.8903) grad_norm 2.1462 (2.3583) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][60/625] eta 0:03:33 lr 0.000249 wd 0.0500 time 0.3661 (0.3783) data time 0.0009 (0.0081) model time 0.3652 (0.3721) loss 2.1989 (2.8938) grad_norm 2.3329 (2.3559) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][70/625] eta 0:03:29 lr 0.000249 wd 0.0500 time 0.3666 (0.3772) data time 0.0013 (0.0072) model time 0.3653 (0.3706) loss 2.9238 (2.9101) grad_norm 2.0863 (2.3212) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][80/625] eta 0:03:25 lr 0.000249 wd 0.0500 time 0.3673 (0.3762) data time 0.0009 (0.0064) model time 0.3664 (0.3699) loss 2.1244 (2.8936) grad_norm 5.2742 (2.3462) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][90/625] eta 0:03:22 lr 0.000249 wd 0.0500 time 0.3319 (0.3779) data time 0.0010 (0.0058) model time 0.3309 (0.3751) loss 2.8596 (2.8710) grad_norm 2.2944 (2.3420) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][100/625] eta 0:03:20 lr 0.000249 wd 0.0500 time 0.3720 (0.3816) data time 0.0011 (0.0054) model time 0.3709 (0.3827) loss 2.9179 (2.8513) grad_norm 2.3769 (2.3630) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][110/625] eta 0:03:16 lr 0.000249 wd 0.0500 time 0.3745 (0.3807) data time 0.0012 (0.0050) model time 0.3733 (0.3808) loss 2.7968 (2.8389) grad_norm 2.6442 (2.3592) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][120/625] eta 0:03:11 lr 0.000249 wd 0.0500 time 0.3765 (0.3801) data time 0.0012 (0.0047) model time 0.3753 (0.3795) loss 3.0123 (2.8275) grad_norm 2.6028 (2.3379) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][130/625] eta 0:03:07 lr 0.000249 wd 0.0500 time 0.3770 (0.3793) data time 0.0012 (0.0044) model time 0.3758 (0.3781) loss 2.6552 (2.8271) grad_norm 2.0437 (2.3947) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][140/625] eta 0:03:03 lr 0.000248 wd 0.0500 time 0.3703 (0.3788) data time 0.0009 (0.0042) model time 0.3693 (0.3773) loss 3.0867 (2.8168) grad_norm 2.4596 (2.4008) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][150/625] eta 0:02:59 lr 0.000248 wd 0.0500 time 0.3692 (0.3783) data time 0.0008 (0.0040) model time 0.3683 (0.3767) loss 2.2294 (2.8135) grad_norm 2.4621 (2.4136) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][160/625] eta 0:02:55 lr 0.000248 wd 0.0500 time 0.3708 (0.3778) data time 0.0011 (0.0038) model time 0.3697 (0.3760) loss 3.3663 (2.7881) grad_norm 1.9904 (2.3909) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][170/625] eta 0:02:51 lr 0.000248 wd 0.0500 time 0.3715 (0.3773) data time 0.0008 (0.0037) model time 0.3706 (0.3753) loss 2.0015 (2.7838) grad_norm 2.3341 (2.3727) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][180/625] eta 0:02:47 lr 0.000248 wd 0.0500 time 0.3670 (0.3767) data time 0.0008 (0.0035) model time 0.3662 (0.3746) loss 3.2762 (2.7726) grad_norm 1.9403 (2.3581) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][190/625] eta 0:02:43 lr 0.000248 wd 0.0500 time 0.3765 (0.3765) data time 0.0011 (0.0034) model time 0.3754 (0.3743) loss 3.0197 (2.7653) grad_norm 1.7741 (2.3563) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][200/625] eta 0:02:39 lr 0.000248 wd 0.0500 time 0.3791 (0.3764) data time 0.0009 (0.0033) model time 0.3781 (0.3742) loss 3.0224 (2.7684) grad_norm 2.8467 (2.3644) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][210/625] eta 0:02:36 lr 0.000248 wd 0.0500 time 0.3729 (0.3771) data time 0.0011 (0.0032) model time 0.3719 (0.3752) loss 3.0893 (2.7634) grad_norm 1.5368 (2.3540) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][220/625] eta 0:02:32 lr 0.000248 wd 0.0500 time 0.3684 (0.3770) data time 0.0008 (0.0031) model time 0.3675 (0.3751) loss 2.4855 (2.7697) grad_norm 1.7747 (2.3426) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][230/625] eta 0:02:28 lr 0.000247 wd 0.0500 time 0.3838 (0.3769) data time 0.0011 (0.0030) model time 0.3827 (0.3751) loss 2.9093 (2.7526) grad_norm 2.0269 (2.3383) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][240/625] eta 0:02:25 lr 0.000247 wd 0.0500 time 0.3742 (0.3767) data time 0.0011 (0.0029) model time 0.3730 (0.3748) loss 2.9003 (2.7539) grad_norm 2.3606 (2.3359) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][250/625] eta 0:02:21 lr 0.000247 wd 0.0500 time 0.3694 (0.3764) data time 0.0012 (0.0029) model time 0.3682 (0.3744) loss 2.8905 (2.7565) grad_norm 2.3172 (2.3230) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][260/625] eta 0:02:17 lr 0.000247 wd 0.0500 time 0.3645 (0.3762) data time 0.0012 (0.0028) model time 0.3632 (0.3742) loss 2.8508 (2.7653) grad_norm 2.2597 (2.3221) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][270/625] eta 0:02:13 lr 0.000247 wd 0.0500 time 0.3648 (0.3760) data time 0.0011 (0.0028) model time 0.3637 (0.3741) loss 3.0609 (2.7733) grad_norm 3.1666 (2.3234) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][280/625] eta 0:02:09 lr 0.000247 wd 0.0500 time 0.3728 (0.3758) data time 0.0011 (0.0027) model time 0.3717 (0.3739) loss 1.9002 (2.7592) grad_norm 2.0811 (2.3204) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][290/625] eta 0:02:05 lr 0.000247 wd 0.0500 time 0.3672 (0.3757) data time 0.0009 (0.0026) model time 0.3663 (0.3738) loss 3.0770 (2.7639) grad_norm 2.3611 (2.3482) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][300/625] eta 0:02:02 lr 0.000247 wd 0.0500 time 0.3628 (0.3755) data time 0.0011 (0.0026) model time 0.3617 (0.3735) loss 2.8974 (2.7663) grad_norm 2.1527 (2.3631) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][310/625] eta 0:01:58 lr 0.000247 wd 0.0500 time 0.3698 (0.3767) data time 0.0009 (0.0026) model time 0.3688 (0.3751) loss 3.2531 (2.7741) grad_norm 1.8760 (2.3606) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][320/625] eta 0:01:54 lr 0.000246 wd 0.0500 time 0.3796 (0.3766) data time 0.0012 (0.0025) model time 0.3784 (0.3749) loss 1.9061 (2.7691) grad_norm 2.0516 (2.3545) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][330/625] eta 0:01:51 lr 0.000246 wd 0.0500 time 0.3703 (0.3764) data time 0.0008 (0.0025) model time 0.3694 (0.3747) loss 2.3846 (2.7653) grad_norm 2.5656 (2.3553) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][340/625] eta 0:01:47 lr 0.000246 wd 0.0500 time 0.3678 (0.3763) data time 0.0012 (0.0024) model time 0.3666 (0.3746) loss 3.0408 (2.7651) grad_norm 5.9361 (2.3634) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][350/625] eta 0:01:43 lr 0.000246 wd 0.0500 time 0.3629 (0.3761) data time 0.0009 (0.0024) model time 0.3620 (0.3744) loss 2.9122 (2.7705) grad_norm 3.1628 (2.3757) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][360/625] eta 0:01:39 lr 0.000246 wd 0.0500 time 0.3745 (0.3760) data time 0.0009 (0.0024) model time 0.3736 (0.3742) loss 2.7924 (2.7690) grad_norm 1.8594 (2.3721) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][370/625] eta 0:01:35 lr 0.000246 wd 0.0500 time 0.3750 (0.3759) data time 0.0011 (0.0023) model time 0.3739 (0.3741) loss 2.9875 (2.7720) grad_norm 2.4616 (2.3694) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][380/625] eta 0:01:32 lr 0.000246 wd 0.0500 time 0.3739 (0.3758) data time 0.0014 (0.0023) model time 0.3725 (0.3740) loss 3.1678 (2.7704) grad_norm 1.6509 (2.3796) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][390/625] eta 0:01:28 lr 0.000246 wd 0.0500 time 0.3783 (0.3757) data time 0.0011 (0.0023) model time 0.3772 (0.3739) loss 3.0346 (2.7691) grad_norm 2.7041 (2.3881) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][400/625] eta 0:01:24 lr 0.000245 wd 0.0500 time 0.3769 (0.3756) data time 0.0010 (0.0023) model time 0.3759 (0.3739) loss 2.4685 (2.7756) grad_norm 3.1032 (2.3824) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][410/625] eta 0:01:20 lr 0.000245 wd 0.0500 time 0.3587 (0.3756) data time 0.0011 (0.0022) model time 0.3576 (0.3739) loss 2.3025 (2.7755) grad_norm 2.3230 (2.4505) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][420/625] eta 0:01:17 lr 0.000245 wd 0.0500 time 0.3697 (0.3761) data time 0.0012 (0.0022) model time 0.3685 (0.3744) loss 2.9518 (2.7716) grad_norm 2.3180 (2.4503) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][430/625] eta 0:01:13 lr 0.000245 wd 0.0500 time 0.3700 (0.3759) data time 0.0009 (0.0022) model time 0.3691 (0.3743) loss 3.6260 (2.7724) grad_norm 1.6987 (2.4474) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:17:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][440/625] eta 0:01:09 lr 0.000245 wd 0.0500 time 0.3743 (0.3758) data time 0.0008 (0.0022) model time 0.3735 (0.3742) loss 2.5654 (2.7723) grad_norm 2.9254 (2.4498) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][450/625] eta 0:01:05 lr 0.000245 wd 0.0500 time 0.3643 (0.3757) data time 0.0008 (0.0021) model time 0.3635 (0.3741) loss 3.0579 (2.7732) grad_norm 1.8288 (2.4470) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][460/625] eta 0:01:01 lr 0.000245 wd 0.0500 time 0.3678 (0.3756) data time 0.0013 (0.0021) model time 0.3666 (0.3740) loss 2.7126 (2.7763) grad_norm 1.8379 (2.4442) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][470/625] eta 0:00:58 lr 0.000245 wd 0.0500 time 0.3715 (0.3756) data time 0.0011 (0.0021) model time 0.3703 (0.3739) loss 3.3077 (2.7762) grad_norm 1.7471 (2.4369) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][480/625] eta 0:00:54 lr 0.000245 wd 0.0500 time 0.3781 (0.3755) data time 0.0013 (0.0021) model time 0.3767 (0.3738) loss 3.1939 (2.7744) grad_norm 3.3230 (2.4385) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][490/625] eta 0:00:50 lr 0.000244 wd 0.0500 time 0.3771 (0.3766) data time 0.0012 (0.0021) model time 0.3759 (0.3751) loss 2.9782 (2.7742) grad_norm 2.7792 (2.4417) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][500/625] eta 0:00:47 lr 0.000244 wd 0.0500 time 0.3692 (0.3765) data time 0.0010 (0.0021) model time 0.3683 (0.3750) loss 3.0669 (2.7724) grad_norm 2.2437 (2.4532) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][510/625] eta 0:00:43 lr 0.000244 wd 0.0500 time 0.3748 (0.3764) data time 0.0013 (0.0020) model time 0.3735 (0.3749) loss 3.0931 (2.7730) grad_norm 3.8008 (2.4793) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][520/625] eta 0:00:39 lr 0.000244 wd 0.0500 time 0.3773 (0.3763) data time 0.0009 (0.0020) model time 0.3764 (0.3748) loss 3.2672 (2.7775) grad_norm 2.0883 (2.4825) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][530/625] eta 0:00:35 lr 0.000244 wd 0.0500 time 0.3747 (0.3764) data time 0.0009 (0.0020) model time 0.3738 (0.3748) loss 2.4345 (2.7729) grad_norm 1.9877 (2.4831) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][540/625] eta 0:00:31 lr 0.000244 wd 0.0500 time 0.3812 (0.3763) data time 0.0011 (0.0020) model time 0.3801 (0.3748) loss 3.1325 (2.7758) grad_norm 2.6170 (2.4771) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][550/625] eta 0:00:28 lr 0.000244 wd 0.0500 time 0.3723 (0.3762) data time 0.0009 (0.0020) model time 0.3715 (0.3747) loss 3.4959 (2.7775) grad_norm 2.4758 (2.4724) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][560/625] eta 0:00:24 lr 0.000244 wd 0.0500 time 0.3642 (0.3761) data time 0.0009 (0.0020) model time 0.3633 (0.3746) loss 2.9402 (2.7801) grad_norm 2.5313 (2.4657) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][570/625] eta 0:00:20 lr 0.000244 wd 0.0500 time 0.3710 (0.3761) data time 0.0008 (0.0019) model time 0.3702 (0.3746) loss 3.5701 (2.7796) grad_norm 2.4371 (2.4641) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][580/625] eta 0:00:16 lr 0.000243 wd 0.0500 time 0.3668 (0.3760) data time 0.0011 (0.0019) model time 0.3657 (0.3745) loss 2.2051 (2.7808) grad_norm 2.0415 (2.4576) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][590/625] eta 0:00:13 lr 0.000243 wd 0.0500 time 0.3678 (0.3759) data time 0.0011 (0.0019) model time 0.3667 (0.3744) loss 2.7608 (2.7849) grad_norm 3.3032 (2.4528) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:18:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][600/625] eta 0:00:09 lr 0.000243 wd 0.0500 time 0.6057 (0.3763) data time 0.0009 (0.0019) model time 0.6048 (0.3748) loss 2.1810 (2.7830) grad_norm 2.0063 (2.4488) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][610/625] eta 0:00:05 lr 0.000243 wd 0.0500 time 0.3523 (0.3768) data time 0.0008 (0.0019) model time 0.3515 (0.3753) loss 2.2626 (2.7816) grad_norm 1.9426 (2.4490) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [238/300][620/625] eta 0:00:01 lr 0.000243 wd 0.0500 time 0.3601 (0.3765) data time 0.0008 (0.0019) model time 0.3593 (0.3750) loss 3.3606 (2.7799) grad_norm 1.6874 (2.4393) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:06 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 238 training takes 0:03:55 [2024-09-01 11:19:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:19:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:19:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.518 (0.518) Loss 0.5273 (0.5273) Acc@1 88.574 (88.574) Acc@5 98.389 (98.389) Mem 11225MB [2024-09-01 11:19:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 0.8618 (0.6396) Acc@1 79.834 (86.004) Acc@5 95.947 (97.643) Mem 11225MB [2024-09-01 11:19:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9155 (0.7552) Acc@1 78.369 (83.201) Acc@5 95.068 (96.422) Mem 11225MB [2024-09-01 11:19:10 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 82.947 Acc@5 96.435 [2024-09-01 11:19:10 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 82.9% [2024-09-01 11:19:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.830 (0.830) Loss 0.4888 (0.4888) Acc@1 89.502 (89.502) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 11:19:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.173) Loss 0.8013 (0.5930) Acc@1 80.664 (86.941) Acc@5 96.094 (97.825) Mem 11225MB [2024-09-01 11:19:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.8750 (0.7022) Acc@1 79.248 (84.159) Acc@5 95.459 (96.722) Mem 11225MB [2024-09-01 11:19:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.851 Acc@5 96.747 [2024-09-01 11:19:13 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:19:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][0/625] eta 0:12:43 lr 0.000243 wd 0.0500 time 1.2220 (1.2220) data time 0.4707 (0.4707) model time 0.0000 (0.0000) loss 2.7432 (2.7432) grad_norm 1.4737 (1.4737) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][10/625] eta 0:04:35 lr 0.000243 wd 0.0500 time 0.3675 (0.4487) data time 0.0011 (0.0439) model time 0.0000 (0.0000) loss 3.0417 (2.8175) grad_norm 2.0286 (1.9821) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][20/625] eta 0:04:09 lr 0.000243 wd 0.0500 time 0.3678 (0.4121) data time 0.0009 (0.0238) model time 0.0000 (0.0000) loss 2.3099 (2.7430) grad_norm 1.8768 (2.0666) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][30/625] eta 0:03:56 lr 0.000243 wd 0.0500 time 0.3670 (0.3982) data time 0.0011 (0.0165) model time 0.0000 (0.0000) loss 2.8697 (2.7922) grad_norm 2.9141 (2.2948) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][40/625] eta 0:03:49 lr 0.000243 wd 0.0500 time 0.3707 (0.3917) data time 0.0011 (0.0128) model time 0.0000 (0.0000) loss 2.8691 (2.7596) grad_norm 2.5218 (2.2973) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][50/625] eta 0:03:43 lr 0.000242 wd 0.0500 time 0.3772 (0.3880) data time 0.0008 (0.0106) model time 0.0000 (0.0000) loss 2.1666 (2.7620) grad_norm 3.2891 (2.4562) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][60/625] eta 0:03:37 lr 0.000242 wd 0.0500 time 0.3773 (0.3853) data time 0.0009 (0.0090) model time 0.3764 (0.3702) loss 2.7926 (2.7556) grad_norm 3.6240 (2.4490) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][70/625] eta 0:03:32 lr 0.000242 wd 0.0500 time 0.3661 (0.3832) data time 0.0009 (0.0079) model time 0.3652 (0.3696) loss 2.1067 (2.7498) grad_norm 2.0297 (2.4204) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][80/625] eta 0:03:27 lr 0.000242 wd 0.0500 time 0.3663 (0.3816) data time 0.0010 (0.0071) model time 0.3653 (0.3694) loss 2.9552 (2.7486) grad_norm 1.5544 (2.3708) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][90/625] eta 0:03:23 lr 0.000242 wd 0.0500 time 0.3709 (0.3808) data time 0.0011 (0.0065) model time 0.3698 (0.3703) loss 2.8914 (2.7366) grad_norm 2.1655 (2.3345) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][100/625] eta 0:03:19 lr 0.000242 wd 0.0500 time 0.3710 (0.3802) data time 0.0012 (0.0059) model time 0.3698 (0.3710) loss 2.7567 (2.7471) grad_norm 2.3635 (2.3451) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][110/625] eta 0:03:15 lr 0.000242 wd 0.0500 time 0.3666 (0.3793) data time 0.0012 (0.0055) model time 0.3654 (0.3706) loss 2.8364 (2.7470) grad_norm 1.8596 (2.3234) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:19:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][120/625] eta 0:03:11 lr 0.000242 wd 0.0500 time 0.3676 (0.3800) data time 0.0011 (0.0052) model time 0.3664 (0.3729) loss 3.1795 (2.7545) grad_norm 1.7411 (2.3169) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][130/625] eta 0:03:07 lr 0.000242 wd 0.0500 time 0.3648 (0.3794) data time 0.0011 (0.0049) model time 0.3637 (0.3727) loss 2.8691 (2.7550) grad_norm 1.5197 (2.2892) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][140/625] eta 0:03:03 lr 0.000241 wd 0.0500 time 0.3712 (0.3789) data time 0.0009 (0.0046) model time 0.3703 (0.3726) loss 3.2693 (2.7515) grad_norm 1.6615 (2.2618) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][150/625] eta 0:02:59 lr 0.000241 wd 0.0500 time 0.3704 (0.3784) data time 0.0009 (0.0044) model time 0.3695 (0.3723) loss 2.2692 (2.7390) grad_norm 1.9097 (2.2565) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][160/625] eta 0:02:55 lr 0.000241 wd 0.0500 time 0.3724 (0.3780) data time 0.0014 (0.0042) model time 0.3710 (0.3721) loss 2.8411 (2.7451) grad_norm 2.0226 (2.2430) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][170/625] eta 0:02:52 lr 0.000241 wd 0.0500 time 0.3652 (0.3799) data time 0.0011 (0.0040) model time 0.3641 (0.3753) loss 2.6975 (2.7513) grad_norm 2.4174 (2.2381) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][180/625] eta 0:02:48 lr 0.000241 wd 0.0500 time 0.3701 (0.3794) data time 0.0011 (0.0039) model time 0.3690 (0.3748) loss 3.3399 (2.7554) grad_norm 3.1975 (2.3051) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][190/625] eta 0:02:44 lr 0.000241 wd 0.0500 time 0.3724 (0.3790) data time 0.0010 (0.0037) model time 0.3714 (0.3745) loss 3.1630 (2.7619) grad_norm 2.8065 (2.3259) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][200/625] eta 0:02:40 lr 0.000241 wd 0.0500 time 0.3665 (0.3788) data time 0.0012 (0.0036) model time 0.3653 (0.3744) loss 3.0154 (2.7558) grad_norm 2.1749 (2.3990) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][210/625] eta 0:02:37 lr 0.000241 wd 0.0500 time 0.3676 (0.3786) data time 0.0009 (0.0035) model time 0.3667 (0.3743) loss 1.8896 (2.7480) grad_norm 2.2053 (2.4056) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][220/625] eta 0:02:33 lr 0.000240 wd 0.0500 time 0.3728 (0.3783) data time 0.0012 (0.0034) model time 0.3716 (0.3742) loss 3.2643 (2.7471) grad_norm 2.4884 (2.4162) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][230/625] eta 0:02:29 lr 0.000240 wd 0.0500 time 0.3780 (0.3781) data time 0.0009 (0.0033) model time 0.3771 (0.3741) loss 2.2006 (2.7469) grad_norm 1.7710 (2.4031) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][240/625] eta 0:02:25 lr 0.000240 wd 0.0500 time 0.3696 (0.3787) data time 0.0013 (0.0032) model time 0.3683 (0.3750) loss 2.9849 (2.7456) grad_norm 2.3247 (2.4180) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][250/625] eta 0:02:21 lr 0.000240 wd 0.0500 time 0.3637 (0.3784) data time 0.0011 (0.0031) model time 0.3626 (0.3747) loss 2.3590 (2.7417) grad_norm 2.9806 (2.4159) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][260/625] eta 0:02:18 lr 0.000240 wd 0.0500 time 0.3676 (0.3782) data time 0.0010 (0.0030) model time 0.3666 (0.3746) loss 3.1099 (2.7432) grad_norm 2.8130 (2.4224) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][270/625] eta 0:02:14 lr 0.000240 wd 0.0500 time 0.3676 (0.3779) data time 0.0009 (0.0030) model time 0.3667 (0.3744) loss 3.0867 (2.7459) grad_norm 3.6968 (2.4508) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:20:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][280/625] eta 0:02:10 lr 0.000240 wd 0.0500 time 0.3738 (0.3778) data time 0.0011 (0.0029) model time 0.3727 (0.3743) loss 2.4453 (2.7463) grad_norm 2.2642 (2.5088) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][290/625] eta 0:02:06 lr 0.000240 wd 0.0500 time 0.3684 (0.3790) data time 0.0010 (0.0028) model time 0.3674 (0.3759) loss 2.1178 (2.7385) grad_norm 2.6905 (2.5022) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][300/625] eta 0:02:03 lr 0.000240 wd 0.0500 time 0.3751 (0.3787) data time 0.0009 (0.0028) model time 0.3742 (0.3756) loss 3.4023 (2.7415) grad_norm 1.7520 (2.4797) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][310/625] eta 0:01:59 lr 0.000239 wd 0.0500 time 0.3691 (0.3785) data time 0.0009 (0.0027) model time 0.3683 (0.3755) loss 2.9089 (2.7396) grad_norm 2.2313 (2.4705) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][320/625] eta 0:01:55 lr 0.000239 wd 0.0500 time 0.3850 (0.3783) data time 0.0009 (0.0027) model time 0.3841 (0.3753) loss 2.9446 (2.7379) grad_norm 3.1132 (2.5042) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][330/625] eta 0:01:51 lr 0.000239 wd 0.0500 time 0.3824 (0.3782) data time 0.0012 (0.0027) model time 0.3813 (0.3752) loss 2.5670 (2.7325) grad_norm 3.4271 (2.5062) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][340/625] eta 0:01:47 lr 0.000239 wd 0.0500 time 0.3808 (0.3781) data time 0.0011 (0.0026) model time 0.3797 (0.3751) loss 3.3230 (2.7337) grad_norm 3.2069 (2.4971) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][350/625] eta 0:01:43 lr 0.000239 wd 0.0500 time 0.3779 (0.3780) data time 0.0011 (0.0026) model time 0.3768 (0.3751) loss 3.1490 (2.7375) grad_norm 2.2764 (2.4897) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][360/625] eta 0:01:40 lr 0.000239 wd 0.0500 time 0.3713 (0.3779) data time 0.0009 (0.0025) model time 0.3705 (0.3750) loss 1.8537 (2.7409) grad_norm 4.2140 (2.4880) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][370/625] eta 0:01:36 lr 0.000239 wd 0.0500 time 0.3666 (0.3777) data time 0.0011 (0.0025) model time 0.3654 (0.3748) loss 2.1954 (2.7336) grad_norm 1.9840 (2.5008) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][380/625] eta 0:01:32 lr 0.000239 wd 0.0500 time 0.3816 (0.3776) data time 0.0011 (0.0025) model time 0.3805 (0.3747) loss 3.5683 (2.7341) grad_norm 2.0169 (2.4978) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][390/625] eta 0:01:28 lr 0.000239 wd 0.0500 time 0.3673 (0.3775) data time 0.0010 (0.0024) model time 0.3664 (0.3747) loss 2.7840 (2.7373) grad_norm 1.9908 (2.4820) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][400/625] eta 0:01:24 lr 0.000238 wd 0.0500 time 0.3673 (0.3774) data time 0.0012 (0.0024) model time 0.3661 (0.3746) loss 2.7255 (2.7352) grad_norm 1.9621 (2.4730) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][410/625] eta 0:01:21 lr 0.000238 wd 0.0500 time 0.3731 (0.3774) data time 0.0011 (0.0024) model time 0.3719 (0.3746) loss 2.8035 (2.7390) grad_norm 2.0788 (2.4717) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][420/625] eta 0:01:17 lr 0.000238 wd 0.0500 time 0.3608 (0.3772) data time 0.0010 (0.0023) model time 0.3598 (0.3746) loss 2.9375 (2.7374) grad_norm 2.4676 (2.4682) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][430/625] eta 0:01:13 lr 0.000238 wd 0.0500 time 0.3785 (0.3772) data time 0.0010 (0.0023) model time 0.3775 (0.3745) loss 1.9246 (2.7293) grad_norm 2.4380 (2.4598) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:21:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][440/625] eta 0:01:09 lr 0.000238 wd 0.0500 time 0.3642 (0.3770) data time 0.0009 (0.0023) model time 0.3633 (0.3744) loss 2.8715 (2.7308) grad_norm 1.9463 (2.4552) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][450/625] eta 0:01:06 lr 0.000238 wd 0.0500 time 0.3744 (0.3774) data time 0.0011 (0.0023) model time 0.3733 (0.3748) loss 3.1630 (2.7294) grad_norm 2.2166 (2.4756) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][460/625] eta 0:01:02 lr 0.000238 wd 0.0500 time 0.3694 (0.3773) data time 0.0013 (0.0022) model time 0.3681 (0.3748) loss 2.5749 (2.7256) grad_norm 1.9477 (2.4686) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][470/625] eta 0:00:58 lr 0.000238 wd 0.0500 time 0.3864 (0.3773) data time 0.0009 (0.0022) model time 0.3855 (0.3748) loss 1.6705 (2.7224) grad_norm 4.0336 (2.4742) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][480/625] eta 0:00:54 lr 0.000238 wd 0.0500 time 0.3724 (0.3772) data time 0.0009 (0.0022) model time 0.3716 (0.3747) loss 2.0475 (2.7224) grad_norm 4.3208 (2.5047) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][490/625] eta 0:00:50 lr 0.000237 wd 0.0500 time 0.3710 (0.3770) data time 0.0009 (0.0022) model time 0.3701 (0.3746) loss 3.0900 (2.7225) grad_norm 2.6796 (2.5073) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][500/625] eta 0:00:47 lr 0.000237 wd 0.0500 time 0.3678 (0.3781) data time 0.0011 (0.0022) model time 0.3667 (0.3757) loss 2.6575 (2.7285) grad_norm 1.6465 (2.5106) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][510/625] eta 0:00:43 lr 0.000237 wd 0.0500 time 0.3724 (0.3780) data time 0.0009 (0.0021) model time 0.3715 (0.3757) loss 1.7398 (2.7290) grad_norm 1.8469 (2.5046) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][520/625] eta 0:00:39 lr 0.000237 wd 0.0500 time 0.3688 (0.3779) data time 0.0013 (0.0021) model time 0.3675 (0.3756) loss 3.2407 (2.7300) grad_norm 2.2528 (2.5061) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][530/625] eta 0:00:35 lr 0.000237 wd 0.0500 time 0.3897 (0.3778) data time 0.0011 (0.0021) model time 0.3886 (0.3756) loss 2.8765 (2.7296) grad_norm 1.9350 (2.4995) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][540/625] eta 0:00:32 lr 0.000237 wd 0.0500 time 0.3765 (0.3777) data time 0.0008 (0.0021) model time 0.3757 (0.3754) loss 2.0594 (2.7300) grad_norm 5.5780 (2.4962) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][550/625] eta 0:00:28 lr 0.000237 wd 0.0500 time 0.3834 (0.3777) data time 0.0011 (0.0021) model time 0.3823 (0.3754) loss 3.0120 (2.7348) grad_norm 3.1615 (2.4939) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][560/625] eta 0:00:24 lr 0.000237 wd 0.0500 time 0.3724 (0.3776) data time 0.0011 (0.0021) model time 0.3712 (0.3753) loss 3.1059 (2.7400) grad_norm 1.8875 (2.4888) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][570/625] eta 0:00:20 lr 0.000237 wd 0.0500 time 0.3670 (0.3775) data time 0.0009 (0.0020) model time 0.3661 (0.3753) loss 1.8306 (2.7370) grad_norm 1.9437 (2.4961) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][580/625] eta 0:00:16 lr 0.000236 wd 0.0500 time 0.3704 (0.3774) data time 0.0010 (0.0020) model time 0.3694 (0.3752) loss 2.1812 (2.7391) grad_norm 1.5345 (2.4906) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:22:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][590/625] eta 0:00:13 lr 0.000236 wd 0.0500 time 0.3664 (0.3773) data time 0.0010 (0.0020) model time 0.3655 (0.3751) loss 3.5067 (2.7392) grad_norm 1.4179 (2.4948) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:23:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][600/625] eta 0:00:09 lr 0.000236 wd 0.0500 time 0.3654 (0.3772) data time 0.0009 (0.0020) model time 0.3645 (0.3750) loss 3.1472 (2.7427) grad_norm 1.6069 (2.4907) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:23:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][610/625] eta 0:00:05 lr 0.000236 wd 0.0500 time 0.3557 (0.3771) data time 0.0006 (0.0020) model time 0.3552 (0.3749) loss 1.9451 (2.7421) grad_norm 2.8569 (2.5085) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:23:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [239/300][620/625] eta 0:00:01 lr 0.000236 wd 0.0500 time 0.3608 (0.3769) data time 0.0006 (0.0020) model time 0.3602 (0.3747) loss 2.6681 (2.7419) grad_norm 3.0723 (2.5099) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 239 training takes 0:03:55 [2024-09-01 11:23:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:23:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:23:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.502 (0.502) Loss 0.5142 (0.5142) Acc@1 88.379 (88.379) Acc@5 98.779 (98.779) Mem 11225MB [2024-09-01 11:23:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.142) Loss 0.8579 (0.6294) Acc@1 80.273 (86.226) Acc@5 95.410 (97.643) Mem 11225MB [2024-09-01 11:23:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9253 (0.7464) Acc@1 78.174 (83.261) Acc@5 94.775 (96.394) Mem 11225MB [2024-09-01 11:23:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.011 Acc@5 96.431 [2024-09-01 11:23:13 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.0% [2024-09-01 11:23:13 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.01% [2024-09-01 11:23:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 11:23:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 11:23:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.488 (0.488) Loss 0.4885 (0.4885) Acc@1 89.551 (89.551) Acc@5 98.584 (98.584) Mem 11225MB [2024-09-01 11:23:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.138) Loss 0.8013 (0.5928) Acc@1 80.811 (86.972) Acc@5 95.996 (97.820) Mem 11225MB [2024-09-01 11:23:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.100 (0.117) Loss 0.8726 (0.7020) Acc@1 79.248 (84.203) Acc@5 95.264 (96.729) Mem 11225MB [2024-09-01 11:23:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.895 Acc@5 96.761 [2024-09-01 11:23:17 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:23:17 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.90% [2024-09-01 11:23:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 11:23:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 11:23:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][0/625] eta 0:08:28 lr 0.000236 wd 0.0500 time 0.8133 (0.8133) data time 0.4742 (0.4742) model time 0.0000 (0.0000) loss 2.2964 (2.2964) grad_norm 2.7291 (2.7291) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:23:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][10/625] eta 0:04:13 lr 0.000236 wd 0.0500 time 0.3860 (0.4122) data time 0.0011 (0.0443) model time 0.0000 (0.0000) loss 3.0555 (2.8410) grad_norm 3.2137 (2.3974) loss_scale 128.0000 (128.0000) mem 11225MB [2024-09-01 11:23:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][20/625] eta 0:03:57 lr 0.000236 wd 0.0500 time 0.3728 (0.3918) data time 0.0009 (0.0238) model time 0.0000 (0.0000) loss 2.5598 (2.7074) grad_norm 1.8796 (2.3859) loss_scale 256.0000 (158.4762) mem 11225MB [2024-09-01 11:23:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][30/625] eta 0:03:49 lr 0.000236 wd 0.0500 time 0.3675 (0.3852) data time 0.0009 (0.0166) model time 0.0000 (0.0000) loss 1.9668 (2.6758) grad_norm 2.4303 (2.4346) loss_scale 256.0000 (189.9355) mem 11225MB [2024-09-01 11:23:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][40/625] eta 0:03:43 lr 0.000236 wd 0.0500 time 0.3732 (0.3821) data time 0.0009 (0.0128) model time 0.0000 (0.0000) loss 2.2303 (2.5602) grad_norm 1.9052 (2.3795) loss_scale 256.0000 (206.0488) mem 11225MB [2024-09-01 11:23:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][50/625] eta 0:03:38 lr 0.000235 wd 0.0500 time 0.3902 (0.3802) data time 0.0013 (0.0106) model time 0.0000 (0.0000) loss 3.0691 (2.6431) grad_norm 2.2481 (2.5793) loss_scale 256.0000 (215.8431) mem 11225MB [2024-09-01 11:23:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][60/625] eta 0:03:33 lr 0.000235 wd 0.0500 time 0.3614 (0.3784) data time 0.0011 (0.0090) model time 0.3604 (0.3681) loss 2.9504 (2.6782) grad_norm 3.2875 (2.6268) loss_scale 256.0000 (222.4262) mem 11225MB [2024-09-01 11:23:42 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 11:23:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:23:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:28:16 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 11:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 11:28:31 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 11:28:43 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 11:28:43 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 11:28:46 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 11:28:48 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 11:28:48 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 240) [2024-09-01 11:28:48 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 11:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][70/625] eta 0:18:50 lr 0.000235 wd 0.0500 time 0.3828 (2.0369) data time 0.0008 (0.0927) model time 0.3820 (1.9442) loss 2.8502 (3.0943) grad_norm 2.7645 (2.5744) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 11:29:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:29:17 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:30:53 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 11:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 11:31:07 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 11:31:19 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 11:31:19 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 11:31:21 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 11:31:23 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 11:31:23 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 240) [2024-09-01 11:31:23 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 11:31:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][80/625] eta 0:18:06 lr 0.000235 wd 0.0500 time 0.3422 (1.9930) data time 0.0006 (0.0820) model time 0.3415 (1.9110) loss 2.9716 (3.0438) grad_norm 2.4244 (2.2226) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:31:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][90/625] eta 0:09:36 lr 0.000235 wd 0.0500 time 0.3430 (1.0784) data time 0.0006 (0.0370) model time 0.3424 (1.0414) loss 3.4388 (2.9529) grad_norm 2.2699 (2.4234) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:31:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][100/625] eta 0:07:13 lr 0.000235 wd 0.0500 time 0.3577 (0.8257) data time 0.0008 (0.0241) model time 0.3569 (0.8016) loss 2.9148 (2.9735) grad_norm 1.7099 (2.3370) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:31:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][110/625] eta 0:06:00 lr 0.000235 wd 0.0500 time 0.3459 (0.6997) data time 0.0008 (0.0180) model time 0.3451 (0.6817) loss 2.8610 (2.9126) grad_norm 2.4997 (2.2412) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][120/625] eta 0:05:19 lr 0.000235 wd 0.0500 time 0.3465 (0.6318) data time 0.0007 (0.0145) model time 0.3458 (0.6173) loss 3.0627 (2.8758) grad_norm 2.7815 (2.1979) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][130/625] eta 0:04:48 lr 0.000235 wd 0.0500 time 0.3403 (0.5822) data time 0.0006 (0.0121) model time 0.3397 (0.5701) loss 2.3319 (2.8581) grad_norm 2.6449 (2.2636) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][140/625] eta 0:04:25 lr 0.000234 wd 0.0500 time 0.3445 (0.5474) data time 0.0007 (0.0105) model time 0.3438 (0.5369) loss 2.0774 (2.8349) grad_norm 1.8121 (2.2768) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][150/625] eta 0:04:07 lr 0.000234 wd 0.0500 time 0.3409 (0.5215) data time 0.0006 (0.0093) model time 0.3403 (0.5123) loss 2.3053 (2.8136) grad_norm 1.8415 (2.2676) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][160/625] eta 0:03:53 lr 0.000234 wd 0.0500 time 0.3498 (0.5019) data time 0.0009 (0.0083) model time 0.3490 (0.4935) loss 2.7423 (2.7765) grad_norm 1.8254 (2.3101) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][170/625] eta 0:03:41 lr 0.000234 wd 0.0500 time 0.3434 (0.4862) data time 0.0006 (0.0076) model time 0.3428 (0.4786) loss 2.9171 (2.7892) grad_norm 2.7290 (2.3662) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][180/625] eta 0:03:30 lr 0.000234 wd 0.0500 time 0.3494 (0.4734) data time 0.0006 (0.0070) model time 0.3488 (0.4664) loss 2.1368 (2.8012) grad_norm 1.6641 (2.3382) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][190/625] eta 0:03:21 lr 0.000234 wd 0.0500 time 0.3490 (0.4627) data time 0.0008 (0.0064) model time 0.3482 (0.4562) loss 2.8315 (2.8065) grad_norm 4.1753 (2.3661) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][200/625] eta 0:03:12 lr 0.000234 wd 0.0500 time 0.3476 (0.4536) data time 0.0007 (0.0060) model time 0.3469 (0.4476) loss 3.2194 (2.8013) grad_norm 2.3081 (2.3940) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][210/625] eta 0:03:05 lr 0.000234 wd 0.0500 time 0.3521 (0.4459) data time 0.0008 (0.0056) model time 0.3513 (0.4402) loss 2.6711 (2.7995) grad_norm 2.3064 (2.4350) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][220/625] eta 0:02:57 lr 0.000234 wd 0.0500 time 0.3537 (0.4394) data time 0.0007 (0.0053) model time 0.3531 (0.4340) loss 2.9745 (2.8000) grad_norm 6.5863 (2.4663) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][230/625] eta 0:02:51 lr 0.000233 wd 0.0500 time 0.3531 (0.4335) data time 0.0006 (0.0050) model time 0.3525 (0.4285) loss 2.5473 (2.7971) grad_norm 2.1869 (2.5029) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][240/625] eta 0:02:44 lr 0.000233 wd 0.0500 time 0.3466 (0.4284) data time 0.0009 (0.0048) model time 0.3457 (0.4236) loss 3.1415 (2.8037) grad_norm 2.8995 (2.5063) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][250/625] eta 0:02:38 lr 0.000233 wd 0.0500 time 0.3432 (0.4239) data time 0.0007 (0.0046) model time 0.3425 (0.4193) loss 2.3723 (2.7861) grad_norm 2.8916 (2.6162) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][260/625] eta 0:02:33 lr 0.000233 wd 0.0500 time 0.3472 (0.4198) data time 0.0007 (0.0044) model time 0.3465 (0.4154) loss 2.9451 (2.7796) grad_norm 2.8072 (2.6107) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][270/625] eta 0:02:27 lr 0.000233 wd 0.0500 time 0.3426 (0.4160) data time 0.0009 (0.0042) model time 0.3417 (0.4118) loss 2.5141 (2.7698) grad_norm 2.2196 (2.5920) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][280/625] eta 0:02:22 lr 0.000233 wd 0.0500 time 0.3478 (0.4127) data time 0.0009 (0.0041) model time 0.3470 (0.4087) loss 3.4383 (2.7604) grad_norm 2.3091 (2.5569) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:32:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][290/625] eta 0:02:17 lr 0.000233 wd 0.0500 time 0.3484 (0.4097) data time 0.0009 (0.0039) model time 0.3475 (0.4058) loss 2.2406 (2.7556) grad_norm 2.9283 (2.5498) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][300/625] eta 0:02:12 lr 0.000233 wd 0.0500 time 0.3484 (0.4079) data time 0.0008 (0.0038) model time 0.3476 (0.4042) loss 2.7824 (2.7597) grad_norm 1.7475 (2.5258) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][310/625] eta 0:02:07 lr 0.000233 wd 0.0500 time 0.3551 (0.4062) data time 0.0006 (0.0037) model time 0.3546 (0.4026) loss 3.2248 (2.7601) grad_norm 1.4770 (2.5018) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][320/625] eta 0:02:03 lr 0.000232 wd 0.0500 time 0.3519 (0.4039) data time 0.0009 (0.0035) model time 0.3510 (0.4003) loss 2.3442 (2.7560) grad_norm 2.3170 (2.4857) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][330/625] eta 0:01:58 lr 0.000232 wd 0.0500 time 0.3459 (0.4017) data time 0.0008 (0.0034) model time 0.3450 (0.3983) loss 2.6168 (2.7503) grad_norm 2.3587 (2.4727) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][340/625] eta 0:01:53 lr 0.000232 wd 0.0500 time 0.3475 (0.3997) data time 0.0006 (0.0033) model time 0.3469 (0.3964) loss 3.0099 (2.7470) grad_norm 1.4838 (2.4596) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][350/625] eta 0:01:49 lr 0.000232 wd 0.0500 time 0.3485 (0.3978) data time 0.0009 (0.0033) model time 0.3476 (0.3946) loss 2.1137 (2.7520) grad_norm 2.0993 (2.4558) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][360/625] eta 0:01:44 lr 0.000232 wd 0.0500 time 0.3448 (0.3960) data time 0.0006 (0.0032) model time 0.3441 (0.3928) loss 3.8297 (2.7565) grad_norm 2.3617 (2.4584) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][370/625] eta 0:01:40 lr 0.000232 wd 0.0500 time 0.3449 (0.3943) data time 0.0009 (0.0031) model time 0.3440 (0.3912) loss 2.3584 (2.7430) grad_norm 2.1391 (2.4618) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][380/625] eta 0:01:36 lr 0.000232 wd 0.0500 time 0.3416 (0.3928) data time 0.0006 (0.0030) model time 0.3410 (0.3897) loss 2.4060 (2.7389) grad_norm 1.8587 (2.4838) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][390/625] eta 0:01:31 lr 0.000232 wd 0.0500 time 0.3469 (0.3913) data time 0.0008 (0.0030) model time 0.3461 (0.3883) loss 2.7930 (2.7491) grad_norm 2.0705 (2.5134) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][400/625] eta 0:01:27 lr 0.000232 wd 0.0500 time 0.3484 (0.3900) data time 0.0007 (0.0029) model time 0.3477 (0.3871) loss 3.0898 (2.7575) grad_norm 2.3919 (2.5022) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][410/625] eta 0:01:23 lr 0.000231 wd 0.0500 time 0.3465 (0.3888) data time 0.0008 (0.0029) model time 0.3457 (0.3859) loss 3.1339 (2.7555) grad_norm 2.3590 (2.4885) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][420/625] eta 0:01:19 lr 0.000231 wd 0.0500 time 0.3463 (0.3881) data time 0.0006 (0.0028) model time 0.3457 (0.3853) loss 2.7728 (2.7565) grad_norm 2.3976 (2.5168) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][430/625] eta 0:01:15 lr 0.000231 wd 0.0500 time 0.3451 (0.3869) data time 0.0009 (0.0027) model time 0.3442 (0.3842) loss 2.9871 (2.7570) grad_norm 1.5664 (2.5394) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][440/625] eta 0:01:11 lr 0.000231 wd 0.0500 time 0.3448 (0.3858) data time 0.0007 (0.0027) model time 0.3442 (0.3831) loss 2.5636 (2.7555) grad_norm 2.8391 (2.5258) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][450/625] eta 0:01:07 lr 0.000231 wd 0.0500 time 0.3402 (0.3848) data time 0.0009 (0.0027) model time 0.3392 (0.3822) loss 3.0693 (2.7533) grad_norm 3.4860 (2.5378) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:33:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][460/625] eta 0:01:03 lr 0.000231 wd 0.0500 time 0.3529 (0.3838) data time 0.0007 (0.0026) model time 0.3521 (0.3812) loss 2.1896 (2.7481) grad_norm 3.4410 (2.5438) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][470/625] eta 0:00:59 lr 0.000231 wd 0.0500 time 0.3495 (0.3829) data time 0.0007 (0.0026) model time 0.3489 (0.3803) loss 2.9109 (2.7510) grad_norm 2.4899 (2.5373) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][480/625] eta 0:00:55 lr 0.000231 wd 0.0500 time 0.3520 (0.3820) data time 0.0008 (0.0025) model time 0.3511 (0.3795) loss 3.0521 (2.7575) grad_norm 1.7291 (2.5297) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][490/625] eta 0:00:51 lr 0.000231 wd 0.0500 time 0.3421 (0.3816) data time 0.0007 (0.0025) model time 0.3414 (0.3791) loss 3.0445 (2.7562) grad_norm 2.9955 (2.5395) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][500/625] eta 0:00:47 lr 0.000230 wd 0.0500 time 0.3457 (0.3808) data time 0.0007 (0.0025) model time 0.3450 (0.3784) loss 3.4220 (2.7622) grad_norm 2.6869 (2.5451) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][510/625] eta 0:00:43 lr 0.000230 wd 0.0500 time 0.3473 (0.3800) data time 0.0008 (0.0024) model time 0.3465 (0.3776) loss 3.1667 (2.7662) grad_norm 2.0790 (2.5553) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][520/625] eta 0:00:39 lr 0.000230 wd 0.0500 time 0.3457 (0.3793) data time 0.0010 (0.0024) model time 0.3447 (0.3769) loss 2.3234 (2.7655) grad_norm 2.0184 (2.5496) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][530/625] eta 0:00:35 lr 0.000230 wd 0.0500 time 0.3458 (0.3786) data time 0.0007 (0.0024) model time 0.3452 (0.3763) loss 2.2444 (2.7606) grad_norm 2.6496 (2.5623) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][540/625] eta 0:00:32 lr 0.000230 wd 0.0500 time 0.3513 (0.3780) data time 0.0009 (0.0023) model time 0.3504 (0.3756) loss 2.3457 (2.7571) grad_norm 2.5272 (2.5892) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][550/625] eta 0:00:28 lr 0.000230 wd 0.0500 time 0.3459 (0.3773) data time 0.0008 (0.0023) model time 0.3451 (0.3750) loss 1.9815 (2.7547) grad_norm 2.9824 (2.5862) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][560/625] eta 0:00:24 lr 0.000230 wd 0.0500 time 0.3493 (0.3767) data time 0.0009 (0.0023) model time 0.3484 (0.3744) loss 2.9951 (2.7616) grad_norm 1.7150 (2.5848) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][570/625] eta 0:00:20 lr 0.000230 wd 0.0500 time 0.3426 (0.3760) data time 0.0007 (0.0022) model time 0.3419 (0.3738) loss 3.2727 (2.7618) grad_norm 2.4632 (2.5842) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][580/625] eta 0:00:16 lr 0.000230 wd 0.0500 time 0.3492 (0.3755) data time 0.0007 (0.0022) model time 0.3485 (0.3733) loss 3.5791 (2.7634) grad_norm 2.1167 (2.5969) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][590/625] eta 0:00:13 lr 0.000229 wd 0.0500 time 0.3468 (0.3749) data time 0.0008 (0.0022) model time 0.3460 (0.3727) loss 2.7628 (2.7667) grad_norm 2.8285 (2.6323) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][600/625] eta 0:00:09 lr 0.000229 wd 0.0500 time 0.3461 (0.3743) data time 0.0006 (0.0022) model time 0.3455 (0.3722) loss 1.6333 (2.7619) grad_norm 1.6548 (2.6210) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][610/625] eta 0:00:05 lr 0.000229 wd 0.0500 time 0.3381 (0.3741) data time 0.0004 (0.0021) model time 0.3378 (0.3720) loss 3.1512 (2.7622) grad_norm 2.1997 (2.6648) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [240/300][620/625] eta 0:00:01 lr 0.000229 wd 0.0500 time 0.3353 (0.3735) data time 0.0005 (0.0021) model time 0.3348 (0.3714) loss 3.3274 (2.7623) grad_norm 1.9529 (2.6574) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 11:34:54 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 240 training takes 0:03:26 [2024-09-01 11:34:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:34:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.434 (0.434) Loss 0.5254 (0.5254) Acc@1 88.867 (88.867) Acc@5 98.730 (98.730) Mem 11219MB [2024-09-01 11:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.8511 (0.6292) Acc@1 80.322 (86.390) Acc@5 95.459 (97.661) Mem 11219MB [2024-09-01 11:35:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 0.9263 (0.7483) Acc@1 78.516 (83.429) Acc@5 94.678 (96.394) Mem 11219MB [2024-09-01 11:35:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.061 Acc@5 96.411 [2024-09-01 11:35:03 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 11:35:03 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.06% [2024-09-01 11:35:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 11:35:08 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 11:35:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.431 (0.431) Loss 0.4873 (0.4873) Acc@1 89.551 (89.551) Acc@5 98.682 (98.682) Mem 11219MB [2024-09-01 11:35:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.127) Loss 0.8037 (0.5930) Acc@1 80.811 (86.972) Acc@5 95.947 (97.816) Mem 11219MB [2024-09-01 11:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.110) Loss 0.8721 (0.7021) Acc@1 79.199 (84.224) Acc@5 95.312 (96.735) Mem 11219MB [2024-09-01 11:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.919 Acc@5 96.767 [2024-09-01 11:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.92% [2024-09-01 11:35:10 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 11:35:15 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 11:35:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][0/625] eta 0:08:02 lr 0.000229 wd 0.0500 time 0.7721 (0.7721) data time 0.4296 (0.4296) model time 0.0000 (0.0000) loss 3.2882 (3.2882) grad_norm 1.9099 (1.9099) loss_scale 256.0000 (256.0000) mem 11227MB [2024-09-01 11:35:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][10/625] eta 0:03:56 lr 0.000229 wd 0.0500 time 0.3491 (0.3846) data time 0.0009 (0.0399) model time 0.0000 (0.0000) loss 2.9953 (2.9261) grad_norm 2.6649 (2.9233) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][20/625] eta 0:03:42 lr 0.000229 wd 0.0500 time 0.3450 (0.3671) data time 0.0007 (0.0213) model time 0.0000 (0.0000) loss 2.2509 (2.8906) grad_norm 2.1787 (2.9254) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][30/625] eta 0:03:34 lr 0.000229 wd 0.0500 time 0.3444 (0.3605) data time 0.0009 (0.0148) model time 0.0000 (0.0000) loss 2.9860 (2.9170) grad_norm 3.5394 (3.6405) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][40/625] eta 0:03:28 lr 0.000229 wd 0.0500 time 0.3451 (0.3570) data time 0.0009 (0.0114) model time 0.0000 (0.0000) loss 2.2771 (2.8949) grad_norm 2.6377 (3.3210) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][50/625] eta 0:03:24 lr 0.000229 wd 0.0500 time 0.3446 (0.3549) data time 0.0009 (0.0093) model time 0.0000 (0.0000) loss 2.1304 (2.8297) grad_norm 1.7324 (3.1071) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][60/625] eta 0:03:21 lr 0.000228 wd 0.0500 time 0.3480 (0.3561) data time 0.0008 (0.0080) model time 0.3472 (0.3617) loss 3.1066 (2.8347) grad_norm 2.1287 (2.9584) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][70/625] eta 0:03:16 lr 0.000228 wd 0.0500 time 0.3454 (0.3548) data time 0.0006 (0.0070) model time 0.3448 (0.3539) loss 2.8899 (2.8459) grad_norm 2.1279 (2.8320) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][80/625] eta 0:03:12 lr 0.000228 wd 0.0500 time 0.3441 (0.3538) data time 0.0007 (0.0062) model time 0.3434 (0.3511) loss 3.1578 (2.8355) grad_norm 2.6436 (2.7583) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][90/625] eta 0:03:08 lr 0.000228 wd 0.0500 time 0.3465 (0.3530) data time 0.0007 (0.0056) model time 0.3459 (0.3497) loss 3.4096 (2.8056) grad_norm 2.8629 (2.6989) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][100/625] eta 0:03:04 lr 0.000228 wd 0.0500 time 0.3486 (0.3523) data time 0.0008 (0.0052) model time 0.3478 (0.3487) loss 3.2072 (2.7974) grad_norm 3.5042 (2.7041) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][110/625] eta 0:03:01 lr 0.000228 wd 0.0500 time 0.3477 (0.3518) data time 0.0008 (0.0048) model time 0.3469 (0.3484) loss 2.3840 (2.7724) grad_norm 2.4557 (2.7643) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:35:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][120/625] eta 0:02:57 lr 0.000228 wd 0.0500 time 0.3506 (0.3513) data time 0.0006 (0.0045) model time 0.3500 (0.3479) loss 3.3293 (2.7942) grad_norm 2.3048 (2.7698) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][130/625] eta 0:02:53 lr 0.000228 wd 0.0500 time 0.3465 (0.3508) data time 0.0006 (0.0042) model time 0.3459 (0.3474) loss 2.9860 (2.8018) grad_norm 1.4698 (2.7227) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][140/625] eta 0:02:50 lr 0.000228 wd 0.0500 time 0.3479 (0.3509) data time 0.0008 (0.0040) model time 0.3471 (0.3477) loss 2.8199 (2.7952) grad_norm 1.7319 (2.6724) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][150/625] eta 0:02:46 lr 0.000227 wd 0.0500 time 0.3421 (0.3506) data time 0.0007 (0.0038) model time 0.3414 (0.3475) loss 3.1591 (2.7799) grad_norm 1.9402 (2.7330) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][160/625] eta 0:02:42 lr 0.000227 wd 0.0500 time 0.3461 (0.3504) data time 0.0007 (0.0036) model time 0.3453 (0.3474) loss 2.6962 (2.7714) grad_norm 1.6586 (2.7113) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][170/625] eta 0:02:39 lr 0.000227 wd 0.0500 time 0.3485 (0.3501) data time 0.0007 (0.0034) model time 0.3478 (0.3471) loss 2.8523 (2.7686) grad_norm 1.6989 (2.7055) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][180/625] eta 0:02:35 lr 0.000227 wd 0.0500 time 0.3439 (0.3500) data time 0.0008 (0.0033) model time 0.3431 (0.3471) loss 3.0462 (2.7704) grad_norm 1.6203 (2.6690) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][190/625] eta 0:02:32 lr 0.000227 wd 0.0500 time 0.3455 (0.3498) data time 0.0008 (0.0032) model time 0.3446 (0.3471) loss 2.9985 (2.7854) grad_norm 1.8893 (2.6468) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][200/625] eta 0:02:28 lr 0.000227 wd 0.0500 time 0.3390 (0.3496) data time 0.0010 (0.0031) model time 0.3381 (0.3469) loss 1.9664 (2.7830) grad_norm 2.5616 (2.6332) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][210/625] eta 0:02:25 lr 0.000227 wd 0.0500 time 0.3406 (0.3495) data time 0.0007 (0.0030) model time 0.3399 (0.3469) loss 2.8062 (2.7824) grad_norm 2.8988 (2.6866) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][220/625] eta 0:02:21 lr 0.000227 wd 0.0500 time 0.3531 (0.3504) data time 0.0007 (0.0029) model time 0.3524 (0.3481) loss 3.3151 (2.7870) grad_norm 2.0829 (2.6796) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][230/625] eta 0:02:18 lr 0.000227 wd 0.0500 time 0.3456 (0.3502) data time 0.0008 (0.0028) model time 0.3448 (0.3479) loss 3.0109 (2.7917) grad_norm 1.9594 (2.6726) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][240/625] eta 0:02:14 lr 0.000226 wd 0.0500 time 0.3474 (0.3500) data time 0.0008 (0.0027) model time 0.3465 (0.3478) loss 2.8772 (2.7857) grad_norm 1.7676 (2.6390) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][250/625] eta 0:02:11 lr 0.000226 wd 0.0500 time 0.3461 (0.3500) data time 0.0010 (0.0026) model time 0.3452 (0.3478) loss 1.9540 (2.7764) grad_norm 2.4425 (2.6857) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][260/625] eta 0:02:07 lr 0.000226 wd 0.0500 time 0.3482 (0.3499) data time 0.0007 (0.0026) model time 0.3475 (0.3478) loss 1.9528 (2.7610) grad_norm 1.7325 (2.6650) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][270/625] eta 0:02:04 lr 0.000226 wd 0.0500 time 0.3439 (0.3505) data time 0.0008 (0.0025) model time 0.3431 (0.3485) loss 2.3058 (2.7637) grad_norm 2.6115 (2.6498) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][280/625] eta 0:02:00 lr 0.000226 wd 0.0500 time 0.3512 (0.3504) data time 0.0007 (0.0025) model time 0.3505 (0.3485) loss 2.9322 (2.7584) grad_norm 1.9850 (2.6415) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:36:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][290/625] eta 0:01:57 lr 0.000226 wd 0.0500 time 0.3449 (0.3503) data time 0.0007 (0.0024) model time 0.3443 (0.3484) loss 3.6091 (2.7596) grad_norm 1.2348 (2.6448) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][300/625] eta 0:01:53 lr 0.000226 wd 0.0500 time 0.3447 (0.3503) data time 0.0008 (0.0023) model time 0.3439 (0.3484) loss 2.3332 (2.7560) grad_norm 4.0274 (2.6503) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][310/625] eta 0:01:50 lr 0.000226 wd 0.0500 time 0.3478 (0.3502) data time 0.0008 (0.0023) model time 0.3470 (0.3483) loss 1.9071 (2.7561) grad_norm 2.6808 (2.6348) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][320/625] eta 0:01:46 lr 0.000226 wd 0.0500 time 0.3453 (0.3501) data time 0.0008 (0.0023) model time 0.3445 (0.3482) loss 3.0298 (2.7585) grad_norm 2.1915 (2.6191) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][330/625] eta 0:01:43 lr 0.000226 wd 0.0500 time 0.3444 (0.3499) data time 0.0008 (0.0022) model time 0.3436 (0.3481) loss 3.1369 (2.7543) grad_norm 3.1909 (2.6115) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][340/625] eta 0:01:39 lr 0.000225 wd 0.0500 time 0.3485 (0.3498) data time 0.0006 (0.0022) model time 0.3479 (0.3480) loss 2.6980 (2.7553) grad_norm 4.8192 (2.6213) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][350/625] eta 0:01:36 lr 0.000225 wd 0.0500 time 0.3431 (0.3497) data time 0.0008 (0.0022) model time 0.3423 (0.3479) loss 3.0737 (2.7568) grad_norm 1.9821 (2.6134) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][360/625] eta 0:01:32 lr 0.000225 wd 0.0500 time 0.3446 (0.3495) data time 0.0006 (0.0021) model time 0.3439 (0.3477) loss 3.1073 (2.7536) grad_norm 1.5891 (2.6033) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][370/625] eta 0:01:29 lr 0.000225 wd 0.0500 time 0.3535 (0.3495) data time 0.0009 (0.0021) model time 0.3527 (0.3477) loss 2.9566 (2.7588) grad_norm 2.2855 (2.5954) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][380/625] eta 0:01:25 lr 0.000225 wd 0.0500 time 0.3462 (0.3494) data time 0.0008 (0.0021) model time 0.3455 (0.3476) loss 2.6849 (2.7674) grad_norm 1.9375 (2.5868) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][390/625] eta 0:01:22 lr 0.000225 wd 0.0500 time 0.3424 (0.3493) data time 0.0008 (0.0020) model time 0.3415 (0.3475) loss 2.3842 (2.7621) grad_norm 2.8094 (2.5913) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][400/625] eta 0:01:18 lr 0.000225 wd 0.0500 time 0.3451 (0.3503) data time 0.0007 (0.0020) model time 0.3444 (0.3487) loss 3.0859 (2.7582) grad_norm 1.7614 (2.5897) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][410/625] eta 0:01:15 lr 0.000225 wd 0.0500 time 0.3468 (0.3501) data time 0.0009 (0.0020) model time 0.3459 (0.3486) loss 3.0150 (2.7588) grad_norm 2.4222 (2.5796) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][420/625] eta 0:01:11 lr 0.000225 wd 0.0500 time 0.3529 (0.3500) data time 0.0006 (0.0019) model time 0.3522 (0.3484) loss 2.9440 (2.7626) grad_norm 2.2245 (2.5752) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][430/625] eta 0:01:08 lr 0.000224 wd 0.0500 time 0.3607 (0.3500) data time 0.0007 (0.0019) model time 0.3601 (0.3484) loss 1.5766 (2.7610) grad_norm 2.0730 (2.5651) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][440/625] eta 0:01:04 lr 0.000224 wd 0.0500 time 0.3418 (0.3498) data time 0.0006 (0.0019) model time 0.3412 (0.3483) loss 2.6933 (2.7630) grad_norm 1.9163 (2.5549) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][450/625] eta 0:01:01 lr 0.000224 wd 0.0500 time 0.3452 (0.3502) data time 0.0007 (0.0019) model time 0.3445 (0.3487) loss 3.0089 (2.7630) grad_norm 1.9580 (2.5638) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:37:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][460/625] eta 0:00:57 lr 0.000224 wd 0.0500 time 0.3479 (0.3501) data time 0.0008 (0.0019) model time 0.3471 (0.3486) loss 2.8914 (2.7669) grad_norm 4.8603 (2.5691) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][470/625] eta 0:00:54 lr 0.000224 wd 0.0500 time 0.3362 (0.3500) data time 0.0007 (0.0018) model time 0.3356 (0.3485) loss 1.7649 (2.7641) grad_norm 2.9890 (2.5840) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][480/625] eta 0:00:50 lr 0.000224 wd 0.0500 time 0.3512 (0.3500) data time 0.0008 (0.0018) model time 0.3504 (0.3485) loss 2.1544 (2.7590) grad_norm 1.6200 (2.5914) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][490/625] eta 0:00:47 lr 0.000224 wd 0.0500 time 0.3456 (0.3500) data time 0.0006 (0.0018) model time 0.3449 (0.3485) loss 2.9153 (2.7665) grad_norm 2.7687 (2.5876) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][500/625] eta 0:00:43 lr 0.000224 wd 0.0500 time 0.3431 (0.3500) data time 0.0009 (0.0018) model time 0.3422 (0.3485) loss 2.9310 (2.7665) grad_norm 2.3175 (2.5796) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][510/625] eta 0:00:40 lr 0.000224 wd 0.0500 time 0.3558 (0.3499) data time 0.0006 (0.0018) model time 0.3552 (0.3484) loss 2.9850 (2.7703) grad_norm 4.8679 (2.5780) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][520/625] eta 0:00:36 lr 0.000223 wd 0.0500 time 0.3416 (0.3506) data time 0.0009 (0.0018) model time 0.3407 (0.3492) loss 2.8132 (2.7668) grad_norm 2.6382 (2.5734) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][530/625] eta 0:00:33 lr 0.000223 wd 0.0500 time 0.3476 (0.3505) data time 0.0009 (0.0017) model time 0.3467 (0.3491) loss 3.0826 (2.7671) grad_norm 2.0171 (2.5610) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][540/625] eta 0:00:29 lr 0.000223 wd 0.0500 time 0.3467 (0.3504) data time 0.0009 (0.0017) model time 0.3458 (0.3490) loss 3.3901 (2.7628) grad_norm 2.9026 (2.5514) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][550/625] eta 0:00:26 lr 0.000223 wd 0.0500 time 0.3447 (0.3504) data time 0.0009 (0.0017) model time 0.3438 (0.3490) loss 2.9047 (2.7608) grad_norm 2.3235 (2.5442) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][560/625] eta 0:00:22 lr 0.000223 wd 0.0500 time 0.3465 (0.3503) data time 0.0007 (0.0017) model time 0.3458 (0.3490) loss 3.0454 (2.7632) grad_norm 3.4528 (2.5456) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][570/625] eta 0:00:19 lr 0.000223 wd 0.0500 time 0.3433 (0.3506) data time 0.0008 (0.0017) model time 0.3425 (0.3493) loss 2.4683 (2.7656) grad_norm 2.3408 (2.5408) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][580/625] eta 0:00:15 lr 0.000223 wd 0.0500 time 0.3564 (0.3506) data time 0.0009 (0.0017) model time 0.3555 (0.3493) loss 2.3548 (2.7671) grad_norm 3.0190 (2.5503) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][590/625] eta 0:00:12 lr 0.000223 wd 0.0500 time 0.3469 (0.3506) data time 0.0007 (0.0017) model time 0.3462 (0.3493) loss 3.4671 (2.7679) grad_norm 1.8655 (2.5472) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][600/625] eta 0:00:08 lr 0.000223 wd 0.0500 time 0.3476 (0.3506) data time 0.0007 (0.0016) model time 0.3469 (0.3492) loss 3.2741 (2.7682) grad_norm 1.8082 (2.5436) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][610/625] eta 0:00:05 lr 0.000222 wd 0.0500 time 0.3465 (0.3505) data time 0.0006 (0.0016) model time 0.3459 (0.3492) loss 2.1305 (2.7664) grad_norm 1.9540 (2.5367) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [241/300][620/625] eta 0:00:01 lr 0.000222 wd 0.0500 time 0.3343 (0.3503) data time 0.0004 (0.0016) model time 0.3339 (0.3489) loss 3.0235 (2.7688) grad_norm 3.2210 (2.5510) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:38:54 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 241 training takes 0:03:38 [2024-09-01 11:38:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:38:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.434 (0.434) Loss 0.5088 (0.5088) Acc@1 89.014 (89.014) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 11:38:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.129) Loss 0.8633 (0.6256) Acc@1 80.078 (86.297) Acc@5 95.410 (97.674) Mem 11224MB [2024-09-01 11:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.111) Loss 0.9243 (0.7400) Acc@1 77.734 (83.366) Acc@5 94.775 (96.443) Mem 11224MB [2024-09-01 11:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.099 Acc@5 96.461 [2024-09-01 11:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 11:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.10% [2024-09-01 11:38:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 11:38:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 11:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.422 (0.422) Loss 0.4873 (0.4873) Acc@1 89.600 (89.600) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 11:39:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.128) Loss 0.8052 (0.5931) Acc@1 80.664 (86.967) Acc@5 96.045 (97.834) Mem 11224MB [2024-09-01 11:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.110) Loss 0.8711 (0.7022) Acc@1 79.248 (84.226) Acc@5 95.410 (96.745) Mem 11224MB [2024-09-01 11:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.913 Acc@5 96.777 [2024-09-01 11:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:39:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][0/625] eta 0:12:30 lr 0.000222 wd 0.0500 time 1.2014 (1.2014) data time 0.8297 (0.8297) model time 0.0000 (0.0000) loss 3.1716 (3.1716) grad_norm 2.8299 (2.8299) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][10/625] eta 0:04:23 lr 0.000222 wd 0.0500 time 0.3454 (0.4278) data time 0.0007 (0.0763) model time 0.0000 (0.0000) loss 3.1143 (3.0009) grad_norm 2.5803 (2.3386) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][20/625] eta 0:03:56 lr 0.000222 wd 0.0500 time 0.3493 (0.3911) data time 0.0006 (0.0404) model time 0.0000 (0.0000) loss 3.0367 (2.8641) grad_norm 3.2678 (2.7417) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][30/625] eta 0:03:44 lr 0.000222 wd 0.0500 time 0.3483 (0.3774) data time 0.0009 (0.0276) model time 0.0000 (0.0000) loss 3.0437 (2.8914) grad_norm 1.6036 (2.5615) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][40/625] eta 0:03:36 lr 0.000222 wd 0.0500 time 0.3450 (0.3707) data time 0.0006 (0.0211) model time 0.0000 (0.0000) loss 2.2636 (2.8645) grad_norm 3.2735 (2.5013) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][50/625] eta 0:03:30 lr 0.000222 wd 0.0500 time 0.3467 (0.3666) data time 0.0007 (0.0172) model time 0.0000 (0.0000) loss 1.9325 (2.8232) grad_norm 2.5862 (2.4936) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][60/625] eta 0:03:29 lr 0.000222 wd 0.0500 time 0.3496 (0.3704) data time 0.0009 (0.0145) model time 0.3488 (0.3889) loss 2.9325 (2.8099) grad_norm 2.7063 (2.5299) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][70/625] eta 0:03:23 lr 0.000222 wd 0.0500 time 0.3486 (0.3671) data time 0.0008 (0.0126) model time 0.3477 (0.3675) loss 2.1304 (2.8123) grad_norm 1.9812 (2.6232) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][80/625] eta 0:03:18 lr 0.000221 wd 0.0500 time 0.3467 (0.3648) data time 0.0006 (0.0112) model time 0.3460 (0.3609) loss 3.3996 (2.7964) grad_norm 2.4083 (2.6229) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 11:39:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][90/625] eta 0:03:13 lr 0.000221 wd 0.0500 time 0.3476 (0.3625) data time 0.0008 (0.0100) model time 0.3468 (0.3564) loss 2.4853 (2.8144) grad_norm 2.8803 (inf) loss_scale 128.0000 (241.9341) mem 11224MB [2024-09-01 11:39:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][100/625] eta 0:03:10 lr 0.000221 wd 0.0500 time 0.3449 (0.3625) data time 0.0006 (0.0091) model time 0.3443 (0.3574) loss 3.2298 (2.8265) grad_norm 2.4477 (inf) loss_scale 128.0000 (230.6535) mem 11224MB [2024-09-01 11:39:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][110/625] eta 0:03:05 lr 0.000221 wd 0.0500 time 0.3450 (0.3611) data time 0.0010 (0.0084) model time 0.3440 (0.3556) loss 3.1878 (2.8439) grad_norm 2.3875 (inf) loss_scale 128.0000 (221.4054) mem 11224MB [2024-09-01 11:39:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][120/625] eta 0:03:01 lr 0.000221 wd 0.0500 time 0.3512 (0.3601) data time 0.0007 (0.0078) model time 0.3506 (0.3544) loss 2.3907 (2.8492) grad_norm 2.3341 (inf) loss_scale 128.0000 (213.6860) mem 11224MB [2024-09-01 11:39:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][130/625] eta 0:02:57 lr 0.000221 wd 0.0500 time 0.3660 (0.3594) data time 0.0009 (0.0073) model time 0.3651 (0.3538) loss 2.7934 (2.8441) grad_norm 2.3699 (inf) loss_scale 128.0000 (207.1450) mem 11224MB [2024-09-01 11:39:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][140/625] eta 0:02:53 lr 0.000221 wd 0.0500 time 0.3535 (0.3585) data time 0.0009 (0.0068) model time 0.3526 (0.3530) loss 2.5880 (2.8204) grad_norm 1.9640 (inf) loss_scale 128.0000 (201.5319) mem 11224MB [2024-09-01 11:39:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][150/625] eta 0:02:49 lr 0.000221 wd 0.0500 time 0.3480 (0.3577) data time 0.0006 (0.0064) model time 0.3474 (0.3523) loss 3.2222 (2.8129) grad_norm 3.1896 (inf) loss_scale 128.0000 (196.6623) mem 11224MB [2024-09-01 11:39:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][160/625] eta 0:02:46 lr 0.000221 wd 0.0500 time 0.3471 (0.3571) data time 0.0009 (0.0061) model time 0.3462 (0.3517) loss 3.1255 (2.8068) grad_norm 2.6143 (inf) loss_scale 128.0000 (192.3975) mem 11224MB [2024-09-01 11:40:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][170/625] eta 0:02:42 lr 0.000221 wd 0.0500 time 0.3457 (0.3566) data time 0.0009 (0.0058) model time 0.3447 (0.3514) loss 2.9817 (2.8056) grad_norm 2.9525 (inf) loss_scale 128.0000 (188.6316) mem 11224MB [2024-09-01 11:40:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][180/625] eta 0:02:38 lr 0.000220 wd 0.0500 time 0.3478 (0.3561) data time 0.0008 (0.0055) model time 0.3471 (0.3511) loss 3.5628 (2.8206) grad_norm 1.6973 (inf) loss_scale 128.0000 (185.2818) mem 11224MB [2024-09-01 11:40:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][190/625] eta 0:02:34 lr 0.000220 wd 0.0500 time 0.3453 (0.3556) data time 0.0006 (0.0053) model time 0.3447 (0.3506) loss 2.8767 (2.8298) grad_norm 3.1692 (inf) loss_scale 128.0000 (182.2827) mem 11224MB [2024-09-01 11:40:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][200/625] eta 0:02:30 lr 0.000220 wd 0.0500 time 0.3544 (0.3552) data time 0.0009 (0.0051) model time 0.3535 (0.3504) loss 2.6478 (2.8291) grad_norm 2.1810 (inf) loss_scale 128.0000 (179.5821) mem 11224MB [2024-09-01 11:40:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][210/625] eta 0:02:27 lr 0.000220 wd 0.0500 time 0.3508 (0.3552) data time 0.0007 (0.0049) model time 0.3501 (0.3506) loss 1.8089 (2.8319) grad_norm 2.3007 (inf) loss_scale 128.0000 (177.1374) mem 11224MB [2024-09-01 11:40:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][220/625] eta 0:02:23 lr 0.000220 wd 0.0500 time 0.3480 (0.3548) data time 0.0008 (0.0047) model time 0.3472 (0.3504) loss 3.1325 (2.8277) grad_norm 2.4639 (inf) loss_scale 128.0000 (174.9140) mem 11224MB [2024-09-01 11:40:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][230/625] eta 0:02:20 lr 0.000220 wd 0.0500 time 0.3468 (0.3545) data time 0.0005 (0.0045) model time 0.3462 (0.3502) loss 2.7437 (2.8360) grad_norm 3.6387 (inf) loss_scale 128.0000 (172.8831) mem 11224MB [2024-09-01 11:40:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][240/625] eta 0:02:16 lr 0.000220 wd 0.0500 time 0.3485 (0.3542) data time 0.0009 (0.0044) model time 0.3476 (0.3499) loss 2.3847 (2.8299) grad_norm 1.6070 (inf) loss_scale 128.0000 (171.0207) mem 11224MB [2024-09-01 11:40:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][250/625] eta 0:02:12 lr 0.000220 wd 0.0500 time 0.3468 (0.3539) data time 0.0009 (0.0043) model time 0.3459 (0.3497) loss 2.0020 (2.8231) grad_norm 1.7773 (inf) loss_scale 128.0000 (169.3068) mem 11224MB [2024-09-01 11:40:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][260/625] eta 0:02:09 lr 0.000220 wd 0.0500 time 0.3494 (0.3537) data time 0.0008 (0.0041) model time 0.3486 (0.3496) loss 2.5490 (2.8206) grad_norm 1.8814 (inf) loss_scale 128.0000 (167.7241) mem 11224MB [2024-09-01 11:40:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][270/625] eta 0:02:06 lr 0.000219 wd 0.0500 time 0.3451 (0.3550) data time 0.0009 (0.0040) model time 0.3442 (0.3513) loss 1.8655 (2.8214) grad_norm 2.4878 (inf) loss_scale 128.0000 (166.2583) mem 11224MB [2024-09-01 11:40:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][280/625] eta 0:02:02 lr 0.000219 wd 0.0500 time 0.3451 (0.3552) data time 0.0006 (0.0039) model time 0.3445 (0.3516) loss 3.1431 (2.8224) grad_norm 2.1771 (inf) loss_scale 128.0000 (164.8968) mem 11224MB [2024-09-01 11:40:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][290/625] eta 0:01:58 lr 0.000219 wd 0.0500 time 0.3444 (0.3549) data time 0.0008 (0.0038) model time 0.3436 (0.3514) loss 3.2241 (2.8165) grad_norm 3.9662 (inf) loss_scale 128.0000 (163.6289) mem 11224MB [2024-09-01 11:40:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][300/625] eta 0:01:55 lr 0.000219 wd 0.0500 time 0.3460 (0.3546) data time 0.0009 (0.0037) model time 0.3451 (0.3511) loss 2.6113 (2.8083) grad_norm 2.9219 (inf) loss_scale 128.0000 (162.4452) mem 11224MB [2024-09-01 11:40:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][310/625] eta 0:01:51 lr 0.000219 wd 0.0500 time 0.3497 (0.3543) data time 0.0006 (0.0036) model time 0.3491 (0.3509) loss 2.2557 (2.7974) grad_norm 2.2169 (inf) loss_scale 128.0000 (161.3376) mem 11224MB [2024-09-01 11:40:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][320/625] eta 0:01:47 lr 0.000219 wd 0.0500 time 0.3480 (0.3541) data time 0.0007 (0.0035) model time 0.3473 (0.3508) loss 3.1091 (2.7932) grad_norm 2.2341 (inf) loss_scale 128.0000 (160.2991) mem 11224MB [2024-09-01 11:40:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][330/625] eta 0:01:44 lr 0.000219 wd 0.0500 time 0.3501 (0.3539) data time 0.0009 (0.0035) model time 0.3493 (0.3506) loss 2.8973 (2.7987) grad_norm 1.6063 (inf) loss_scale 128.0000 (159.3233) mem 11224MB [2024-09-01 11:41:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][340/625] eta 0:01:40 lr 0.000219 wd 0.0500 time 0.3504 (0.3537) data time 0.0008 (0.0034) model time 0.3496 (0.3505) loss 2.7907 (2.7995) grad_norm 2.9857 (inf) loss_scale 128.0000 (158.4047) mem 11224MB [2024-09-01 11:41:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][350/625] eta 0:01:37 lr 0.000219 wd 0.0500 time 0.3482 (0.3536) data time 0.0006 (0.0033) model time 0.3476 (0.3505) loss 2.1025 (2.7998) grad_norm 4.0187 (inf) loss_scale 128.0000 (157.5385) mem 11224MB [2024-09-01 11:41:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][360/625] eta 0:01:33 lr 0.000218 wd 0.0500 time 0.3477 (0.3535) data time 0.0008 (0.0032) model time 0.3469 (0.3504) loss 2.9788 (2.7986) grad_norm 3.0633 (inf) loss_scale 128.0000 (156.7202) mem 11224MB [2024-09-01 11:41:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][370/625] eta 0:01:30 lr 0.000218 wd 0.0500 time 0.3484 (0.3533) data time 0.0007 (0.0032) model time 0.3477 (0.3503) loss 3.1534 (2.7947) grad_norm 2.9840 (inf) loss_scale 128.0000 (155.9461) mem 11224MB [2024-09-01 11:41:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][380/625] eta 0:01:26 lr 0.000218 wd 0.0500 time 0.3482 (0.3532) data time 0.0006 (0.0031) model time 0.3476 (0.3501) loss 2.7104 (2.7917) grad_norm 3.6588 (inf) loss_scale 128.0000 (155.2126) mem 11224MB [2024-09-01 11:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][390/625] eta 0:01:22 lr 0.000218 wd 0.0500 time 0.3535 (0.3531) data time 0.0007 (0.0031) model time 0.3528 (0.3501) loss 1.6356 (2.7822) grad_norm 2.3726 (inf) loss_scale 128.0000 (154.5166) mem 11224MB [2024-09-01 11:41:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][400/625] eta 0:01:19 lr 0.000218 wd 0.0500 time 0.3570 (0.3534) data time 0.0008 (0.0030) model time 0.3562 (0.3505) loss 3.2421 (2.7806) grad_norm 2.4548 (inf) loss_scale 128.0000 (153.8554) mem 11224MB [2024-09-01 11:41:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][410/625] eta 0:01:15 lr 0.000218 wd 0.0500 time 0.3560 (0.3532) data time 0.0007 (0.0030) model time 0.3553 (0.3503) loss 3.0291 (2.7812) grad_norm 1.7601 (inf) loss_scale 128.0000 (153.2263) mem 11224MB [2024-09-01 11:41:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][420/625] eta 0:01:12 lr 0.000218 wd 0.0500 time 0.3514 (0.3531) data time 0.0008 (0.0029) model time 0.3506 (0.3503) loss 3.0386 (2.7848) grad_norm 2.3734 (inf) loss_scale 128.0000 (152.6271) mem 11224MB [2024-09-01 11:41:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][430/625] eta 0:01:08 lr 0.000218 wd 0.0500 time 0.3477 (0.3530) data time 0.0006 (0.0029) model time 0.3471 (0.3502) loss 2.0565 (2.7894) grad_norm 1.8994 (inf) loss_scale 128.0000 (152.0557) mem 11224MB [2024-09-01 11:41:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][440/625] eta 0:01:05 lr 0.000218 wd 0.0500 time 0.3533 (0.3529) data time 0.0005 (0.0028) model time 0.3528 (0.3501) loss 3.1420 (2.7903) grad_norm 1.9854 (inf) loss_scale 128.0000 (151.5102) mem 11224MB [2024-09-01 11:41:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][450/625] eta 0:01:01 lr 0.000218 wd 0.0500 time 0.3482 (0.3536) data time 0.0006 (0.0028) model time 0.3476 (0.3510) loss 2.5618 (2.7858) grad_norm 1.5956 (inf) loss_scale 128.0000 (150.9889) mem 11224MB [2024-09-01 11:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][460/625] eta 0:00:58 lr 0.000217 wd 0.0500 time 0.3490 (0.3544) data time 0.0006 (0.0027) model time 0.3484 (0.3519) loss 2.7297 (2.7911) grad_norm 1.8362 (inf) loss_scale 128.0000 (150.4902) mem 11224MB [2024-09-01 11:41:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][470/625] eta 0:00:54 lr 0.000217 wd 0.0500 time 0.3419 (0.3542) data time 0.0009 (0.0027) model time 0.3410 (0.3518) loss 2.1088 (2.7789) grad_norm 2.1866 (inf) loss_scale 128.0000 (150.0127) mem 11224MB [2024-09-01 11:41:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][480/625] eta 0:00:51 lr 0.000217 wd 0.0500 time 0.3455 (0.3541) data time 0.0006 (0.0027) model time 0.3449 (0.3516) loss 1.9803 (2.7738) grad_norm 2.3300 (inf) loss_scale 128.0000 (149.5551) mem 11224MB [2024-09-01 11:41:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][490/625] eta 0:00:47 lr 0.000217 wd 0.0500 time 0.3500 (0.3539) data time 0.0006 (0.0026) model time 0.3495 (0.3515) loss 2.1328 (2.7711) grad_norm 2.8216 (inf) loss_scale 128.0000 (149.1161) mem 11224MB [2024-09-01 11:41:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][500/625] eta 0:00:44 lr 0.000217 wd 0.0500 time 0.3504 (0.3538) data time 0.0009 (0.0026) model time 0.3495 (0.3514) loss 3.2072 (2.7716) grad_norm 1.5998 (inf) loss_scale 128.0000 (148.6946) mem 11224MB [2024-09-01 11:42:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][510/625] eta 0:00:40 lr 0.000217 wd 0.0500 time 0.3522 (0.3537) data time 0.0007 (0.0026) model time 0.3515 (0.3513) loss 2.6914 (2.7724) grad_norm 2.4666 (inf) loss_scale 128.0000 (148.2896) mem 11224MB [2024-09-01 11:42:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][520/625] eta 0:00:37 lr 0.000217 wd 0.0500 time 0.3474 (0.3535) data time 0.0007 (0.0025) model time 0.3467 (0.3512) loss 2.9545 (2.7753) grad_norm 1.8302 (inf) loss_scale 128.0000 (147.9002) mem 11224MB [2024-09-01 11:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][530/625] eta 0:00:33 lr 0.000217 wd 0.0500 time 0.3488 (0.3534) data time 0.0006 (0.0025) model time 0.3481 (0.3511) loss 2.9697 (2.7767) grad_norm 1.9012 (inf) loss_scale 128.0000 (147.5254) mem 11224MB [2024-09-01 11:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][540/625] eta 0:00:30 lr 0.000217 wd 0.0500 time 0.3490 (0.3533) data time 0.0006 (0.0025) model time 0.3484 (0.3510) loss 2.4770 (2.7753) grad_norm 2.1380 (inf) loss_scale 128.0000 (147.1645) mem 11224MB [2024-09-01 11:42:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][550/625] eta 0:00:26 lr 0.000216 wd 0.0500 time 0.3453 (0.3532) data time 0.0009 (0.0025) model time 0.3444 (0.3509) loss 3.1189 (2.7740) grad_norm 2.5884 (inf) loss_scale 128.0000 (146.8167) mem 11224MB [2024-09-01 11:42:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][560/625] eta 0:00:22 lr 0.000216 wd 0.0500 time 0.3538 (0.3531) data time 0.0009 (0.0024) model time 0.3529 (0.3508) loss 3.2823 (2.7744) grad_norm 1.9518 (inf) loss_scale 128.0000 (146.4813) mem 11224MB [2024-09-01 11:42:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][570/625] eta 0:00:19 lr 0.000216 wd 0.0500 time 0.3459 (0.3537) data time 0.0006 (0.0024) model time 0.3452 (0.3515) loss 3.4838 (2.7720) grad_norm 2.5946 (inf) loss_scale 128.0000 (146.1576) mem 11224MB [2024-09-01 11:42:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][580/625] eta 0:00:15 lr 0.000216 wd 0.0500 time 0.3483 (0.3540) data time 0.0007 (0.0024) model time 0.3477 (0.3518) loss 2.2853 (2.7668) grad_norm 2.4598 (inf) loss_scale 128.0000 (145.8451) mem 11224MB [2024-09-01 11:42:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][590/625] eta 0:00:12 lr 0.000216 wd 0.0500 time 0.3394 (0.3539) data time 0.0007 (0.0023) model time 0.3387 (0.3517) loss 2.9752 (2.7704) grad_norm 2.7951 (inf) loss_scale 128.0000 (145.5431) mem 11224MB [2024-09-01 11:42:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][600/625] eta 0:00:08 lr 0.000216 wd 0.0500 time 0.3405 (0.3537) data time 0.0010 (0.0023) model time 0.3395 (0.3516) loss 2.2255 (2.7721) grad_norm 2.4309 (inf) loss_scale 128.0000 (145.2512) mem 11224MB [2024-09-01 11:42:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][610/625] eta 0:00:05 lr 0.000216 wd 0.0500 time 0.3404 (0.3539) data time 0.0006 (0.0023) model time 0.3398 (0.3518) loss 2.9710 (2.7699) grad_norm 2.3233 (inf) loss_scale 128.0000 (144.9689) mem 11224MB [2024-09-01 11:42:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [242/300][620/625] eta 0:00:01 lr 0.000216 wd 0.0500 time 0.3379 (0.3536) data time 0.0004 (0.0023) model time 0.3374 (0.3515) loss 2.9531 (2.7739) grad_norm 2.0217 (inf) loss_scale 128.0000 (144.6957) mem 11224MB [2024-09-01 11:42:42 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 242 training takes 0:03:40 [2024-09-01 11:42:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:42:43 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:42:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.433 (0.433) Loss 0.5181 (0.5181) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 11:42:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.129) Loss 0.8574 (0.6241) Acc@1 80.420 (86.506) Acc@5 95.557 (97.727) Mem 11224MB [2024-09-01 11:42:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 0.9346 (0.7427) Acc@1 77.539 (83.540) Acc@5 95.068 (96.491) Mem 11224MB [2024-09-01 11:42:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.235 Acc@5 96.499 [2024-09-01 11:42:46 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.2% [2024-09-01 11:42:46 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.24% [2024-09-01 11:42:46 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 11:42:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 11:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.428 (0.428) Loss 0.4878 (0.4878) Acc@1 89.600 (89.600) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 11:42:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.128) Loss 0.8052 (0.5929) Acc@1 80.713 (86.990) Acc@5 96.143 (97.843) Mem 11224MB [2024-09-01 11:42:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.110) Loss 0.8706 (0.7023) Acc@1 79.199 (84.224) Acc@5 95.508 (96.752) Mem 11224MB [2024-09-01 11:42:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.919 Acc@5 96.787 [2024-09-01 11:42:49 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:42:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][0/625] eta 0:11:48 lr 0.000216 wd 0.0500 time 1.1335 (1.1335) data time 0.7395 (0.7395) model time 0.0000 (0.0000) loss 2.7492 (2.7492) grad_norm 2.9043 (2.9043) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:42:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][10/625] eta 0:04:18 lr 0.000216 wd 0.0500 time 0.3495 (0.4211) data time 0.0006 (0.0681) model time 0.0000 (0.0000) loss 2.8392 (2.6786) grad_norm 1.7849 (2.6259) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:42:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][20/625] eta 0:03:54 lr 0.000215 wd 0.0500 time 0.3499 (0.3869) data time 0.0009 (0.0361) model time 0.0000 (0.0000) loss 2.8580 (2.7141) grad_norm 1.6000 (2.3600) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][30/625] eta 0:03:42 lr 0.000215 wd 0.0500 time 0.3522 (0.3746) data time 0.0009 (0.0248) model time 0.0000 (0.0000) loss 2.8786 (2.8146) grad_norm 3.4921 (2.6976) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][40/625] eta 0:03:35 lr 0.000215 wd 0.0500 time 0.3487 (0.3687) data time 0.0009 (0.0190) model time 0.0000 (0.0000) loss 2.6419 (2.7163) grad_norm 2.9760 (2.6732) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][50/625] eta 0:03:29 lr 0.000215 wd 0.0500 time 0.3510 (0.3647) data time 0.0007 (0.0154) model time 0.0000 (0.0000) loss 3.4121 (2.6966) grad_norm 1.9244 (2.7751) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][60/625] eta 0:03:24 lr 0.000215 wd 0.0500 time 0.3477 (0.3621) data time 0.0007 (0.0131) model time 0.3470 (0.3477) loss 2.5781 (2.7115) grad_norm 2.0656 (2.7590) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][70/625] eta 0:03:19 lr 0.000215 wd 0.0500 time 0.3471 (0.3600) data time 0.0007 (0.0113) model time 0.3464 (0.3471) loss 3.4640 (2.6996) grad_norm 1.9192 (2.7225) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][80/625] eta 0:03:15 lr 0.000215 wd 0.0500 time 0.3452 (0.3586) data time 0.0007 (0.0101) model time 0.3445 (0.3472) loss 1.6891 (2.7124) grad_norm 1.7921 (2.6561) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][90/625] eta 0:03:14 lr 0.000215 wd 0.0500 time 0.3459 (0.3633) data time 0.0006 (0.0090) model time 0.3453 (0.3605) loss 3.3012 (2.7062) grad_norm 1.9893 (2.5951) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][100/625] eta 0:03:09 lr 0.000215 wd 0.0500 time 0.3418 (0.3614) data time 0.0007 (0.0082) model time 0.3412 (0.3571) loss 3.1683 (2.7095) grad_norm 1.5884 (2.5379) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][110/625] eta 0:03:05 lr 0.000214 wd 0.0500 time 0.3394 (0.3599) data time 0.0007 (0.0076) model time 0.3388 (0.3548) loss 2.2394 (2.7158) grad_norm 1.5641 (2.4883) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][120/625] eta 0:03:01 lr 0.000214 wd 0.0500 time 0.3502 (0.3586) data time 0.0008 (0.0070) model time 0.3494 (0.3533) loss 2.7863 (2.6858) grad_norm 1.8400 (2.4626) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][130/625] eta 0:02:57 lr 0.000214 wd 0.0500 time 0.3134 (0.3587) data time 0.0007 (0.0066) model time 0.3126 (0.3539) loss 2.7449 (2.6797) grad_norm 4.4479 (2.5437) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][140/625] eta 0:02:53 lr 0.000214 wd 0.0500 time 0.3472 (0.3579) data time 0.0008 (0.0062) model time 0.3464 (0.3530) loss 2.6848 (2.6838) grad_norm 2.6120 (2.5463) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][150/625] eta 0:02:49 lr 0.000214 wd 0.0500 time 0.3511 (0.3572) data time 0.0007 (0.0059) model time 0.3504 (0.3523) loss 1.8150 (2.6756) grad_norm 1.8372 (2.5234) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][160/625] eta 0:02:45 lr 0.000214 wd 0.0500 time 0.3518 (0.3565) data time 0.0008 (0.0056) model time 0.3510 (0.3517) loss 1.9387 (2.6761) grad_norm 1.9020 (2.5238) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][170/625] eta 0:02:41 lr 0.000214 wd 0.0500 time 0.3487 (0.3559) data time 0.0009 (0.0053) model time 0.3479 (0.3512) loss 2.7457 (2.6655) grad_norm 2.5237 (2.5302) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][180/625] eta 0:02:38 lr 0.000214 wd 0.0500 time 0.3487 (0.3554) data time 0.0006 (0.0050) model time 0.3481 (0.3507) loss 3.1732 (2.6632) grad_norm 1.9894 (2.5070) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:43:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][190/625] eta 0:02:34 lr 0.000214 wd 0.0500 time 0.3438 (0.3549) data time 0.0008 (0.0048) model time 0.3430 (0.3503) loss 2.3532 (2.6595) grad_norm 3.0371 (2.5089) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][200/625] eta 0:02:30 lr 0.000214 wd 0.0500 time 0.3460 (0.3545) data time 0.0009 (0.0046) model time 0.3451 (0.3500) loss 2.8902 (2.6626) grad_norm 1.9397 (2.4785) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][210/625] eta 0:02:26 lr 0.000213 wd 0.0500 time 0.3473 (0.3541) data time 0.0010 (0.0045) model time 0.3464 (0.3497) loss 2.1605 (2.6777) grad_norm 1.5554 (2.4557) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][220/625] eta 0:02:23 lr 0.000213 wd 0.0500 time 0.3504 (0.3538) data time 0.0009 (0.0043) model time 0.3495 (0.3496) loss 2.9471 (2.6823) grad_norm 2.7169 (2.4481) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][230/625] eta 0:02:19 lr 0.000213 wd 0.0500 time 0.3520 (0.3536) data time 0.0008 (0.0042) model time 0.3512 (0.3494) loss 2.9752 (2.6922) grad_norm 2.1402 (2.4483) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][240/625] eta 0:02:16 lr 0.000213 wd 0.0500 time 0.3474 (0.3533) data time 0.0006 (0.0040) model time 0.3468 (0.3493) loss 2.8450 (2.6971) grad_norm 1.7446 (2.4503) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][250/625] eta 0:02:12 lr 0.000213 wd 0.0500 time 0.3473 (0.3531) data time 0.0007 (0.0039) model time 0.3466 (0.3491) loss 3.2278 (2.6968) grad_norm 1.9989 (2.4476) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][260/625] eta 0:02:08 lr 0.000213 wd 0.0500 time 0.3490 (0.3529) data time 0.0009 (0.0038) model time 0.3481 (0.3490) loss 2.7799 (2.6886) grad_norm 3.3488 (2.4454) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][270/625] eta 0:02:05 lr 0.000213 wd 0.0500 time 0.3538 (0.3526) data time 0.0007 (0.0037) model time 0.3531 (0.3488) loss 2.9512 (2.6956) grad_norm 2.6237 (2.4482) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][280/625] eta 0:02:01 lr 0.000213 wd 0.0500 time 0.3469 (0.3524) data time 0.0006 (0.0036) model time 0.3463 (0.3487) loss 2.5581 (2.6897) grad_norm 3.6487 (2.4732) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:44:28 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 11:44:28 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:44:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:46:17 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 11:46:19 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 11:46:33 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 11:46:44 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 11:46:44 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 11:46:46 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 11:46:49 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 11:46:49 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 243) [2024-09-01 11:46:49 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 11:47:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][290/625] eta 0:10:09 lr 0.000213 wd 0.0500 time 0.3796 (1.8193) data time 0.0012 (0.0926) model time 0.3784 (1.7266) loss 2.9755 (3.0758) grad_norm 2.0048 (2.4992) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][300/625] eta 0:05:56 lr 0.000212 wd 0.0500 time 0.3753 (1.0961) data time 0.0009 (0.0470) model time 0.3745 (1.0491) loss 3.4557 (2.9872) grad_norm 1.5004 (2.4454) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][310/625] eta 0:04:31 lr 0.000212 wd 0.0500 time 0.3654 (0.8626) data time 0.0012 (0.0317) model time 0.3642 (0.8309) loss 3.2756 (3.0226) grad_norm 2.7500 (3.4395) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][320/625] eta 0:03:45 lr 0.000212 wd 0.0500 time 0.3785 (0.7403) data time 0.0009 (0.0241) model time 0.3776 (0.7162) loss 2.2691 (2.9584) grad_norm 2.2969 (3.1693) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][330/625] eta 0:03:18 lr 0.000212 wd 0.0500 time 0.3729 (0.6727) data time 0.0013 (0.0195) model time 0.3716 (0.6531) loss 2.4591 (2.9263) grad_norm 2.4796 (3.1209) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][340/625] eta 0:02:57 lr 0.000212 wd 0.0500 time 0.3721 (0.6226) data time 0.0010 (0.0165) model time 0.3711 (0.6061) loss 3.1236 (2.9045) grad_norm 1.9425 (2.9388) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][350/625] eta 0:02:41 lr 0.000212 wd 0.0500 time 0.3769 (0.5869) data time 0.0009 (0.0143) model time 0.3760 (0.5727) loss 2.1228 (2.8726) grad_norm 1.8951 (2.8001) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][360/625] eta 0:02:28 lr 0.000212 wd 0.0500 time 0.3721 (0.5604) data time 0.0011 (0.0127) model time 0.3710 (0.5477) loss 2.7824 (2.8592) grad_norm 2.5179 (2.7215) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][370/625] eta 0:02:17 lr 0.000212 wd 0.0500 time 0.3760 (0.5397) data time 0.0009 (0.0114) model time 0.3752 (0.5283) loss 3.2269 (2.8431) grad_norm 2.4426 (2.9449) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][380/625] eta 0:02:08 lr 0.000212 wd 0.0500 time 0.3784 (0.5228) data time 0.0011 (0.0104) model time 0.3773 (0.5125) loss 3.1073 (2.8513) grad_norm 2.3134 (2.9027) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][390/625] eta 0:01:59 lr 0.000212 wd 0.0500 time 0.3711 (0.5091) data time 0.0011 (0.0095) model time 0.3700 (0.4995) loss 2.3104 (2.8490) grad_norm 2.1873 (2.8798) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][400/625] eta 0:01:51 lr 0.000211 wd 0.0500 time 0.3674 (0.4976) data time 0.0009 (0.0088) model time 0.3665 (0.4887) loss 3.4446 (2.8511) grad_norm 1.8102 (2.8840) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:47:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][410/625] eta 0:01:44 lr 0.000211 wd 0.0500 time 0.3718 (0.4880) data time 0.0009 (0.0082) model time 0.3709 (0.4798) loss 3.0989 (2.8402) grad_norm 2.5151 (2.8907) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][420/625] eta 0:01:38 lr 0.000211 wd 0.0500 time 0.3728 (0.4796) data time 0.0009 (0.0077) model time 0.3720 (0.4719) loss 1.8632 (2.8333) grad_norm 1.6596 (2.8508) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][430/625] eta 0:01:32 lr 0.000211 wd 0.0500 time 0.3745 (0.4725) data time 0.0011 (0.0073) model time 0.3734 (0.4652) loss 3.0069 (2.8270) grad_norm 2.0041 (2.8316) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][440/625] eta 0:01:26 lr 0.000211 wd 0.0500 time 0.3718 (0.4660) data time 0.0013 (0.0069) model time 0.3705 (0.4591) loss 2.8570 (2.8199) grad_norm 3.3224 (2.8998) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][450/625] eta 0:01:20 lr 0.000211 wd 0.0500 time 0.3687 (0.4605) data time 0.0009 (0.0066) model time 0.3678 (0.4539) loss 2.1241 (2.8207) grad_norm 2.9598 (2.8873) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][460/625] eta 0:01:15 lr 0.000211 wd 0.0500 time 0.3742 (0.4556) data time 0.0008 (0.0063) model time 0.3734 (0.4493) loss 2.0258 (2.8053) grad_norm 2.5355 (2.8494) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][470/625] eta 0:01:09 lr 0.000211 wd 0.0500 time 0.3743 (0.4514) data time 0.0009 (0.0060) model time 0.3734 (0.4453) loss 2.7930 (2.8148) grad_norm 3.1257 (2.8294) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][480/625] eta 0:01:04 lr 0.000211 wd 0.0500 time 0.3658 (0.4474) data time 0.0011 (0.0058) model time 0.3647 (0.4416) loss 3.3480 (2.8053) grad_norm 3.2907 (2.8202) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][490/625] eta 0:00:59 lr 0.000210 wd 0.0500 time 0.3699 (0.4438) data time 0.0008 (0.0056) model time 0.3691 (0.4382) loss 2.9563 (2.7964) grad_norm 2.7534 (2.8319) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][500/625] eta 0:00:55 lr 0.000210 wd 0.0500 time 0.3646 (0.4405) data time 0.0011 (0.0054) model time 0.3636 (0.4352) loss 2.4800 (2.7885) grad_norm 2.8780 (2.8245) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][510/625] eta 0:00:50 lr 0.000210 wd 0.0500 time 0.3735 (0.4385) data time 0.0011 (0.0052) model time 0.3724 (0.4333) loss 3.3557 (2.7895) grad_norm 1.7027 (2.7982) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][520/625] eta 0:00:45 lr 0.000210 wd 0.0500 time 0.3730 (0.4365) data time 0.0011 (0.0050) model time 0.3718 (0.4315) loss 3.0218 (2.7820) grad_norm 3.0519 (2.7812) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][530/625] eta 0:00:41 lr 0.000210 wd 0.0500 time 0.3688 (0.4339) data time 0.0010 (0.0049) model time 0.3678 (0.4290) loss 2.1930 (2.7756) grad_norm 2.7431 (2.7706) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][540/625] eta 0:00:36 lr 0.000210 wd 0.0500 time 0.3867 (0.4317) data time 0.0011 (0.0047) model time 0.3855 (0.4270) loss 2.2784 (2.7672) grad_norm 2.1244 (2.7678) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][550/625] eta 0:00:32 lr 0.000210 wd 0.0500 time 0.3708 (0.4296) data time 0.0008 (0.0046) model time 0.3700 (0.4250) loss 3.9100 (2.7654) grad_norm 2.1131 (2.9118) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][560/625] eta 0:00:27 lr 0.000210 wd 0.0500 time 0.3667 (0.4275) data time 0.0011 (0.0045) model time 0.3656 (0.4231) loss 2.9859 (2.7736) grad_norm 2.3008 (2.9034) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:48:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][570/625] eta 0:00:23 lr 0.000210 wd 0.0500 time 0.3555 (0.4256) data time 0.0011 (0.0043) model time 0.3544 (0.4212) loss 2.1305 (2.7683) grad_norm 2.5485 (2.8769) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][580/625] eta 0:00:19 lr 0.000210 wd 0.0500 time 0.3618 (0.4237) data time 0.0008 (0.0042) model time 0.3610 (0.4194) loss 2.5340 (2.7574) grad_norm 2.1903 (2.8437) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:49:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][590/625] eta 0:00:14 lr 0.000209 wd 0.0500 time 0.3724 (0.4221) data time 0.0012 (0.0041) model time 0.3712 (0.4179) loss 2.5912 (2.7563) grad_norm 45.9270 (2.9656) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:49:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][600/625] eta 0:00:10 lr 0.000209 wd 0.0500 time 0.3793 (0.4206) data time 0.0013 (0.0040) model time 0.3780 (0.4165) loss 2.3610 (2.7656) grad_norm 2.8351 (2.9543) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][610/625] eta 0:00:06 lr 0.000209 wd 0.0500 time 0.3661 (0.4191) data time 0.0006 (0.0040) model time 0.3656 (0.4151) loss 2.9599 (2.7698) grad_norm 1.9622 (2.9468) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:49:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [243/300][620/625] eta 0:00:02 lr 0.000209 wd 0.0500 time 0.3566 (0.4173) data time 0.0008 (0.0039) model time 0.3558 (0.4134) loss 2.7542 (2.7704) grad_norm 2.3062 (2.9241) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 11:49:17 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 243 training takes 0:02:23 [2024-09-01 11:49:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:49:22 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:49:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.484 (0.484) Loss 0.5225 (0.5225) Acc@1 89.209 (89.209) Acc@5 98.730 (98.730) Mem 11219MB [2024-09-01 11:49:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.141) Loss 0.8652 (0.6256) Acc@1 80.469 (86.648) Acc@5 95.605 (97.781) Mem 11219MB [2024-09-01 11:49:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 0.9155 (0.7409) Acc@1 78.516 (83.650) Acc@5 95.020 (96.519) Mem 11219MB [2024-09-01 11:49:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.309 Acc@5 96.521 [2024-09-01 11:49:26 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 11:49:26 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.31% [2024-09-01 11:49:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 11:49:31 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 11:49:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.4873 (0.4873) Acc@1 89.551 (89.551) Acc@5 98.682 (98.682) Mem 11219MB [2024-09-01 11:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.143) Loss 0.8057 (0.5928) Acc@1 80.664 (86.972) Acc@5 96.143 (97.843) Mem 11219MB [2024-09-01 11:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.120) Loss 0.8716 (0.7022) Acc@1 79.199 (84.203) Acc@5 95.459 (96.756) Mem 11219MB [2024-09-01 11:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.903 Acc@5 96.803 [2024-09-01 11:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.90% [2024-09-01 11:49:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 11:49:39 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 11:49:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][0/625] eta 0:08:48 lr 0.000209 wd 0.0500 time 0.8454 (0.8454) data time 0.4544 (0.4544) model time 0.0000 (0.0000) loss 2.1717 (2.1717) grad_norm 2.3458 (2.3458) loss_scale 128.0000 (128.0000) mem 11227MB [2024-09-01 11:49:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][10/625] eta 0:04:16 lr 0.000209 wd 0.0500 time 0.3723 (0.4169) data time 0.0011 (0.0424) model time 0.0000 (0.0000) loss 2.7440 (2.8300) grad_norm 2.0009 (2.2006) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][20/625] eta 0:03:59 lr 0.000209 wd 0.0500 time 0.3670 (0.3957) data time 0.0011 (0.0228) model time 0.0000 (0.0000) loss 2.5401 (2.7305) grad_norm 2.4005 (2.4614) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:49:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][30/625] eta 0:03:51 lr 0.000209 wd 0.0500 time 0.3766 (0.3888) data time 0.0008 (0.0159) model time 0.0000 (0.0000) loss 2.3988 (2.7576) grad_norm 3.0749 (2.5747) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:49:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][40/625] eta 0:03:45 lr 0.000209 wd 0.0500 time 0.3682 (0.3847) data time 0.0011 (0.0123) model time 0.0000 (0.0000) loss 2.2709 (2.6873) grad_norm 1.7986 (2.7035) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:49:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][50/625] eta 0:03:39 lr 0.000209 wd 0.0500 time 0.3693 (0.3818) data time 0.0011 (0.0101) model time 0.0000 (0.0000) loss 2.8105 (2.7210) grad_norm 1.3761 (2.6508) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][60/625] eta 0:03:34 lr 0.000208 wd 0.0500 time 0.3692 (0.3797) data time 0.0010 (0.0086) model time 0.3682 (0.3679) loss 2.7067 (2.7507) grad_norm 2.1309 (2.6699) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][70/625] eta 0:03:29 lr 0.000208 wd 0.0500 time 0.3667 (0.3780) data time 0.0011 (0.0076) model time 0.3657 (0.3671) loss 2.8299 (2.7709) grad_norm 2.1713 (2.5987) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][80/625] eta 0:03:25 lr 0.000208 wd 0.0500 time 0.3692 (0.3768) data time 0.0009 (0.0068) model time 0.3683 (0.3673) loss 3.4727 (2.7677) grad_norm 2.3095 (2.5124) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][90/625] eta 0:03:21 lr 0.000208 wd 0.0500 time 0.3693 (0.3761) data time 0.0008 (0.0062) model time 0.3685 (0.3678) loss 3.1140 (2.7956) grad_norm 2.6687 (2.6090) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][100/625] eta 0:03:17 lr 0.000208 wd 0.0500 time 0.3738 (0.3757) data time 0.0013 (0.0057) model time 0.3725 (0.3684) loss 2.6018 (2.7966) grad_norm 3.4310 (2.6128) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][110/625] eta 0:03:13 lr 0.000208 wd 0.0500 time 0.3749 (0.3753) data time 0.0009 (0.0053) model time 0.3741 (0.3686) loss 2.7523 (2.7857) grad_norm 1.7938 (2.5504) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][120/625] eta 0:03:09 lr 0.000208 wd 0.0500 time 0.3719 (0.3751) data time 0.0009 (0.0049) model time 0.3710 (0.3691) loss 2.7987 (2.7623) grad_norm 2.6225 (2.5079) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][130/625] eta 0:03:05 lr 0.000208 wd 0.0500 time 0.3655 (0.3749) data time 0.0009 (0.0046) model time 0.3646 (0.3693) loss 2.7293 (2.7381) grad_norm 2.9259 (2.5196) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][140/625] eta 0:03:01 lr 0.000208 wd 0.0500 time 0.3742 (0.3748) data time 0.0009 (0.0044) model time 0.3733 (0.3697) loss 2.8555 (2.7434) grad_norm 2.7590 (2.5514) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][150/625] eta 0:02:57 lr 0.000208 wd 0.0500 time 0.3691 (0.3746) data time 0.0008 (0.0042) model time 0.3682 (0.3698) loss 3.0959 (2.7414) grad_norm 2.2458 (2.6333) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][160/625] eta 0:02:54 lr 0.000207 wd 0.0500 time 0.3727 (0.3746) data time 0.0011 (0.0040) model time 0.3717 (0.3701) loss 3.0233 (2.7403) grad_norm 2.9333 (2.6249) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][170/625] eta 0:02:50 lr 0.000207 wd 0.0500 time 0.3583 (0.3746) data time 0.0010 (0.0038) model time 0.3572 (0.3703) loss 2.5935 (2.7554) grad_norm 2.2647 (2.6114) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][180/625] eta 0:02:46 lr 0.000207 wd 0.0500 time 0.3677 (0.3742) data time 0.0010 (0.0037) model time 0.3667 (0.3700) loss 3.0854 (2.7448) grad_norm 2.0399 (2.5911) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][190/625] eta 0:02:42 lr 0.000207 wd 0.0500 time 0.3815 (0.3740) data time 0.0009 (0.0035) model time 0.3806 (0.3700) loss 2.6375 (2.7384) grad_norm 2.4261 (2.5852) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][200/625] eta 0:02:38 lr 0.000207 wd 0.0500 time 0.3814 (0.3739) data time 0.0009 (0.0034) model time 0.3805 (0.3701) loss 2.1603 (2.7387) grad_norm 2.0703 (2.5870) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:50:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][210/625] eta 0:02:35 lr 0.000207 wd 0.0500 time 0.3776 (0.3758) data time 0.0011 (0.0033) model time 0.3765 (0.3728) loss 2.8464 (2.7507) grad_norm 2.3041 (2.5965) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][220/625] eta 0:02:32 lr 0.000207 wd 0.0500 time 0.3888 (0.3757) data time 0.0012 (0.0032) model time 0.3876 (0.3727) loss 2.8382 (2.7594) grad_norm 2.1144 (2.5890) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][230/625] eta 0:02:28 lr 0.000207 wd 0.0500 time 0.3621 (0.3754) data time 0.0011 (0.0031) model time 0.3610 (0.3724) loss 2.9464 (2.7676) grad_norm 2.0735 (2.5889) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][240/625] eta 0:02:24 lr 0.000207 wd 0.0500 time 0.3771 (0.3752) data time 0.0011 (0.0031) model time 0.3760 (0.3722) loss 2.6140 (2.7662) grad_norm 1.6160 (2.5705) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][250/625] eta 0:02:20 lr 0.000206 wd 0.0500 time 0.3736 (0.3750) data time 0.0008 (0.0030) model time 0.3728 (0.3721) loss 3.2148 (2.7678) grad_norm 2.2794 (2.5771) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][260/625] eta 0:02:16 lr 0.000206 wd 0.0500 time 0.3695 (0.3749) data time 0.0009 (0.0029) model time 0.3686 (0.3721) loss 3.4572 (2.7596) grad_norm 2.1298 (2.5569) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][270/625] eta 0:02:13 lr 0.000206 wd 0.0500 time 0.3699 (0.3748) data time 0.0009 (0.0029) model time 0.3690 (0.3720) loss 3.1143 (2.7616) grad_norm 2.3032 (2.5669) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][280/625] eta 0:02:09 lr 0.000206 wd 0.0500 time 0.3585 (0.3746) data time 0.0011 (0.0028) model time 0.3574 (0.3718) loss 2.8872 (2.7691) grad_norm 1.5330 (2.5484) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][290/625] eta 0:02:05 lr 0.000206 wd 0.0500 time 0.3718 (0.3745) data time 0.0012 (0.0027) model time 0.3705 (0.3717) loss 2.5945 (2.7677) grad_norm 1.9714 (2.5290) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][300/625] eta 0:02:01 lr 0.000206 wd 0.0500 time 0.3695 (0.3743) data time 0.0009 (0.0027) model time 0.3686 (0.3716) loss 2.1156 (2.7603) grad_norm 2.1847 (2.5177) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][310/625] eta 0:01:57 lr 0.000206 wd 0.0500 time 0.3594 (0.3742) data time 0.0009 (0.0026) model time 0.3585 (0.3716) loss 2.9289 (2.7616) grad_norm 2.3194 (2.5114) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][320/625] eta 0:01:54 lr 0.000206 wd 0.0500 time 0.3735 (0.3742) data time 0.0009 (0.0026) model time 0.3727 (0.3716) loss 2.8729 (2.7556) grad_norm 1.8364 (2.4995) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][330/625] eta 0:01:50 lr 0.000206 wd 0.0500 time 0.3748 (0.3742) data time 0.0008 (0.0025) model time 0.3740 (0.3717) loss 2.9319 (2.7660) grad_norm 3.1254 (2.4955) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][340/625] eta 0:01:46 lr 0.000206 wd 0.0500 time 0.3744 (0.3741) data time 0.0013 (0.0025) model time 0.3732 (0.3716) loss 2.8045 (2.7659) grad_norm 1.9094 (2.4914) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][350/625] eta 0:01:42 lr 0.000205 wd 0.0500 time 0.3783 (0.3740) data time 0.0011 (0.0025) model time 0.3773 (0.3716) loss 2.6961 (2.7616) grad_norm 1.8812 (2.4779) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][360/625] eta 0:01:39 lr 0.000205 wd 0.0500 time 0.3719 (0.3739) data time 0.0009 (0.0024) model time 0.3710 (0.3715) loss 3.2517 (2.7566) grad_norm 12.6616 (2.5000) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:51:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][370/625] eta 0:01:35 lr 0.000205 wd 0.0500 time 0.3755 (0.3739) data time 0.0012 (0.0024) model time 0.3743 (0.3714) loss 2.8155 (2.7483) grad_norm 2.8150 (2.4982) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][380/625] eta 0:01:31 lr 0.000205 wd 0.0500 time 0.3630 (0.3738) data time 0.0008 (0.0024) model time 0.3622 (0.3714) loss 2.4022 (2.7466) grad_norm 2.0634 (2.5046) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][390/625] eta 0:01:27 lr 0.000205 wd 0.0500 time 0.3703 (0.3743) data time 0.0009 (0.0023) model time 0.3695 (0.3720) loss 3.2172 (2.7517) grad_norm 1.8361 (2.4983) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][400/625] eta 0:01:24 lr 0.000205 wd 0.0500 time 0.3831 (0.3753) data time 0.0011 (0.0023) model time 0.3820 (0.3732) loss 2.8162 (2.7526) grad_norm 2.3319 (2.5045) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][410/625] eta 0:01:20 lr 0.000205 wd 0.0500 time 0.3791 (0.3753) data time 0.0012 (0.0023) model time 0.3779 (0.3732) loss 2.2440 (2.7519) grad_norm 2.1014 (2.4953) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][420/625] eta 0:01:16 lr 0.000205 wd 0.0500 time 0.3704 (0.3751) data time 0.0012 (0.0023) model time 0.3692 (0.3730) loss 2.9658 (2.7586) grad_norm 2.9110 (2.5031) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][430/625] eta 0:01:13 lr 0.000205 wd 0.0500 time 0.3677 (0.3750) data time 0.0011 (0.0022) model time 0.3666 (0.3729) loss 2.2394 (2.7621) grad_norm 2.4112 (2.5184) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][440/625] eta 0:01:09 lr 0.000205 wd 0.0500 time 0.3700 (0.3749) data time 0.0011 (0.0022) model time 0.3689 (0.3729) loss 2.0257 (2.7615) grad_norm 2.5698 (2.5403) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][450/625] eta 0:01:05 lr 0.000204 wd 0.0500 time 0.3761 (0.3750) data time 0.0009 (0.0022) model time 0.3752 (0.3730) loss 2.5304 (2.7615) grad_norm 3.1302 (2.5426) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][460/625] eta 0:01:01 lr 0.000204 wd 0.0500 time 0.3728 (0.3750) data time 0.0009 (0.0022) model time 0.3719 (0.3730) loss 3.1823 (2.7604) grad_norm 3.5545 (2.5812) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][470/625] eta 0:00:58 lr 0.000204 wd 0.0500 time 0.3622 (0.3749) data time 0.0010 (0.0021) model time 0.3613 (0.3729) loss 2.8590 (2.7535) grad_norm 1.3298 (2.5689) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][480/625] eta 0:00:54 lr 0.000204 wd 0.0500 time 0.3734 (0.3748) data time 0.0010 (0.0021) model time 0.3724 (0.3728) loss 1.9353 (2.7526) grad_norm 2.0994 (2.5630) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][490/625] eta 0:00:50 lr 0.000204 wd 0.0500 time 0.3708 (0.3748) data time 0.0012 (0.0021) model time 0.3696 (0.3728) loss 2.2078 (2.7478) grad_norm 2.5817 (2.5647) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][500/625] eta 0:00:46 lr 0.000204 wd 0.0500 time 0.3803 (0.3747) data time 0.0009 (0.0021) model time 0.3795 (0.3727) loss 1.9633 (2.7465) grad_norm 2.6731 (2.5718) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][510/625] eta 0:00:43 lr 0.000204 wd 0.0500 time 0.5838 (0.3754) data time 0.0008 (0.0021) model time 0.5829 (0.3735) loss 3.2528 (2.7476) grad_norm 2.7515 (2.5698) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][520/625] eta 0:00:39 lr 0.000204 wd 0.0500 time 0.3705 (0.3758) data time 0.0011 (0.0020) model time 0.3694 (0.3739) loss 2.5967 (2.7461) grad_norm 2.5844 (2.5643) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:52:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][530/625] eta 0:00:35 lr 0.000204 wd 0.0500 time 0.3762 (0.3757) data time 0.0008 (0.0020) model time 0.3754 (0.3739) loss 2.6861 (2.7484) grad_norm 2.6647 (2.5594) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][540/625] eta 0:00:31 lr 0.000203 wd 0.0500 time 0.3697 (0.3757) data time 0.0010 (0.0020) model time 0.3687 (0.3739) loss 2.3944 (2.7447) grad_norm 1.9635 (2.5564) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][550/625] eta 0:00:28 lr 0.000203 wd 0.0500 time 0.3742 (0.3756) data time 0.0011 (0.0020) model time 0.3731 (0.3738) loss 2.7958 (2.7449) grad_norm 2.6867 (2.5549) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][560/625] eta 0:00:24 lr 0.000203 wd 0.0500 time 0.3698 (0.3756) data time 0.0009 (0.0020) model time 0.3689 (0.3738) loss 3.3055 (2.7474) grad_norm 1.8115 (2.5461) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][570/625] eta 0:00:20 lr 0.000203 wd 0.0500 time 0.3743 (0.3756) data time 0.0008 (0.0020) model time 0.3735 (0.3738) loss 3.7958 (2.7469) grad_norm 2.6355 (2.5424) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][580/625] eta 0:00:16 lr 0.000203 wd 0.0500 time 0.3808 (0.3756) data time 0.0009 (0.0020) model time 0.3799 (0.3738) loss 3.2140 (2.7519) grad_norm 2.0912 (2.5463) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][590/625] eta 0:00:13 lr 0.000203 wd 0.0500 time 0.3736 (0.3756) data time 0.0008 (0.0019) model time 0.3728 (0.3739) loss 1.8592 (2.7524) grad_norm 1.5851 (2.5426) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][600/625] eta 0:00:09 lr 0.000203 wd 0.0500 time 0.3692 (0.3757) data time 0.0008 (0.0019) model time 0.3684 (0.3739) loss 3.1664 (2.7481) grad_norm 3.2887 (2.5563) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][610/625] eta 0:00:05 lr 0.000203 wd 0.0500 time 0.3638 (0.3756) data time 0.0008 (0.0019) model time 0.3630 (0.3739) loss 2.8700 (2.7438) grad_norm 2.1304 (2.5538) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [244/300][620/625] eta 0:00:01 lr 0.000203 wd 0.0500 time 0.3540 (0.3753) data time 0.0008 (0.0019) model time 0.3532 (0.3735) loss 2.3143 (2.7435) grad_norm 1.9677 (2.5574) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:33 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 244 training takes 0:03:54 [2024-09-01 11:53:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:53:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:53:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.5210 (0.5210) Acc@1 88.916 (88.916) Acc@5 98.438 (98.438) Mem 11224MB [2024-09-01 11:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 0.8472 (0.6180) Acc@1 79.883 (86.506) Acc@5 95.508 (97.683) Mem 11224MB [2024-09-01 11:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9175 (0.7361) Acc@1 77.686 (83.431) Acc@5 95.166 (96.470) Mem 11224MB [2024-09-01 11:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.145 Acc@5 96.471 [2024-09-01 11:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 11:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.831 (0.831) Loss 0.4875 (0.4875) Acc@1 89.600 (89.600) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 11:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.176) Loss 0.8062 (0.5927) Acc@1 80.664 (87.025) Acc@5 96.045 (97.843) Mem 11224MB [2024-09-01 11:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8711 (0.7020) Acc@1 79.004 (84.212) Acc@5 95.459 (96.766) Mem 11224MB [2024-09-01 11:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.917 Acc@5 96.807 [2024-09-01 11:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.92% [2024-09-01 11:53:40 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 11:53:42 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 11:53:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][0/625] eta 0:07:57 lr 0.000203 wd 0.0500 time 0.7634 (0.7634) data time 0.4031 (0.4031) model time 0.0000 (0.0000) loss 2.8278 (2.8278) grad_norm 2.3286 (2.3286) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][10/625] eta 0:04:09 lr 0.000202 wd 0.0500 time 0.3644 (0.4055) data time 0.0008 (0.0377) model time 0.0000 (0.0000) loss 3.5878 (2.7738) grad_norm 4.9488 (2.5298) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][20/625] eta 0:03:56 lr 0.000202 wd 0.0500 time 0.3798 (0.3904) data time 0.0012 (0.0203) model time 0.0000 (0.0000) loss 2.8357 (2.7425) grad_norm 2.4410 (2.4505) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][30/625] eta 0:03:49 lr 0.000202 wd 0.0500 time 0.3731 (0.3849) data time 0.0009 (0.0142) model time 0.0000 (0.0000) loss 2.5245 (2.7460) grad_norm 2.7401 (2.4249) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][40/625] eta 0:03:45 lr 0.000202 wd 0.0500 time 0.3712 (0.3852) data time 0.0013 (0.0110) model time 0.0000 (0.0000) loss 3.0421 (2.7649) grad_norm 2.7523 (2.6047) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][50/625] eta 0:03:40 lr 0.000202 wd 0.0500 time 0.3725 (0.3831) data time 0.0009 (0.0090) model time 0.0000 (0.0000) loss 3.0503 (2.7548) grad_norm 2.0148 (2.5532) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][60/625] eta 0:03:39 lr 0.000202 wd 0.0500 time 0.3777 (0.3878) data time 0.0009 (0.0077) model time 0.3769 (0.4105) loss 2.3796 (2.7082) grad_norm 2.9241 (2.9364) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][70/625] eta 0:03:34 lr 0.000202 wd 0.0500 time 0.3752 (0.3857) data time 0.0008 (0.0068) model time 0.3744 (0.3910) loss 2.1409 (2.7243) grad_norm 1.8103 (2.9022) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][80/625] eta 0:03:29 lr 0.000202 wd 0.0500 time 0.3774 (0.3840) data time 0.0011 (0.0061) model time 0.3763 (0.3845) loss 3.3277 (2.7357) grad_norm 2.1616 (2.9218) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][90/625] eta 0:03:24 lr 0.000202 wd 0.0500 time 0.3619 (0.3826) data time 0.0010 (0.0056) model time 0.3609 (0.3808) loss 2.5323 (2.7441) grad_norm 2.3862 (2.8328) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][100/625] eta 0:03:20 lr 0.000202 wd 0.0500 time 0.3740 (0.3821) data time 0.0009 (0.0051) model time 0.3731 (0.3800) loss 2.1891 (2.7312) grad_norm 1.7216 (2.7898) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][110/625] eta 0:03:16 lr 0.000201 wd 0.0500 time 0.3713 (0.3817) data time 0.0008 (0.0048) model time 0.3704 (0.3793) loss 2.6554 (2.7333) grad_norm 2.0408 (2.7448) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][120/625] eta 0:03:12 lr 0.000201 wd 0.0500 time 0.3804 (0.3813) data time 0.0008 (0.0045) model time 0.3795 (0.3787) loss 3.1074 (2.7192) grad_norm 3.9384 (2.7166) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][130/625] eta 0:03:08 lr 0.000201 wd 0.0500 time 0.3727 (0.3808) data time 0.0011 (0.0042) model time 0.3716 (0.3782) loss 3.1224 (2.7222) grad_norm 11.8249 (2.7855) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][140/625] eta 0:03:04 lr 0.000201 wd 0.0500 time 0.3756 (0.3807) data time 0.0010 (0.0040) model time 0.3746 (0.3781) loss 3.2285 (2.7358) grad_norm 2.7395 (2.7710) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][150/625] eta 0:03:00 lr 0.000201 wd 0.0500 time 0.3799 (0.3805) data time 0.0010 (0.0038) model time 0.3789 (0.3779) loss 2.6334 (2.7513) grad_norm 1.3960 (2.7316) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][160/625] eta 0:02:56 lr 0.000201 wd 0.0500 time 0.3704 (0.3803) data time 0.0014 (0.0037) model time 0.3690 (0.3777) loss 2.9530 (2.7513) grad_norm 3.5774 (2.7041) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][170/625] eta 0:02:52 lr 0.000201 wd 0.0500 time 0.3807 (0.3801) data time 0.0010 (0.0035) model time 0.3797 (0.3776) loss 3.2126 (2.7532) grad_norm 1.8923 (2.7083) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][180/625] eta 0:02:49 lr 0.000201 wd 0.0500 time 0.3728 (0.3801) data time 0.0011 (0.0034) model time 0.3718 (0.3777) loss 2.6747 (2.7517) grad_norm 1.8246 (2.6802) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][190/625] eta 0:02:45 lr 0.000201 wd 0.0500 time 0.3704 (0.3799) data time 0.0008 (0.0033) model time 0.3696 (0.3775) loss 2.1162 (2.7453) grad_norm 2.3923 (2.6679) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:54:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][200/625] eta 0:02:41 lr 0.000201 wd 0.0500 time 0.3642 (0.3796) data time 0.0010 (0.0032) model time 0.3632 (0.3773) loss 3.1585 (2.7461) grad_norm 2.3333 (2.6461) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:55:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][210/625] eta 0:02:37 lr 0.000200 wd 0.0500 time 0.3968 (0.3795) data time 0.0010 (0.0031) model time 0.3957 (0.3772) loss 3.1476 (2.7562) grad_norm 2.2297 (2.6132) loss_scale 256.0000 (131.0332) mem 11224MB [2024-09-01 11:55:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][220/625] eta 0:02:33 lr 0.000200 wd 0.0500 time 0.5465 (0.3801) data time 0.0009 (0.0030) model time 0.5456 (0.3780) loss 3.3025 (2.7661) grad_norm 1.9169 (2.5925) loss_scale 256.0000 (136.6878) mem 11224MB [2024-09-01 11:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][230/625] eta 0:02:29 lr 0.000200 wd 0.0500 time 0.3697 (0.3796) data time 0.0009 (0.0029) model time 0.3688 (0.3774) loss 3.3272 (2.7692) grad_norm 1.9301 (2.5829) loss_scale 256.0000 (141.8528) mem 11224MB [2024-09-01 11:55:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][240/625] eta 0:02:26 lr 0.000200 wd 0.0500 time 0.3690 (0.3794) data time 0.0012 (0.0028) model time 0.3678 (0.3772) loss 2.5889 (2.7694) grad_norm 2.4635 (2.5730) loss_scale 256.0000 (146.5892) mem 11224MB [2024-09-01 11:55:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][250/625] eta 0:02:22 lr 0.000200 wd 0.0500 time 0.3717 (0.3792) data time 0.0010 (0.0028) model time 0.3707 (0.3771) loss 2.7836 (2.7662) grad_norm 2.9478 (2.5631) loss_scale 256.0000 (150.9482) mem 11224MB [2024-09-01 11:55:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][260/625] eta 0:02:18 lr 0.000200 wd 0.0500 time 0.3651 (0.3791) data time 0.0011 (0.0027) model time 0.3640 (0.3770) loss 2.6822 (2.7612) grad_norm 2.6590 (2.6008) loss_scale 256.0000 (154.9732) mem 11224MB [2024-09-01 11:55:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][270/625] eta 0:02:15 lr 0.000200 wd 0.0500 time 0.3585 (0.3807) data time 0.0009 (0.0026) model time 0.3576 (0.3790) loss 2.8051 (2.7606) grad_norm 2.0869 (2.5883) loss_scale 256.0000 (158.7011) mem 11224MB [2024-09-01 11:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][280/625] eta 0:02:11 lr 0.000200 wd 0.0500 time 0.3656 (0.3804) data time 0.0011 (0.0026) model time 0.3645 (0.3786) loss 1.8863 (2.7511) grad_norm 3.9731 (2.5882) loss_scale 256.0000 (162.1637) mem 11224MB [2024-09-01 11:55:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][290/625] eta 0:02:07 lr 0.000200 wd 0.0500 time 0.3741 (0.3801) data time 0.0011 (0.0025) model time 0.3729 (0.3783) loss 3.0803 (2.7538) grad_norm 2.4820 (2.5943) loss_scale 256.0000 (165.3883) mem 11224MB [2024-09-01 11:55:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][300/625] eta 0:02:03 lr 0.000200 wd 0.0500 time 0.3722 (0.3798) data time 0.0011 (0.0025) model time 0.3711 (0.3779) loss 2.8221 (2.7580) grad_norm 1.5851 (2.5984) loss_scale 256.0000 (168.3987) mem 11224MB [2024-09-01 11:55:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][310/625] eta 0:01:59 lr 0.000199 wd 0.0500 time 0.3695 (0.3796) data time 0.0009 (0.0025) model time 0.3686 (0.3777) loss 3.5528 (2.7616) grad_norm 2.0342 (2.5801) loss_scale 256.0000 (171.2154) mem 11224MB [2024-09-01 11:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][320/625] eta 0:01:55 lr 0.000199 wd 0.0500 time 0.3747 (0.3795) data time 0.0011 (0.0024) model time 0.3736 (0.3777) loss 2.7462 (2.7635) grad_norm 2.8905 (2.6027) loss_scale 256.0000 (173.8567) mem 11224MB [2024-09-01 11:55:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][330/625] eta 0:01:51 lr 0.000199 wd 0.0500 time 0.3727 (0.3794) data time 0.0011 (0.0024) model time 0.3717 (0.3776) loss 2.8617 (2.7656) grad_norm 3.3787 (2.6318) loss_scale 256.0000 (176.3384) mem 11224MB [2024-09-01 11:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][340/625] eta 0:01:48 lr 0.000199 wd 0.0500 time 0.6033 (0.3800) data time 0.0008 (0.0023) model time 0.6025 (0.3783) loss 3.1346 (2.7642) grad_norm 2.5873 (2.6388) loss_scale 256.0000 (178.6745) mem 11224MB [2024-09-01 11:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][350/625] eta 0:01:44 lr 0.000199 wd 0.0500 time 0.3743 (0.3798) data time 0.0008 (0.0023) model time 0.3735 (0.3780) loss 3.1695 (2.7576) grad_norm 2.5252 (2.6375) loss_scale 256.0000 (180.8775) mem 11224MB [2024-09-01 11:55:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][360/625] eta 0:01:40 lr 0.000199 wd 0.0500 time 0.3565 (0.3796) data time 0.0012 (0.0023) model time 0.3553 (0.3779) loss 3.2744 (2.7579) grad_norm inf (inf) loss_scale 128.0000 (182.6039) mem 11224MB [2024-09-01 11:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][370/625] eta 0:01:36 lr 0.000199 wd 0.0500 time 0.3739 (0.3795) data time 0.0011 (0.0022) model time 0.3728 (0.3778) loss 2.5775 (2.7564) grad_norm 1.9271 (inf) loss_scale 128.0000 (181.1321) mem 11224MB [2024-09-01 11:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][380/625] eta 0:01:32 lr 0.000199 wd 0.0500 time 0.3742 (0.3795) data time 0.0009 (0.0022) model time 0.3733 (0.3777) loss 3.0311 (2.7563) grad_norm 2.3948 (inf) loss_scale 128.0000 (179.7375) mem 11224MB [2024-09-01 11:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][390/625] eta 0:01:29 lr 0.000199 wd 0.0500 time 0.3735 (0.3794) data time 0.0011 (0.0022) model time 0.3724 (0.3777) loss 2.9145 (2.7617) grad_norm 4.6126 (inf) loss_scale 128.0000 (178.4143) mem 11224MB [2024-09-01 11:56:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][400/625] eta 0:01:25 lr 0.000199 wd 0.0500 time 0.3721 (0.3793) data time 0.0010 (0.0022) model time 0.3711 (0.3776) loss 2.6199 (2.7649) grad_norm 2.4151 (inf) loss_scale 128.0000 (177.1571) mem 11224MB [2024-09-01 11:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][410/625] eta 0:01:21 lr 0.000198 wd 0.0500 time 0.3769 (0.3793) data time 0.0011 (0.0021) model time 0.3758 (0.3776) loss 2.7653 (2.7647) grad_norm 1.8849 (inf) loss_scale 128.0000 (175.9611) mem 11224MB [2024-09-01 11:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][420/625] eta 0:01:17 lr 0.000198 wd 0.0500 time 0.3710 (0.3793) data time 0.0008 (0.0021) model time 0.3702 (0.3776) loss 2.8113 (2.7634) grad_norm 2.3010 (inf) loss_scale 128.0000 (174.8219) mem 11224MB [2024-09-01 11:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][430/625] eta 0:01:13 lr 0.000198 wd 0.0500 time 0.3703 (0.3792) data time 0.0013 (0.0021) model time 0.3690 (0.3775) loss 2.8939 (2.7636) grad_norm 2.4143 (inf) loss_scale 128.0000 (173.7355) mem 11224MB [2024-09-01 11:56:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][440/625] eta 0:01:10 lr 0.000198 wd 0.0500 time 0.3719 (0.3791) data time 0.0010 (0.0021) model time 0.3709 (0.3774) loss 3.1093 (2.7687) grad_norm 2.9405 (inf) loss_scale 128.0000 (172.6984) mem 11224MB [2024-09-01 11:56:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][450/625] eta 0:01:06 lr 0.000198 wd 0.0500 time 0.3729 (0.3798) data time 0.0013 (0.0021) model time 0.3716 (0.3782) loss 3.0402 (2.7670) grad_norm 4.8861 (inf) loss_scale 128.0000 (171.7073) mem 11224MB [2024-09-01 11:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][460/625] eta 0:01:02 lr 0.000198 wd 0.0500 time 0.3745 (0.3796) data time 0.0008 (0.0020) model time 0.3737 (0.3780) loss 3.3655 (2.7678) grad_norm 3.4637 (inf) loss_scale 128.0000 (170.7592) mem 11224MB [2024-09-01 11:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][470/625] eta 0:00:58 lr 0.000198 wd 0.0500 time 0.3773 (0.3796) data time 0.0011 (0.0020) model time 0.3762 (0.3780) loss 2.3925 (2.7670) grad_norm 1.9850 (inf) loss_scale 128.0000 (169.8514) mem 11224MB [2024-09-01 11:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][480/625] eta 0:00:55 lr 0.000198 wd 0.0500 time 0.3802 (0.3795) data time 0.0008 (0.0020) model time 0.3794 (0.3779) loss 2.4850 (2.7692) grad_norm 2.4034 (inf) loss_scale 128.0000 (168.9813) mem 11224MB [2024-09-01 11:56:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][490/625] eta 0:00:51 lr 0.000198 wd 0.0500 time 0.3798 (0.3795) data time 0.0008 (0.0020) model time 0.3790 (0.3779) loss 2.8131 (2.7718) grad_norm 2.7231 (inf) loss_scale 128.0000 (168.1466) mem 11224MB [2024-09-01 11:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][500/625] eta 0:00:47 lr 0.000197 wd 0.0500 time 0.4051 (0.3795) data time 0.0011 (0.0020) model time 0.4040 (0.3779) loss 2.8823 (2.7715) grad_norm 2.1086 (inf) loss_scale 128.0000 (167.3453) mem 11224MB [2024-09-01 11:56:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][510/625] eta 0:00:43 lr 0.000197 wd 0.0500 time 0.3866 (0.3794) data time 0.0009 (0.0020) model time 0.3857 (0.3779) loss 2.1349 (2.7657) grad_norm 1.5063 (inf) loss_scale 128.0000 (166.5753) mem 11224MB [2024-09-01 11:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][520/625] eta 0:00:39 lr 0.000197 wd 0.0500 time 0.3927 (0.3795) data time 0.0010 (0.0019) model time 0.3916 (0.3780) loss 1.7040 (2.7589) grad_norm 2.7075 (inf) loss_scale 128.0000 (165.8349) mem 11224MB [2024-09-01 11:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][530/625] eta 0:00:36 lr 0.000197 wd 0.0500 time 0.3897 (0.3795) data time 0.0011 (0.0019) model time 0.3886 (0.3779) loss 3.0968 (2.7585) grad_norm 3.5145 (inf) loss_scale 128.0000 (165.1224) mem 11224MB [2024-09-01 11:57:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][540/625] eta 0:00:32 lr 0.000197 wd 0.0500 time 0.3718 (0.3794) data time 0.0011 (0.0019) model time 0.3707 (0.3779) loss 1.8382 (2.7599) grad_norm 2.1935 (inf) loss_scale 128.0000 (164.4362) mem 11224MB [2024-09-01 11:57:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][550/625] eta 0:00:28 lr 0.000197 wd 0.0500 time 0.5575 (0.3797) data time 0.0008 (0.0019) model time 0.5567 (0.3782) loss 2.6331 (2.7628) grad_norm 2.8285 (inf) loss_scale 128.0000 (163.7750) mem 11224MB [2024-09-01 11:57:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][560/625] eta 0:00:24 lr 0.000197 wd 0.0500 time 0.3690 (0.3797) data time 0.0010 (0.0019) model time 0.3679 (0.3782) loss 2.7809 (2.7639) grad_norm 3.4998 (inf) loss_scale 128.0000 (163.1373) mem 11224MB [2024-09-01 11:57:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][570/625] eta 0:00:20 lr 0.000197 wd 0.0500 time 0.3711 (0.3803) data time 0.0008 (0.0019) model time 0.3703 (0.3789) loss 3.2732 (2.7636) grad_norm 1.8767 (inf) loss_scale 128.0000 (162.5219) mem 11224MB [2024-09-01 11:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][580/625] eta 0:00:17 lr 0.000197 wd 0.0500 time 0.3778 (0.3802) data time 0.0010 (0.0019) model time 0.3768 (0.3787) loss 2.8188 (2.7589) grad_norm 2.9563 (inf) loss_scale 128.0000 (161.9277) mem 11224MB [2024-09-01 11:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][590/625] eta 0:00:13 lr 0.000197 wd 0.0500 time 0.3743 (0.3801) data time 0.0012 (0.0019) model time 0.3731 (0.3786) loss 2.0136 (2.7558) grad_norm 1.5121 (inf) loss_scale 128.0000 (161.3536) mem 11224MB [2024-09-01 11:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][600/625] eta 0:00:09 lr 0.000196 wd 0.0500 time 0.3724 (0.3800) data time 0.0008 (0.0018) model time 0.3716 (0.3785) loss 2.4476 (2.7513) grad_norm 1.9023 (inf) loss_scale 128.0000 (160.7987) mem 11224MB [2024-09-01 11:57:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][610/625] eta 0:00:05 lr 0.000196 wd 0.0500 time 0.3568 (0.3798) data time 0.0008 (0.0018) model time 0.3560 (0.3783) loss 2.3964 (2.7514) grad_norm 2.3375 (inf) loss_scale 128.0000 (160.2619) mem 11224MB [2024-09-01 11:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [245/300][620/625] eta 0:00:01 lr 0.000196 wd 0.0500 time 0.3712 (0.3795) data time 0.0005 (0.0018) model time 0.3707 (0.3780) loss 3.0104 (2.7531) grad_norm 11.3828 (inf) loss_scale 128.0000 (159.7424) mem 11224MB [2024-09-01 11:57:39 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 245 training takes 0:03:57 [2024-09-01 11:57:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 11:57:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 11:57:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.561 (0.561) Loss 0.5200 (0.5200) Acc@1 89.404 (89.404) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 11:57:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.149) Loss 0.8501 (0.6190) Acc@1 79.443 (86.546) Acc@5 95.605 (97.714) Mem 11224MB [2024-09-01 11:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.123) Loss 0.9136 (0.7372) Acc@1 78.809 (83.664) Acc@5 94.873 (96.515) Mem 11224MB [2024-09-01 11:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.331 Acc@5 96.489 [2024-09-01 11:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 11:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.33% [2024-09-01 11:57:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 11:57:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 11:57:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.548 (0.548) Loss 0.4880 (0.4880) Acc@1 89.600 (89.600) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 11:57:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.147) Loss 0.8062 (0.5923) Acc@1 80.518 (87.016) Acc@5 95.947 (97.852) Mem 11224MB [2024-09-01 11:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 0.8721 (0.7019) Acc@1 79.102 (84.201) Acc@5 95.410 (96.773) Mem 11224MB [2024-09-01 11:57:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.915 Acc@5 96.805 [2024-09-01 11:57:47 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 11:57:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][0/625] eta 0:11:53 lr 0.000196 wd 0.0500 time 1.1420 (1.1420) data time 0.7291 (0.7291) model time 0.0000 (0.0000) loss 2.7383 (2.7383) grad_norm 2.5750 (2.5750) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:57:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][10/625] eta 0:04:32 lr 0.000196 wd 0.0500 time 0.3723 (0.4430) data time 0.0013 (0.0674) model time 0.0000 (0.0000) loss 3.0591 (2.9761) grad_norm 2.4135 (2.4804) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:57:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][20/625] eta 0:04:08 lr 0.000196 wd 0.0500 time 0.3740 (0.4101) data time 0.0008 (0.0358) model time 0.0000 (0.0000) loss 2.4646 (2.8616) grad_norm 2.4913 (2.4374) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:57:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][30/625] eta 0:03:56 lr 0.000196 wd 0.0500 time 0.3726 (0.3978) data time 0.0009 (0.0247) model time 0.0000 (0.0000) loss 1.8626 (2.7999) grad_norm 1.7580 (2.4237) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][40/625] eta 0:03:48 lr 0.000196 wd 0.0500 time 0.3716 (0.3914) data time 0.0011 (0.0189) model time 0.0000 (0.0000) loss 2.6345 (2.8131) grad_norm 2.3187 (2.3948) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][50/625] eta 0:03:43 lr 0.000196 wd 0.0500 time 0.3786 (0.3878) data time 0.0008 (0.0154) model time 0.0000 (0.0000) loss 2.6011 (2.7667) grad_norm 1.6912 (2.3519) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][60/625] eta 0:03:39 lr 0.000196 wd 0.0500 time 0.3767 (0.3883) data time 0.0009 (0.0131) model time 0.3758 (0.3892) loss 2.1816 (2.7094) grad_norm 1.7705 (2.4021) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][70/625] eta 0:03:34 lr 0.000196 wd 0.0500 time 0.3744 (0.3858) data time 0.0009 (0.0114) model time 0.3735 (0.3796) loss 2.6370 (2.6758) grad_norm 2.2555 (2.4110) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][80/625] eta 0:03:29 lr 0.000195 wd 0.0500 time 0.3720 (0.3843) data time 0.0008 (0.0101) model time 0.3712 (0.3772) loss 2.9369 (2.6815) grad_norm 1.7675 (2.4087) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][90/625] eta 0:03:26 lr 0.000195 wd 0.0500 time 0.3630 (0.3869) data time 0.0013 (0.0091) model time 0.3616 (0.3845) loss 3.3959 (2.6998) grad_norm 3.3471 (2.4135) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][100/625] eta 0:03:22 lr 0.000195 wd 0.0500 time 0.3807 (0.3856) data time 0.0011 (0.0083) model time 0.3797 (0.3823) loss 2.8022 (2.7262) grad_norm 1.8859 (2.4032) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][110/625] eta 0:03:17 lr 0.000195 wd 0.0500 time 0.3672 (0.3843) data time 0.0008 (0.0077) model time 0.3664 (0.3802) loss 3.3484 (2.7579) grad_norm 3.1039 (2.4389) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][120/625] eta 0:03:13 lr 0.000195 wd 0.0500 time 0.3766 (0.3834) data time 0.0008 (0.0071) model time 0.3757 (0.3790) loss 1.9758 (2.7500) grad_norm 1.8387 (2.4170) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][130/625] eta 0:03:09 lr 0.000195 wd 0.0500 time 0.3659 (0.3827) data time 0.0008 (0.0067) model time 0.3651 (0.3783) loss 3.0754 (2.7536) grad_norm 2.1711 (2.3843) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][140/625] eta 0:03:05 lr 0.000195 wd 0.0500 time 0.3846 (0.3823) data time 0.0011 (0.0063) model time 0.3834 (0.3780) loss 1.8822 (2.7538) grad_norm 1.8221 (2.3574) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][150/625] eta 0:03:01 lr 0.000195 wd 0.0500 time 0.3721 (0.3818) data time 0.0013 (0.0060) model time 0.3708 (0.3776) loss 3.1518 (2.7569) grad_norm 1.6179 (2.3339) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][160/625] eta 0:02:57 lr 0.000195 wd 0.0500 time 0.3730 (0.3813) data time 0.0009 (0.0057) model time 0.3721 (0.3771) loss 1.9668 (2.7355) grad_norm 2.4134 (2.3239) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][170/625] eta 0:02:53 lr 0.000195 wd 0.0500 time 0.3799 (0.3808) data time 0.0011 (0.0054) model time 0.3788 (0.3767) loss 2.2656 (2.7366) grad_norm 1.7692 (2.3233) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:58:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][180/625] eta 0:02:49 lr 0.000194 wd 0.0500 time 0.3747 (0.3814) data time 0.0011 (0.0052) model time 0.3736 (0.3778) loss 3.0510 (2.7451) grad_norm 2.4725 (2.3526) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][190/625] eta 0:02:45 lr 0.000194 wd 0.0500 time 0.3669 (0.3810) data time 0.0012 (0.0050) model time 0.3657 (0.3774) loss 2.5978 (2.7424) grad_norm 2.1182 (2.4412) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][200/625] eta 0:02:41 lr 0.000194 wd 0.0500 time 0.3709 (0.3807) data time 0.0010 (0.0048) model time 0.3699 (0.3771) loss 3.1524 (2.7532) grad_norm 2.1385 (2.4178) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][210/625] eta 0:02:37 lr 0.000194 wd 0.0500 time 0.3754 (0.3804) data time 0.0009 (0.0046) model time 0.3745 (0.3768) loss 2.2878 (2.7528) grad_norm 1.7554 (2.4019) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][220/625] eta 0:02:33 lr 0.000194 wd 0.0500 time 0.3739 (0.3801) data time 0.0013 (0.0045) model time 0.3726 (0.3766) loss 2.8068 (2.7451) grad_norm 1.8195 (2.3999) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][230/625] eta 0:02:30 lr 0.000194 wd 0.0500 time 0.3738 (0.3798) data time 0.0011 (0.0043) model time 0.3727 (0.3764) loss 2.7451 (2.7481) grad_norm 3.1352 (2.4474) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][240/625] eta 0:02:26 lr 0.000194 wd 0.0500 time 0.3727 (0.3797) data time 0.0011 (0.0042) model time 0.3716 (0.3764) loss 3.1439 (2.7547) grad_norm 1.9826 (2.4635) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][250/625] eta 0:02:22 lr 0.000194 wd 0.0500 time 0.3740 (0.3795) data time 0.0007 (0.0041) model time 0.3732 (0.3763) loss 2.5709 (2.7466) grad_norm 1.5053 (2.4526) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][260/625] eta 0:02:18 lr 0.000194 wd 0.0500 time 0.3782 (0.3794) data time 0.0008 (0.0040) model time 0.3774 (0.3762) loss 2.4230 (2.7394) grad_norm 2.4624 (2.4713) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][270/625] eta 0:02:14 lr 0.000193 wd 0.0500 time 0.3700 (0.3793) data time 0.0009 (0.0038) model time 0.3691 (0.3761) loss 3.5479 (2.7413) grad_norm 11.9214 (2.5094) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][280/625] eta 0:02:10 lr 0.000193 wd 0.0500 time 0.3677 (0.3791) data time 0.0010 (0.0037) model time 0.3667 (0.3761) loss 2.9204 (2.7394) grad_norm 1.9213 (2.4981) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][290/625] eta 0:02:06 lr 0.000193 wd 0.0500 time 0.3758 (0.3789) data time 0.0008 (0.0037) model time 0.3750 (0.3758) loss 3.5340 (2.7420) grad_norm 2.1642 (2.4872) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][300/625] eta 0:02:03 lr 0.000193 wd 0.0500 time 0.3591 (0.3800) data time 0.0011 (0.0036) model time 0.3580 (0.3773) loss 2.7385 (2.7364) grad_norm 2.6846 (2.4839) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][310/625] eta 0:01:59 lr 0.000193 wd 0.0500 time 0.3745 (0.3798) data time 0.0012 (0.0035) model time 0.3733 (0.3770) loss 2.7867 (2.7342) grad_norm 2.0515 (2.4716) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][320/625] eta 0:01:55 lr 0.000193 wd 0.0500 time 0.3707 (0.3795) data time 0.0011 (0.0034) model time 0.3697 (0.3768) loss 1.7373 (2.7290) grad_norm 1.8085 (2.4738) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][330/625] eta 0:01:51 lr 0.000193 wd 0.0500 time 0.3723 (0.3793) data time 0.0008 (0.0034) model time 0.3715 (0.3765) loss 3.1950 (2.7223) grad_norm 3.5836 (2.4869) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 11:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][340/625] eta 0:01:48 lr 0.000193 wd 0.0500 time 0.3742 (0.3792) data time 0.0013 (0.0033) model time 0.3729 (0.3765) loss 2.8014 (2.7199) grad_norm 2.0445 (2.4825) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][350/625] eta 0:01:44 lr 0.000193 wd 0.0500 time 0.3708 (0.3791) data time 0.0008 (0.0032) model time 0.3700 (0.3764) loss 2.9764 (2.7202) grad_norm 2.4231 (2.4796) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][360/625] eta 0:01:40 lr 0.000193 wd 0.0500 time 0.3675 (0.3789) data time 0.0008 (0.0032) model time 0.3667 (0.3763) loss 2.6954 (2.7107) grad_norm 3.0671 (2.4906) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][370/625] eta 0:01:36 lr 0.000192 wd 0.0500 time 0.3771 (0.3787) data time 0.0008 (0.0031) model time 0.3762 (0.3761) loss 2.9364 (2.7109) grad_norm 2.1233 (2.4982) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][380/625] eta 0:01:32 lr 0.000192 wd 0.0500 time 0.3695 (0.3786) data time 0.0011 (0.0031) model time 0.3685 (0.3760) loss 2.7643 (2.7086) grad_norm 2.1222 (2.5005) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][390/625] eta 0:01:29 lr 0.000192 wd 0.0500 time 0.3785 (0.3789) data time 0.0009 (0.0030) model time 0.3776 (0.3765) loss 3.2877 (2.7036) grad_norm 2.3633 (2.6018) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][400/625] eta 0:01:25 lr 0.000192 wd 0.0500 time 0.3671 (0.3788) data time 0.0008 (0.0030) model time 0.3662 (0.3763) loss 1.9331 (2.7020) grad_norm 2.4118 (2.6043) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][410/625] eta 0:01:21 lr 0.000192 wd 0.0500 time 0.3769 (0.3787) data time 0.0010 (0.0029) model time 0.3760 (0.3762) loss 3.2200 (2.7076) grad_norm 1.7291 (2.5953) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][420/625] eta 0:01:17 lr 0.000192 wd 0.0500 time 0.3720 (0.3785) data time 0.0008 (0.0029) model time 0.3712 (0.3761) loss 2.1295 (2.7115) grad_norm 2.2997 (2.5884) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][430/625] eta 0:01:13 lr 0.000192 wd 0.0500 time 0.3648 (0.3784) data time 0.0011 (0.0028) model time 0.3637 (0.3760) loss 3.2365 (2.7137) grad_norm 3.3125 (2.5799) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][440/625] eta 0:01:09 lr 0.000192 wd 0.0500 time 0.3705 (0.3783) data time 0.0008 (0.0028) model time 0.3696 (0.3759) loss 2.1118 (2.7190) grad_norm 2.2402 (2.6243) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][450/625] eta 0:01:06 lr 0.000192 wd 0.0500 time 0.3741 (0.3782) data time 0.0011 (0.0028) model time 0.3730 (0.3758) loss 2.8788 (2.7212) grad_norm 3.5654 (2.6999) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][460/625] eta 0:01:02 lr 0.000192 wd 0.0500 time 0.3650 (0.3781) data time 0.0008 (0.0027) model time 0.3641 (0.3758) loss 2.5829 (2.7228) grad_norm 3.1642 (2.7146) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][470/625] eta 0:00:58 lr 0.000191 wd 0.0500 time 0.3654 (0.3781) data time 0.0009 (0.0027) model time 0.3645 (0.3757) loss 2.9309 (2.7209) grad_norm 2.7583 (2.7080) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][480/625] eta 0:00:54 lr 0.000191 wd 0.0500 time 0.5926 (0.3788) data time 0.0008 (0.0027) model time 0.5917 (0.3766) loss 2.7028 (2.7225) grad_norm 1.7403 (2.6953) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][490/625] eta 0:00:51 lr 0.000191 wd 0.0500 time 0.3679 (0.3787) data time 0.0009 (0.0026) model time 0.3671 (0.3765) loss 3.2378 (2.7209) grad_norm 1.8658 (2.6864) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][500/625] eta 0:00:47 lr 0.000191 wd 0.0500 time 0.3688 (0.3786) data time 0.0009 (0.0026) model time 0.3679 (0.3764) loss 3.5625 (2.7236) grad_norm 1.8563 (2.6803) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][510/625] eta 0:00:43 lr 0.000191 wd 0.0500 time 0.3697 (0.3784) data time 0.0011 (0.0026) model time 0.3686 (0.3762) loss 3.2293 (2.7214) grad_norm 4.0247 (2.6737) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][520/625] eta 0:00:39 lr 0.000191 wd 0.0500 time 0.3659 (0.3783) data time 0.0013 (0.0026) model time 0.3646 (0.3761) loss 2.9692 (2.7227) grad_norm 1.9150 (2.6679) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][530/625] eta 0:00:35 lr 0.000191 wd 0.0500 time 0.3675 (0.3782) data time 0.0011 (0.0025) model time 0.3664 (0.3760) loss 3.1257 (2.7245) grad_norm 4.6638 (2.6650) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][540/625] eta 0:00:32 lr 0.000191 wd 0.0500 time 0.3647 (0.3781) data time 0.0011 (0.0025) model time 0.3636 (0.3759) loss 3.0306 (2.7270) grad_norm 2.5673 (2.6708) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][550/625] eta 0:00:28 lr 0.000191 wd 0.0500 time 0.3722 (0.3779) data time 0.0008 (0.0025) model time 0.3713 (0.3758) loss 3.1519 (2.7268) grad_norm 1.9440 (2.6693) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][560/625] eta 0:00:24 lr 0.000191 wd 0.0500 time 0.3684 (0.3779) data time 0.0012 (0.0025) model time 0.3673 (0.3757) loss 2.6033 (2.7267) grad_norm 2.3451 (2.6608) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][570/625] eta 0:00:20 lr 0.000190 wd 0.0500 time 0.3642 (0.3780) data time 0.0009 (0.0024) model time 0.3634 (0.3759) loss 2.6520 (2.7284) grad_norm 2.2229 (2.6626) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][580/625] eta 0:00:17 lr 0.000190 wd 0.0500 time 0.3732 (0.3779) data time 0.0008 (0.0024) model time 0.3723 (0.3758) loss 1.7962 (2.7282) grad_norm 2.4524 (2.6592) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][590/625] eta 0:00:13 lr 0.000190 wd 0.0500 time 0.3749 (0.3778) data time 0.0012 (0.0024) model time 0.3738 (0.3757) loss 2.7520 (2.7284) grad_norm 1.6923 (2.6560) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][600/625] eta 0:00:09 lr 0.000190 wd 0.0500 time 0.3720 (0.3784) data time 0.0011 (0.0024) model time 0.3710 (0.3764) loss 2.7006 (2.7324) grad_norm 2.1014 (2.6502) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][610/625] eta 0:00:05 lr 0.000190 wd 0.0500 time 0.3521 (0.3783) data time 0.0008 (0.0024) model time 0.3513 (0.3762) loss 2.4999 (2.7283) grad_norm 2.6211 (2.6413) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [246/300][620/625] eta 0:00:01 lr 0.000190 wd 0.0500 time 0.3633 (0.3780) data time 0.0008 (0.0023) model time 0.3626 (0.3760) loss 2.8973 (2.7281) grad_norm 1.8266 (2.6350) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:43 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 246 training takes 0:03:56 [2024-09-01 12:01:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:01:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 0.5093 (0.5093) Acc@1 89.160 (89.160) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 12:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.8345 (0.6168) Acc@1 79.785 (86.581) Acc@5 95.605 (97.687) Mem 11224MB [2024-09-01 12:01:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9463 (0.7336) Acc@1 77.295 (83.496) Acc@5 94.727 (96.498) Mem 11224MB [2024-09-01 12:01:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.211 Acc@5 96.493 [2024-09-01 12:01:47 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.2% [2024-09-01 12:01:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.803 (0.803) Loss 0.4873 (0.4873) Acc@1 89.600 (89.600) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 12:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.178) Loss 0.8052 (0.5923) Acc@1 80.469 (87.034) Acc@5 95.996 (97.860) Mem 11224MB [2024-09-01 12:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.138) Loss 0.8730 (0.7020) Acc@1 79.248 (84.217) Acc@5 95.361 (96.763) Mem 11224MB [2024-09-01 12:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.925 Acc@5 96.793 [2024-09-01 12:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 12:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.93% [2024-09-01 12:01:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 12:01:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 12:01:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][0/625] eta 0:08:31 lr 0.000190 wd 0.0500 time 0.8177 (0.8177) data time 0.4817 (0.4817) model time 0.0000 (0.0000) loss 3.1620 (3.1620) grad_norm 2.6429 (2.6429) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:01:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][10/625] eta 0:04:13 lr 0.000190 wd 0.0500 time 0.3797 (0.4129) data time 0.0008 (0.0448) model time 0.0000 (0.0000) loss 3.0068 (2.6694) grad_norm 3.0014 (4.8228) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][20/625] eta 0:03:57 lr 0.000190 wd 0.0500 time 0.3744 (0.3932) data time 0.0010 (0.0240) model time 0.0000 (0.0000) loss 3.0778 (2.5969) grad_norm 1.5420 (3.5041) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][30/625] eta 0:03:53 lr 0.000190 wd 0.0500 time 0.3623 (0.3933) data time 0.0013 (0.0166) model time 0.0000 (0.0000) loss 2.3923 (2.6069) grad_norm 2.5966 (3.0283) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][40/625] eta 0:03:47 lr 0.000190 wd 0.0500 time 0.3732 (0.3884) data time 0.0013 (0.0129) model time 0.0000 (0.0000) loss 2.1917 (2.6041) grad_norm 2.7069 (2.8442) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][50/625] eta 0:03:41 lr 0.000189 wd 0.0500 time 0.3794 (0.3853) data time 0.0008 (0.0106) model time 0.0000 (0.0000) loss 2.2027 (2.6668) grad_norm 2.4529 (2.7793) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][60/625] eta 0:03:36 lr 0.000189 wd 0.0500 time 0.3685 (0.3832) data time 0.0012 (0.0090) model time 0.3673 (0.3716) loss 2.8004 (2.6755) grad_norm 2.1867 (2.8512) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][70/625] eta 0:03:31 lr 0.000189 wd 0.0500 time 0.3822 (0.3819) data time 0.0009 (0.0079) model time 0.3813 (0.3723) loss 3.5392 (2.6872) grad_norm 2.0962 (2.7561) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][80/625] eta 0:03:27 lr 0.000189 wd 0.0500 time 0.3667 (0.3808) data time 0.0013 (0.0071) model time 0.3654 (0.3721) loss 1.9393 (2.6919) grad_norm 2.3998 (2.8617) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][90/625] eta 0:03:23 lr 0.000189 wd 0.0500 time 0.3687 (0.3797) data time 0.0009 (0.0064) model time 0.3678 (0.3713) loss 2.1550 (2.6862) grad_norm 2.3250 (2.8426) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][100/625] eta 0:03:18 lr 0.000189 wd 0.0500 time 0.3690 (0.3789) data time 0.0011 (0.0059) model time 0.3679 (0.3711) loss 2.8782 (2.6979) grad_norm 1.9688 (2.9000) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][110/625] eta 0:03:16 lr 0.000189 wd 0.0500 time 0.3681 (0.3816) data time 0.0011 (0.0055) model time 0.3670 (0.3772) loss 3.1940 (2.7073) grad_norm 1.9704 (2.8515) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][120/625] eta 0:03:12 lr 0.000189 wd 0.0500 time 0.3760 (0.3808) data time 0.0014 (0.0052) model time 0.3746 (0.3763) loss 1.6372 (2.7073) grad_norm 2.0092 (2.7989) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][130/625] eta 0:03:08 lr 0.000189 wd 0.0500 time 0.3716 (0.3801) data time 0.0008 (0.0049) model time 0.3708 (0.3756) loss 3.3783 (2.7118) grad_norm 3.8579 (2.8101) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][140/625] eta 0:03:04 lr 0.000189 wd 0.0500 time 0.3772 (0.3796) data time 0.0009 (0.0046) model time 0.3763 (0.3752) loss 1.7485 (2.6830) grad_norm 3.0713 (2.7786) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][150/625] eta 0:03:00 lr 0.000188 wd 0.0500 time 0.3738 (0.3791) data time 0.0010 (0.0044) model time 0.3727 (0.3747) loss 3.1574 (2.6791) grad_norm 3.3696 (2.7768) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][160/625] eta 0:02:55 lr 0.000188 wd 0.0500 time 0.3650 (0.3784) data time 0.0010 (0.0042) model time 0.3640 (0.3741) loss 2.8984 (2.6872) grad_norm 2.1854 (2.7292) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:02:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][170/625] eta 0:02:52 lr 0.000188 wd 0.0500 time 0.3649 (0.3782) data time 0.0010 (0.0040) model time 0.3638 (0.3740) loss 2.1755 (2.6866) grad_norm 2.5453 (2.7126) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][180/625] eta 0:02:48 lr 0.000188 wd 0.0500 time 0.3753 (0.3779) data time 0.0011 (0.0038) model time 0.3742 (0.3738) loss 2.5460 (2.6850) grad_norm 2.4978 (2.6955) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][190/625] eta 0:02:44 lr 0.000188 wd 0.0500 time 0.3793 (0.3777) data time 0.0014 (0.0037) model time 0.3779 (0.3738) loss 2.5789 (2.7018) grad_norm 2.8693 (2.6806) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][200/625] eta 0:02:40 lr 0.000188 wd 0.0500 time 0.3696 (0.3775) data time 0.0009 (0.0036) model time 0.3687 (0.3737) loss 3.1578 (2.6920) grad_norm 2.8040 (2.6832) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][210/625] eta 0:02:36 lr 0.000188 wd 0.0500 time 0.3696 (0.3772) data time 0.0011 (0.0035) model time 0.3685 (0.3734) loss 2.4938 (2.6935) grad_norm 1.5731 (2.6722) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][220/625] eta 0:02:32 lr 0.000188 wd 0.0500 time 0.3847 (0.3770) data time 0.0009 (0.0034) model time 0.3838 (0.3733) loss 3.2715 (2.6984) grad_norm 2.2170 (2.6489) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][230/625] eta 0:02:28 lr 0.000188 wd 0.0500 time 0.3782 (0.3768) data time 0.0009 (0.0033) model time 0.3773 (0.3732) loss 3.4051 (2.7062) grad_norm 2.0659 (2.6483) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][240/625] eta 0:02:25 lr 0.000188 wd 0.0500 time 0.3751 (0.3774) data time 0.0012 (0.0032) model time 0.3738 (0.3741) loss 2.7344 (2.7082) grad_norm 1.3414 (2.6255) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][250/625] eta 0:02:21 lr 0.000187 wd 0.0500 time 0.3741 (0.3773) data time 0.0009 (0.0031) model time 0.3733 (0.3741) loss 2.6966 (2.7027) grad_norm 2.2176 (2.6058) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][260/625] eta 0:02:17 lr 0.000187 wd 0.0500 time 0.3676 (0.3771) data time 0.0011 (0.0030) model time 0.3665 (0.3739) loss 2.2305 (2.7004) grad_norm 2.1716 (2.5985) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][270/625] eta 0:02:13 lr 0.000187 wd 0.0500 time 0.3713 (0.3769) data time 0.0009 (0.0029) model time 0.3704 (0.3738) loss 3.5376 (2.7045) grad_norm 2.4642 (2.5986) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][280/625] eta 0:02:10 lr 0.000187 wd 0.0500 time 0.3852 (0.3769) data time 0.0011 (0.0029) model time 0.3841 (0.3739) loss 2.6750 (2.6995) grad_norm 3.1036 (2.6144) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][290/625] eta 0:02:06 lr 0.000187 wd 0.0500 time 0.3741 (0.3768) data time 0.0011 (0.0028) model time 0.3730 (0.3738) loss 2.5507 (2.6973) grad_norm 1.9965 (2.5952) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][300/625] eta 0:02:02 lr 0.000187 wd 0.0500 time 0.3766 (0.3767) data time 0.0010 (0.0028) model time 0.3755 (0.3738) loss 2.6068 (2.7014) grad_norm 2.0522 (2.5838) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][310/625] eta 0:01:58 lr 0.000187 wd 0.0500 time 0.5937 (0.3773) data time 0.0010 (0.0027) model time 0.5927 (0.3746) loss 2.3713 (2.7080) grad_norm 2.0269 (2.5838) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][320/625] eta 0:01:55 lr 0.000187 wd 0.0500 time 0.3642 (0.3779) data time 0.0008 (0.0027) model time 0.3634 (0.3753) loss 2.9366 (2.7090) grad_norm 3.0561 (2.5850) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:03:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][330/625] eta 0:01:51 lr 0.000187 wd 0.0500 time 0.3666 (0.3777) data time 0.0013 (0.0026) model time 0.3653 (0.3751) loss 2.6329 (2.7097) grad_norm 2.2026 (2.5778) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][340/625] eta 0:01:47 lr 0.000187 wd 0.0500 time 0.3761 (0.3776) data time 0.0009 (0.0026) model time 0.3752 (0.3750) loss 3.2111 (2.7199) grad_norm 2.4123 (2.5656) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][350/625] eta 0:01:43 lr 0.000186 wd 0.0500 time 0.3772 (0.3773) data time 0.0011 (0.0025) model time 0.3760 (0.3748) loss 3.4390 (2.7249) grad_norm 3.3996 (2.5654) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][360/625] eta 0:01:39 lr 0.000186 wd 0.0500 time 0.3725 (0.3771) data time 0.0010 (0.0025) model time 0.3714 (0.3746) loss 2.8765 (2.7266) grad_norm 3.5503 (2.5681) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][370/625] eta 0:01:36 lr 0.000186 wd 0.0500 time 0.3609 (0.3770) data time 0.0009 (0.0025) model time 0.3600 (0.3745) loss 2.5279 (2.7234) grad_norm 4.1894 (2.5655) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][380/625] eta 0:01:32 lr 0.000186 wd 0.0500 time 0.3703 (0.3768) data time 0.0009 (0.0024) model time 0.3694 (0.3743) loss 3.4508 (2.7260) grad_norm 2.0657 (2.5531) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][390/625] eta 0:01:28 lr 0.000186 wd 0.0500 time 0.3952 (0.3768) data time 0.0008 (0.0024) model time 0.3944 (0.3743) loss 2.5312 (2.7223) grad_norm 1.8988 (2.5480) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][400/625] eta 0:01:24 lr 0.000186 wd 0.0500 time 0.3747 (0.3767) data time 0.0008 (0.0024) model time 0.3739 (0.3742) loss 3.0040 (2.7249) grad_norm 5.7242 (2.5555) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][410/625] eta 0:01:20 lr 0.000186 wd 0.0500 time 0.3677 (0.3765) data time 0.0012 (0.0023) model time 0.3665 (0.3741) loss 2.3354 (2.7220) grad_norm 3.8536 (2.5549) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][420/625] eta 0:01:17 lr 0.000186 wd 0.0500 time 0.6014 (0.3769) data time 0.0011 (0.0023) model time 0.6003 (0.3746) loss 3.0574 (2.7291) grad_norm 2.1191 (2.5493) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][430/625] eta 0:01:13 lr 0.000186 wd 0.0500 time 0.3723 (0.3768) data time 0.0009 (0.0023) model time 0.3714 (0.3745) loss 1.8269 (2.7328) grad_norm 2.1180 (2.5709) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][440/625] eta 0:01:09 lr 0.000186 wd 0.0500 time 0.3646 (0.3766) data time 0.0013 (0.0023) model time 0.3633 (0.3743) loss 3.5049 (2.7334) grad_norm 2.0828 (2.5620) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][450/625] eta 0:01:05 lr 0.000185 wd 0.0500 time 0.3679 (0.3766) data time 0.0014 (0.0022) model time 0.3665 (0.3743) loss 3.0462 (2.7355) grad_norm 2.1220 (2.5552) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][460/625] eta 0:01:02 lr 0.000185 wd 0.0500 time 0.3653 (0.3765) data time 0.0011 (0.0022) model time 0.3642 (0.3742) loss 3.0471 (2.7397) grad_norm 2.3583 (2.5520) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][470/625] eta 0:00:58 lr 0.000185 wd 0.0500 time 0.3733 (0.3764) data time 0.0011 (0.0022) model time 0.3722 (0.3741) loss 2.5700 (2.7428) grad_norm 2.5242 (2.5504) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][480/625] eta 0:00:54 lr 0.000185 wd 0.0500 time 0.3712 (0.3763) data time 0.0012 (0.0022) model time 0.3700 (0.3741) loss 2.8539 (2.7452) grad_norm 1.5654 (2.6313) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:04:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][490/625] eta 0:00:50 lr 0.000185 wd 0.0500 time 0.3732 (0.3762) data time 0.0012 (0.0022) model time 0.3720 (0.3740) loss 2.8068 (2.7491) grad_norm 2.0272 (2.6356) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][500/625] eta 0:00:47 lr 0.000185 wd 0.0500 time 0.3829 (0.3774) data time 0.0009 (0.0021) model time 0.3820 (0.3754) loss 1.8090 (2.7481) grad_norm 1.7957 (2.6292) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][510/625] eta 0:00:43 lr 0.000185 wd 0.0500 time 0.3709 (0.3773) data time 0.0009 (0.0021) model time 0.3700 (0.3752) loss 2.8651 (2.7525) grad_norm 2.2981 (2.6235) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][520/625] eta 0:00:39 lr 0.000185 wd 0.0500 time 0.3635 (0.3772) data time 0.0008 (0.0021) model time 0.3627 (0.3751) loss 2.4094 (2.7466) grad_norm 1.9938 (2.6349) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][530/625] eta 0:00:35 lr 0.000185 wd 0.0500 time 0.3655 (0.3771) data time 0.0011 (0.0021) model time 0.3644 (0.3750) loss 2.7319 (2.7460) grad_norm 2.6751 (2.6520) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][540/625] eta 0:00:32 lr 0.000185 wd 0.0500 time 0.3405 (0.3773) data time 0.0012 (0.0021) model time 0.3393 (0.3753) loss 2.6646 (2.7428) grad_norm 1.8705 (2.6560) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][550/625] eta 0:00:28 lr 0.000185 wd 0.0500 time 0.3709 (0.3773) data time 0.0008 (0.0021) model time 0.3701 (0.3753) loss 3.7072 (2.7468) grad_norm 3.3712 (2.6483) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][560/625] eta 0:00:24 lr 0.000184 wd 0.0500 time 0.3682 (0.3772) data time 0.0008 (0.0020) model time 0.3674 (0.3752) loss 1.9425 (2.7459) grad_norm 2.5935 (2.6729) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][570/625] eta 0:00:20 lr 0.000184 wd 0.0500 time 0.3766 (0.3771) data time 0.0013 (0.0020) model time 0.3753 (0.3752) loss 2.7507 (2.7443) grad_norm 1.9415 (2.6727) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][580/625] eta 0:00:16 lr 0.000184 wd 0.0500 time 0.3724 (0.3771) data time 0.0011 (0.0020) model time 0.3713 (0.3751) loss 2.6072 (2.7423) grad_norm 3.4607 (2.6685) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][590/625] eta 0:00:13 lr 0.000184 wd 0.0500 time 0.3829 (0.3770) data time 0.0009 (0.0020) model time 0.3820 (0.3751) loss 2.9413 (2.7426) grad_norm 2.2024 (2.6639) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][600/625] eta 0:00:09 lr 0.000184 wd 0.0500 time 0.3765 (0.3770) data time 0.0008 (0.0020) model time 0.3757 (0.3750) loss 3.7284 (2.7435) grad_norm 2.4490 (2.6591) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][610/625] eta 0:00:05 lr 0.000184 wd 0.0500 time 0.3748 (0.3769) data time 0.0008 (0.0020) model time 0.3741 (0.3750) loss 3.0010 (2.7415) grad_norm 2.0251 (2.6489) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [247/300][620/625] eta 0:00:01 lr 0.000184 wd 0.0500 time 0.3583 (0.3777) data time 0.0006 (0.0020) model time 0.3577 (0.3758) loss 3.2920 (2.7420) grad_norm 11.3507 (2.6580) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:05:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 247 training takes 0:03:55 [2024-09-01 12:05:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:05:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:05:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.503 (0.503) Loss 0.5024 (0.5024) Acc@1 89.160 (89.160) Acc@5 98.779 (98.779) Mem 11224MB [2024-09-01 12:05:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.099 (0.144) Loss 0.8120 (0.6140) Acc@1 81.006 (86.497) Acc@5 95.654 (97.701) Mem 11224MB [2024-09-01 12:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 0.9297 (0.7332) Acc@1 77.393 (83.468) Acc@5 95.361 (96.498) Mem 11224MB [2024-09-01 12:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.145 Acc@5 96.479 [2024-09-01 12:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.1% [2024-09-01 12:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.786 (0.786) Loss 0.4868 (0.4868) Acc@1 89.502 (89.502) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 12:05:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.173) Loss 0.8066 (0.5925) Acc@1 80.566 (87.034) Acc@5 95.996 (97.865) Mem 11224MB [2024-09-01 12:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.135) Loss 0.8750 (0.7023) Acc@1 79.150 (84.191) Acc@5 95.264 (96.766) Mem 11224MB [2024-09-01 12:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.895 Acc@5 96.789 [2024-09-01 12:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 12:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][0/625] eta 0:13:08 lr 0.000184 wd 0.0500 time 1.2617 (1.2617) data time 0.5077 (0.5077) model time 0.0000 (0.0000) loss 3.0933 (3.0933) grad_norm 5.4097 (5.4097) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][10/625] eta 0:04:38 lr 0.000184 wd 0.0500 time 0.3707 (0.4534) data time 0.0009 (0.0473) model time 0.0000 (0.0000) loss 2.3108 (2.6354) grad_norm 2.0127 (2.7433) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][20/625] eta 0:04:11 lr 0.000184 wd 0.0500 time 0.3658 (0.4149) data time 0.0011 (0.0253) model time 0.0000 (0.0000) loss 2.8540 (2.5210) grad_norm 2.6471 (2.7602) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][30/625] eta 0:03:58 lr 0.000183 wd 0.0500 time 0.3692 (0.4005) data time 0.0009 (0.0176) model time 0.0000 (0.0000) loss 2.6828 (2.5440) grad_norm 2.0377 (2.7257) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][40/625] eta 0:03:50 lr 0.000183 wd 0.0500 time 0.3789 (0.3935) data time 0.0013 (0.0136) model time 0.0000 (0.0000) loss 2.4615 (2.5670) grad_norm 2.5069 (2.6691) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][50/625] eta 0:03:44 lr 0.000183 wd 0.0500 time 0.3734 (0.3897) data time 0.0008 (0.0112) model time 0.0000 (0.0000) loss 2.9394 (2.5997) grad_norm 2.4106 (3.2455) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][60/625] eta 0:03:38 lr 0.000183 wd 0.0500 time 0.3951 (0.3871) data time 0.0008 (0.0096) model time 0.3943 (0.3726) loss 2.1427 (2.6179) grad_norm 2.6531 (3.0905) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][70/625] eta 0:03:33 lr 0.000183 wd 0.0500 time 0.3667 (0.3850) data time 0.0011 (0.0084) model time 0.3656 (0.3717) loss 2.8534 (2.6297) grad_norm 2.3158 (3.0519) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][80/625] eta 0:03:29 lr 0.000183 wd 0.0500 time 0.3927 (0.3838) data time 0.0010 (0.0075) model time 0.3917 (0.3725) loss 2.5966 (2.6266) grad_norm 2.5512 (2.9789) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][90/625] eta 0:03:24 lr 0.000183 wd 0.0500 time 0.3777 (0.3824) data time 0.0008 (0.0068) model time 0.3769 (0.3718) loss 3.0665 (2.6750) grad_norm 2.8624 (2.9389) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][100/625] eta 0:03:21 lr 0.000183 wd 0.0500 time 0.3655 (0.3835) data time 0.0011 (0.0062) model time 0.3644 (0.3760) loss 2.7544 (2.6786) grad_norm 3.1341 (2.9295) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][110/625] eta 0:03:16 lr 0.000183 wd 0.0500 time 0.3717 (0.3823) data time 0.0011 (0.0058) model time 0.3706 (0.3747) loss 2.9509 (2.6683) grad_norm 2.6030 (2.9003) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][120/625] eta 0:03:12 lr 0.000183 wd 0.0500 time 0.3616 (0.3814) data time 0.0008 (0.0054) model time 0.3608 (0.3742) loss 2.1618 (2.6718) grad_norm 2.1924 (2.8601) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][130/625] eta 0:03:08 lr 0.000183 wd 0.0500 time 0.3728 (0.3807) data time 0.0013 (0.0051) model time 0.3716 (0.3738) loss 2.9150 (2.6891) grad_norm 1.6831 (2.8706) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][140/625] eta 0:03:04 lr 0.000182 wd 0.0500 time 0.3690 (0.3802) data time 0.0011 (0.0048) model time 0.3679 (0.3735) loss 2.0761 (2.6849) grad_norm 3.8342 (2.8598) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][150/625] eta 0:03:00 lr 0.000182 wd 0.0500 time 0.3639 (0.3796) data time 0.0008 (0.0046) model time 0.3631 (0.3733) loss 1.7361 (2.6985) grad_norm 4.4289 (3.0609) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][160/625] eta 0:02:56 lr 0.000182 wd 0.0500 time 0.3685 (0.3792) data time 0.0013 (0.0044) model time 0.3672 (0.3731) loss 2.7183 (2.6982) grad_norm 2.5603 (3.0563) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:06:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][170/625] eta 0:02:52 lr 0.000182 wd 0.0500 time 0.3683 (0.3788) data time 0.0008 (0.0042) model time 0.3675 (0.3730) loss 2.5124 (2.6979) grad_norm 2.3246 (3.0492) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][180/625] eta 0:02:48 lr 0.000182 wd 0.0500 time 0.3757 (0.3795) data time 0.0011 (0.0040) model time 0.3746 (0.3743) loss 1.9514 (2.6927) grad_norm 2.3231 (3.0085) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][190/625] eta 0:02:44 lr 0.000182 wd 0.0500 time 0.3638 (0.3791) data time 0.0010 (0.0039) model time 0.3628 (0.3740) loss 3.0025 (2.7000) grad_norm 1.6488 (3.0337) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][200/625] eta 0:02:41 lr 0.000182 wd 0.0500 time 0.3818 (0.3808) data time 0.0010 (0.0037) model time 0.3809 (0.3765) loss 3.0821 (2.7019) grad_norm 2.0135 (2.9970) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][210/625] eta 0:02:37 lr 0.000182 wd 0.0500 time 0.3720 (0.3804) data time 0.0012 (0.0036) model time 0.3709 (0.3763) loss 3.0408 (2.7084) grad_norm 4.1837 (2.9507) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][220/625] eta 0:02:33 lr 0.000182 wd 0.0500 time 0.3868 (0.3801) data time 0.0011 (0.0035) model time 0.3857 (0.3761) loss 2.5972 (2.7013) grad_norm 1.9285 (2.9273) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][230/625] eta 0:02:30 lr 0.000182 wd 0.0500 time 0.3697 (0.3798) data time 0.0008 (0.0034) model time 0.3689 (0.3758) loss 2.0083 (2.7006) grad_norm 3.2047 (2.9525) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][240/625] eta 0:02:26 lr 0.000181 wd 0.0500 time 0.3747 (0.3797) data time 0.0008 (0.0033) model time 0.3739 (0.3758) loss 2.0210 (2.6960) grad_norm 2.5047 (2.9394) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][250/625] eta 0:02:22 lr 0.000181 wd 0.0500 time 0.3733 (0.3793) data time 0.0011 (0.0032) model time 0.3722 (0.3755) loss 3.0824 (2.7039) grad_norm 4.0634 (2.9265) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][260/625] eta 0:02:18 lr 0.000181 wd 0.0500 time 0.3802 (0.3791) data time 0.0011 (0.0031) model time 0.3792 (0.3753) loss 2.8177 (2.7008) grad_norm 3.3292 (2.9123) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][270/625] eta 0:02:14 lr 0.000181 wd 0.0500 time 0.3720 (0.3790) data time 0.0011 (0.0031) model time 0.3709 (0.3753) loss 2.4997 (2.7058) grad_norm 2.4485 (2.8991) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][280/625] eta 0:02:10 lr 0.000181 wd 0.0500 time 0.3622 (0.3794) data time 0.0008 (0.0030) model time 0.3613 (0.3759) loss 3.4241 (2.7112) grad_norm 3.9872 (2.9011) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][290/625] eta 0:02:07 lr 0.000181 wd 0.0500 time 0.3787 (0.3793) data time 0.0011 (0.0029) model time 0.3776 (0.3759) loss 2.8760 (2.7092) grad_norm 2.8096 (2.8817) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][300/625] eta 0:02:03 lr 0.000181 wd 0.0500 time 0.3766 (0.3798) data time 0.0009 (0.0029) model time 0.3757 (0.3766) loss 3.0686 (2.7006) grad_norm 2.3635 (2.8737) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][310/625] eta 0:01:59 lr 0.000181 wd 0.0500 time 0.3743 (0.3795) data time 0.0008 (0.0028) model time 0.3735 (0.3763) loss 2.1383 (2.6987) grad_norm 4.5542 (2.8818) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:07:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][320/625] eta 0:01:55 lr 0.000181 wd 0.0500 time 0.3693 (0.3792) data time 0.0013 (0.0028) model time 0.3680 (0.3760) loss 3.1590 (2.6992) grad_norm 2.5310 (2.8755) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][330/625] eta 0:01:51 lr 0.000181 wd 0.0500 time 0.3763 (0.3790) data time 0.0009 (0.0027) model time 0.3754 (0.3759) loss 1.6024 (2.6942) grad_norm 2.3418 (2.8543) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][340/625] eta 0:01:47 lr 0.000180 wd 0.0500 time 0.3833 (0.3789) data time 0.0009 (0.0027) model time 0.3824 (0.3758) loss 1.7182 (2.6955) grad_norm 2.3064 (2.8349) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][350/625] eta 0:01:44 lr 0.000180 wd 0.0500 time 0.3722 (0.3788) data time 0.0010 (0.0026) model time 0.3711 (0.3758) loss 2.6225 (2.6947) grad_norm 1.7502 (2.8438) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][360/625] eta 0:01:40 lr 0.000180 wd 0.0500 time 0.3719 (0.3788) data time 0.0014 (0.0026) model time 0.3705 (0.3758) loss 2.8883 (2.6994) grad_norm 3.5045 (2.8377) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][370/625] eta 0:01:36 lr 0.000180 wd 0.0500 time 0.3688 (0.3787) data time 0.0010 (0.0026) model time 0.3678 (0.3757) loss 2.5270 (2.7029) grad_norm 2.7688 (2.8322) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][380/625] eta 0:01:32 lr 0.000180 wd 0.0500 time 0.3691 (0.3787) data time 0.0012 (0.0025) model time 0.3678 (0.3758) loss 2.7574 (2.6994) grad_norm 2.3492 (2.8240) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][390/625] eta 0:01:29 lr 0.000180 wd 0.0500 time 0.3756 (0.3796) data time 0.0010 (0.0025) model time 0.3746 (0.3769) loss 3.0737 (2.7068) grad_norm 3.5292 (2.8242) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][400/625] eta 0:01:25 lr 0.000180 wd 0.0500 time 0.3665 (0.3800) data time 0.0011 (0.0025) model time 0.3654 (0.3774) loss 2.1891 (2.7095) grad_norm 2.8716 (2.8172) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][410/625] eta 0:01:21 lr 0.000180 wd 0.0500 time 0.3754 (0.3798) data time 0.0012 (0.0024) model time 0.3742 (0.3772) loss 2.7295 (2.7091) grad_norm 2.4246 (2.8165) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][420/625] eta 0:01:17 lr 0.000180 wd 0.0500 time 0.3829 (0.3797) data time 0.0011 (0.0024) model time 0.3818 (0.3771) loss 2.5529 (2.7115) grad_norm 3.5360 (2.8157) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][430/625] eta 0:01:13 lr 0.000180 wd 0.0500 time 0.3675 (0.3795) data time 0.0008 (0.0024) model time 0.3667 (0.3769) loss 2.9868 (2.7102) grad_norm 2.1732 (2.8306) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][440/625] eta 0:01:10 lr 0.000179 wd 0.0500 time 0.3762 (0.3793) data time 0.0008 (0.0023) model time 0.3753 (0.3768) loss 1.5516 (2.7053) grad_norm 2.4037 (2.8290) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][450/625] eta 0:01:06 lr 0.000179 wd 0.0500 time 0.3757 (0.3792) data time 0.0011 (0.0023) model time 0.3746 (0.3767) loss 3.2981 (2.7113) grad_norm 5.6654 (2.8308) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][460/625] eta 0:01:02 lr 0.000179 wd 0.0500 time 0.3672 (0.3791) data time 0.0008 (0.0023) model time 0.3663 (0.3766) loss 3.8289 (2.7131) grad_norm 2.3660 (2.8234) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][470/625] eta 0:00:58 lr 0.000179 wd 0.0500 time 0.3746 (0.3789) data time 0.0010 (0.0023) model time 0.3736 (0.3764) loss 3.1182 (2.7175) grad_norm 1.8306 (2.8104) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:08:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][480/625] eta 0:00:54 lr 0.000179 wd 0.0500 time 0.3738 (0.3787) data time 0.0013 (0.0022) model time 0.3724 (0.3763) loss 3.1086 (2.7137) grad_norm 1.7152 (2.7971) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:09:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][490/625] eta 0:00:51 lr 0.000179 wd 0.0500 time 0.3724 (0.3787) data time 0.0010 (0.0022) model time 0.3714 (0.3762) loss 1.9771 (2.7086) grad_norm 3.5396 (2.7946) loss_scale 256.0000 (129.5642) mem 11224MB [2024-09-01 12:09:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][500/625] eta 0:00:47 lr 0.000179 wd 0.0500 time 0.5874 (0.3790) data time 0.0011 (0.0022) model time 0.5863 (0.3766) loss 3.1430 (2.7084) grad_norm 2.3723 (2.7913) loss_scale 256.0000 (132.0878) mem 11224MB [2024-09-01 12:09:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][510/625] eta 0:00:43 lr 0.000179 wd 0.0500 time 0.3744 (0.3792) data time 0.0011 (0.0022) model time 0.3733 (0.3769) loss 1.6804 (2.7060) grad_norm 2.4471 (2.7862) loss_scale 256.0000 (134.5127) mem 11224MB [2024-09-01 12:09:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][520/625] eta 0:00:39 lr 0.000179 wd 0.0500 time 0.3694 (0.3791) data time 0.0009 (0.0022) model time 0.3685 (0.3768) loss 2.1768 (2.7040) grad_norm 2.7393 (2.7945) loss_scale 256.0000 (136.8445) mem 11224MB [2024-09-01 12:09:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][530/625] eta 0:00:36 lr 0.000179 wd 0.0500 time 0.3825 (0.3790) data time 0.0009 (0.0021) model time 0.3816 (0.3767) loss 2.7505 (2.7062) grad_norm 2.1020 (2.7994) loss_scale 256.0000 (139.0885) mem 11224MB [2024-09-01 12:09:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][540/625] eta 0:00:32 lr 0.000179 wd 0.0500 time 0.3724 (0.3789) data time 0.0008 (0.0021) model time 0.3716 (0.3766) loss 3.2396 (2.7094) grad_norm 2.8593 (2.7933) loss_scale 256.0000 (141.2495) mem 11224MB [2024-09-01 12:09:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][550/625] eta 0:00:28 lr 0.000178 wd 0.0500 time 0.3653 (0.3788) data time 0.0012 (0.0021) model time 0.3642 (0.3765) loss 2.4065 (2.7103) grad_norm 1.8634 (2.7841) loss_scale 256.0000 (143.3321) mem 11224MB [2024-09-01 12:09:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][560/625] eta 0:00:24 lr 0.000178 wd 0.0500 time 0.3801 (0.3786) data time 0.0010 (0.0021) model time 0.3790 (0.3763) loss 2.8793 (2.7073) grad_norm 5.9762 (2.7866) loss_scale 256.0000 (145.3405) mem 11224MB [2024-09-01 12:09:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][570/625] eta 0:00:20 lr 0.000178 wd 0.0500 time 0.3749 (0.3785) data time 0.0012 (0.0021) model time 0.3738 (0.3763) loss 2.7543 (2.7076) grad_norm 1.5743 (2.7726) loss_scale 256.0000 (147.2785) mem 11224MB [2024-09-01 12:09:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][580/625] eta 0:00:17 lr 0.000178 wd 0.0500 time 0.3735 (0.3784) data time 0.0009 (0.0021) model time 0.3726 (0.3762) loss 2.9902 (2.7079) grad_norm 2.7939 (2.7651) loss_scale 256.0000 (149.1497) mem 11224MB [2024-09-01 12:09:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][590/625] eta 0:00:13 lr 0.000178 wd 0.0500 time 0.3726 (0.3783) data time 0.0010 (0.0020) model time 0.3716 (0.3761) loss 2.5665 (2.7062) grad_norm 1.8737 (2.7657) loss_scale 256.0000 (150.9577) mem 11224MB [2024-09-01 12:09:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][600/625] eta 0:00:09 lr 0.000178 wd 0.0500 time 0.3691 (0.3783) data time 0.0009 (0.0020) model time 0.3683 (0.3761) loss 2.9562 (2.7064) grad_norm 2.0128 (2.7618) loss_scale 256.0000 (152.7055) mem 11224MB [2024-09-01 12:09:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][610/625] eta 0:00:05 lr 0.000178 wd 0.0500 time 0.3531 (0.3785) data time 0.0008 (0.0020) model time 0.3523 (0.3763) loss 2.9101 (2.7061) grad_norm 2.0632 (2.7628) loss_scale 256.0000 (154.3961) mem 11224MB [2024-09-01 12:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [248/300][620/625] eta 0:00:01 lr 0.000178 wd 0.0500 time 0.3669 (0.3782) data time 0.0005 (0.0020) model time 0.3663 (0.3760) loss 2.0721 (2.7059) grad_norm 1.6735 (2.7532) loss_scale 256.0000 (156.0322) mem 11224MB [2024-09-01 12:09:51 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 248 training takes 0:03:56 [2024-09-01 12:09:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:09:52 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:09:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.564 (0.564) Loss 0.5352 (0.5352) Acc@1 88.086 (88.086) Acc@5 98.389 (98.389) Mem 11224MB [2024-09-01 12:09:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.150) Loss 0.8540 (0.6260) Acc@1 80.078 (86.448) Acc@5 95.605 (97.678) Mem 11224MB [2024-09-01 12:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.123) Loss 0.9136 (0.7424) Acc@1 78.467 (83.566) Acc@5 95.264 (96.468) Mem 11224MB [2024-09-01 12:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.251 Acc@5 96.465 [2024-09-01 12:09:55 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 12:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.782 (0.782) Loss 0.4868 (0.4868) Acc@1 89.355 (89.355) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 12:09:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.174) Loss 0.8042 (0.5923) Acc@1 80.566 (86.981) Acc@5 95.898 (97.860) Mem 11224MB [2024-09-01 12:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.8726 (0.7020) Acc@1 79.443 (84.177) Acc@5 95.264 (96.761) Mem 11224MB [2024-09-01 12:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.893 Acc@5 96.781 [2024-09-01 12:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 12:09:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][0/625] eta 0:12:35 lr 0.000178 wd 0.0500 time 1.2081 (1.2081) data time 0.6413 (0.6413) model time 0.0000 (0.0000) loss 2.8857 (2.8857) grad_norm 3.1553 (3.1553) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][10/625] eta 0:04:36 lr 0.000178 wd 0.0500 time 0.3566 (0.4492) data time 0.0009 (0.0594) model time 0.0000 (0.0000) loss 2.7819 (2.6209) grad_norm 2.9692 (3.1844) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][20/625] eta 0:04:08 lr 0.000178 wd 0.0500 time 0.3603 (0.4104) data time 0.0011 (0.0316) model time 0.0000 (0.0000) loss 2.6009 (2.6703) grad_norm 2.2273 (3.1233) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][30/625] eta 0:03:57 lr 0.000177 wd 0.0500 time 0.3778 (0.3984) data time 0.0011 (0.0218) model time 0.0000 (0.0000) loss 3.3509 (2.6595) grad_norm 3.3668 (2.9905) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][40/625] eta 0:03:49 lr 0.000177 wd 0.0500 time 0.3731 (0.3929) data time 0.0011 (0.0168) model time 0.0000 (0.0000) loss 1.7864 (2.7184) grad_norm 1.9230 (2.9219) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][50/625] eta 0:03:45 lr 0.000177 wd 0.0500 time 0.3719 (0.3922) data time 0.0011 (0.0137) model time 0.0000 (0.0000) loss 2.7701 (2.6730) grad_norm 3.5555 (2.9037) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][60/625] eta 0:03:39 lr 0.000177 wd 0.0500 time 0.3789 (0.3891) data time 0.0011 (0.0117) model time 0.3778 (0.3717) loss 2.8686 (2.6652) grad_norm 1.9240 (3.1684) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][70/625] eta 0:03:36 lr 0.000177 wd 0.0500 time 0.3964 (0.3901) data time 0.0009 (0.0102) model time 0.3955 (0.3833) loss 1.9308 (2.6584) grad_norm 1.8011 (3.0282) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][80/625] eta 0:03:31 lr 0.000177 wd 0.0500 time 0.3706 (0.3879) data time 0.0010 (0.0091) model time 0.3697 (0.3793) loss 2.8981 (2.6536) grad_norm 2.4745 (3.0463) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][90/625] eta 0:03:26 lr 0.000177 wd 0.0500 time 0.3658 (0.3862) data time 0.0009 (0.0082) model time 0.3649 (0.3775) loss 3.4136 (2.6819) grad_norm 2.4586 (3.0317) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][100/625] eta 0:03:22 lr 0.000177 wd 0.0500 time 0.3715 (0.3848) data time 0.0008 (0.0075) model time 0.3707 (0.3761) loss 2.2244 (2.6891) grad_norm 1.8977 (2.9667) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][110/625] eta 0:03:17 lr 0.000177 wd 0.0500 time 0.3774 (0.3836) data time 0.0012 (0.0069) model time 0.3762 (0.3751) loss 2.6596 (2.7036) grad_norm 1.7023 (2.9023) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][120/625] eta 0:03:13 lr 0.000177 wd 0.0500 time 0.3669 (0.3826) data time 0.0009 (0.0065) model time 0.3661 (0.3745) loss 2.3466 (2.7102) grad_norm 1.8638 (2.8540) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][130/625] eta 0:03:09 lr 0.000176 wd 0.0500 time 0.3663 (0.3818) data time 0.0012 (0.0061) model time 0.3651 (0.3740) loss 2.8201 (2.6999) grad_norm 2.3766 (2.8253) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][140/625] eta 0:03:04 lr 0.000176 wd 0.0500 time 0.3659 (0.3811) data time 0.0009 (0.0057) model time 0.3649 (0.3737) loss 3.0089 (2.6955) grad_norm 2.9262 (2.7929) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][150/625] eta 0:03:00 lr 0.000176 wd 0.0500 time 0.3810 (0.3805) data time 0.0011 (0.0054) model time 0.3799 (0.3734) loss 2.7617 (2.7085) grad_norm 2.3649 (2.7807) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][160/625] eta 0:02:57 lr 0.000176 wd 0.0500 time 0.3626 (0.3810) data time 0.0013 (0.0051) model time 0.3613 (0.3747) loss 3.0627 (2.7292) grad_norm 2.9891 (2.7445) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][170/625] eta 0:02:53 lr 0.000176 wd 0.0500 time 0.3674 (0.3804) data time 0.0013 (0.0049) model time 0.3661 (0.3743) loss 1.9716 (2.7209) grad_norm 2.0679 (2.7328) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][180/625] eta 0:02:49 lr 0.000176 wd 0.0500 time 0.3672 (0.3799) data time 0.0010 (0.0047) model time 0.3662 (0.3739) loss 3.0243 (2.7304) grad_norm 2.8518 (2.7381) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][190/625] eta 0:02:45 lr 0.000176 wd 0.0500 time 0.3751 (0.3796) data time 0.0012 (0.0045) model time 0.3739 (0.3738) loss 2.8557 (2.7299) grad_norm 2.6481 (2.7339) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][200/625] eta 0:02:41 lr 0.000176 wd 0.0500 time 0.3787 (0.3792) data time 0.0008 (0.0044) model time 0.3779 (0.3735) loss 2.6661 (2.7335) grad_norm 1.9956 (2.7170) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][210/625] eta 0:02:37 lr 0.000176 wd 0.0500 time 0.3737 (0.3789) data time 0.0012 (0.0042) model time 0.3725 (0.3735) loss 2.6618 (2.7317) grad_norm 1.9052 (2.7619) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][220/625] eta 0:02:33 lr 0.000176 wd 0.0500 time 0.3688 (0.3787) data time 0.0010 (0.0041) model time 0.3678 (0.3734) loss 3.0209 (2.7343) grad_norm 2.1706 (2.7434) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][230/625] eta 0:02:29 lr 0.000175 wd 0.0500 time 0.3723 (0.3785) data time 0.0009 (0.0040) model time 0.3714 (0.3734) loss 2.8509 (2.7253) grad_norm 5.3776 (2.7603) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][240/625] eta 0:02:25 lr 0.000175 wd 0.0500 time 0.3817 (0.3784) data time 0.0009 (0.0038) model time 0.3808 (0.3734) loss 2.8639 (2.7307) grad_norm 1.7251 (2.7335) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][250/625] eta 0:02:22 lr 0.000175 wd 0.0500 time 0.5693 (0.3789) data time 0.0008 (0.0037) model time 0.5685 (0.3743) loss 2.2373 (2.7321) grad_norm 1.8484 (2.7113) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][260/625] eta 0:02:18 lr 0.000175 wd 0.0500 time 0.3693 (0.3804) data time 0.0013 (0.0036) model time 0.3681 (0.3763) loss 2.7786 (2.7372) grad_norm 1.6634 (2.6868) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][270/625] eta 0:02:14 lr 0.000175 wd 0.0500 time 0.3771 (0.3801) data time 0.0011 (0.0035) model time 0.3761 (0.3761) loss 3.1057 (2.7396) grad_norm 2.1689 (2.7035) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][280/625] eta 0:02:11 lr 0.000175 wd 0.0500 time 0.3781 (0.3799) data time 0.0011 (0.0035) model time 0.3771 (0.3760) loss 2.6509 (2.7379) grad_norm 2.1576 (2.7245) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][290/625] eta 0:02:07 lr 0.000175 wd 0.0500 time 0.3872 (0.3798) data time 0.0011 (0.0034) model time 0.3861 (0.3760) loss 3.1272 (2.7417) grad_norm 2.8161 (2.7149) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][300/625] eta 0:02:03 lr 0.000175 wd 0.0500 time 0.3664 (0.3795) data time 0.0008 (0.0033) model time 0.3656 (0.3757) loss 3.5490 (2.7479) grad_norm 6.5546 (2.7167) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:11:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][310/625] eta 0:01:59 lr 0.000175 wd 0.0500 time 0.3829 (0.3793) data time 0.0008 (0.0032) model time 0.3821 (0.3756) loss 2.0988 (2.7437) grad_norm 3.1126 (2.7059) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][320/625] eta 0:01:55 lr 0.000175 wd 0.0500 time 0.3709 (0.3791) data time 0.0010 (0.0032) model time 0.3699 (0.3754) loss 3.0114 (2.7474) grad_norm 1.7747 (2.7171) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][330/625] eta 0:01:51 lr 0.000175 wd 0.0500 time 0.3699 (0.3788) data time 0.0009 (0.0031) model time 0.3690 (0.3752) loss 2.5297 (2.7484) grad_norm 4.5625 (2.7157) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][340/625] eta 0:01:48 lr 0.000174 wd 0.0500 time 0.3589 (0.3792) data time 0.0011 (0.0031) model time 0.3579 (0.3757) loss 2.6568 (2.7471) grad_norm 1.8542 (2.7050) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][350/625] eta 0:01:44 lr 0.000174 wd 0.0500 time 0.3722 (0.3790) data time 0.0011 (0.0030) model time 0.3711 (0.3756) loss 2.3406 (2.7431) grad_norm 3.7071 (2.6934) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][360/625] eta 0:01:40 lr 0.000174 wd 0.0500 time 0.3743 (0.3789) data time 0.0009 (0.0030) model time 0.3735 (0.3755) loss 3.2826 (2.7479) grad_norm 2.1570 (2.6812) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][370/625] eta 0:01:36 lr 0.000174 wd 0.0500 time 0.5898 (0.3798) data time 0.0011 (0.0029) model time 0.5887 (0.3767) loss 2.9303 (2.7401) grad_norm 2.6131 (2.6773) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][380/625] eta 0:01:33 lr 0.000174 wd 0.0500 time 0.3660 (0.3796) data time 0.0008 (0.0029) model time 0.3652 (0.3765) loss 3.2683 (2.7423) grad_norm 5.5881 (2.6716) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][390/625] eta 0:01:29 lr 0.000174 wd 0.0500 time 0.3712 (0.3793) data time 0.0013 (0.0028) model time 0.3700 (0.3762) loss 2.2648 (2.7410) grad_norm 4.5979 (2.6757) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][400/625] eta 0:01:25 lr 0.000174 wd 0.0500 time 0.3709 (0.3791) data time 0.0011 (0.0028) model time 0.3698 (0.3760) loss 2.8900 (2.7396) grad_norm 2.3114 (2.6776) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][410/625] eta 0:01:21 lr 0.000174 wd 0.0500 time 0.3828 (0.3790) data time 0.0011 (0.0027) model time 0.3817 (0.3759) loss 2.4874 (2.7353) grad_norm 2.7071 (2.6827) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][420/625] eta 0:01:17 lr 0.000174 wd 0.0500 time 0.3776 (0.3788) data time 0.0008 (0.0027) model time 0.3768 (0.3758) loss 2.6869 (2.7359) grad_norm 1.7846 (2.6729) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][430/625] eta 0:01:13 lr 0.000174 wd 0.0500 time 0.3692 (0.3787) data time 0.0010 (0.0027) model time 0.3682 (0.3757) loss 1.5041 (2.7305) grad_norm 2.0193 (2.6639) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][440/625] eta 0:01:10 lr 0.000173 wd 0.0500 time 0.3770 (0.3786) data time 0.0008 (0.0026) model time 0.3762 (0.3756) loss 2.8129 (2.7311) grad_norm 2.2667 (2.6555) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][450/625] eta 0:01:06 lr 0.000173 wd 0.0500 time 0.3744 (0.3789) data time 0.0009 (0.0026) model time 0.3734 (0.3761) loss 2.4270 (2.7339) grad_norm 2.3081 (2.6542) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][460/625] eta 0:01:02 lr 0.000173 wd 0.0500 time 0.3697 (0.3792) data time 0.0008 (0.0026) model time 0.3689 (0.3764) loss 1.8006 (2.7334) grad_norm 2.4752 (2.6574) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][470/625] eta 0:00:58 lr 0.000173 wd 0.0500 time 0.3604 (0.3791) data time 0.0012 (0.0025) model time 0.3592 (0.3763) loss 2.1338 (2.7297) grad_norm 1.6715 (2.6481) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][480/625] eta 0:00:54 lr 0.000173 wd 0.0500 time 0.3777 (0.3789) data time 0.0008 (0.0025) model time 0.3769 (0.3762) loss 2.8732 (2.7255) grad_norm 5.1041 (2.6459) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][490/625] eta 0:00:51 lr 0.000173 wd 0.0500 time 0.3643 (0.3788) data time 0.0010 (0.0025) model time 0.3633 (0.3761) loss 2.2459 (2.7263) grad_norm 2.3328 (2.6447) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][500/625] eta 0:00:47 lr 0.000173 wd 0.0500 time 0.3715 (0.3787) data time 0.0009 (0.0025) model time 0.3707 (0.3760) loss 2.9072 (2.7296) grad_norm 2.3221 (2.6476) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][510/625] eta 0:00:43 lr 0.000173 wd 0.0500 time 0.3711 (0.3785) data time 0.0012 (0.0024) model time 0.3699 (0.3758) loss 2.9568 (2.7311) grad_norm 2.6105 (2.6486) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][520/625] eta 0:00:39 lr 0.000173 wd 0.0500 time 0.3811 (0.3784) data time 0.0014 (0.0024) model time 0.3797 (0.3758) loss 2.7928 (2.7287) grad_norm 2.3683 (2.6440) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][530/625] eta 0:00:35 lr 0.000173 wd 0.0500 time 0.3779 (0.3784) data time 0.0011 (0.0024) model time 0.3768 (0.3757) loss 2.2700 (2.7280) grad_norm 2.2787 (2.6477) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][540/625] eta 0:00:32 lr 0.000173 wd 0.0500 time 0.3747 (0.3783) data time 0.0008 (0.0024) model time 0.3739 (0.3756) loss 2.9143 (2.7253) grad_norm 2.9635 (2.6439) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][550/625] eta 0:00:28 lr 0.000172 wd 0.0500 time 0.3734 (0.3782) data time 0.0011 (0.0023) model time 0.3723 (0.3756) loss 2.6067 (2.7268) grad_norm 3.0182 (2.6381) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][560/625] eta 0:00:24 lr 0.000172 wd 0.0500 time 0.3634 (0.3781) data time 0.0010 (0.0023) model time 0.3624 (0.3755) loss 2.6276 (2.7259) grad_norm 3.0587 (2.6407) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][570/625] eta 0:00:20 lr 0.000172 wd 0.0500 time 0.3668 (0.3784) data time 0.0014 (0.0023) model time 0.3654 (0.3759) loss 2.6445 (2.7276) grad_norm 2.8774 (2.6391) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][580/625] eta 0:00:17 lr 0.000172 wd 0.0500 time 0.3724 (0.3787) data time 0.0011 (0.0023) model time 0.3713 (0.3762) loss 3.1711 (2.7329) grad_norm 2.3902 (2.6298) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][590/625] eta 0:00:13 lr 0.000172 wd 0.0500 time 0.3738 (0.3790) data time 0.0011 (0.0023) model time 0.3726 (0.3766) loss 2.7092 (2.7349) grad_norm 2.3164 (2.6278) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][600/625] eta 0:00:09 lr 0.000172 wd 0.0500 time 0.3809 (0.3789) data time 0.0013 (0.0023) model time 0.3797 (0.3765) loss 2.1110 (2.7311) grad_norm 1.6886 (2.6245) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][610/625] eta 0:00:05 lr 0.000172 wd 0.0500 time 0.3670 (0.3789) data time 0.0005 (0.0022) model time 0.3665 (0.3764) loss 2.5779 (2.7300) grad_norm 1.6156 (2.6161) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [249/300][620/625] eta 0:00:01 lr 0.000172 wd 0.0500 time 0.3553 (0.3786) data time 0.0005 (0.0022) model time 0.3547 (0.3762) loss 2.4791 (2.7280) grad_norm 2.6670 (2.6141) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:13:55 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 249 training takes 0:03:56 [2024-09-01 12:13:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:13:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:13:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.519 (0.519) Loss 0.5063 (0.5063) Acc@1 88.672 (88.672) Acc@5 98.730 (98.730) Mem 11224MB [2024-09-01 12:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 0.8506 (0.6198) Acc@1 80.420 (86.577) Acc@5 95.508 (97.732) Mem 11224MB [2024-09-01 12:13:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9072 (0.7354) Acc@1 78.369 (83.647) Acc@5 95.068 (96.536) Mem 11224MB [2024-09-01 12:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.365 Acc@5 96.553 [2024-09-01 12:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 12:13:59 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.37% [2024-09-01 12:13:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 12:14:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 12:14:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.492 (0.492) Loss 0.4885 (0.4885) Acc@1 89.453 (89.453) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 12:14:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.141) Loss 0.8042 (0.5926) Acc@1 80.518 (86.998) Acc@5 95.850 (97.865) Mem 11224MB [2024-09-01 12:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.8730 (0.7023) Acc@1 79.492 (84.194) Acc@5 95.312 (96.749) Mem 11224MB [2024-09-01 12:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.903 Acc@5 96.775 [2024-09-01 12:14:03 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 12:14:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][0/625] eta 0:13:32 lr 0.000172 wd 0.0500 time 1.3005 (1.3005) data time 0.5414 (0.5414) model time 0.0000 (0.0000) loss 2.7431 (2.7431) grad_norm 2.7237 (2.7237) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][10/625] eta 0:04:42 lr 0.000172 wd 0.0500 time 0.3796 (0.4588) data time 0.0009 (0.0502) model time 0.0000 (0.0000) loss 1.7326 (2.4329) grad_norm 2.2390 (2.4520) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][20/625] eta 0:04:12 lr 0.000172 wd 0.0500 time 0.3698 (0.4173) data time 0.0009 (0.0269) model time 0.0000 (0.0000) loss 2.7601 (2.6350) grad_norm 1.5157 (2.4972) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][30/625] eta 0:04:00 lr 0.000171 wd 0.0500 time 0.3672 (0.4042) data time 0.0011 (0.0186) model time 0.0000 (0.0000) loss 2.8313 (2.6407) grad_norm 2.6641 (2.5890) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][40/625] eta 0:03:52 lr 0.000171 wd 0.0500 time 0.3700 (0.3968) data time 0.0008 (0.0143) model time 0.0000 (0.0000) loss 2.9812 (2.6459) grad_norm 1.9659 (2.5098) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][50/625] eta 0:03:45 lr 0.000171 wd 0.0500 time 0.3710 (0.3918) data time 0.0015 (0.0118) model time 0.0000 (0.0000) loss 2.6490 (2.6555) grad_norm 3.2147 (2.4983) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][60/625] eta 0:03:40 lr 0.000171 wd 0.0500 time 0.3990 (0.3897) data time 0.0009 (0.0100) model time 0.3981 (0.3780) loss 3.4123 (2.6745) grad_norm 2.4035 (2.5006) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][70/625] eta 0:03:35 lr 0.000171 wd 0.0500 time 0.3667 (0.3876) data time 0.0008 (0.0087) model time 0.3659 (0.3756) loss 3.0676 (2.6782) grad_norm 3.0466 (2.5828) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][80/625] eta 0:03:30 lr 0.000171 wd 0.0500 time 0.3750 (0.3860) data time 0.0011 (0.0078) model time 0.3739 (0.3750) loss 2.4002 (2.6967) grad_norm 4.3662 (2.7485) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][90/625] eta 0:03:26 lr 0.000171 wd 0.0500 time 0.3761 (0.3866) data time 0.0010 (0.0071) model time 0.3751 (0.3790) loss 2.8476 (2.7117) grad_norm 3.4380 (2.7584) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][100/625] eta 0:03:22 lr 0.000171 wd 0.0500 time 0.3756 (0.3854) data time 0.0013 (0.0065) model time 0.3743 (0.3778) loss 2.5837 (2.6875) grad_norm 1.9724 (2.7596) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][110/625] eta 0:03:17 lr 0.000171 wd 0.0500 time 0.3767 (0.3843) data time 0.0010 (0.0060) model time 0.3757 (0.3769) loss 2.6966 (2.6935) grad_norm 1.6726 (2.7438) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][120/625] eta 0:03:13 lr 0.000171 wd 0.0500 time 0.3691 (0.3834) data time 0.0008 (0.0056) model time 0.3683 (0.3762) loss 2.7646 (2.6985) grad_norm 2.8168 (2.7183) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][130/625] eta 0:03:09 lr 0.000171 wd 0.0500 time 0.3634 (0.3823) data time 0.0009 (0.0053) model time 0.3626 (0.3751) loss 3.1344 (2.7028) grad_norm 1.8074 (2.7290) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:14:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][140/625] eta 0:03:05 lr 0.000170 wd 0.0500 time 0.3787 (0.3815) data time 0.0009 (0.0050) model time 0.3778 (0.3745) loss 2.6014 (2.6899) grad_norm 3.2043 (2.7410) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][150/625] eta 0:03:00 lr 0.000170 wd 0.0500 time 0.3758 (0.3810) data time 0.0011 (0.0047) model time 0.3747 (0.3743) loss 1.9719 (2.6923) grad_norm 2.1933 (2.7728) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][160/625] eta 0:02:58 lr 0.000170 wd 0.0500 time 0.3741 (0.3830) data time 0.0014 (0.0045) model time 0.3727 (0.3777) loss 2.2335 (2.6968) grad_norm 2.2471 (2.7479) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][170/625] eta 0:02:54 lr 0.000170 wd 0.0500 time 0.3742 (0.3834) data time 0.0011 (0.0043) model time 0.3732 (0.3786) loss 3.1402 (2.6989) grad_norm 2.5844 (2.7395) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][180/625] eta 0:02:50 lr 0.000170 wd 0.0500 time 0.3775 (0.3829) data time 0.0012 (0.0042) model time 0.3762 (0.3782) loss 2.5770 (2.6923) grad_norm 2.5468 (2.7420) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][190/625] eta 0:02:46 lr 0.000170 wd 0.0500 time 0.3709 (0.3823) data time 0.0013 (0.0040) model time 0.3697 (0.3776) loss 3.0318 (2.6952) grad_norm 2.4631 (2.7635) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][200/625] eta 0:02:42 lr 0.000170 wd 0.0500 time 0.3776 (0.3818) data time 0.0012 (0.0039) model time 0.3764 (0.3772) loss 3.0427 (2.6963) grad_norm 2.4084 (2.7773) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][210/625] eta 0:02:38 lr 0.000170 wd 0.0500 time 0.3782 (0.3814) data time 0.0009 (0.0037) model time 0.3773 (0.3769) loss 2.7290 (2.6913) grad_norm 2.6271 (2.7865) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][220/625] eta 0:02:34 lr 0.000170 wd 0.0500 time 0.3726 (0.3811) data time 0.0009 (0.0036) model time 0.3718 (0.3767) loss 2.7906 (2.6904) grad_norm 2.4971 (2.7702) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][230/625] eta 0:02:30 lr 0.000170 wd 0.0500 time 0.3630 (0.3808) data time 0.0011 (0.0035) model time 0.3619 (0.3765) loss 3.1446 (2.7025) grad_norm 3.2656 (2.7608) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][240/625] eta 0:02:26 lr 0.000169 wd 0.0500 time 0.3692 (0.3804) data time 0.0013 (0.0034) model time 0.3679 (0.3762) loss 2.8927 (2.6964) grad_norm 2.1849 (2.7425) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][250/625] eta 0:02:22 lr 0.000169 wd 0.0500 time 0.3851 (0.3802) data time 0.0008 (0.0033) model time 0.3842 (0.3760) loss 2.3096 (2.6913) grad_norm 2.1433 (2.7322) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][260/625] eta 0:02:18 lr 0.000169 wd 0.0500 time 0.3729 (0.3801) data time 0.0011 (0.0032) model time 0.3718 (0.3761) loss 2.8527 (2.6904) grad_norm 2.5539 (2.7265) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][270/625] eta 0:02:14 lr 0.000169 wd 0.0500 time 0.3774 (0.3799) data time 0.0015 (0.0032) model time 0.3759 (0.3760) loss 2.7495 (2.6865) grad_norm 2.1339 (2.7087) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][280/625] eta 0:02:10 lr 0.000169 wd 0.0500 time 0.3709 (0.3796) data time 0.0008 (0.0031) model time 0.3701 (0.3757) loss 2.8421 (2.6925) grad_norm 3.1851 (2.6967) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][290/625] eta 0:02:07 lr 0.000169 wd 0.0500 time 0.3674 (0.3801) data time 0.0009 (0.0030) model time 0.3665 (0.3764) loss 3.3256 (2.6870) grad_norm 4.9721 (2.6923) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:15:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][300/625] eta 0:02:03 lr 0.000169 wd 0.0500 time 0.3829 (0.3806) data time 0.0011 (0.0030) model time 0.3818 (0.3771) loss 2.6167 (2.6908) grad_norm 3.1199 (2.7042) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][310/625] eta 0:01:59 lr 0.000169 wd 0.0500 time 0.3799 (0.3804) data time 0.0013 (0.0029) model time 0.3786 (0.3769) loss 2.4529 (2.6962) grad_norm 3.5288 (2.7116) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][320/625] eta 0:01:55 lr 0.000169 wd 0.0500 time 0.3785 (0.3801) data time 0.0008 (0.0029) model time 0.3776 (0.3767) loss 1.9817 (2.6931) grad_norm 2.2554 (2.7041) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][330/625] eta 0:01:52 lr 0.000169 wd 0.0500 time 0.3785 (0.3800) data time 0.0008 (0.0028) model time 0.3777 (0.3766) loss 3.0203 (2.6961) grad_norm 1.9698 (2.6896) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][340/625] eta 0:01:48 lr 0.000169 wd 0.0500 time 0.3625 (0.3802) data time 0.0009 (0.0028) model time 0.3616 (0.3770) loss 1.7042 (2.6947) grad_norm 2.9235 (2.6901) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][350/625] eta 0:01:44 lr 0.000168 wd 0.0500 time 0.3641 (0.3804) data time 0.0008 (0.0027) model time 0.3633 (0.3773) loss 3.1579 (2.7004) grad_norm 2.2430 (2.7360) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][360/625] eta 0:01:40 lr 0.000168 wd 0.0500 time 0.3712 (0.3803) data time 0.0010 (0.0027) model time 0.3701 (0.3773) loss 2.8220 (2.6989) grad_norm 2.4071 (2.7299) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][370/625] eta 0:01:36 lr 0.000168 wd 0.0500 time 0.3731 (0.3802) data time 0.0008 (0.0026) model time 0.3723 (0.3771) loss 3.3549 (2.7019) grad_norm 11.2112 (2.7467) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][380/625] eta 0:01:33 lr 0.000168 wd 0.0500 time 0.3701 (0.3799) data time 0.0009 (0.0026) model time 0.3692 (0.3769) loss 2.0280 (2.7087) grad_norm 3.2339 (2.7403) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][390/625] eta 0:01:29 lr 0.000168 wd 0.0500 time 0.3668 (0.3798) data time 0.0009 (0.0026) model time 0.3659 (0.3768) loss 2.9488 (2.7114) grad_norm 1.7511 (2.7414) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][400/625] eta 0:01:25 lr 0.000168 wd 0.0500 time 0.3701 (0.3796) data time 0.0010 (0.0025) model time 0.3691 (0.3766) loss 2.8309 (2.7042) grad_norm 2.1833 (2.7355) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][410/625] eta 0:01:21 lr 0.000168 wd 0.0500 time 0.3710 (0.3794) data time 0.0014 (0.0025) model time 0.3696 (0.3765) loss 2.9169 (2.7055) grad_norm 3.8231 (2.7460) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][420/625] eta 0:01:17 lr 0.000168 wd 0.0500 time 0.3689 (0.3792) data time 0.0013 (0.0025) model time 0.3676 (0.3763) loss 1.8512 (2.7070) grad_norm 1.6449 (2.7378) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][430/625] eta 0:01:13 lr 0.000168 wd 0.0500 time 0.3730 (0.3790) data time 0.0011 (0.0024) model time 0.3719 (0.3761) loss 2.1008 (2.7070) grad_norm 1.5869 (2.7302) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][440/625] eta 0:01:10 lr 0.000168 wd 0.0500 time 0.3732 (0.3789) data time 0.0009 (0.0024) model time 0.3723 (0.3760) loss 3.6796 (2.7110) grad_norm 4.4128 (2.7170) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][450/625] eta 0:01:06 lr 0.000168 wd 0.0500 time 0.3785 (0.3788) data time 0.0011 (0.0024) model time 0.3775 (0.3759) loss 2.0112 (2.7080) grad_norm 3.1245 (2.7218) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][460/625] eta 0:01:02 lr 0.000167 wd 0.0500 time 0.3711 (0.3791) data time 0.0009 (0.0024) model time 0.3701 (0.3763) loss 2.9984 (2.7123) grad_norm 1.5668 (2.7155) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][470/625] eta 0:00:58 lr 0.000167 wd 0.0500 time 0.3702 (0.3794) data time 0.0011 (0.0023) model time 0.3691 (0.3767) loss 2.5577 (2.7134) grad_norm 2.1476 (2.7086) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][480/625] eta 0:00:55 lr 0.000167 wd 0.0500 time 0.3705 (0.3797) data time 0.0011 (0.0023) model time 0.3695 (0.3771) loss 2.0062 (2.7095) grad_norm 1.7532 (2.7028) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][490/625] eta 0:00:51 lr 0.000167 wd 0.0500 time 0.3709 (0.3795) data time 0.0008 (0.0023) model time 0.3700 (0.3768) loss 3.0796 (2.7153) grad_norm 2.0521 (2.6931) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][500/625] eta 0:00:47 lr 0.000167 wd 0.0500 time 0.3855 (0.3797) data time 0.0008 (0.0023) model time 0.3846 (0.3772) loss 2.3775 (2.7151) grad_norm 2.3452 (2.6832) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][510/625] eta 0:00:43 lr 0.000167 wd 0.0500 time 0.3626 (0.3795) data time 0.0012 (0.0022) model time 0.3614 (0.3770) loss 2.5262 (2.7168) grad_norm 1.5020 (2.6724) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][520/625] eta 0:00:39 lr 0.000167 wd 0.0500 time 0.3737 (0.3794) data time 0.0008 (0.0022) model time 0.3729 (0.3769) loss 2.9678 (2.7145) grad_norm 3.3385 (2.6686) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][530/625] eta 0:00:36 lr 0.000167 wd 0.0500 time 0.3620 (0.3793) data time 0.0010 (0.0022) model time 0.3610 (0.3768) loss 2.0265 (2.7133) grad_norm 4.3329 (2.6731) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][540/625] eta 0:00:32 lr 0.000167 wd 0.0500 time 0.3776 (0.3793) data time 0.0011 (0.0022) model time 0.3765 (0.3768) loss 2.7437 (2.7138) grad_norm 2.0497 (2.6861) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][550/625] eta 0:00:28 lr 0.000167 wd 0.0500 time 0.3642 (0.3791) data time 0.0008 (0.0022) model time 0.3633 (0.3766) loss 2.0899 (2.7155) grad_norm 2.7040 (2.6811) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][560/625] eta 0:00:24 lr 0.000166 wd 0.0500 time 0.3950 (0.3791) data time 0.0011 (0.0021) model time 0.3939 (0.3766) loss 2.7654 (2.7175) grad_norm 3.0499 (2.6826) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][570/625] eta 0:00:20 lr 0.000166 wd 0.0500 time 0.3719 (0.3789) data time 0.0010 (0.0021) model time 0.3708 (0.3765) loss 2.5934 (2.7189) grad_norm 2.7516 (2.6780) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][580/625] eta 0:00:17 lr 0.000166 wd 0.0500 time 0.3704 (0.3788) data time 0.0011 (0.0021) model time 0.3693 (0.3764) loss 3.3236 (2.7221) grad_norm 1.9859 (2.6728) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][590/625] eta 0:00:13 lr 0.000166 wd 0.0500 time 0.3766 (0.3787) data time 0.0011 (0.0021) model time 0.3755 (0.3763) loss 2.2475 (2.7209) grad_norm 1.9646 (2.6617) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][600/625] eta 0:00:09 lr 0.000166 wd 0.0500 time 0.3860 (0.3790) data time 0.0010 (0.0021) model time 0.3850 (0.3766) loss 2.5612 (2.7238) grad_norm 2.5984 (2.6596) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][610/625] eta 0:00:05 lr 0.000166 wd 0.0500 time 0.3657 (0.3789) data time 0.0008 (0.0021) model time 0.3649 (0.3766) loss 2.5594 (2.7273) grad_norm 3.5235 (2.6586) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:17:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [250/300][620/625] eta 0:00:01 lr 0.000166 wd 0.0500 time 0.3591 (0.3786) data time 0.0008 (0.0021) model time 0.3583 (0.3762) loss 2.7423 (2.7267) grad_norm 2.5598 (2.6546) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:00 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 250 training takes 0:03:56 [2024-09-01 12:18:00 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:18:01 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:18:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.517 (0.517) Loss 0.5210 (0.5210) Acc@1 89.062 (89.062) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 12:18:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.144) Loss 0.8521 (0.6209) Acc@1 80.176 (86.630) Acc@5 95.557 (97.723) Mem 11224MB [2024-09-01 12:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.8965 (0.7364) Acc@1 78.955 (83.703) Acc@5 95.410 (96.545) Mem 11224MB [2024-09-01 12:18:04 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.369 Acc@5 96.533 [2024-09-01 12:18:04 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 12:18:04 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.37% [2024-09-01 12:18:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 12:18:05 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 12:18:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.487 (0.487) Loss 0.4885 (0.4885) Acc@1 89.502 (89.502) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 12:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.147) Loss 0.8052 (0.5928) Acc@1 80.420 (87.007) Acc@5 95.850 (97.874) Mem 11224MB [2024-09-01 12:18:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 0.8726 (0.7024) Acc@1 79.395 (84.205) Acc@5 95.264 (96.749) Mem 11224MB [2024-09-01 12:18:08 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.915 Acc@5 96.777 [2024-09-01 12:18:08 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 12:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][0/625] eta 0:13:14 lr 0.000166 wd 0.0500 time 1.2706 (1.2706) data time 0.7549 (0.7549) model time 0.0000 (0.0000) loss 2.9279 (2.9279) grad_norm 3.6219 (3.6219) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][10/625] eta 0:04:48 lr 0.000166 wd 0.0500 time 0.3769 (0.4691) data time 0.0010 (0.0696) model time 0.0000 (0.0000) loss 2.5001 (2.6476) grad_norm 1.9865 (2.6809) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][20/625] eta 0:04:15 lr 0.000166 wd 0.0500 time 0.3689 (0.4228) data time 0.0013 (0.0371) model time 0.0000 (0.0000) loss 2.6237 (2.6921) grad_norm 1.8148 (2.3706) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][30/625] eta 0:04:09 lr 0.000166 wd 0.0500 time 0.3723 (0.4193) data time 0.0010 (0.0255) model time 0.0000 (0.0000) loss 2.7377 (2.6892) grad_norm 3.4516 (2.3669) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][40/625] eta 0:03:58 lr 0.000166 wd 0.0500 time 0.3754 (0.4079) data time 0.0013 (0.0196) model time 0.0000 (0.0000) loss 2.6531 (2.6675) grad_norm 2.4826 (2.3530) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][50/625] eta 0:03:50 lr 0.000165 wd 0.0500 time 0.3791 (0.4010) data time 0.0008 (0.0159) model time 0.0000 (0.0000) loss 2.1351 (2.6639) grad_norm 2.4792 (2.3982) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][60/625] eta 0:03:43 lr 0.000165 wd 0.0500 time 0.3662 (0.3962) data time 0.0011 (0.0135) model time 0.3651 (0.3709) loss 2.7115 (2.6528) grad_norm 2.1331 (2.3731) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][70/625] eta 0:03:38 lr 0.000165 wd 0.0500 time 0.3713 (0.3933) data time 0.0011 (0.0118) model time 0.3702 (0.3723) loss 2.8288 (2.6630) grad_norm 1.9091 (2.3086) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][80/625] eta 0:03:32 lr 0.000165 wd 0.0500 time 0.3720 (0.3905) data time 0.0009 (0.0105) model time 0.3711 (0.3715) loss 2.6213 (2.6746) grad_norm 3.2386 (2.2836) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][90/625] eta 0:03:27 lr 0.000165 wd 0.0500 time 0.3763 (0.3885) data time 0.0008 (0.0094) model time 0.3755 (0.3715) loss 2.2841 (2.6825) grad_norm 2.3564 (2.3024) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][100/625] eta 0:03:23 lr 0.000165 wd 0.0500 time 0.3728 (0.3871) data time 0.0008 (0.0086) model time 0.3719 (0.3718) loss 2.7852 (2.6956) grad_norm 2.0106 (2.2816) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][110/625] eta 0:03:19 lr 0.000165 wd 0.0500 time 0.3723 (0.3878) data time 0.0011 (0.0079) model time 0.3712 (0.3755) loss 2.7533 (2.6694) grad_norm 2.9650 (2.3553) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][120/625] eta 0:03:16 lr 0.000165 wd 0.0500 time 0.3355 (0.3881) data time 0.0012 (0.0074) model time 0.3344 (0.3777) loss 2.1903 (2.6733) grad_norm 2.4828 (2.4743) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:18:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][130/625] eta 0:03:11 lr 0.000165 wd 0.0500 time 0.3798 (0.3871) data time 0.0011 (0.0069) model time 0.3787 (0.3770) loss 3.0605 (2.6723) grad_norm 1.9281 (2.4655) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][140/625] eta 0:03:07 lr 0.000165 wd 0.0500 time 0.3775 (0.3860) data time 0.0009 (0.0065) model time 0.3766 (0.3763) loss 1.6312 (2.6586) grad_norm 2.7349 (2.4537) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][150/625] eta 0:03:02 lr 0.000164 wd 0.0500 time 0.3776 (0.3852) data time 0.0009 (0.0062) model time 0.3767 (0.3760) loss 2.6408 (2.6501) grad_norm 1.9929 (2.4270) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][160/625] eta 0:02:58 lr 0.000164 wd 0.0500 time 0.3625 (0.3843) data time 0.0012 (0.0058) model time 0.3613 (0.3754) loss 2.2439 (2.6422) grad_norm 2.6274 (2.4290) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][170/625] eta 0:02:54 lr 0.000164 wd 0.0500 time 0.3612 (0.3836) data time 0.0008 (0.0056) model time 0.3603 (0.3750) loss 2.7591 (2.6462) grad_norm 5.8738 (2.5028) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][180/625] eta 0:02:50 lr 0.000164 wd 0.0500 time 0.3714 (0.3829) data time 0.0009 (0.0053) model time 0.3706 (0.3746) loss 2.8822 (2.6477) grad_norm 3.2881 (2.5360) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][190/625] eta 0:02:46 lr 0.000164 wd 0.0500 time 0.3763 (0.3824) data time 0.0010 (0.0051) model time 0.3754 (0.3745) loss 2.5329 (2.6501) grad_norm 3.5508 (2.5399) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][200/625] eta 0:02:42 lr 0.000164 wd 0.0500 time 0.3696 (0.3819) data time 0.0011 (0.0049) model time 0.3685 (0.3743) loss 2.4300 (2.6506) grad_norm 2.7675 (2.5167) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][210/625] eta 0:02:38 lr 0.000164 wd 0.0500 time 0.3741 (0.3816) data time 0.0009 (0.0048) model time 0.3733 (0.3742) loss 3.1255 (2.6508) grad_norm 2.0279 (2.5115) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][220/625] eta 0:02:34 lr 0.000164 wd 0.0500 time 0.3724 (0.3812) data time 0.0010 (0.0046) model time 0.3715 (0.3741) loss 1.7677 (2.6575) grad_norm 2.5379 (2.5162) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][230/625] eta 0:02:30 lr 0.000164 wd 0.0500 time 0.3657 (0.3809) data time 0.0010 (0.0045) model time 0.3647 (0.3740) loss 3.0581 (2.6662) grad_norm 2.2178 (2.5116) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][240/625] eta 0:02:26 lr 0.000164 wd 0.0500 time 0.3776 (0.3816) data time 0.0012 (0.0043) model time 0.3764 (0.3751) loss 2.8990 (2.6696) grad_norm 2.6687 (2.5072) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][250/625] eta 0:02:22 lr 0.000164 wd 0.0500 time 0.3692 (0.3813) data time 0.0011 (0.0042) model time 0.3681 (0.3750) loss 2.4976 (2.6652) grad_norm 2.6911 (2.5204) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][260/625] eta 0:02:19 lr 0.000163 wd 0.0500 time 0.3765 (0.3810) data time 0.0011 (0.0041) model time 0.3754 (0.3749) loss 2.4679 (2.6755) grad_norm 2.3657 (2.5296) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][270/625] eta 0:02:15 lr 0.000163 wd 0.0500 time 0.3812 (0.3807) data time 0.0009 (0.0040) model time 0.3803 (0.3747) loss 2.5923 (2.6812) grad_norm 1.5177 (2.5212) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][280/625] eta 0:02:11 lr 0.000163 wd 0.0500 time 0.3743 (0.3804) data time 0.0008 (0.0039) model time 0.3734 (0.3746) loss 3.4419 (2.6873) grad_norm 2.1044 (2.5017) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:19:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][290/625] eta 0:02:07 lr 0.000163 wd 0.0500 time 0.3730 (0.3801) data time 0.0009 (0.0038) model time 0.3722 (0.3745) loss 2.5675 (2.6828) grad_norm 2.6211 (2.5111) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][300/625] eta 0:02:03 lr 0.000163 wd 0.0500 time 0.3732 (0.3799) data time 0.0008 (0.0037) model time 0.3724 (0.3744) loss 1.7288 (2.6869) grad_norm 2.9300 (2.5340) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][310/625] eta 0:01:59 lr 0.000163 wd 0.0500 time 0.3873 (0.3797) data time 0.0008 (0.0036) model time 0.3864 (0.3743) loss 3.0971 (2.6796) grad_norm 2.1947 (2.5238) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][320/625] eta 0:01:55 lr 0.000163 wd 0.0500 time 0.3740 (0.3802) data time 0.0010 (0.0035) model time 0.3729 (0.3750) loss 3.1194 (2.6820) grad_norm 1.9914 (2.5133) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][330/625] eta 0:01:52 lr 0.000163 wd 0.0500 time 0.3374 (0.3804) data time 0.0008 (0.0035) model time 0.3366 (0.3754) loss 3.3423 (2.6843) grad_norm 2.3050 (2.5048) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][340/625] eta 0:01:48 lr 0.000163 wd 0.0500 time 0.3751 (0.3802) data time 0.0008 (0.0034) model time 0.3743 (0.3753) loss 3.2871 (2.6857) grad_norm 2.2512 (2.5631) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][350/625] eta 0:01:44 lr 0.000163 wd 0.0500 time 0.3728 (0.3799) data time 0.0008 (0.0033) model time 0.3720 (0.3751) loss 1.7696 (2.6828) grad_norm 1.6349 (2.5573) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][360/625] eta 0:01:40 lr 0.000163 wd 0.0500 time 0.3735 (0.3797) data time 0.0012 (0.0033) model time 0.3722 (0.3750) loss 2.7636 (2.6759) grad_norm 2.0428 (2.5625) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][370/625] eta 0:01:36 lr 0.000162 wd 0.0500 time 0.3764 (0.3795) data time 0.0011 (0.0032) model time 0.3753 (0.3749) loss 2.4091 (2.6756) grad_norm 3.6429 (2.5575) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][380/625] eta 0:01:32 lr 0.000162 wd 0.0500 time 0.3785 (0.3793) data time 0.0011 (0.0032) model time 0.3775 (0.3748) loss 2.9930 (2.6743) grad_norm 2.8781 (2.5555) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][390/625] eta 0:01:29 lr 0.000162 wd 0.0500 time 0.3697 (0.3791) data time 0.0011 (0.0031) model time 0.3686 (0.3747) loss 2.5900 (2.6767) grad_norm 3.4849 (2.5510) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][400/625] eta 0:01:25 lr 0.000162 wd 0.0500 time 0.3722 (0.3790) data time 0.0011 (0.0031) model time 0.3711 (0.3746) loss 2.6812 (2.6755) grad_norm 2.4104 (2.5559) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][410/625] eta 0:01:21 lr 0.000162 wd 0.0500 time 0.3758 (0.3788) data time 0.0009 (0.0030) model time 0.3749 (0.3745) loss 3.1101 (2.6775) grad_norm 2.6670 (2.5513) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][420/625] eta 0:01:17 lr 0.000162 wd 0.0500 time 0.3662 (0.3797) data time 0.0010 (0.0030) model time 0.3652 (0.3755) loss 3.0430 (2.6791) grad_norm 2.6336 (2.5506) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][430/625] eta 0:01:13 lr 0.000162 wd 0.0500 time 0.3708 (0.3795) data time 0.0009 (0.0029) model time 0.3698 (0.3754) loss 3.0540 (2.6816) grad_norm 1.8940 (2.5648) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][440/625] eta 0:01:10 lr 0.000162 wd 0.0500 time 0.3714 (0.3793) data time 0.0011 (0.0029) model time 0.3703 (0.3753) loss 2.3616 (2.6840) grad_norm 21.1464 (2.6444) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:20:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][450/625] eta 0:01:06 lr 0.000162 wd 0.0500 time 0.3660 (0.3791) data time 0.0008 (0.0029) model time 0.3652 (0.3752) loss 2.1406 (2.6867) grad_norm 2.1406 (2.6382) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][460/625] eta 0:01:02 lr 0.000162 wd 0.0500 time 0.3685 (0.3790) data time 0.0009 (0.0028) model time 0.3676 (0.3751) loss 3.4633 (2.6892) grad_norm 2.3804 (2.6347) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][470/625] eta 0:00:58 lr 0.000162 wd 0.0500 time 0.3734 (0.3788) data time 0.0011 (0.0028) model time 0.3723 (0.3750) loss 2.1549 (2.6863) grad_norm 1.7266 (2.6274) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][480/625] eta 0:00:54 lr 0.000161 wd 0.0500 time 0.3631 (0.3786) data time 0.0011 (0.0027) model time 0.3620 (0.3748) loss 2.7537 (2.6897) grad_norm 2.1905 (2.6264) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][490/625] eta 0:00:51 lr 0.000161 wd 0.0500 time 0.3684 (0.3785) data time 0.0008 (0.0027) model time 0.3677 (0.3747) loss 3.3251 (2.6946) grad_norm 1.7977 (2.6320) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][500/625] eta 0:00:47 lr 0.000161 wd 0.0500 time 0.3745 (0.3789) data time 0.0010 (0.0027) model time 0.3734 (0.3752) loss 3.0006 (2.6960) grad_norm 2.0012 (2.6290) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][510/625] eta 0:00:43 lr 0.000161 wd 0.0500 time 0.3650 (0.3787) data time 0.0009 (0.0027) model time 0.3641 (0.3751) loss 3.1040 (2.6940) grad_norm 3.2600 (2.6214) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][520/625] eta 0:00:39 lr 0.000161 wd 0.0500 time 0.3715 (0.3789) data time 0.0011 (0.0026) model time 0.3705 (0.3753) loss 3.3127 (2.6983) grad_norm 3.9231 (2.6251) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][530/625] eta 0:00:35 lr 0.000161 wd 0.0500 time 0.3798 (0.3787) data time 0.0009 (0.0026) model time 0.3790 (0.3752) loss 3.5802 (2.6981) grad_norm 2.6494 (2.6347) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][540/625] eta 0:00:32 lr 0.000161 wd 0.0500 time 0.3762 (0.3794) data time 0.0012 (0.0026) model time 0.3750 (0.3760) loss 2.9825 (2.6994) grad_norm 2.1732 (2.6353) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][550/625] eta 0:00:28 lr 0.000161 wd 0.0500 time 0.3787 (0.3794) data time 0.0008 (0.0025) model time 0.3778 (0.3760) loss 3.1267 (2.7005) grad_norm 1.7381 (2.6423) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][560/625] eta 0:00:24 lr 0.000161 wd 0.0500 time 0.3711 (0.3793) data time 0.0008 (0.0025) model time 0.3703 (0.3760) loss 2.9521 (2.7003) grad_norm 2.4959 (2.6359) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][570/625] eta 0:00:20 lr 0.000161 wd 0.0500 time 0.3700 (0.3792) data time 0.0010 (0.0025) model time 0.3689 (0.3759) loss 2.6884 (2.7020) grad_norm 3.2479 (2.6303) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][580/625] eta 0:00:17 lr 0.000161 wd 0.0500 time 0.3635 (0.3791) data time 0.0011 (0.0025) model time 0.3624 (0.3758) loss 3.2984 (2.7030) grad_norm 2.4806 (2.6384) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][590/625] eta 0:00:13 lr 0.000160 wd 0.0500 time 0.3630 (0.3789) data time 0.0009 (0.0025) model time 0.3621 (0.3757) loss 3.3999 (2.7057) grad_norm 2.6484 (2.6482) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][600/625] eta 0:00:09 lr 0.000160 wd 0.0500 time 0.3677 (0.3788) data time 0.0011 (0.0024) model time 0.3666 (0.3756) loss 1.8333 (2.7016) grad_norm 4.3136 (2.6678) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:21:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][610/625] eta 0:00:05 lr 0.000160 wd 0.0500 time 0.3568 (0.3787) data time 0.0008 (0.0024) model time 0.3560 (0.3755) loss 2.7924 (2.7018) grad_norm 1.5995 (2.6774) loss_scale 512.0000 (256.4190) mem 11224MB [2024-09-01 12:22:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [251/300][620/625] eta 0:00:01 lr 0.000160 wd 0.0500 time 0.3590 (0.3787) data time 0.0006 (0.0024) model time 0.3584 (0.3755) loss 3.2249 (2.7025) grad_norm 3.5123 (2.6790) loss_scale 512.0000 (260.5346) mem 11224MB [2024-09-01 12:22:04 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 251 training takes 0:03:56 [2024-09-01 12:22:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:22:06 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:22:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.477 (0.477) Loss 0.5103 (0.5103) Acc@1 89.258 (89.258) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 12:22:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 0.8481 (0.6194) Acc@1 79.932 (86.537) Acc@5 95.605 (97.749) Mem 11224MB [2024-09-01 12:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9458 (0.7358) Acc@1 77.490 (83.633) Acc@5 94.922 (96.529) Mem 11224MB [2024-09-01 12:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.313 Acc@5 96.527 [2024-09-01 12:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.3% [2024-09-01 12:22:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.778 (0.778) Loss 0.4880 (0.4880) Acc@1 89.551 (89.551) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 12:22:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.171) Loss 0.8052 (0.5926) Acc@1 80.762 (87.078) Acc@5 95.850 (97.883) Mem 11224MB [2024-09-01 12:22:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.135) Loss 0.8750 (0.7024) Acc@1 79.395 (84.263) Acc@5 95.312 (96.740) Mem 11224MB [2024-09-01 12:22:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.971 Acc@5 96.763 [2024-09-01 12:22:12 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 12:22:12 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.97% [2024-09-01 12:22:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 12:22:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 12:22:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][0/625] eta 0:08:38 lr 0.000160 wd 0.0500 time 0.8290 (0.8290) data time 0.4924 (0.4924) model time 0.0000 (0.0000) loss 3.1145 (3.1145) grad_norm 1.9541 (1.9541) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][10/625] eta 0:04:14 lr 0.000160 wd 0.0500 time 0.3748 (0.4140) data time 0.0011 (0.0458) model time 0.0000 (0.0000) loss 3.4654 (2.7948) grad_norm 1.8797 (2.3158) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][20/625] eta 0:03:58 lr 0.000160 wd 0.0500 time 0.3664 (0.3947) data time 0.0008 (0.0245) model time 0.0000 (0.0000) loss 2.1945 (2.6709) grad_norm 2.0176 (2.4042) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][30/625] eta 0:03:50 lr 0.000160 wd 0.0500 time 0.3737 (0.3874) data time 0.0008 (0.0170) model time 0.0000 (0.0000) loss 2.7640 (2.6498) grad_norm 3.0942 (2.6881) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][40/625] eta 0:03:44 lr 0.000160 wd 0.0500 time 0.3694 (0.3834) data time 0.0011 (0.0131) model time 0.0000 (0.0000) loss 3.0347 (2.6401) grad_norm 2.8634 (2.5969) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][50/625] eta 0:03:39 lr 0.000160 wd 0.0500 time 0.3673 (0.3810) data time 0.0010 (0.0108) model time 0.0000 (0.0000) loss 2.9420 (2.6535) grad_norm 2.3414 (2.5578) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][60/625] eta 0:03:34 lr 0.000160 wd 0.0500 time 0.3629 (0.3793) data time 0.0011 (0.0092) model time 0.3618 (0.3694) loss 3.1205 (2.6905) grad_norm 2.7755 (2.5166) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][70/625] eta 0:03:31 lr 0.000159 wd 0.0500 time 0.3700 (0.3812) data time 0.0009 (0.0081) model time 0.3691 (0.3802) loss 3.1022 (2.7118) grad_norm 1.6981 (2.4389) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][80/625] eta 0:03:28 lr 0.000159 wd 0.0500 time 0.3649 (0.3822) data time 0.0010 (0.0072) model time 0.3638 (0.3829) loss 2.3350 (2.6658) grad_norm 2.6332 (2.4195) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][90/625] eta 0:03:23 lr 0.000159 wd 0.0500 time 0.3696 (0.3810) data time 0.0010 (0.0066) model time 0.3686 (0.3798) loss 3.1400 (2.6745) grad_norm 2.5728 (2.4019) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][100/625] eta 0:03:19 lr 0.000159 wd 0.0500 time 0.3694 (0.3798) data time 0.0012 (0.0060) model time 0.3682 (0.3773) loss 3.1366 (2.6867) grad_norm 6.5760 (2.4725) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][110/625] eta 0:03:15 lr 0.000159 wd 0.0500 time 0.3703 (0.3791) data time 0.0010 (0.0056) model time 0.3693 (0.3762) loss 2.6917 (2.6938) grad_norm 2.7907 (2.4608) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:22:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][120/625] eta 0:03:11 lr 0.000159 wd 0.0500 time 0.3703 (0.3785) data time 0.0011 (0.0052) model time 0.3693 (0.3754) loss 2.8316 (2.7058) grad_norm 2.2022 (2.4570) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][130/625] eta 0:03:07 lr 0.000159 wd 0.0500 time 0.3698 (0.3782) data time 0.0012 (0.0050) model time 0.3686 (0.3751) loss 2.9943 (2.7054) grad_norm 3.0200 (2.4890) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][140/625] eta 0:03:03 lr 0.000159 wd 0.0500 time 0.3757 (0.3778) data time 0.0012 (0.0047) model time 0.3744 (0.3748) loss 1.8149 (2.6940) grad_norm 2.5939 (2.4906) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][150/625] eta 0:02:59 lr 0.000159 wd 0.0500 time 0.3749 (0.3773) data time 0.0011 (0.0045) model time 0.3738 (0.3742) loss 2.9987 (2.7031) grad_norm 2.8250 (2.5052) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][160/625] eta 0:02:55 lr 0.000159 wd 0.0500 time 0.3692 (0.3770) data time 0.0008 (0.0043) model time 0.3684 (0.3738) loss 3.3632 (2.7078) grad_norm 1.6859 (2.5160) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][170/625] eta 0:02:51 lr 0.000159 wd 0.0500 time 0.3724 (0.3766) data time 0.0011 (0.0041) model time 0.3712 (0.3735) loss 2.3164 (2.6982) grad_norm 3.0265 (2.5239) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][180/625] eta 0:02:47 lr 0.000158 wd 0.0500 time 0.3738 (0.3775) data time 0.0008 (0.0039) model time 0.3730 (0.3749) loss 3.5342 (2.7073) grad_norm 2.8672 (2.5230) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][190/625] eta 0:02:44 lr 0.000158 wd 0.0500 time 0.3754 (0.3773) data time 0.0011 (0.0038) model time 0.3743 (0.3747) loss 1.5641 (2.6935) grad_norm 2.9007 (2.5187) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][200/625] eta 0:02:40 lr 0.000158 wd 0.0500 time 0.3757 (0.3783) data time 0.0009 (0.0036) model time 0.3747 (0.3761) loss 1.9861 (2.6918) grad_norm 2.1006 (2.5357) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][210/625] eta 0:02:36 lr 0.000158 wd 0.0500 time 0.3762 (0.3781) data time 0.0011 (0.0035) model time 0.3751 (0.3760) loss 3.0634 (2.6896) grad_norm 2.4053 (2.5327) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][220/625] eta 0:02:33 lr 0.000158 wd 0.0500 time 0.3755 (0.3779) data time 0.0011 (0.0034) model time 0.3743 (0.3758) loss 2.3347 (2.6910) grad_norm 2.5842 (2.5595) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][230/625] eta 0:02:29 lr 0.000158 wd 0.0500 time 0.3746 (0.3776) data time 0.0009 (0.0033) model time 0.3738 (0.3755) loss 1.9782 (2.6865) grad_norm 28.1355 (2.7011) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][240/625] eta 0:02:25 lr 0.000158 wd 0.0500 time 0.3755 (0.3775) data time 0.0011 (0.0032) model time 0.3744 (0.3753) loss 2.6616 (2.6853) grad_norm 2.2523 (2.6900) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][250/625] eta 0:02:21 lr 0.000158 wd 0.0500 time 0.3725 (0.3773) data time 0.0009 (0.0031) model time 0.3716 (0.3752) loss 3.3778 (2.6904) grad_norm 2.3233 (2.6722) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][260/625] eta 0:02:17 lr 0.000158 wd 0.0500 time 0.3681 (0.3772) data time 0.0008 (0.0031) model time 0.3673 (0.3751) loss 2.0754 (2.6844) grad_norm 2.2061 (2.6753) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][270/625] eta 0:02:13 lr 0.000158 wd 0.0500 time 0.3751 (0.3770) data time 0.0008 (0.0030) model time 0.3743 (0.3749) loss 3.0925 (2.6883) grad_norm 2.3269 (2.6754) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:23:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][280/625] eta 0:02:10 lr 0.000158 wd 0.0500 time 0.5853 (0.3781) data time 0.0008 (0.0029) model time 0.5845 (0.3763) loss 3.7350 (2.6918) grad_norm 3.1857 (2.6727) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:24:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][290/625] eta 0:02:06 lr 0.000158 wd 0.0500 time 0.3693 (0.3787) data time 0.0011 (0.0029) model time 0.3683 (0.3770) loss 2.9031 (2.6996) grad_norm 2.2290 (2.7021) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:24:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][300/625] eta 0:02:02 lr 0.000157 wd 0.0500 time 0.3633 (0.3784) data time 0.0009 (0.0028) model time 0.3625 (0.3767) loss 1.6640 (2.6970) grad_norm 1.7121 (2.7144) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:24:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][310/625] eta 0:01:59 lr 0.000157 wd 0.0500 time 0.3642 (0.3782) data time 0.0011 (0.0028) model time 0.3631 (0.3765) loss 2.9746 (2.6970) grad_norm 2.9132 (2.7154) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:24:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][320/625] eta 0:01:55 lr 0.000157 wd 0.0500 time 0.3683 (0.3781) data time 0.0008 (0.0027) model time 0.3675 (0.3763) loss 2.5408 (2.6986) grad_norm 1.4513 (2.7437) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:24:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][330/625] eta 0:01:51 lr 0.000157 wd 0.0500 time 0.3752 (0.3779) data time 0.0008 (0.0027) model time 0.3744 (0.3762) loss 3.4545 (2.6988) grad_norm 2.6760 (2.7573) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:24:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][340/625] eta 0:01:47 lr 0.000157 wd 0.0500 time 0.3683 (0.3778) data time 0.0012 (0.0026) model time 0.3671 (0.3760) loss 2.7377 (2.7022) grad_norm 2.5180 (2.7794) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:24:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][350/625] eta 0:01:43 lr 0.000157 wd 0.0500 time 0.3688 (0.3777) data time 0.0009 (0.0026) model time 0.3679 (0.3759) loss 1.8599 (2.7033) grad_norm 1.7709 (2.7852) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 12:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][360/625] eta 0:01:40 lr 0.000157 wd 0.0500 time 0.3716 (0.3775) data time 0.0010 (0.0025) model time 0.3705 (0.3757) loss 2.7886 (2.7063) grad_norm 2.6049 (inf) loss_scale 256.0000 (510.5817) mem 11224MB [2024-09-01 12:24:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][370/625] eta 0:01:36 lr 0.000157 wd 0.0500 time 0.3779 (0.3778) data time 0.0008 (0.0025) model time 0.3770 (0.3761) loss 3.0675 (2.7042) grad_norm 2.9129 (inf) loss_scale 256.0000 (503.7197) mem 11224MB [2024-09-01 12:24:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][380/625] eta 0:01:32 lr 0.000157 wd 0.0500 time 0.3891 (0.3778) data time 0.0011 (0.0025) model time 0.3880 (0.3761) loss 2.6154 (2.7012) grad_norm 3.2483 (inf) loss_scale 256.0000 (497.2178) mem 11224MB [2024-09-01 12:24:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][390/625] eta 0:01:28 lr 0.000157 wd 0.0500 time 0.3750 (0.3782) data time 0.0011 (0.0024) model time 0.3739 (0.3766) loss 2.9366 (2.7016) grad_norm 2.0070 (inf) loss_scale 256.0000 (491.0486) mem 11224MB [2024-09-01 12:24:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][400/625] eta 0:01:25 lr 0.000157 wd 0.0500 time 0.3867 (0.3781) data time 0.0011 (0.0024) model time 0.3856 (0.3765) loss 2.5531 (2.6993) grad_norm 2.1384 (inf) loss_scale 256.0000 (485.1870) mem 11224MB [2024-09-01 12:24:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][410/625] eta 0:01:21 lr 0.000156 wd 0.0500 time 0.3729 (0.3780) data time 0.0011 (0.0024) model time 0.3718 (0.3764) loss 2.8992 (2.7025) grad_norm 2.0297 (inf) loss_scale 256.0000 (479.6107) mem 11224MB [2024-09-01 12:24:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][420/625] eta 0:01:17 lr 0.000156 wd 0.0500 time 0.3660 (0.3778) data time 0.0010 (0.0023) model time 0.3650 (0.3762) loss 2.0713 (2.7054) grad_norm 2.2120 (inf) loss_scale 256.0000 (474.2993) mem 11224MB [2024-09-01 12:24:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][430/625] eta 0:01:13 lr 0.000156 wd 0.0500 time 0.3805 (0.3776) data time 0.0011 (0.0023) model time 0.3794 (0.3760) loss 1.9044 (2.7027) grad_norm 3.0538 (inf) loss_scale 256.0000 (469.2343) mem 11224MB [2024-09-01 12:24:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][440/625] eta 0:01:09 lr 0.000156 wd 0.0500 time 0.3710 (0.3775) data time 0.0010 (0.0023) model time 0.3700 (0.3758) loss 2.9007 (2.7002) grad_norm 2.0056 (inf) loss_scale 256.0000 (464.3991) mem 11224MB [2024-09-01 12:25:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][450/625] eta 0:01:06 lr 0.000156 wd 0.0500 time 0.3722 (0.3773) data time 0.0009 (0.0023) model time 0.3713 (0.3757) loss 3.3548 (2.7003) grad_norm 2.0450 (inf) loss_scale 256.0000 (459.7783) mem 11224MB [2024-09-01 12:25:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][460/625] eta 0:01:02 lr 0.000156 wd 0.0500 time 0.3752 (0.3772) data time 0.0008 (0.0022) model time 0.3745 (0.3755) loss 3.1557 (2.7002) grad_norm 1.8200 (inf) loss_scale 256.0000 (455.3579) mem 11224MB [2024-09-01 12:25:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][470/625] eta 0:00:58 lr 0.000156 wd 0.0500 time 0.3682 (0.3779) data time 0.0009 (0.0022) model time 0.3673 (0.3763) loss 2.4971 (2.7002) grad_norm 2.4580 (inf) loss_scale 256.0000 (451.1253) mem 11224MB [2024-09-01 12:25:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][480/625] eta 0:00:54 lr 0.000156 wd 0.0500 time 0.5975 (0.3782) data time 0.0011 (0.0022) model time 0.5963 (0.3767) loss 2.4051 (2.7003) grad_norm 1.7467 (inf) loss_scale 256.0000 (447.0686) mem 11224MB [2024-09-01 12:25:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][490/625] eta 0:00:51 lr 0.000156 wd 0.0500 time 0.3674 (0.3780) data time 0.0013 (0.0022) model time 0.3661 (0.3765) loss 2.6793 (2.6995) grad_norm 2.8044 (inf) loss_scale 256.0000 (443.1772) mem 11224MB [2024-09-01 12:25:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][500/625] eta 0:00:47 lr 0.000156 wd 0.0500 time 0.5833 (0.3783) data time 0.0008 (0.0022) model time 0.5825 (0.3768) loss 3.1814 (2.7006) grad_norm 2.4851 (inf) loss_scale 256.0000 (439.4411) mem 11224MB [2024-09-01 12:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][510/625] eta 0:00:43 lr 0.000156 wd 0.0500 time 0.3810 (0.3781) data time 0.0010 (0.0021) model time 0.3800 (0.3766) loss 2.9517 (2.7002) grad_norm 2.3175 (inf) loss_scale 256.0000 (435.8513) mem 11224MB [2024-09-01 12:25:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][520/625] eta 0:00:39 lr 0.000155 wd 0.0500 time 0.3687 (0.3780) data time 0.0009 (0.0021) model time 0.3679 (0.3765) loss 3.6907 (2.7069) grad_norm 2.2659 (inf) loss_scale 256.0000 (432.3992) mem 11224MB [2024-09-01 12:25:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][530/625] eta 0:00:35 lr 0.000155 wd 0.0500 time 0.3678 (0.3779) data time 0.0009 (0.0021) model time 0.3668 (0.3764) loss 3.0827 (2.7147) grad_norm 2.3697 (inf) loss_scale 256.0000 (429.0772) mem 11224MB [2024-09-01 12:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][540/625] eta 0:00:32 lr 0.000155 wd 0.0500 time 0.3708 (0.3777) data time 0.0008 (0.0021) model time 0.3699 (0.3762) loss 3.4635 (2.7158) grad_norm 2.7490 (inf) loss_scale 256.0000 (425.8780) mem 11224MB [2024-09-01 12:25:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][550/625] eta 0:00:28 lr 0.000155 wd 0.0500 time 0.3735 (0.3776) data time 0.0009 (0.0021) model time 0.3726 (0.3761) loss 2.2720 (2.7142) grad_norm 2.6058 (inf) loss_scale 256.0000 (422.7949) mem 11224MB [2024-09-01 12:25:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][560/625] eta 0:00:24 lr 0.000155 wd 0.0500 time 0.3723 (0.3775) data time 0.0012 (0.0020) model time 0.3711 (0.3760) loss 2.8779 (2.7114) grad_norm 1.7749 (inf) loss_scale 256.0000 (419.8217) mem 11224MB [2024-09-01 12:25:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][570/625] eta 0:00:20 lr 0.000155 wd 0.0500 time 0.3711 (0.3774) data time 0.0011 (0.0020) model time 0.3700 (0.3759) loss 2.6668 (2.7137) grad_norm 3.3679 (inf) loss_scale 256.0000 (416.9527) mem 11224MB [2024-09-01 12:25:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][580/625] eta 0:00:16 lr 0.000155 wd 0.0500 time 0.3751 (0.3773) data time 0.0011 (0.0020) model time 0.3740 (0.3758) loss 3.2792 (2.7138) grad_norm 4.3250 (inf) loss_scale 256.0000 (414.1824) mem 11224MB [2024-09-01 12:25:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][590/625] eta 0:00:13 lr 0.000155 wd 0.0500 time 0.3738 (0.3776) data time 0.0011 (0.0020) model time 0.3727 (0.3761) loss 2.9811 (2.7160) grad_norm 1.9132 (inf) loss_scale 256.0000 (411.5059) mem 11224MB [2024-09-01 12:26:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][600/625] eta 0:00:09 lr 0.000155 wd 0.0500 time 0.3731 (0.3775) data time 0.0012 (0.0020) model time 0.3719 (0.3760) loss 2.3361 (2.7182) grad_norm 2.2760 (inf) loss_scale 256.0000 (408.9185) mem 11224MB [2024-09-01 12:26:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][610/625] eta 0:00:05 lr 0.000155 wd 0.0500 time 0.3528 (0.3774) data time 0.0005 (0.0020) model time 0.3523 (0.3759) loss 2.1039 (2.7180) grad_norm 2.2662 (inf) loss_scale 256.0000 (406.4157) mem 11224MB [2024-09-01 12:26:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [252/300][620/625] eta 0:00:01 lr 0.000155 wd 0.0500 time 0.3567 (0.3770) data time 0.0005 (0.0020) model time 0.3562 (0.3755) loss 3.2342 (2.7162) grad_norm 4.4828 (inf) loss_scale 256.0000 (403.9936) mem 11224MB [2024-09-01 12:26:09 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 252 training takes 0:03:55 [2024-09-01 12:26:09 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:26:10 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:26:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.540 (0.540) Loss 0.5195 (0.5195) Acc@1 88.721 (88.721) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 12:26:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.146) Loss 0.8560 (0.6146) Acc@1 80.078 (86.776) Acc@5 95.312 (97.701) Mem 11224MB [2024-09-01 12:26:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 0.9380 (0.7314) Acc@1 77.832 (83.745) Acc@5 94.873 (96.515) Mem 11224MB [2024-09-01 12:26:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.435 Acc@5 96.533 [2024-09-01 12:26:13 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 12:26:13 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.44% [2024-09-01 12:26:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 12:26:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 12:26:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.483 (0.483) Loss 0.4890 (0.4890) Acc@1 89.404 (89.404) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 12:26:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.098 (0.141) Loss 0.8052 (0.5926) Acc@1 80.762 (87.100) Acc@5 95.801 (97.887) Mem 11224MB [2024-09-01 12:26:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 0.8765 (0.7026) Acc@1 79.443 (84.275) Acc@5 95.410 (96.745) Mem 11224MB [2024-09-01 12:26:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.981 Acc@5 96.767 [2024-09-01 12:26:17 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 12:26:17 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.98% [2024-09-01 12:26:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 12:26:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 12:26:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][0/625] eta 0:08:06 lr 0.000154 wd 0.0500 time 0.7788 (0.7788) data time 0.4323 (0.4323) model time 0.0000 (0.0000) loss 2.3487 (2.3487) grad_norm 2.8122 (2.8122) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][10/625] eta 0:04:11 lr 0.000154 wd 0.0500 time 0.3756 (0.4093) data time 0.0009 (0.0403) model time 0.0000 (0.0000) loss 2.9820 (2.8087) grad_norm 3.6093 (3.1238) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][20/625] eta 0:03:57 lr 0.000154 wd 0.0500 time 0.3738 (0.3931) data time 0.0009 (0.0217) model time 0.0000 (0.0000) loss 2.8288 (2.7864) grad_norm 2.6907 (2.9729) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][30/625] eta 0:03:50 lr 0.000154 wd 0.0500 time 0.3691 (0.3870) data time 0.0011 (0.0151) model time 0.0000 (0.0000) loss 2.7553 (2.7274) grad_norm 21.9517 (3.3965) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][40/625] eta 0:03:44 lr 0.000154 wd 0.0500 time 0.3746 (0.3834) data time 0.0011 (0.0117) model time 0.0000 (0.0000) loss 2.8551 (2.7255) grad_norm 2.9978 (3.4176) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][50/625] eta 0:03:41 lr 0.000154 wd 0.0500 time 0.3732 (0.3853) data time 0.0011 (0.0096) model time 0.0000 (0.0000) loss 1.7313 (2.6789) grad_norm 3.6390 (3.8996) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][60/625] eta 0:03:36 lr 0.000154 wd 0.0500 time 0.3789 (0.3832) data time 0.0008 (0.0082) model time 0.3780 (0.3711) loss 3.4108 (2.6875) grad_norm 2.3959 (3.7481) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][70/625] eta 0:03:31 lr 0.000154 wd 0.0500 time 0.3798 (0.3820) data time 0.0011 (0.0072) model time 0.3787 (0.3723) loss 2.9031 (2.6709) grad_norm 2.0587 (3.5750) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][80/625] eta 0:03:27 lr 0.000154 wd 0.0500 time 0.3650 (0.3812) data time 0.0008 (0.0065) model time 0.3641 (0.3730) loss 2.7739 (2.6743) grad_norm 2.7310 (3.4956) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][90/625] eta 0:03:23 lr 0.000154 wd 0.0500 time 0.3644 (0.3803) data time 0.0008 (0.0059) model time 0.3635 (0.3729) loss 2.6388 (2.6775) grad_norm 2.6633 (3.4068) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:26:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][100/625] eta 0:03:20 lr 0.000154 wd 0.0500 time 0.3723 (0.3823) data time 0.0009 (0.0054) model time 0.3714 (0.3781) loss 2.6512 (2.6945) grad_norm 1.6982 (3.3609) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][110/625] eta 0:03:16 lr 0.000154 wd 0.0500 time 0.3704 (0.3819) data time 0.0008 (0.0051) model time 0.3695 (0.3778) loss 1.9584 (2.6906) grad_norm 2.3690 (3.2938) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][120/625] eta 0:03:12 lr 0.000153 wd 0.0500 time 0.3757 (0.3813) data time 0.0008 (0.0047) model time 0.3748 (0.3772) loss 2.8990 (2.7049) grad_norm 1.9184 (3.1954) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:27:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][130/625] eta 0:03:08 lr 0.000153 wd 0.0500 time 0.3720 (0.3809) data time 0.0008 (0.0045) model time 0.3712 (0.3769) loss 2.1040 (2.6930) grad_norm 3.1626 (3.1057) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:27:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][140/625] eta 0:03:04 lr 0.000153 wd 0.0500 time 0.3757 (0.3803) data time 0.0013 (0.0042) model time 0.3744 (0.3763) loss 2.3197 (2.6849) grad_norm 2.7461 (3.0427) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 12:27:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][150/625] eta 0:03:00 lr 0.000153 wd 0.0500 time 0.3749 (0.3798) data time 0.0009 (0.0040) model time 0.3741 (0.3759) loss 2.7329 (2.6899) grad_norm 2.8821 (inf) loss_scale 128.0000 (250.9139) mem 11224MB [2024-09-01 12:27:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][160/625] eta 0:02:56 lr 0.000153 wd 0.0500 time 0.3785 (0.3795) data time 0.0010 (0.0038) model time 0.3775 (0.3756) loss 2.3404 (2.6841) grad_norm 2.1804 (inf) loss_scale 128.0000 (243.2795) mem 11224MB [2024-09-01 12:27:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][170/625] eta 0:02:52 lr 0.000153 wd 0.0500 time 0.3787 (0.3802) data time 0.0011 (0.0037) model time 0.3776 (0.3769) loss 3.0946 (2.6743) grad_norm 1.9452 (inf) loss_scale 128.0000 (236.5380) mem 11224MB [2024-09-01 12:27:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][180/625] eta 0:02:49 lr 0.000153 wd 0.0500 time 0.3781 (0.3799) data time 0.0014 (0.0035) model time 0.3767 (0.3767) loss 2.0918 (2.6697) grad_norm 2.9280 (inf) loss_scale 128.0000 (230.5414) mem 11224MB [2024-09-01 12:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][190/625] eta 0:02:45 lr 0.000153 wd 0.0500 time 0.3737 (0.3796) data time 0.0012 (0.0034) model time 0.3725 (0.3764) loss 2.2120 (2.6707) grad_norm 2.9428 (inf) loss_scale 128.0000 (225.1728) mem 11224MB [2024-09-01 12:27:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][200/625] eta 0:02:41 lr 0.000153 wd 0.0500 time 0.3678 (0.3792) data time 0.0011 (0.0033) model time 0.3667 (0.3759) loss 2.3595 (2.6797) grad_norm 2.1359 (inf) loss_scale 128.0000 (220.3383) mem 11224MB [2024-09-01 12:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][210/625] eta 0:02:37 lr 0.000153 wd 0.0500 time 0.3812 (0.3789) data time 0.0009 (0.0032) model time 0.3803 (0.3757) loss 2.9989 (2.6947) grad_norm 1.7757 (inf) loss_scale 128.0000 (215.9621) mem 11224MB [2024-09-01 12:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][220/625] eta 0:02:33 lr 0.000153 wd 0.0500 time 0.3764 (0.3785) data time 0.0011 (0.0031) model time 0.3753 (0.3753) loss 2.9758 (2.6954) grad_norm 2.2802 (inf) loss_scale 128.0000 (211.9819) mem 11224MB [2024-09-01 12:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][230/625] eta 0:02:29 lr 0.000152 wd 0.0500 time 0.3715 (0.3782) data time 0.0009 (0.0030) model time 0.3706 (0.3751) loss 2.4324 (2.6996) grad_norm 4.1626 (inf) loss_scale 128.0000 (208.3463) mem 11224MB [2024-09-01 12:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][240/625] eta 0:02:25 lr 0.000152 wd 0.0500 time 0.3794 (0.3781) data time 0.0013 (0.0029) model time 0.3782 (0.3750) loss 1.6261 (2.6880) grad_norm 2.4600 (inf) loss_scale 128.0000 (205.0124) mem 11224MB [2024-09-01 12:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][250/625] eta 0:02:21 lr 0.000152 wd 0.0500 time 0.3717 (0.3779) data time 0.0009 (0.0029) model time 0.3708 (0.3748) loss 2.7204 (2.6872) grad_norm 2.5568 (inf) loss_scale 128.0000 (201.9442) mem 11224MB [2024-09-01 12:27:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][260/625] eta 0:02:18 lr 0.000152 wd 0.0500 time 0.3772 (0.3786) data time 0.0013 (0.0028) model time 0.3759 (0.3758) loss 2.9958 (2.6851) grad_norm 2.5737 (inf) loss_scale 128.0000 (199.1111) mem 11224MB [2024-09-01 12:28:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][270/625] eta 0:02:14 lr 0.000152 wd 0.0500 time 0.3833 (0.3784) data time 0.0008 (0.0027) model time 0.3825 (0.3757) loss 2.8301 (2.6856) grad_norm 1.8600 (inf) loss_scale 128.0000 (196.4871) mem 11224MB [2024-09-01 12:28:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][280/625] eta 0:02:10 lr 0.000152 wd 0.0500 time 0.3698 (0.3784) data time 0.0011 (0.0027) model time 0.3687 (0.3758) loss 2.5768 (2.6978) grad_norm 13.4933 (inf) loss_scale 128.0000 (194.0498) mem 11224MB [2024-09-01 12:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][290/625] eta 0:02:06 lr 0.000152 wd 0.0500 time 0.3652 (0.3788) data time 0.0011 (0.0026) model time 0.3641 (0.3762) loss 2.2265 (2.6984) grad_norm 2.1988 (inf) loss_scale 128.0000 (191.7801) mem 11224MB [2024-09-01 12:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][300/625] eta 0:02:03 lr 0.000152 wd 0.0500 time 0.3700 (0.3786) data time 0.0010 (0.0026) model time 0.3690 (0.3760) loss 3.0717 (2.7123) grad_norm 2.4813 (inf) loss_scale 128.0000 (189.6611) mem 11224MB [2024-09-01 12:28:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][310/625] eta 0:01:59 lr 0.000152 wd 0.0500 time 0.3727 (0.3790) data time 0.0008 (0.0025) model time 0.3719 (0.3766) loss 2.9544 (2.7111) grad_norm 2.2363 (inf) loss_scale 128.0000 (187.6785) mem 11224MB [2024-09-01 12:28:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][320/625] eta 0:01:55 lr 0.000152 wd 0.0500 time 0.3776 (0.3788) data time 0.0011 (0.0025) model time 0.3765 (0.3764) loss 2.7853 (2.7146) grad_norm 2.4515 (inf) loss_scale 128.0000 (185.8193) mem 11224MB [2024-09-01 12:28:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][330/625] eta 0:01:51 lr 0.000152 wd 0.0500 time 0.3616 (0.3785) data time 0.0011 (0.0025) model time 0.3605 (0.3761) loss 2.7268 (2.7231) grad_norm 1.8475 (inf) loss_scale 128.0000 (184.0725) mem 11224MB [2024-09-01 12:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][340/625] eta 0:01:47 lr 0.000151 wd 0.0500 time 0.3587 (0.3783) data time 0.0009 (0.0024) model time 0.3578 (0.3759) loss 2.5714 (2.7317) grad_norm 1.6242 (inf) loss_scale 128.0000 (182.4282) mem 11224MB [2024-09-01 12:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][350/625] eta 0:01:43 lr 0.000151 wd 0.0500 time 0.3745 (0.3781) data time 0.0008 (0.0024) model time 0.3738 (0.3758) loss 2.1193 (2.7300) grad_norm 1.9221 (inf) loss_scale 128.0000 (180.8775) mem 11224MB [2024-09-01 12:28:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][360/625] eta 0:01:40 lr 0.000151 wd 0.0500 time 0.3685 (0.3780) data time 0.0009 (0.0024) model time 0.3676 (0.3756) loss 1.8930 (2.7301) grad_norm 1.7108 (inf) loss_scale 128.0000 (179.4127) mem 11224MB [2024-09-01 12:28:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][370/625] eta 0:01:36 lr 0.000151 wd 0.0500 time 0.3759 (0.3778) data time 0.0011 (0.0023) model time 0.3748 (0.3755) loss 2.7970 (2.7325) grad_norm 2.3620 (inf) loss_scale 128.0000 (178.0270) mem 11224MB [2024-09-01 12:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][380/625] eta 0:01:32 lr 0.000151 wd 0.0500 time 0.3700 (0.3777) data time 0.0009 (0.0023) model time 0.3691 (0.3754) loss 1.8776 (2.7300) grad_norm 3.2555 (inf) loss_scale 128.0000 (176.7139) mem 11224MB [2024-09-01 12:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][390/625] eta 0:01:28 lr 0.000151 wd 0.0500 time 0.3706 (0.3776) data time 0.0008 (0.0023) model time 0.3697 (0.3753) loss 2.9306 (2.7253) grad_norm 2.3823 (inf) loss_scale 128.0000 (175.4680) mem 11224MB [2024-09-01 12:28:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][400/625] eta 0:01:24 lr 0.000151 wd 0.0500 time 0.3701 (0.3776) data time 0.0013 (0.0023) model time 0.3688 (0.3753) loss 2.9670 (2.7236) grad_norm 2.1143 (inf) loss_scale 128.0000 (174.2843) mem 11224MB [2024-09-01 12:28:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][410/625] eta 0:01:21 lr 0.000151 wd 0.0500 time 0.3695 (0.3775) data time 0.0013 (0.0022) model time 0.3682 (0.3752) loss 2.2117 (2.7268) grad_norm 2.0944 (inf) loss_scale 128.0000 (173.1582) mem 11224MB [2024-09-01 12:28:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][420/625] eta 0:01:17 lr 0.000151 wd 0.0500 time 0.3771 (0.3774) data time 0.0009 (0.0022) model time 0.3763 (0.3751) loss 2.9609 (2.7243) grad_norm 2.3258 (inf) loss_scale 128.0000 (172.0855) mem 11224MB [2024-09-01 12:29:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][430/625] eta 0:01:13 lr 0.000151 wd 0.0500 time 0.3782 (0.3773) data time 0.0008 (0.0022) model time 0.3774 (0.3751) loss 3.1513 (2.7296) grad_norm 2.1752 (inf) loss_scale 128.0000 (171.0626) mem 11224MB [2024-09-01 12:29:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][440/625] eta 0:01:09 lr 0.000151 wd 0.0500 time 0.3966 (0.3778) data time 0.0013 (0.0022) model time 0.3952 (0.3757) loss 2.5861 (2.7312) grad_norm 3.4087 (inf) loss_scale 128.0000 (170.0862) mem 11224MB [2024-09-01 12:29:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][450/625] eta 0:01:06 lr 0.000150 wd 0.0500 time 0.3688 (0.3777) data time 0.0012 (0.0021) model time 0.3676 (0.3756) loss 3.6280 (2.7361) grad_norm 4.0004 (inf) loss_scale 128.0000 (169.1530) mem 11224MB [2024-09-01 12:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][460/625] eta 0:01:02 lr 0.000150 wd 0.0500 time 0.3682 (0.3776) data time 0.0010 (0.0021) model time 0.3672 (0.3754) loss 2.5278 (2.7350) grad_norm 2.5413 (inf) loss_scale 128.0000 (168.2603) mem 11224MB [2024-09-01 12:29:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][470/625] eta 0:00:58 lr 0.000150 wd 0.0500 time 0.3633 (0.3775) data time 0.0009 (0.0021) model time 0.3624 (0.3753) loss 2.9338 (2.7353) grad_norm 2.7279 (inf) loss_scale 128.0000 (167.4055) mem 11224MB [2024-09-01 12:29:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][480/625] eta 0:00:54 lr 0.000150 wd 0.0500 time 0.3750 (0.3774) data time 0.0011 (0.0021) model time 0.3739 (0.3753) loss 1.9592 (2.7303) grad_norm 6.3361 (inf) loss_scale 128.0000 (166.5863) mem 11224MB [2024-09-01 12:29:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][490/625] eta 0:00:51 lr 0.000150 wd 0.0500 time 0.3754 (0.3782) data time 0.0008 (0.0021) model time 0.3746 (0.3762) loss 2.9326 (2.7286) grad_norm 2.5008 (inf) loss_scale 128.0000 (165.8004) mem 11224MB [2024-09-01 12:29:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][500/625] eta 0:00:47 lr 0.000150 wd 0.0500 time 0.3679 (0.3784) data time 0.0008 (0.0020) model time 0.3671 (0.3764) loss 2.6943 (2.7250) grad_norm 2.1486 (inf) loss_scale 128.0000 (165.0459) mem 11224MB [2024-09-01 12:29:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][510/625] eta 0:00:43 lr 0.000150 wd 0.0500 time 0.3794 (0.3783) data time 0.0014 (0.0020) model time 0.3780 (0.3764) loss 2.3954 (2.7218) grad_norm 3.2969 (inf) loss_scale 128.0000 (164.3209) mem 11224MB [2024-09-01 12:29:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][520/625] eta 0:00:39 lr 0.000150 wd 0.0500 time 0.3770 (0.3782) data time 0.0009 (0.0020) model time 0.3761 (0.3763) loss 2.2270 (2.7216) grad_norm 2.5475 (inf) loss_scale 128.0000 (163.6238) mem 11224MB [2024-09-01 12:29:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][530/625] eta 0:00:35 lr 0.000150 wd 0.0500 time 0.3659 (0.3781) data time 0.0009 (0.0020) model time 0.3650 (0.3761) loss 1.7217 (2.7201) grad_norm 1.5628 (inf) loss_scale 128.0000 (162.9529) mem 11224MB [2024-09-01 12:29:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][540/625] eta 0:00:32 lr 0.000150 wd 0.0500 time 0.3672 (0.3780) data time 0.0009 (0.0020) model time 0.3664 (0.3760) loss 2.7363 (2.7242) grad_norm 2.1801 (inf) loss_scale 128.0000 (162.3068) mem 11224MB [2024-09-01 12:29:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][550/625] eta 0:00:28 lr 0.000150 wd 0.0500 time 0.3716 (0.3779) data time 0.0008 (0.0020) model time 0.3708 (0.3759) loss 2.4754 (2.7238) grad_norm 2.1327 (inf) loss_scale 128.0000 (161.6842) mem 11224MB [2024-09-01 12:29:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][560/625] eta 0:00:24 lr 0.000150 wd 0.0500 time 0.3634 (0.3781) data time 0.0009 (0.0020) model time 0.3625 (0.3762) loss 3.0956 (2.7232) grad_norm 2.0100 (inf) loss_scale 128.0000 (161.0838) mem 11224MB [2024-09-01 12:29:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][570/625] eta 0:00:20 lr 0.000149 wd 0.0500 time 0.3845 (0.3781) data time 0.0009 (0.0019) model time 0.3836 (0.3762) loss 3.2556 (2.7229) grad_norm 2.4519 (inf) loss_scale 128.0000 (160.5044) mem 11224MB [2024-09-01 12:29:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][580/625] eta 0:00:17 lr 0.000149 wd 0.0500 time 0.3847 (0.3780) data time 0.0011 (0.0019) model time 0.3836 (0.3761) loss 3.0433 (2.7238) grad_norm 3.8475 (inf) loss_scale 128.0000 (159.9449) mem 11224MB [2024-09-01 12:30:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][590/625] eta 0:00:13 lr 0.000149 wd 0.0500 time 0.3634 (0.3780) data time 0.0009 (0.0019) model time 0.3625 (0.3761) loss 2.0033 (2.7234) grad_norm 2.6042 (inf) loss_scale 128.0000 (159.4044) mem 11224MB [2024-09-01 12:30:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][600/625] eta 0:00:09 lr 0.000149 wd 0.0500 time 0.3614 (0.3778) data time 0.0008 (0.0019) model time 0.3606 (0.3760) loss 2.0794 (2.7237) grad_norm 2.4492 (inf) loss_scale 128.0000 (158.8819) mem 11224MB [2024-09-01 12:30:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][610/625] eta 0:00:05 lr 0.000149 wd 0.0500 time 0.3570 (0.3784) data time 0.0008 (0.0019) model time 0.3562 (0.3765) loss 2.3962 (2.7232) grad_norm 2.4896 (inf) loss_scale 128.0000 (158.3764) mem 11224MB [2024-09-01 12:30:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [253/300][620/625] eta 0:00:01 lr 0.000149 wd 0.0500 time 0.3558 (0.3780) data time 0.0006 (0.0019) model time 0.3553 (0.3762) loss 2.1283 (2.7242) grad_norm 4.0664 (inf) loss_scale 128.0000 (157.8873) mem 11224MB [2024-09-01 12:30:14 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 253 training takes 0:03:56 [2024-09-01 12:30:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:30:15 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.5186 (0.5186) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 12:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 0.8296 (0.6077) Acc@1 80.859 (86.683) Acc@5 95.410 (97.749) Mem 11224MB [2024-09-01 12:30:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.8926 (0.7235) Acc@1 78.320 (83.738) Acc@5 95.166 (96.526) Mem 11224MB [2024-09-01 12:30:18 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.385 Acc@5 96.525 [2024-09-01 12:30:18 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 12:30:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.820 (0.820) Loss 0.4897 (0.4897) Acc@1 89.404 (89.404) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 12:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.175) Loss 0.8071 (0.5927) Acc@1 80.713 (87.123) Acc@5 95.801 (97.878) Mem 11224MB [2024-09-01 12:30:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8774 (0.7029) Acc@1 79.395 (84.310) Acc@5 95.410 (96.749) Mem 11224MB [2024-09-01 12:30:21 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.017 Acc@5 96.769 [2024-09-01 12:30:21 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 12:30:21 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 84.02% [2024-09-01 12:30:21 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 12:30:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 12:30:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][0/625] eta 0:07:53 lr 0.000149 wd 0.0500 time 0.7579 (0.7579) data time 0.4096 (0.4096) model time 0.0000 (0.0000) loss 3.1022 (3.1022) grad_norm 4.4199 (4.4199) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][10/625] eta 0:04:20 lr 0.000149 wd 0.0500 time 0.3705 (0.4240) data time 0.0011 (0.0383) model time 0.0000 (0.0000) loss 2.6274 (2.6755) grad_norm 3.2874 (2.9826) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][20/625] eta 0:04:01 lr 0.000149 wd 0.0500 time 0.3716 (0.3999) data time 0.0010 (0.0207) model time 0.0000 (0.0000) loss 3.0783 (2.6350) grad_norm 2.2510 (3.4495) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][30/625] eta 0:03:52 lr 0.000149 wd 0.0500 time 0.3720 (0.3905) data time 0.0008 (0.0143) model time 0.0000 (0.0000) loss 3.0476 (2.7226) grad_norm 3.0107 (3.1546) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][40/625] eta 0:03:45 lr 0.000149 wd 0.0500 time 0.3753 (0.3863) data time 0.0012 (0.0111) model time 0.0000 (0.0000) loss 2.6601 (2.6607) grad_norm 2.3081 (2.9411) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][50/625] eta 0:03:40 lr 0.000149 wd 0.0500 time 0.3681 (0.3832) data time 0.0011 (0.0092) model time 0.0000 (0.0000) loss 3.1759 (2.6883) grad_norm 2.2867 (2.7932) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][60/625] eta 0:03:35 lr 0.000148 wd 0.0500 time 0.3753 (0.3814) data time 0.0009 (0.0079) model time 0.3743 (0.3710) loss 1.6004 (2.6655) grad_norm 2.4864 (2.7275) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][70/625] eta 0:03:31 lr 0.000148 wd 0.0500 time 0.3774 (0.3805) data time 0.0008 (0.0069) model time 0.3765 (0.3726) loss 2.5421 (2.6705) grad_norm 2.4286 (2.6265) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][80/625] eta 0:03:28 lr 0.000148 wd 0.0500 time 0.5683 (0.3817) data time 0.0011 (0.0062) model time 0.5673 (0.3780) loss 2.0366 (2.6542) grad_norm 2.7938 (2.6058) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][90/625] eta 0:03:23 lr 0.000148 wd 0.0500 time 0.3712 (0.3808) data time 0.0008 (0.0056) model time 0.3704 (0.3766) loss 2.6043 (2.6511) grad_norm 1.8925 (2.5933) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][100/625] eta 0:03:19 lr 0.000148 wd 0.0500 time 0.3856 (0.3803) data time 0.0009 (0.0052) model time 0.3847 (0.3762) loss 3.0241 (2.6429) grad_norm 2.6866 (2.5548) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][110/625] eta 0:03:15 lr 0.000148 wd 0.0500 time 0.3665 (0.3799) data time 0.0009 (0.0048) model time 0.3656 (0.3759) loss 2.9551 (2.6651) grad_norm 1.7004 (2.5894) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][120/625] eta 0:03:11 lr 0.000148 wd 0.0500 time 0.3667 (0.3792) data time 0.0013 (0.0045) model time 0.3654 (0.3751) loss 2.7468 (2.6674) grad_norm 1.7012 (2.7382) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][130/625] eta 0:03:08 lr 0.000148 wd 0.0500 time 0.3732 (0.3802) data time 0.0013 (0.0043) model time 0.3719 (0.3772) loss 3.2967 (2.6938) grad_norm 2.6331 (2.7143) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][140/625] eta 0:03:04 lr 0.000148 wd 0.0500 time 0.3741 (0.3796) data time 0.0012 (0.0041) model time 0.3728 (0.3764) loss 2.8659 (2.6925) grad_norm 4.5922 (2.6886) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][150/625] eta 0:03:00 lr 0.000148 wd 0.0500 time 0.3742 (0.3791) data time 0.0009 (0.0039) model time 0.3733 (0.3758) loss 2.8958 (2.6764) grad_norm 3.1007 (2.6968) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][160/625] eta 0:02:56 lr 0.000148 wd 0.0500 time 0.3743 (0.3786) data time 0.0011 (0.0037) model time 0.3732 (0.3753) loss 2.5768 (2.6652) grad_norm 2.5143 (2.7047) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][170/625] eta 0:02:53 lr 0.000147 wd 0.0500 time 0.3696 (0.3805) data time 0.0008 (0.0036) model time 0.3688 (0.3782) loss 2.1788 (2.6642) grad_norm 2.4227 (2.7958) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][180/625] eta 0:02:49 lr 0.000147 wd 0.0500 time 0.3700 (0.3800) data time 0.0011 (0.0034) model time 0.3689 (0.3776) loss 3.0491 (2.6625) grad_norm 1.9814 (2.7659) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][190/625] eta 0:02:45 lr 0.000147 wd 0.0500 time 0.3682 (0.3796) data time 0.0008 (0.0033) model time 0.3674 (0.3772) loss 2.9167 (2.6627) grad_norm 3.0864 (2.7629) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][200/625] eta 0:02:41 lr 0.000147 wd 0.0500 time 0.3690 (0.3793) data time 0.0011 (0.0032) model time 0.3679 (0.3768) loss 2.5501 (2.6724) grad_norm 30.8326 (2.9564) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][210/625] eta 0:02:37 lr 0.000147 wd 0.0500 time 0.3698 (0.3791) data time 0.0010 (0.0031) model time 0.3688 (0.3766) loss 2.8292 (2.6687) grad_norm 2.5941 (2.9575) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][220/625] eta 0:02:33 lr 0.000147 wd 0.0500 time 0.3742 (0.3787) data time 0.0008 (0.0030) model time 0.3734 (0.3761) loss 2.0798 (2.6666) grad_norm 1.9103 (2.9485) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][230/625] eta 0:02:29 lr 0.000147 wd 0.0500 time 0.3762 (0.3785) data time 0.0011 (0.0029) model time 0.3751 (0.3760) loss 3.0448 (2.6736) grad_norm 2.9615 (3.0502) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][240/625] eta 0:02:25 lr 0.000147 wd 0.0500 time 0.3690 (0.3782) data time 0.0013 (0.0029) model time 0.3677 (0.3757) loss 2.9995 (2.6720) grad_norm 2.0816 (3.0191) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][250/625] eta 0:02:21 lr 0.000147 wd 0.0500 time 0.3709 (0.3779) data time 0.0009 (0.0028) model time 0.3700 (0.3754) loss 1.9669 (2.6744) grad_norm 1.9162 (3.0021) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][260/625] eta 0:02:17 lr 0.000147 wd 0.0500 time 0.3664 (0.3777) data time 0.0008 (0.0027) model time 0.3656 (0.3752) loss 1.8775 (2.6658) grad_norm 2.4925 (3.0033) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][270/625] eta 0:02:13 lr 0.000147 wd 0.0500 time 0.3757 (0.3775) data time 0.0008 (0.0027) model time 0.3749 (0.3750) loss 2.7226 (2.6648) grad_norm 2.1750 (2.9811) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][280/625] eta 0:02:10 lr 0.000147 wd 0.0500 time 0.3731 (0.3773) data time 0.0009 (0.0026) model time 0.3722 (0.3748) loss 2.6811 (2.6633) grad_norm 2.7368 (3.0010) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][290/625] eta 0:02:07 lr 0.000146 wd 0.0500 time 0.5882 (0.3793) data time 0.0009 (0.0026) model time 0.5872 (0.3773) loss 2.0784 (2.6617) grad_norm 4.5206 (3.0067) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][300/625] eta 0:02:03 lr 0.000146 wd 0.0500 time 0.3751 (0.3790) data time 0.0011 (0.0025) model time 0.3740 (0.3770) loss 3.0061 (2.6646) grad_norm 1.8571 (2.9910) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][310/625] eta 0:01:59 lr 0.000146 wd 0.0500 time 0.3728 (0.3789) data time 0.0010 (0.0025) model time 0.3718 (0.3768) loss 1.9513 (2.6610) grad_norm 2.3065 (2.9843) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][320/625] eta 0:01:55 lr 0.000146 wd 0.0500 time 0.3710 (0.3787) data time 0.0008 (0.0025) model time 0.3702 (0.3766) loss 3.2938 (2.6596) grad_norm 2.2628 (2.9621) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][330/625] eta 0:01:51 lr 0.000146 wd 0.0500 time 0.3723 (0.3785) data time 0.0013 (0.0024) model time 0.3710 (0.3764) loss 1.9994 (2.6607) grad_norm 2.5810 (2.9585) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][340/625] eta 0:01:47 lr 0.000146 wd 0.0500 time 0.3731 (0.3789) data time 0.0011 (0.0024) model time 0.3719 (0.3769) loss 2.6902 (2.6653) grad_norm 3.1620 (2.9641) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][350/625] eta 0:01:44 lr 0.000146 wd 0.0500 time 0.3796 (0.3787) data time 0.0012 (0.0024) model time 0.3784 (0.3768) loss 1.8790 (2.6618) grad_norm 3.9178 (2.9597) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][360/625] eta 0:01:40 lr 0.000146 wd 0.0500 time 0.3727 (0.3785) data time 0.0008 (0.0023) model time 0.3719 (0.3766) loss 2.1360 (2.6565) grad_norm 2.6476 (2.9457) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][370/625] eta 0:01:36 lr 0.000146 wd 0.0500 time 0.3693 (0.3783) data time 0.0011 (0.0023) model time 0.3683 (0.3763) loss 2.9263 (2.6580) grad_norm 2.4639 (2.9363) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][380/625] eta 0:01:32 lr 0.000146 wd 0.0500 time 0.3793 (0.3782) data time 0.0009 (0.0023) model time 0.3785 (0.3762) loss 2.0518 (2.6587) grad_norm 1.7431 (2.9130) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][390/625] eta 0:01:28 lr 0.000146 wd 0.0500 time 0.3663 (0.3781) data time 0.0009 (0.0022) model time 0.3655 (0.3762) loss 3.2184 (2.6630) grad_norm 1.6287 (2.8924) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][400/625] eta 0:01:25 lr 0.000145 wd 0.0500 time 0.3781 (0.3781) data time 0.0009 (0.0022) model time 0.3772 (0.3762) loss 2.4188 (2.6619) grad_norm 1.9466 (2.8851) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:32:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][410/625] eta 0:01:21 lr 0.000145 wd 0.0500 time 0.3678 (0.3780) data time 0.0010 (0.0022) model time 0.3668 (0.3760) loss 2.4447 (2.6559) grad_norm 4.1350 (2.8854) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][420/625] eta 0:01:17 lr 0.000145 wd 0.0500 time 0.3861 (0.3780) data time 0.0013 (0.0022) model time 0.3848 (0.3761) loss 3.3362 (2.6559) grad_norm 1.7736 (2.8722) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][430/625] eta 0:01:13 lr 0.000145 wd 0.0500 time 0.3741 (0.3780) data time 0.0013 (0.0021) model time 0.3727 (0.3761) loss 1.6820 (2.6539) grad_norm 2.6028 (2.8781) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][440/625] eta 0:01:09 lr 0.000145 wd 0.0500 time 0.4060 (0.3779) data time 0.0009 (0.0021) model time 0.4051 (0.3760) loss 2.7330 (2.6576) grad_norm 2.6422 (2.8615) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][450/625] eta 0:01:06 lr 0.000145 wd 0.0500 time 0.3707 (0.3778) data time 0.0010 (0.0021) model time 0.3697 (0.3759) loss 2.9644 (2.6547) grad_norm 2.8115 (2.8541) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][460/625] eta 0:01:02 lr 0.000145 wd 0.0500 time 0.3695 (0.3776) data time 0.0009 (0.0021) model time 0.3686 (0.3757) loss 1.8652 (2.6537) grad_norm 3.2690 (2.8577) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][470/625] eta 0:00:58 lr 0.000145 wd 0.0500 time 0.3664 (0.3775) data time 0.0009 (0.0021) model time 0.3656 (0.3756) loss 2.5724 (2.6590) grad_norm 2.5039 (2.8744) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][480/625] eta 0:00:54 lr 0.000145 wd 0.0500 time 0.3971 (0.3781) data time 0.0013 (0.0021) model time 0.3958 (0.3762) loss 2.1120 (2.6559) grad_norm 2.1780 (2.9142) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][490/625] eta 0:00:51 lr 0.000145 wd 0.0500 time 0.3868 (0.3780) data time 0.0011 (0.0020) model time 0.3857 (0.3762) loss 2.8053 (2.6558) grad_norm 1.9757 (2.9117) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][500/625] eta 0:00:47 lr 0.000145 wd 0.0500 time 0.3620 (0.3790) data time 0.0010 (0.0020) model time 0.3611 (0.3773) loss 1.8541 (2.6598) grad_norm 10.3066 (2.9211) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][510/625] eta 0:00:43 lr 0.000145 wd 0.0500 time 0.3846 (0.3790) data time 0.0010 (0.0020) model time 0.3836 (0.3773) loss 2.8121 (2.6630) grad_norm 2.0291 (2.9049) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][520/625] eta 0:00:39 lr 0.000144 wd 0.0500 time 0.3804 (0.3793) data time 0.0011 (0.0020) model time 0.3793 (0.3777) loss 2.7241 (2.6612) grad_norm 2.5964 (2.9004) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][530/625] eta 0:00:36 lr 0.000144 wd 0.0500 time 0.3710 (0.3792) data time 0.0012 (0.0020) model time 0.3698 (0.3776) loss 2.8386 (2.6622) grad_norm 2.9035 (2.8959) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][540/625] eta 0:00:32 lr 0.000144 wd 0.0500 time 0.3677 (0.3791) data time 0.0011 (0.0020) model time 0.3666 (0.3775) loss 2.0769 (2.6646) grad_norm 3.0230 (2.8991) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][550/625] eta 0:00:28 lr 0.000144 wd 0.0500 time 0.3672 (0.3790) data time 0.0011 (0.0019) model time 0.3661 (0.3774) loss 2.8417 (2.6643) grad_norm 3.6237 (2.8979) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][560/625] eta 0:00:24 lr 0.000144 wd 0.0500 time 0.3632 (0.3789) data time 0.0009 (0.0019) model time 0.3623 (0.3772) loss 3.2551 (2.6718) grad_norm 2.4428 (2.9175) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:33:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][570/625] eta 0:00:20 lr 0.000144 wd 0.0500 time 0.3672 (0.3787) data time 0.0009 (0.0019) model time 0.3662 (0.3771) loss 2.7369 (2.6764) grad_norm 4.3123 (2.9227) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][580/625] eta 0:00:17 lr 0.000144 wd 0.0500 time 0.3717 (0.3786) data time 0.0012 (0.0019) model time 0.3705 (0.3770) loss 2.7887 (2.6776) grad_norm 2.2940 (2.9266) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][590/625] eta 0:00:13 lr 0.000144 wd 0.0500 time 0.3722 (0.3785) data time 0.0012 (0.0019) model time 0.3711 (0.3769) loss 1.8783 (2.6744) grad_norm 1.6711 (2.9219) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][600/625] eta 0:00:09 lr 0.000144 wd 0.0500 time 0.3754 (0.3789) data time 0.0008 (0.0019) model time 0.3746 (0.3773) loss 3.4900 (2.6758) grad_norm 2.5209 (2.9149) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][610/625] eta 0:00:05 lr 0.000144 wd 0.0500 time 0.3642 (0.3788) data time 0.0006 (0.0019) model time 0.3637 (0.3771) loss 3.2292 (2.6769) grad_norm 6.6825 (2.9114) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [254/300][620/625] eta 0:00:01 lr 0.000144 wd 0.0500 time 0.3588 (0.3785) data time 0.0006 (0.0019) model time 0.3582 (0.3769) loss 3.2983 (2.6754) grad_norm 2.0036 (2.9009) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:19 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 254 training takes 0:03:56 [2024-09-01 12:34:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:34:21 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:34:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.507 (0.507) Loss 0.5093 (0.5093) Acc@1 89.453 (89.453) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 12:34:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.8428 (0.6213) Acc@1 80.176 (86.657) Acc@5 95.801 (97.705) Mem 11224MB [2024-09-01 12:34:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 0.9092 (0.7394) Acc@1 78.760 (83.750) Acc@5 95.215 (96.473) Mem 11224MB [2024-09-01 12:34:24 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.429 Acc@5 96.489 [2024-09-01 12:34:24 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 12:34:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.828 (0.828) Loss 0.4905 (0.4905) Acc@1 89.404 (89.404) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 12:34:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.177) Loss 0.8062 (0.5926) Acc@1 80.713 (87.132) Acc@5 95.801 (97.860) Mem 11224MB [2024-09-01 12:34:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8774 (0.7030) Acc@1 79.297 (84.315) Acc@5 95.312 (96.740) Mem 11224MB [2024-09-01 12:34:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.017 Acc@5 96.761 [2024-09-01 12:34:27 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 12:34:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][0/625] eta 0:12:25 lr 0.000144 wd 0.0500 time 1.1924 (1.1924) data time 0.6570 (0.6570) model time 0.0000 (0.0000) loss 2.5715 (2.5715) grad_norm 1.6208 (1.6208) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][10/625] eta 0:04:35 lr 0.000143 wd 0.0500 time 0.3655 (0.4483) data time 0.0011 (0.0608) model time 0.0000 (0.0000) loss 2.4079 (2.5463) grad_norm 1.8897 (2.6676) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][20/625] eta 0:04:09 lr 0.000143 wd 0.0500 time 0.3785 (0.4116) data time 0.0008 (0.0324) model time 0.0000 (0.0000) loss 2.4341 (2.6339) grad_norm 3.4921 (2.6876) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][30/625] eta 0:03:56 lr 0.000143 wd 0.0500 time 0.3673 (0.3981) data time 0.0009 (0.0223) model time 0.0000 (0.0000) loss 3.1962 (2.6605) grad_norm 2.6066 (2.5390) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][40/625] eta 0:03:49 lr 0.000143 wd 0.0500 time 0.3684 (0.3916) data time 0.0009 (0.0171) model time 0.0000 (0.0000) loss 3.0680 (2.6943) grad_norm 2.0785 (2.4764) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][50/625] eta 0:03:43 lr 0.000143 wd 0.0500 time 0.3780 (0.3879) data time 0.0007 (0.0140) model time 0.0000 (0.0000) loss 2.0044 (2.7127) grad_norm 1.8669 (2.4307) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][60/625] eta 0:03:37 lr 0.000143 wd 0.0500 time 0.3751 (0.3853) data time 0.0013 (0.0119) model time 0.3739 (0.3708) loss 3.1523 (2.7076) grad_norm 1.5594 (2.7007) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][70/625] eta 0:03:36 lr 0.000143 wd 0.0500 time 0.3710 (0.3896) data time 0.0011 (0.0104) model time 0.3699 (0.3928) loss 2.8985 (2.6950) grad_norm 2.1796 (2.8050) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][80/625] eta 0:03:31 lr 0.000143 wd 0.0500 time 0.3769 (0.3877) data time 0.0011 (0.0092) model time 0.3758 (0.3862) loss 2.6058 (2.7001) grad_norm 2.0633 (2.7525) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 12:34:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:35:00 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:48:28 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 12:48:29 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 12:48:36 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 12:48:52 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 12:48:52 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 12:48:54 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 12:48:56 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 12:48:57 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 255) [2024-09-01 12:48:57 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 12:49:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][90/625] eta 0:16:59 lr 0.000143 wd 0.0500 time 0.3424 (1.9059) data time 0.0007 (0.0823) model time 0.3417 (1.8236) loss 2.6992 (3.0768) grad_norm 1.8703 (2.3837) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][100/625] eta 0:09:28 lr 0.000143 wd 0.0500 time 0.3449 (1.0831) data time 0.0007 (0.0394) model time 0.3442 (1.0436) loss 2.8719 (2.9043) grad_norm 5.4742 (2.5744) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][110/625] eta 0:07:09 lr 0.000143 wd 0.0500 time 0.3426 (0.8343) data time 0.0006 (0.0261) model time 0.3421 (0.8082) loss 3.1445 (2.9322) grad_norm 2.3048 (2.8376) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][120/625] eta 0:05:57 lr 0.000142 wd 0.0500 time 0.3383 (0.7082) data time 0.0008 (0.0196) model time 0.3375 (0.6886) loss 2.7734 (2.8646) grad_norm 2.2335 (2.7384) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][130/625] eta 0:05:16 lr 0.000142 wd 0.0500 time 0.3400 (0.6392) data time 0.0008 (0.0158) model time 0.3391 (0.6234) loss 2.4859 (2.8414) grad_norm 2.6494 (2.6259) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][140/625] eta 0:04:45 lr 0.000142 wd 0.0500 time 0.3436 (0.5891) data time 0.0006 (0.0133) model time 0.3430 (0.5758) loss 2.2195 (2.8010) grad_norm 2.2641 (2.5510) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][150/625] eta 0:04:22 lr 0.000142 wd 0.0500 time 0.3488 (0.5537) data time 0.0008 (0.0115) model time 0.3480 (0.5422) loss 2.6149 (2.7802) grad_norm 2.1478 (2.5149) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][160/625] eta 0:04:05 lr 0.000142 wd 0.0500 time 0.3423 (0.5270) data time 0.0008 (0.0101) model time 0.3415 (0.5169) loss 2.6632 (2.7611) grad_norm 1.7851 (2.5530) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][170/625] eta 0:03:50 lr 0.000142 wd 0.0500 time 0.3438 (0.5065) data time 0.0006 (0.0091) model time 0.3431 (0.4974) loss 2.7310 (2.7459) grad_norm 2.3342 (2.5252) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][180/625] eta 0:03:38 lr 0.000142 wd 0.0500 time 0.3430 (0.4899) data time 0.0008 (0.0082) model time 0.3422 (0.4817) loss 3.0436 (2.7556) grad_norm 2.8173 (2.5052) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][190/625] eta 0:03:27 lr 0.000142 wd 0.0500 time 0.3433 (0.4763) data time 0.0006 (0.0076) model time 0.3427 (0.4687) loss 3.0365 (2.7644) grad_norm 2.5387 (2.5150) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:49:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][200/625] eta 0:03:17 lr 0.000142 wd 0.0500 time 0.3535 (0.4650) data time 0.0007 (0.0070) model time 0.3528 (0.4580) loss 3.0106 (2.7611) grad_norm 2.6770 (2.5056) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][210/625] eta 0:03:09 lr 0.000142 wd 0.0500 time 0.3365 (0.4554) data time 0.0006 (0.0065) model time 0.3360 (0.4489) loss 2.6669 (2.7528) grad_norm 2.2307 (2.7022) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][220/625] eta 0:03:01 lr 0.000142 wd 0.0500 time 0.3437 (0.4475) data time 0.0006 (0.0061) model time 0.3431 (0.4414) loss 2.9913 (2.7527) grad_norm 2.0906 (2.7410) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][230/625] eta 0:02:53 lr 0.000142 wd 0.0500 time 0.3381 (0.4404) data time 0.0006 (0.0058) model time 0.3375 (0.4346) loss 2.1980 (2.7397) grad_norm 2.8233 (2.7467) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][240/625] eta 0:02:47 lr 0.000141 wd 0.0500 time 0.3457 (0.4343) data time 0.0006 (0.0054) model time 0.3452 (0.4288) loss 3.5268 (2.7433) grad_norm 2.0743 (2.7475) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][250/625] eta 0:02:40 lr 0.000141 wd 0.0500 time 0.3427 (0.4289) data time 0.0008 (0.0052) model time 0.3419 (0.4237) loss 2.9890 (2.7495) grad_norm 3.5765 (2.8380) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][260/625] eta 0:02:34 lr 0.000141 wd 0.0500 time 0.3398 (0.4241) data time 0.0006 (0.0049) model time 0.3391 (0.4191) loss 2.8156 (2.7302) grad_norm 2.3656 (2.8044) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][270/625] eta 0:02:29 lr 0.000141 wd 0.0500 time 0.3407 (0.4198) data time 0.0006 (0.0047) model time 0.3401 (0.4151) loss 3.2400 (2.7320) grad_norm 3.0543 (2.7746) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][280/625] eta 0:02:23 lr 0.000141 wd 0.0500 time 0.3383 (0.4160) data time 0.0007 (0.0046) model time 0.3376 (0.4114) loss 2.0686 (2.7202) grad_norm 2.5067 (2.7767) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][290/625] eta 0:02:18 lr 0.000141 wd 0.0500 time 0.3463 (0.4126) data time 0.0006 (0.0044) model time 0.3457 (0.4082) loss 2.6237 (2.7132) grad_norm 2.6049 (2.8768) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][300/625] eta 0:02:13 lr 0.000141 wd 0.0500 time 0.3479 (0.4095) data time 0.0006 (0.0042) model time 0.3473 (0.4053) loss 3.1264 (2.7140) grad_norm 2.3602 (2.8730) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][310/625] eta 0:02:08 lr 0.000141 wd 0.0500 time 0.3453 (0.4085) data time 0.0005 (0.0041) model time 0.3448 (0.4044) loss 2.1227 (2.7184) grad_norm 2.3164 (2.8454) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][320/625] eta 0:02:03 lr 0.000141 wd 0.0500 time 0.3408 (0.4065) data time 0.0006 (0.0039) model time 0.3402 (0.4026) loss 1.8848 (2.7168) grad_norm 1.5776 (2.8396) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][330/625] eta 0:01:59 lr 0.000141 wd 0.0500 time 0.3456 (0.4040) data time 0.0007 (0.0038) model time 0.3449 (0.4002) loss 2.3652 (2.7123) grad_norm 3.1599 (2.8393) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][340/625] eta 0:01:54 lr 0.000141 wd 0.0500 time 0.3477 (0.4018) data time 0.0009 (0.0037) model time 0.3468 (0.3981) loss 3.2855 (2.7100) grad_norm 3.4642 (2.8724) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][350/625] eta 0:01:49 lr 0.000141 wd 0.0500 time 0.3398 (0.3997) data time 0.0007 (0.0036) model time 0.3391 (0.3961) loss 2.0742 (2.6994) grad_norm 2.5699 (2.8583) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][360/625] eta 0:01:45 lr 0.000140 wd 0.0500 time 0.3424 (0.3977) data time 0.0008 (0.0035) model time 0.3417 (0.3942) loss 2.6884 (2.7058) grad_norm 2.3818 (2.8794) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][370/625] eta 0:01:40 lr 0.000140 wd 0.0500 time 0.3481 (0.3959) data time 0.0009 (0.0034) model time 0.3472 (0.3925) loss 2.4235 (2.7049) grad_norm 1.5798 (2.8552) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:50:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][380/625] eta 0:01:36 lr 0.000140 wd 0.0500 time 0.3404 (0.3942) data time 0.0009 (0.0033) model time 0.3396 (0.3909) loss 3.2070 (2.6940) grad_norm 2.1990 (2.8442) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][390/625] eta 0:01:32 lr 0.000140 wd 0.0500 time 0.3426 (0.3927) data time 0.0008 (0.0032) model time 0.3417 (0.3894) loss 3.2908 (2.6902) grad_norm 3.3896 (2.8756) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][400/625] eta 0:01:27 lr 0.000140 wd 0.0500 time 0.3385 (0.3911) data time 0.0006 (0.0032) model time 0.3379 (0.3879) loss 2.9650 (2.6962) grad_norm 2.9713 (2.9322) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][410/625] eta 0:01:23 lr 0.000140 wd 0.0500 time 0.3437 (0.3896) data time 0.0007 (0.0031) model time 0.3429 (0.3865) loss 1.8126 (2.6983) grad_norm 2.2913 (2.9132) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][420/625] eta 0:01:19 lr 0.000140 wd 0.0500 time 0.3426 (0.3883) data time 0.0008 (0.0030) model time 0.3418 (0.3852) loss 2.4501 (2.7026) grad_norm 2.1062 (2.8983) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][430/625] eta 0:01:15 lr 0.000140 wd 0.0500 time 0.3394 (0.3883) data time 0.0006 (0.0030) model time 0.3388 (0.3853) loss 2.7643 (2.7030) grad_norm 2.3829 (2.8900) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][440/625] eta 0:01:11 lr 0.000140 wd 0.0500 time 0.3414 (0.3870) data time 0.0006 (0.0029) model time 0.3407 (0.3841) loss 2.6207 (2.7019) grad_norm 3.3723 (2.9065) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][450/625] eta 0:01:07 lr 0.000140 wd 0.0500 time 0.3403 (0.3859) data time 0.0008 (0.0029) model time 0.3395 (0.3830) loss 2.9740 (2.6991) grad_norm 1.9592 (2.8949) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][460/625] eta 0:01:03 lr 0.000140 wd 0.0500 time 0.3380 (0.3847) data time 0.0006 (0.0028) model time 0.3375 (0.3819) loss 3.2228 (2.6971) grad_norm 2.6762 (2.8781) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][470/625] eta 0:00:59 lr 0.000140 wd 0.0500 time 0.3437 (0.3836) data time 0.0006 (0.0028) model time 0.3432 (0.3809) loss 2.7608 (2.6905) grad_norm 3.1588 (2.8652) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][480/625] eta 0:00:55 lr 0.000139 wd 0.0500 time 0.3394 (0.3827) data time 0.0007 (0.0027) model time 0.3387 (0.3800) loss 3.0257 (2.6928) grad_norm 2.2631 (2.8577) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][490/625] eta 0:00:51 lr 0.000139 wd 0.0500 time 0.3481 (0.3817) data time 0.0008 (0.0027) model time 0.3473 (0.3791) loss 2.9009 (2.6957) grad_norm 2.5915 (2.8671) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][500/625] eta 0:00:47 lr 0.000139 wd 0.0500 time 0.3402 (0.3812) data time 0.0007 (0.0026) model time 0.3395 (0.3786) loss 2.0506 (2.6944) grad_norm 2.6338 (2.8498) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][510/625] eta 0:00:43 lr 0.000139 wd 0.0500 time 0.3380 (0.3804) data time 0.0006 (0.0026) model time 0.3374 (0.3778) loss 2.6534 (2.6994) grad_norm 4.5327 (2.8628) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][520/625] eta 0:00:39 lr 0.000139 wd 0.0500 time 0.3424 (0.3795) data time 0.0008 (0.0026) model time 0.3416 (0.3770) loss 2.6724 (2.7035) grad_norm 2.3431 (2.8560) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][530/625] eta 0:00:35 lr 0.000139 wd 0.0500 time 0.3407 (0.3787) data time 0.0008 (0.0025) model time 0.3399 (0.3762) loss 2.8072 (2.7022) grad_norm 2.8250 (2.8538) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][540/625] eta 0:00:32 lr 0.000139 wd 0.0500 time 0.3401 (0.3779) data time 0.0008 (0.0025) model time 0.3393 (0.3754) loss 2.3059 (2.6962) grad_norm 2.7102 (2.8507) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:51:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][550/625] eta 0:00:28 lr 0.000139 wd 0.0500 time 0.3452 (0.3772) data time 0.0006 (0.0024) model time 0.3447 (0.3747) loss 1.5707 (2.6900) grad_norm 2.1745 (2.8425) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:52:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][560/625] eta 0:00:24 lr 0.000139 wd 0.0500 time 0.3411 (0.3765) data time 0.0007 (0.0024) model time 0.3404 (0.3741) loss 2.8907 (2.6895) grad_norm 3.9469 (2.8850) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:52:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][570/625] eta 0:00:20 lr 0.000139 wd 0.0500 time 0.3369 (0.3758) data time 0.0009 (0.0024) model time 0.3360 (0.3735) loss 2.8201 (2.6935) grad_norm 3.6765 (2.8964) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:52:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][580/625] eta 0:00:16 lr 0.000139 wd 0.0500 time 0.3557 (0.3752) data time 0.0007 (0.0023) model time 0.3550 (0.3729) loss 2.3974 (2.6923) grad_norm 1.7935 (2.8875) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:52:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][590/625] eta 0:00:13 lr 0.000139 wd 0.0500 time 0.3424 (0.3746) data time 0.0006 (0.0023) model time 0.3418 (0.3723) loss 3.3341 (2.6930) grad_norm 2.2969 (2.8741) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:52:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][600/625] eta 0:00:09 lr 0.000138 wd 0.0500 time 0.3497 (0.3740) data time 0.0010 (0.0023) model time 0.3487 (0.3717) loss 1.6882 (2.6943) grad_norm 2.5698 (2.8633) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:52:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][610/625] eta 0:00:05 lr 0.000138 wd 0.0500 time 0.3358 (0.3734) data time 0.0004 (0.0023) model time 0.3353 (0.3711) loss 3.0501 (2.6878) grad_norm 2.5322 (2.8560) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:52:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [255/300][620/625] eta 0:00:01 lr 0.000138 wd 0.0500 time 0.3456 (0.3730) data time 0.0006 (0.0022) model time 0.3451 (0.3708) loss 2.9459 (2.6875) grad_norm 3.3297 (2.8492) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 12:52:23 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 255 training takes 0:03:22 [2024-09-01 12:52:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:52:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:52:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.431 (0.431) Loss 0.5166 (0.5166) Acc@1 88.721 (88.721) Acc@5 98.633 (98.633) Mem 11219MB [2024-09-01 12:52:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.8213 (0.6114) Acc@1 80.225 (86.581) Acc@5 95.703 (97.772) Mem 11219MB [2024-09-01 12:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.111) Loss 0.8975 (0.7284) Acc@1 79.248 (83.777) Acc@5 95.117 (96.561) Mem 11219MB [2024-09-01 12:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.451 Acc@5 96.559 [2024-09-01 12:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 12:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.45% [2024-09-01 12:52:30 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 12:52:32 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 12:52:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.429 (0.429) Loss 0.4910 (0.4910) Acc@1 89.453 (89.453) Acc@5 98.682 (98.682) Mem 11219MB [2024-09-01 12:52:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.099 (0.128) Loss 0.8062 (0.5926) Acc@1 80.859 (87.154) Acc@5 95.850 (97.869) Mem 11219MB [2024-09-01 12:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.089 (0.110) Loss 0.8770 (0.7031) Acc@1 79.443 (84.373) Acc@5 95.264 (96.749) Mem 11219MB [2024-09-01 12:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.071 Acc@5 96.771 [2024-09-01 12:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.1% [2024-09-01 12:52:35 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 84.07% [2024-09-01 12:52:35 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 12:52:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 12:52:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][0/625] eta 0:07:32 lr 0.000138 wd 0.0500 time 0.7246 (0.7246) data time 0.3818 (0.3818) model time 0.0000 (0.0000) loss 2.9051 (2.9051) grad_norm 2.6616 (2.6616) loss_scale 128.0000 (128.0000) mem 11227MB [2024-09-01 12:52:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][10/625] eta 0:04:14 lr 0.000138 wd 0.0500 time 0.3422 (0.4136) data time 0.0006 (0.0355) model time 0.0000 (0.0000) loss 3.1876 (2.8111) grad_norm 2.2372 (2.2683) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:52:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][20/625] eta 0:03:49 lr 0.000138 wd 0.0500 time 0.3379 (0.3796) data time 0.0008 (0.0190) model time 0.0000 (0.0000) loss 2.9956 (2.8147) grad_norm 1.8937 (2.2002) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:52:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][30/625] eta 0:03:38 lr 0.000138 wd 0.0500 time 0.3440 (0.3674) data time 0.0006 (0.0131) model time 0.0000 (0.0000) loss 3.0614 (2.7738) grad_norm 1.9875 (2.2457) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:52:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][40/625] eta 0:03:31 lr 0.000138 wd 0.0500 time 0.3541 (0.3620) data time 0.0006 (0.0101) model time 0.0000 (0.0000) loss 2.1390 (2.7602) grad_norm 2.4417 (2.2621) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:52:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][50/625] eta 0:03:25 lr 0.000138 wd 0.0500 time 0.3432 (0.3580) data time 0.0006 (0.0083) model time 0.0000 (0.0000) loss 3.0308 (2.7678) grad_norm 2.4250 (2.3303) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][60/625] eta 0:03:20 lr 0.000138 wd 0.0500 time 0.3425 (0.3556) data time 0.0008 (0.0071) model time 0.3417 (0.3425) loss 2.9431 (2.7356) grad_norm 2.6549 (2.5095) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][70/625] eta 0:03:16 lr 0.000138 wd 0.0500 time 0.3499 (0.3539) data time 0.0007 (0.0063) model time 0.3492 (0.3426) loss 3.0892 (2.7412) grad_norm 2.4295 (2.5689) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][80/625] eta 0:03:12 lr 0.000138 wd 0.0500 time 0.3381 (0.3526) data time 0.0006 (0.0056) model time 0.3375 (0.3425) loss 2.8009 (2.7680) grad_norm 1.7545 (2.9060) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][90/625] eta 0:03:08 lr 0.000137 wd 0.0500 time 0.3477 (0.3516) data time 0.0008 (0.0051) model time 0.3469 (0.3426) loss 2.5214 (2.7705) grad_norm 1.6072 (2.9585) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][100/625] eta 0:03:04 lr 0.000137 wd 0.0500 time 0.3419 (0.3523) data time 0.0006 (0.0047) model time 0.3413 (0.3456) loss 3.0036 (2.7532) grad_norm 4.6133 (2.9457) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][110/625] eta 0:03:00 lr 0.000137 wd 0.0500 time 0.3363 (0.3513) data time 0.0008 (0.0043) model time 0.3355 (0.3448) loss 3.0586 (2.7445) grad_norm 2.7248 (2.9482) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][120/625] eta 0:02:57 lr 0.000137 wd 0.0500 time 0.3468 (0.3506) data time 0.0008 (0.0040) model time 0.3460 (0.3443) loss 3.0186 (2.7202) grad_norm 2.4146 (3.0086) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][130/625] eta 0:02:53 lr 0.000137 wd 0.0500 time 0.3420 (0.3499) data time 0.0006 (0.0038) model time 0.3414 (0.3438) loss 3.0879 (2.7343) grad_norm 2.9464 (2.9568) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][140/625] eta 0:02:49 lr 0.000137 wd 0.0500 time 0.3395 (0.3494) data time 0.0008 (0.0036) model time 0.3387 (0.3437) loss 3.0732 (2.7270) grad_norm 1.9852 (2.9115) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][150/625] eta 0:02:45 lr 0.000137 wd 0.0500 time 0.3369 (0.3489) data time 0.0008 (0.0034) model time 0.3360 (0.3434) loss 3.1874 (2.7204) grad_norm 3.2508 (3.0032) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][160/625] eta 0:02:42 lr 0.000137 wd 0.0500 time 0.3442 (0.3485) data time 0.0008 (0.0032) model time 0.3434 (0.3433) loss 3.1266 (2.7091) grad_norm 1.8337 (2.9796) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][170/625] eta 0:02:38 lr 0.000137 wd 0.0500 time 0.3361 (0.3481) data time 0.0006 (0.0031) model time 0.3355 (0.3431) loss 2.0182 (2.6888) grad_norm 3.0048 (2.9459) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][180/625] eta 0:02:34 lr 0.000137 wd 0.0500 time 0.3473 (0.3480) data time 0.0007 (0.0030) model time 0.3466 (0.3432) loss 2.3364 (2.6897) grad_norm 16.8149 (3.0184) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][190/625] eta 0:02:31 lr 0.000137 wd 0.0500 time 0.3436 (0.3477) data time 0.0008 (0.0029) model time 0.3428 (0.3431) loss 2.8796 (2.6949) grad_norm 2.6892 (3.1158) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][200/625] eta 0:02:27 lr 0.000137 wd 0.0500 time 0.3398 (0.3475) data time 0.0006 (0.0028) model time 0.3392 (0.3430) loss 1.9952 (2.6985) grad_norm 2.3750 (3.1522) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][210/625] eta 0:02:24 lr 0.000136 wd 0.0500 time 0.3423 (0.3474) data time 0.0008 (0.0027) model time 0.3415 (0.3432) loss 1.7894 (2.6919) grad_norm 2.2600 (3.1109) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][220/625] eta 0:02:21 lr 0.000136 wd 0.0500 time 0.3395 (0.3493) data time 0.0008 (0.0026) model time 0.3387 (0.3458) loss 2.7373 (2.7037) grad_norm 2.8759 (3.1008) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:53:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][230/625] eta 0:02:17 lr 0.000136 wd 0.0500 time 0.3448 (0.3491) data time 0.0008 (0.0025) model time 0.3440 (0.3457) loss 3.0189 (2.7161) grad_norm 2.1145 (3.0828) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:54:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][240/625] eta 0:02:14 lr 0.000136 wd 0.0500 time 0.3447 (0.3488) data time 0.0006 (0.0025) model time 0.3440 (0.3455) loss 2.3492 (2.7124) grad_norm 2.7706 (3.0539) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:54:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][250/625] eta 0:02:10 lr 0.000136 wd 0.0500 time 0.3416 (0.3486) data time 0.0008 (0.0024) model time 0.3408 (0.3454) loss 3.0260 (2.7183) grad_norm 1.7041 (3.0576) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:54:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][260/625] eta 0:02:07 lr 0.000136 wd 0.0500 time 0.3423 (0.3485) data time 0.0009 (0.0023) model time 0.3414 (0.3453) loss 2.3885 (2.7059) grad_norm 3.5299 (3.0319) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:54:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][270/625] eta 0:02:03 lr 0.000136 wd 0.0500 time 0.3419 (0.3483) data time 0.0008 (0.0023) model time 0.3410 (0.3452) loss 2.8877 (2.6942) grad_norm 4.9540 (3.0336) loss_scale 256.0000 (128.4723) mem 11224MB [2024-09-01 12:54:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][280/625] eta 0:02:00 lr 0.000136 wd 0.0500 time 0.3447 (0.3482) data time 0.0008 (0.0022) model time 0.3438 (0.3452) loss 3.1647 (2.6969) grad_norm 1.9619 (3.0035) loss_scale 256.0000 (133.0107) mem 11224MB [2024-09-01 12:54:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][290/625] eta 0:01:56 lr 0.000136 wd 0.0500 time 0.3386 (0.3480) data time 0.0006 (0.0022) model time 0.3379 (0.3450) loss 2.3898 (2.6918) grad_norm 2.6713 (2.9786) loss_scale 256.0000 (137.2371) mem 11224MB [2024-09-01 12:54:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][300/625] eta 0:01:53 lr 0.000136 wd 0.0500 time 0.3469 (0.3478) data time 0.0006 (0.0021) model time 0.3463 (0.3449) loss 2.1014 (2.6884) grad_norm 2.7818 (3.1375) loss_scale 256.0000 (141.1827) mem 11224MB [2024-09-01 12:54:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][310/625] eta 0:01:49 lr 0.000136 wd 0.0500 time 0.3404 (0.3477) data time 0.0008 (0.0021) model time 0.3396 (0.3448) loss 2.7759 (2.6863) grad_norm 2.1502 (3.1175) loss_scale 256.0000 (144.8746) mem 11224MB [2024-09-01 12:54:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][320/625] eta 0:01:46 lr 0.000136 wd 0.0500 time 0.3431 (0.3475) data time 0.0008 (0.0021) model time 0.3423 (0.3447) loss 3.2489 (2.6854) grad_norm 2.4614 (3.1190) loss_scale 256.0000 (148.3364) mem 11224MB [2024-09-01 12:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][330/625] eta 0:01:42 lr 0.000135 wd 0.0500 time 0.3502 (0.3475) data time 0.0008 (0.0020) model time 0.3494 (0.3447) loss 2.5446 (2.6849) grad_norm 3.3052 (3.0986) loss_scale 256.0000 (151.5891) mem 11224MB [2024-09-01 12:54:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][340/625] eta 0:01:39 lr 0.000135 wd 0.0500 time 0.3518 (0.3474) data time 0.0007 (0.0020) model time 0.3511 (0.3446) loss 3.2796 (2.6834) grad_norm 3.4193 (3.1521) loss_scale 256.0000 (154.6510) mem 11224MB [2024-09-01 12:54:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][350/625] eta 0:01:35 lr 0.000135 wd 0.0500 time 0.3383 (0.3473) data time 0.0009 (0.0020) model time 0.3375 (0.3446) loss 2.9291 (2.6829) grad_norm 2.4939 (3.1389) loss_scale 256.0000 (157.5385) mem 11224MB [2024-09-01 12:54:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][360/625] eta 0:01:32 lr 0.000135 wd 0.0500 time 0.3429 (0.3473) data time 0.0007 (0.0019) model time 0.3422 (0.3446) loss 2.0472 (2.6812) grad_norm 3.1413 (3.1391) loss_scale 256.0000 (160.2659) mem 11224MB [2024-09-01 12:54:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][370/625] eta 0:01:28 lr 0.000135 wd 0.0500 time 0.3371 (0.3472) data time 0.0006 (0.0019) model time 0.3365 (0.3446) loss 2.5277 (2.6788) grad_norm 3.6018 (3.1473) loss_scale 256.0000 (162.8464) mem 11224MB [2024-09-01 12:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][380/625] eta 0:01:25 lr 0.000135 wd 0.0500 time 0.3407 (0.3471) data time 0.0006 (0.0019) model time 0.3401 (0.3446) loss 3.2202 (2.6841) grad_norm 2.4257 (3.1274) loss_scale 256.0000 (165.2913) mem 11224MB [2024-09-01 12:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][390/625] eta 0:01:21 lr 0.000135 wd 0.0500 time 0.3396 (0.3470) data time 0.0008 (0.0018) model time 0.3387 (0.3445) loss 2.3436 (2.6879) grad_norm 2.1378 (3.1050) loss_scale 256.0000 (167.6113) mem 11224MB [2024-09-01 12:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][400/625] eta 0:01:18 lr 0.000135 wd 0.0500 time 0.5751 (0.3479) data time 0.0008 (0.0018) model time 0.5743 (0.3455) loss 2.3955 (2.6813) grad_norm 2.7526 (3.1004) loss_scale 256.0000 (169.8155) mem 11224MB [2024-09-01 12:55:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][410/625] eta 0:01:14 lr 0.000135 wd 0.0500 time 0.3494 (0.3478) data time 0.0007 (0.0018) model time 0.3487 (0.3455) loss 2.0304 (2.6741) grad_norm 2.0685 (3.0975) loss_scale 256.0000 (171.9124) mem 11224MB [2024-09-01 12:55:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][420/625] eta 0:01:11 lr 0.000135 wd 0.0500 time 0.3400 (0.3477) data time 0.0007 (0.0018) model time 0.3394 (0.3454) loss 2.5534 (2.6769) grad_norm 1.8354 (3.0793) loss_scale 256.0000 (173.9097) mem 11224MB [2024-09-01 12:55:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][430/625] eta 0:01:07 lr 0.000135 wd 0.0500 time 0.3400 (0.3480) data time 0.0008 (0.0018) model time 0.3392 (0.3458) loss 2.8010 (2.6805) grad_norm 5.2386 (3.0686) loss_scale 256.0000 (175.8144) mem 11224MB [2024-09-01 12:55:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][440/625] eta 0:01:04 lr 0.000135 wd 0.0500 time 0.3450 (0.3479) data time 0.0008 (0.0017) model time 0.3441 (0.3457) loss 2.9525 (2.6791) grad_norm 1.7742 (3.0481) loss_scale 256.0000 (177.6327) mem 11224MB [2024-09-01 12:55:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][450/625] eta 0:01:00 lr 0.000134 wd 0.0500 time 0.3412 (0.3478) data time 0.0007 (0.0017) model time 0.3405 (0.3456) loss 2.5017 (2.6791) grad_norm 1.4746 (3.0267) loss_scale 256.0000 (179.3703) mem 11224MB [2024-09-01 12:55:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][460/625] eta 0:00:57 lr 0.000134 wd 0.0500 time 0.3451 (0.3477) data time 0.0006 (0.0017) model time 0.3445 (0.3456) loss 2.7957 (2.6804) grad_norm 2.8297 (3.0177) loss_scale 256.0000 (181.0325) mem 11224MB [2024-09-01 12:55:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][470/625] eta 0:00:53 lr 0.000134 wd 0.0500 time 0.3387 (0.3476) data time 0.0005 (0.0017) model time 0.3382 (0.3455) loss 2.8691 (2.6838) grad_norm 5.7328 (3.0220) loss_scale 256.0000 (182.6242) mem 11224MB [2024-09-01 12:55:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][480/625] eta 0:00:50 lr 0.000134 wd 0.0500 time 0.3525 (0.3476) data time 0.0008 (0.0017) model time 0.3517 (0.3455) loss 2.1991 (2.6799) grad_norm 2.5186 (3.0292) loss_scale 256.0000 (184.1497) mem 11224MB [2024-09-01 12:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][490/625] eta 0:00:46 lr 0.000134 wd 0.0500 time 0.3453 (0.3476) data time 0.0007 (0.0016) model time 0.3447 (0.3455) loss 2.7625 (2.6778) grad_norm 2.3876 (3.0190) loss_scale 256.0000 (185.6130) mem 11224MB [2024-09-01 12:55:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][500/625] eta 0:00:43 lr 0.000134 wd 0.0500 time 0.3481 (0.3476) data time 0.0008 (0.0016) model time 0.3473 (0.3456) loss 1.9770 (2.6811) grad_norm 8.4998 (3.0122) loss_scale 256.0000 (187.0180) mem 11224MB [2024-09-01 12:55:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][510/625] eta 0:00:39 lr 0.000134 wd 0.0500 time 0.3500 (0.3476) data time 0.0009 (0.0016) model time 0.3491 (0.3456) loss 2.8944 (2.6805) grad_norm 2.6473 (3.0072) loss_scale 256.0000 (188.3679) mem 11224MB [2024-09-01 12:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][520/625] eta 0:00:36 lr 0.000134 wd 0.0500 time 0.3453 (0.3484) data time 0.0008 (0.0016) model time 0.3445 (0.3465) loss 2.8625 (2.6835) grad_norm 2.7129 (2.9970) loss_scale 256.0000 (189.6660) mem 11224MB [2024-09-01 12:55:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][530/625] eta 0:00:33 lr 0.000134 wd 0.0500 time 0.3525 (0.3484) data time 0.0008 (0.0016) model time 0.3517 (0.3465) loss 2.9747 (2.6796) grad_norm 2.3614 (2.9842) loss_scale 256.0000 (190.9153) mem 11224MB [2024-09-01 12:55:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][540/625] eta 0:00:29 lr 0.000134 wd 0.0500 time 0.3438 (0.3484) data time 0.0008 (0.0016) model time 0.3430 (0.3465) loss 2.1499 (2.6801) grad_norm 2.5097 (2.9708) loss_scale 256.0000 (192.1183) mem 11224MB [2024-09-01 12:55:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][550/625] eta 0:00:26 lr 0.000134 wd 0.0500 time 0.3398 (0.3482) data time 0.0008 (0.0016) model time 0.3390 (0.3464) loss 2.3570 (2.6781) grad_norm 2.7269 (inf) loss_scale 128.0000 (192.1162) mem 11224MB [2024-09-01 12:55:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][560/625] eta 0:00:22 lr 0.000134 wd 0.0500 time 0.3491 (0.3482) data time 0.0008 (0.0015) model time 0.3483 (0.3464) loss 2.9642 (2.6779) grad_norm 2.4054 (inf) loss_scale 128.0000 (190.9733) mem 11224MB [2024-09-01 12:55:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][570/625] eta 0:00:19 lr 0.000133 wd 0.0500 time 0.3471 (0.3483) data time 0.0008 (0.0015) model time 0.3464 (0.3464) loss 2.9074 (2.6780) grad_norm 2.6559 (inf) loss_scale 128.0000 (189.8704) mem 11224MB [2024-09-01 12:56:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][580/625] eta 0:00:15 lr 0.000133 wd 0.0500 time 0.3515 (0.3483) data time 0.0009 (0.0015) model time 0.3506 (0.3465) loss 1.9625 (2.6788) grad_norm 2.2050 (inf) loss_scale 128.0000 (188.8055) mem 11224MB [2024-09-01 12:56:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][590/625] eta 0:00:12 lr 0.000133 wd 0.0500 time 0.3506 (0.3484) data time 0.0006 (0.0015) model time 0.3500 (0.3466) loss 2.8019 (2.6806) grad_norm 3.2730 (inf) loss_scale 128.0000 (187.7766) mem 11224MB [2024-09-01 12:56:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][600/625] eta 0:00:08 lr 0.000133 wd 0.0500 time 0.3566 (0.3484) data time 0.0008 (0.0015) model time 0.3558 (0.3467) loss 3.1631 (2.6827) grad_norm 2.6007 (inf) loss_scale 128.0000 (186.7820) mem 11224MB [2024-09-01 12:56:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][610/625] eta 0:00:05 lr 0.000133 wd 0.0500 time 0.3333 (0.3487) data time 0.0004 (0.0015) model time 0.3329 (0.3470) loss 2.1659 (2.6818) grad_norm 3.9716 (inf) loss_scale 128.0000 (185.8200) mem 11224MB [2024-09-01 12:56:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [256/300][620/625] eta 0:00:01 lr 0.000133 wd 0.0500 time 0.3335 (0.3485) data time 0.0003 (0.0015) model time 0.3331 (0.3468) loss 1.9928 (2.6793) grad_norm 1.9004 (inf) loss_scale 128.0000 (184.8889) mem 11224MB [2024-09-01 12:56:15 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 256 training takes 0:03:37 [2024-09-01 12:56:15 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 12:56:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 12:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.434 (0.434) Loss 0.5176 (0.5176) Acc@1 88.770 (88.770) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 12:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.128) Loss 0.8418 (0.6154) Acc@1 80.029 (86.741) Acc@5 95.801 (97.741) Mem 11224MB [2024-09-01 12:56:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.111) Loss 0.9316 (0.7305) Acc@1 77.930 (83.929) Acc@5 95.166 (96.540) Mem 11224MB [2024-09-01 12:56:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.623 Acc@5 96.563 [2024-09-01 12:56:19 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 12:56:19 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.62% [2024-09-01 12:56:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 12:56:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 12:56:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.434 (0.434) Loss 0.4912 (0.4912) Acc@1 89.502 (89.502) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 12:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.128) Loss 0.8052 (0.5930) Acc@1 80.859 (87.167) Acc@5 95.850 (97.860) Mem 11224MB [2024-09-01 12:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.110) Loss 0.8779 (0.7033) Acc@1 79.492 (84.382) Acc@5 95.166 (96.738) Mem 11224MB [2024-09-01 12:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.085 Acc@5 96.759 [2024-09-01 12:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.1% [2024-09-01 12:56:22 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 84.09% [2024-09-01 12:56:22 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 12:56:23 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 12:56:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][0/625] eta 0:07:00 lr 0.000133 wd 0.0500 time 0.6729 (0.6729) data time 0.3587 (0.3587) model time 0.0000 (0.0000) loss 2.9460 (2.9460) grad_norm 2.5037 (2.5037) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][10/625] eta 0:03:51 lr 0.000133 wd 0.0500 time 0.3478 (0.3768) data time 0.0007 (0.0333) model time 0.0000 (0.0000) loss 2.9639 (3.0373) grad_norm 2.3953 (2.3553) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][20/625] eta 0:03:39 lr 0.000133 wd 0.0500 time 0.3493 (0.3623) data time 0.0008 (0.0179) model time 0.0000 (0.0000) loss 2.7744 (2.9928) grad_norm 1.7939 (2.3643) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][30/625] eta 0:03:32 lr 0.000133 wd 0.0500 time 0.3489 (0.3568) data time 0.0008 (0.0124) model time 0.0000 (0.0000) loss 2.8937 (2.9064) grad_norm 1.6939 (2.3555) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][40/625] eta 0:03:26 lr 0.000133 wd 0.0500 time 0.3428 (0.3537) data time 0.0006 (0.0096) model time 0.0000 (0.0000) loss 2.3918 (2.8685) grad_norm 2.6960 (2.3205) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][50/625] eta 0:03:22 lr 0.000133 wd 0.0500 time 0.3480 (0.3519) data time 0.0007 (0.0079) model time 0.0000 (0.0000) loss 1.5749 (2.7966) grad_norm 3.1178 (3.1812) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][60/625] eta 0:03:21 lr 0.000133 wd 0.0500 time 0.5430 (0.3573) data time 0.0006 (0.0067) model time 0.5424 (0.3840) loss 2.4490 (2.7622) grad_norm 3.9816 (3.1697) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][70/625] eta 0:03:17 lr 0.000132 wd 0.0500 time 0.3454 (0.3553) data time 0.0008 (0.0059) model time 0.3446 (0.3632) loss 2.6956 (2.7600) grad_norm 2.2489 (3.1567) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][80/625] eta 0:03:12 lr 0.000132 wd 0.0500 time 0.3433 (0.3537) data time 0.0008 (0.0053) model time 0.3425 (0.3560) loss 1.6326 (2.7413) grad_norm 2.6066 (3.0878) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][90/625] eta 0:03:09 lr 0.000132 wd 0.0500 time 0.3389 (0.3545) data time 0.0008 (0.0048) model time 0.3381 (0.3570) loss 3.0538 (2.7351) grad_norm 2.2669 (3.0143) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][100/625] eta 0:03:05 lr 0.000132 wd 0.0500 time 0.3449 (0.3535) data time 0.0008 (0.0044) model time 0.3441 (0.3542) loss 3.1838 (2.7444) grad_norm 1.7285 (2.9368) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][110/625] eta 0:03:01 lr 0.000132 wd 0.0500 time 0.3522 (0.3529) data time 0.0007 (0.0041) model time 0.3515 (0.3528) loss 2.5169 (2.7485) grad_norm 2.7665 (2.8851) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][120/625] eta 0:02:57 lr 0.000132 wd 0.0500 time 0.3467 (0.3522) data time 0.0008 (0.0038) model time 0.3459 (0.3516) loss 2.2758 (2.7574) grad_norm 2.9412 (2.8559) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][130/625] eta 0:02:54 lr 0.000132 wd 0.0500 time 0.3464 (0.3517) data time 0.0007 (0.0036) model time 0.3457 (0.3508) loss 2.7175 (2.7546) grad_norm 3.7819 (2.8438) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][140/625] eta 0:02:50 lr 0.000132 wd 0.0500 time 0.3496 (0.3513) data time 0.0006 (0.0034) model time 0.3490 (0.3502) loss 2.0420 (2.7514) grad_norm 2.2266 (2.8053) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][150/625] eta 0:02:46 lr 0.000132 wd 0.0500 time 0.3537 (0.3510) data time 0.0007 (0.0032) model time 0.3530 (0.3497) loss 2.4428 (2.7239) grad_norm 1.8635 (2.7791) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][160/625] eta 0:02:43 lr 0.000132 wd 0.0500 time 0.3444 (0.3507) data time 0.0009 (0.0031) model time 0.3436 (0.3493) loss 2.8508 (2.7228) grad_norm 11.3868 (2.8861) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][170/625] eta 0:02:39 lr 0.000132 wd 0.0500 time 0.3471 (0.3507) data time 0.0006 (0.0029) model time 0.3466 (0.3493) loss 3.5211 (2.7268) grad_norm 2.2209 (2.8535) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][180/625] eta 0:02:35 lr 0.000132 wd 0.0500 time 0.3456 (0.3505) data time 0.0006 (0.0028) model time 0.3449 (0.3491) loss 3.0757 (2.7263) grad_norm 2.0941 (2.8235) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][190/625] eta 0:02:32 lr 0.000131 wd 0.0500 time 0.3453 (0.3502) data time 0.0008 (0.0027) model time 0.3445 (0.3487) loss 2.4985 (2.7369) grad_norm 1.8650 (2.7986) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][200/625] eta 0:02:28 lr 0.000131 wd 0.0500 time 0.3440 (0.3500) data time 0.0009 (0.0026) model time 0.3431 (0.3485) loss 2.6369 (2.7423) grad_norm 5.8682 (2.8000) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][210/625] eta 0:02:25 lr 0.000131 wd 0.0500 time 0.3436 (0.3498) data time 0.0008 (0.0025) model time 0.3427 (0.3482) loss 3.2616 (2.7469) grad_norm 3.7344 (2.8131) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][220/625] eta 0:02:21 lr 0.000131 wd 0.0500 time 0.3480 (0.3496) data time 0.0006 (0.0025) model time 0.3473 (0.3480) loss 3.2852 (2.7443) grad_norm 3.0857 (2.8084) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][230/625] eta 0:02:18 lr 0.000131 wd 0.0500 time 0.3431 (0.3494) data time 0.0006 (0.0024) model time 0.3425 (0.3479) loss 2.3735 (2.7397) grad_norm 2.5066 (2.7916) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][240/625] eta 0:02:14 lr 0.000131 wd 0.0500 time 0.3428 (0.3494) data time 0.0006 (0.0023) model time 0.3421 (0.3478) loss 2.8686 (2.7454) grad_norm 2.1610 (2.7818) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][250/625] eta 0:02:10 lr 0.000131 wd 0.0500 time 0.3423 (0.3493) data time 0.0007 (0.0023) model time 0.3416 (0.3477) loss 2.8239 (2.7435) grad_norm 2.5950 (2.7728) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][260/625] eta 0:02:07 lr 0.000131 wd 0.0500 time 0.3457 (0.3491) data time 0.0006 (0.0022) model time 0.3452 (0.3476) loss 2.8930 (2.7433) grad_norm 2.7039 (2.7526) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:57:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][270/625] eta 0:02:04 lr 0.000131 wd 0.0500 time 0.7743 (0.3514) data time 0.0008 (0.0022) model time 0.7735 (0.3504) loss 2.0956 (2.7404) grad_norm 1.7104 (2.7351) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][280/625] eta 0:02:01 lr 0.000131 wd 0.0500 time 0.3362 (0.3510) data time 0.0007 (0.0021) model time 0.3355 (0.3499) loss 2.1322 (2.7404) grad_norm 11.8733 (2.7868) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][290/625] eta 0:01:57 lr 0.000131 wd 0.0500 time 0.3394 (0.3507) data time 0.0008 (0.0021) model time 0.3386 (0.3496) loss 1.9764 (2.7393) grad_norm 2.5468 (2.7930) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][300/625] eta 0:01:53 lr 0.000131 wd 0.0500 time 0.3419 (0.3504) data time 0.0008 (0.0020) model time 0.3411 (0.3492) loss 3.0007 (2.7401) grad_norm 2.0272 (2.8364) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][310/625] eta 0:01:50 lr 0.000130 wd 0.0500 time 0.3487 (0.3501) data time 0.0006 (0.0020) model time 0.3481 (0.3490) loss 2.4707 (2.7310) grad_norm 2.6406 (2.8168) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][320/625] eta 0:01:46 lr 0.000130 wd 0.0500 time 0.3477 (0.3500) data time 0.0006 (0.0020) model time 0.3471 (0.3488) loss 3.1953 (2.7210) grad_norm 2.3434 (2.8230) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][330/625] eta 0:01:43 lr 0.000130 wd 0.0500 time 0.3475 (0.3498) data time 0.0006 (0.0019) model time 0.3468 (0.3486) loss 2.9707 (2.7154) grad_norm 1.6350 (2.8268) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][340/625] eta 0:01:39 lr 0.000130 wd 0.0500 time 0.3501 (0.3497) data time 0.0008 (0.0019) model time 0.3493 (0.3485) loss 2.3596 (2.7176) grad_norm 2.6240 (2.8196) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][350/625] eta 0:01:36 lr 0.000130 wd 0.0500 time 0.3448 (0.3495) data time 0.0007 (0.0019) model time 0.3440 (0.3483) loss 2.7568 (2.7173) grad_norm 1.9132 (2.8108) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][360/625] eta 0:01:32 lr 0.000130 wd 0.0500 time 0.3510 (0.3494) data time 0.0008 (0.0018) model time 0.3502 (0.3482) loss 3.0297 (2.7179) grad_norm 3.2167 (2.8040) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][370/625] eta 0:01:29 lr 0.000130 wd 0.0500 time 0.3462 (0.3493) data time 0.0008 (0.0018) model time 0.3455 (0.3480) loss 2.7668 (2.7163) grad_norm 3.8702 (2.8110) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][380/625] eta 0:01:25 lr 0.000130 wd 0.0500 time 0.3453 (0.3493) data time 0.0008 (0.0018) model time 0.3445 (0.3480) loss 1.8041 (2.7103) grad_norm 3.4175 (2.9411) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][390/625] eta 0:01:22 lr 0.000130 wd 0.0500 time 0.3106 (0.3496) data time 0.0008 (0.0018) model time 0.3098 (0.3484) loss 2.7639 (2.7130) grad_norm 2.0134 (2.9304) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][400/625] eta 0:01:18 lr 0.000130 wd 0.0500 time 0.3508 (0.3494) data time 0.0008 (0.0017) model time 0.3499 (0.3482) loss 3.0007 (2.7043) grad_norm 2.0710 (2.9230) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][410/625] eta 0:01:15 lr 0.000130 wd 0.0500 time 0.3478 (0.3493) data time 0.0008 (0.0017) model time 0.3470 (0.3481) loss 2.4782 (2.7043) grad_norm 2.4075 (2.9214) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][420/625] eta 0:01:11 lr 0.000130 wd 0.0500 time 0.3464 (0.3492) data time 0.0006 (0.0017) model time 0.3458 (0.3480) loss 3.2639 (2.7048) grad_norm 4.0764 (2.9261) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][430/625] eta 0:01:08 lr 0.000129 wd 0.0500 time 0.3432 (0.3491) data time 0.0008 (0.0017) model time 0.3424 (0.3479) loss 3.0500 (2.7070) grad_norm 3.7653 (2.9354) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:58:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][440/625] eta 0:01:04 lr 0.000129 wd 0.0500 time 0.3452 (0.3490) data time 0.0009 (0.0017) model time 0.3443 (0.3478) loss 2.8638 (2.7090) grad_norm 2.5961 (2.9330) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][450/625] eta 0:01:01 lr 0.000129 wd 0.0500 time 0.5463 (0.3498) data time 0.0008 (0.0016) model time 0.5455 (0.3487) loss 2.1549 (2.7088) grad_norm 4.3998 (2.9314) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][460/625] eta 0:00:57 lr 0.000129 wd 0.0500 time 0.3488 (0.3506) data time 0.0008 (0.0016) model time 0.3480 (0.3496) loss 3.2626 (2.7069) grad_norm 2.1347 (2.9230) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][470/625] eta 0:00:54 lr 0.000129 wd 0.0500 time 0.3405 (0.3506) data time 0.0006 (0.0016) model time 0.3400 (0.3496) loss 2.0114 (2.7086) grad_norm 2.8979 (2.9194) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][480/625] eta 0:00:50 lr 0.000129 wd 0.0500 time 0.3406 (0.3504) data time 0.0006 (0.0016) model time 0.3400 (0.3494) loss 3.1187 (2.6999) grad_norm 2.2792 (2.9393) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][490/625] eta 0:00:47 lr 0.000129 wd 0.0500 time 0.3463 (0.3503) data time 0.0006 (0.0016) model time 0.3457 (0.3493) loss 2.8081 (2.6936) grad_norm 2.4083 (2.9292) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][500/625] eta 0:00:43 lr 0.000129 wd 0.0500 time 0.3443 (0.3502) data time 0.0006 (0.0016) model time 0.3436 (0.3491) loss 3.2927 (2.6934) grad_norm 2.0565 (2.9227) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][510/625] eta 0:00:40 lr 0.000129 wd 0.0500 time 0.3448 (0.3501) data time 0.0006 (0.0015) model time 0.3442 (0.3490) loss 2.6373 (2.6912) grad_norm 2.8929 (2.9171) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][520/625] eta 0:00:36 lr 0.000129 wd 0.0500 time 0.3436 (0.3500) data time 0.0008 (0.0015) model time 0.3429 (0.3489) loss 3.5258 (2.6953) grad_norm 1.7191 (2.9224) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][530/625] eta 0:00:33 lr 0.000129 wd 0.0500 time 0.3483 (0.3499) data time 0.0008 (0.0015) model time 0.3476 (0.3489) loss 3.1846 (2.6991) grad_norm 1.8724 (2.9177) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][540/625] eta 0:00:29 lr 0.000129 wd 0.0500 time 0.3449 (0.3498) data time 0.0006 (0.0015) model time 0.3444 (0.3488) loss 2.3585 (2.7005) grad_norm 1.9772 (2.9450) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][550/625] eta 0:00:26 lr 0.000129 wd 0.0500 time 0.3448 (0.3497) data time 0.0006 (0.0015) model time 0.3442 (0.3487) loss 3.0394 (2.6977) grad_norm 2.3945 (2.9437) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][560/625] eta 0:00:22 lr 0.000128 wd 0.0500 time 0.3485 (0.3497) data time 0.0008 (0.0015) model time 0.3477 (0.3486) loss 2.9558 (2.6992) grad_norm 1.7673 (2.9447) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][570/625] eta 0:00:19 lr 0.000128 wd 0.0500 time 0.3473 (0.3503) data time 0.0008 (0.0015) model time 0.3465 (0.3493) loss 2.6560 (2.6987) grad_norm 1.7593 (2.9552) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][580/625] eta 0:00:15 lr 0.000128 wd 0.0500 time 0.3441 (0.3509) data time 0.0006 (0.0015) model time 0.3436 (0.3499) loss 1.7897 (2.6973) grad_norm 2.4767 (2.9488) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][590/625] eta 0:00:12 lr 0.000128 wd 0.0500 time 0.3439 (0.3508) data time 0.0008 (0.0014) model time 0.3432 (0.3498) loss 3.1311 (2.6944) grad_norm 3.1864 (2.9519) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][600/625] eta 0:00:08 lr 0.000128 wd 0.0500 time 0.3439 (0.3510) data time 0.0008 (0.0014) model time 0.3431 (0.3500) loss 2.9140 (2.6978) grad_norm 2.8004 (2.9398) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 12:59:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][610/625] eta 0:00:05 lr 0.000128 wd 0.0500 time 0.3332 (0.3508) data time 0.0006 (0.0014) model time 0.3327 (0.3498) loss 2.1618 (2.6974) grad_norm 2.5490 (2.9304) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [257/300][620/625] eta 0:00:01 lr 0.000128 wd 0.0500 time 0.3456 (0.3505) data time 0.0004 (0.0014) model time 0.3453 (0.3496) loss 2.5418 (2.6964) grad_norm 2.1988 (2.9217) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:02 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 257 training takes 0:03:39 [2024-09-01 13:00:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:00:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.447 (0.447) Loss 0.5156 (0.5156) Acc@1 88.477 (88.477) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 13:00:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.8403 (0.6109) Acc@1 80.371 (86.679) Acc@5 95.508 (97.701) Mem 11224MB [2024-09-01 13:00:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.111) Loss 0.9180 (0.7288) Acc@1 79.102 (83.791) Acc@5 94.775 (96.494) Mem 11224MB [2024-09-01 13:00:06 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.457 Acc@5 96.503 [2024-09-01 13:00:06 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 13:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.967 (0.967) Loss 0.4922 (0.4922) Acc@1 89.404 (89.404) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 13:00:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.178) Loss 0.8052 (0.5930) Acc@1 80.908 (87.163) Acc@5 95.850 (97.865) Mem 11224MB [2024-09-01 13:00:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.089 (0.136) Loss 0.8804 (0.7035) Acc@1 79.443 (84.370) Acc@5 95.215 (96.749) Mem 11224MB [2024-09-01 13:00:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.069 Acc@5 96.767 [2024-09-01 13:00:09 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.1% [2024-09-01 13:00:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][0/625] eta 0:12:31 lr 0.000128 wd 0.0500 time 1.2017 (1.2017) data time 0.4567 (0.4567) model time 0.0000 (0.0000) loss 3.0936 (3.0936) grad_norm 1.9614 (1.9614) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][10/625] eta 0:04:19 lr 0.000128 wd 0.0500 time 0.3478 (0.4220) data time 0.0008 (0.0423) model time 0.0000 (0.0000) loss 2.6060 (2.7960) grad_norm 2.7453 (2.7505) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][20/625] eta 0:03:53 lr 0.000128 wd 0.0500 time 0.3406 (0.3864) data time 0.0006 (0.0226) model time 0.0000 (0.0000) loss 2.5042 (2.6898) grad_norm 2.3248 (2.5297) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][30/625] eta 0:03:42 lr 0.000128 wd 0.0500 time 0.3470 (0.3737) data time 0.0006 (0.0155) model time 0.0000 (0.0000) loss 2.5183 (2.7064) grad_norm 3.1468 (2.5028) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][40/625] eta 0:03:34 lr 0.000128 wd 0.0500 time 0.3448 (0.3671) data time 0.0008 (0.0119) model time 0.0000 (0.0000) loss 2.4596 (2.7311) grad_norm 2.6208 (2.5906) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][50/625] eta 0:03:28 lr 0.000128 wd 0.0500 time 0.3424 (0.3631) data time 0.0008 (0.0098) model time 0.0000 (0.0000) loss 2.6217 (2.6936) grad_norm 3.0731 (2.5795) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][60/625] eta 0:03:23 lr 0.000127 wd 0.0500 time 0.3401 (0.3601) data time 0.0006 (0.0083) model time 0.3394 (0.3442) loss 1.8689 (2.6476) grad_norm 7.2316 (2.6493) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][70/625] eta 0:03:18 lr 0.000127 wd 0.0500 time 0.3478 (0.3582) data time 0.0008 (0.0072) model time 0.3470 (0.3449) loss 1.6546 (2.6437) grad_norm 2.3743 (2.6649) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][80/625] eta 0:03:14 lr 0.000127 wd 0.0500 time 0.3474 (0.3568) data time 0.0006 (0.0064) model time 0.3468 (0.3454) loss 2.9420 (2.6508) grad_norm 1.7848 (3.0739) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][90/625] eta 0:03:14 lr 0.000127 wd 0.0500 time 0.5335 (0.3635) data time 0.0006 (0.0058) model time 0.5329 (0.3632) loss 3.1351 (2.6714) grad_norm 3.3887 (3.0274) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][100/625] eta 0:03:09 lr 0.000127 wd 0.0500 time 0.3416 (0.3615) data time 0.0010 (0.0053) model time 0.3406 (0.3590) loss 2.4166 (2.6712) grad_norm 2.2958 (2.9893) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][110/625] eta 0:03:05 lr 0.000127 wd 0.0500 time 0.3460 (0.3600) data time 0.0008 (0.0049) model time 0.3453 (0.3566) loss 2.9650 (2.6754) grad_norm 1.8819 (2.9019) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][120/625] eta 0:03:01 lr 0.000127 wd 0.0500 time 0.3449 (0.3589) data time 0.0006 (0.0046) model time 0.3443 (0.3550) loss 1.7281 (2.6713) grad_norm 2.7425 (2.8609) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][130/625] eta 0:02:57 lr 0.000127 wd 0.0500 time 0.3500 (0.3579) data time 0.0007 (0.0043) model time 0.3492 (0.3538) loss 2.2518 (2.6433) grad_norm 2.5318 (2.8283) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][140/625] eta 0:02:53 lr 0.000127 wd 0.0500 time 0.3505 (0.3570) data time 0.0008 (0.0041) model time 0.3496 (0.3528) loss 2.7987 (2.6336) grad_norm 2.1182 (2.8224) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:01:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][150/625] eta 0:02:49 lr 0.000127 wd 0.0500 time 0.5351 (0.3575) data time 0.0007 (0.0039) model time 0.5344 (0.3538) loss 2.5807 (2.6311) grad_norm 2.1715 (2.7987) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][160/625] eta 0:02:45 lr 0.000127 wd 0.0500 time 0.3531 (0.3568) data time 0.0007 (0.0037) model time 0.3523 (0.3531) loss 1.7060 (2.6141) grad_norm 2.7756 (2.7969) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:01:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][170/625] eta 0:02:42 lr 0.000127 wd 0.0500 time 0.3460 (0.3563) data time 0.0006 (0.0035) model time 0.3454 (0.3525) loss 1.5408 (2.6149) grad_norm 2.5835 (2.7878) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:01:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][180/625] eta 0:02:38 lr 0.000126 wd 0.0500 time 0.3481 (0.3557) data time 0.0006 (0.0033) model time 0.3475 (0.3519) loss 3.2328 (2.6180) grad_norm 2.1313 (2.7705) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:01:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][190/625] eta 0:02:34 lr 0.000126 wd 0.0500 time 0.3500 (0.3553) data time 0.0005 (0.0032) model time 0.3494 (0.3517) loss 1.7446 (2.6073) grad_norm 2.3793 (2.7834) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:01:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][200/625] eta 0:02:30 lr 0.000126 wd 0.0500 time 0.3481 (0.3550) data time 0.0007 (0.0031) model time 0.3474 (0.3514) loss 2.9174 (2.6131) grad_norm 2.6032 (2.7585) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:01:24 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 13:01:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:01:24 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 13:10:04 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 13:10:08 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 13:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 13:10:26 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 13:10:28 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 13:10:30 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 13:10:30 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 258) [2024-09-01 13:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 13:10:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][210/625] eta 0:47:21 lr 0.000126 wd 0.0500 time 0.4690 (6.8469) data time 0.0009 (0.3549) model time 0.4680 (6.4921) loss 3.2092 (3.2284) grad_norm 2.3798 (5.9629) loss_scale 128.0000 (128.0000) mem 11217MB [2024-09-01 13:10:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][220/625] eta 0:09:38 lr 0.000126 wd 0.0500 time 0.3494 (1.4295) data time 0.0006 (0.0598) model time 0.3488 (1.3697) loss 2.2527 (2.8547) grad_norm 2.6639 (3.9178) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:10:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][230/625] eta 0:06:09 lr 0.000126 wd 0.0500 time 0.3519 (0.9360) data time 0.0008 (0.0331) model time 0.3511 (0.9029) loss 2.6386 (2.8737) grad_norm 2.5304 (3.6735) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:10:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][240/625] eta 0:04:52 lr 0.000126 wd 0.0500 time 0.3412 (0.7585) data time 0.0006 (0.0230) model time 0.3406 (0.7355) loss 2.8119 (2.8784) grad_norm 2.1436 (3.3229) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][250/625] eta 0:04:10 lr 0.000126 wd 0.0500 time 0.3407 (0.6671) data time 0.0008 (0.0177) model time 0.3399 (0.6494) loss 3.0072 (2.8438) grad_norm 2.7501 (3.0704) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][260/625] eta 0:03:40 lr 0.000126 wd 0.0500 time 0.3470 (0.6054) data time 0.0006 (0.0145) model time 0.3463 (0.5910) loss 3.0612 (2.8322) grad_norm 2.2085 (3.5047) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][270/625] eta 0:03:20 lr 0.000126 wd 0.0500 time 0.3470 (0.5636) data time 0.0007 (0.0123) model time 0.3462 (0.5513) loss 3.1237 (2.7930) grad_norm 2.0670 (3.3064) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][280/625] eta 0:03:03 lr 0.000126 wd 0.0500 time 0.3392 (0.5333) data time 0.0009 (0.0107) model time 0.3383 (0.5226) loss 2.6460 (2.7595) grad_norm 2.2310 (3.1732) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][290/625] eta 0:02:50 lr 0.000126 wd 0.0500 time 0.3474 (0.5103) data time 0.0009 (0.0095) model time 0.3465 (0.5008) loss 2.7471 (2.7502) grad_norm 1.8875 (3.0811) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][300/625] eta 0:02:39 lr 0.000125 wd 0.0500 time 0.3462 (0.4923) data time 0.0007 (0.0085) model time 0.3454 (0.4837) loss 1.6914 (2.7305) grad_norm 2.0539 (2.9969) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][310/625] eta 0:02:30 lr 0.000125 wd 0.0500 time 0.3388 (0.4778) data time 0.0007 (0.0078) model time 0.3381 (0.4700) loss 3.0273 (2.7515) grad_norm 2.3239 (2.9474) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][320/625] eta 0:02:22 lr 0.000125 wd 0.0500 time 0.3427 (0.4660) data time 0.0009 (0.0072) model time 0.3418 (0.4588) loss 3.1317 (2.7598) grad_norm 1.4825 (2.9384) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][330/625] eta 0:02:14 lr 0.000125 wd 0.0500 time 0.3383 (0.4559) data time 0.0007 (0.0066) model time 0.3376 (0.4493) loss 2.8663 (2.7609) grad_norm 1.8345 (2.9254) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][340/625] eta 0:02:07 lr 0.000125 wd 0.0500 time 0.3379 (0.4477) data time 0.0008 (0.0062) model time 0.3370 (0.4415) loss 3.1783 (2.7587) grad_norm 2.8421 (2.9102) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][350/625] eta 0:02:01 lr 0.000125 wd 0.0500 time 0.3450 (0.4404) data time 0.0009 (0.0058) model time 0.3441 (0.4345) loss 2.8727 (2.7514) grad_norm 2.7543 (2.8979) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][360/625] eta 0:01:55 lr 0.000125 wd 0.0500 time 0.3459 (0.4341) data time 0.0008 (0.0055) model time 0.3451 (0.4286) loss 3.1041 (2.7430) grad_norm 2.4577 (2.8675) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][370/625] eta 0:01:49 lr 0.000125 wd 0.0500 time 0.3466 (0.4286) data time 0.0008 (0.0052) model time 0.3458 (0.4234) loss 2.6877 (2.7414) grad_norm 1.6691 (2.8358) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][380/625] eta 0:01:43 lr 0.000125 wd 0.0500 time 0.3469 (0.4238) data time 0.0008 (0.0050) model time 0.3460 (0.4188) loss 2.6834 (2.7368) grad_norm 2.6534 (2.8416) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][390/625] eta 0:01:38 lr 0.000125 wd 0.0500 time 0.3456 (0.4194) data time 0.0008 (0.0047) model time 0.3448 (0.4147) loss 3.3611 (2.7284) grad_norm 1.7356 (2.8258) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][400/625] eta 0:01:33 lr 0.000125 wd 0.0500 time 0.3468 (0.4157) data time 0.0008 (0.0045) model time 0.3460 (0.4111) loss 2.8985 (2.7287) grad_norm 3.9907 (2.8716) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:11:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][410/625] eta 0:01:28 lr 0.000125 wd 0.0500 time 0.3468 (0.4123) data time 0.0006 (0.0043) model time 0.3462 (0.4079) loss 3.1269 (2.7225) grad_norm 1.7849 (2.8551) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][420/625] eta 0:01:23 lr 0.000125 wd 0.0500 time 0.3434 (0.4091) data time 0.0008 (0.0042) model time 0.3427 (0.4050) loss 2.9889 (2.7176) grad_norm 1.8788 (2.8384) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][430/625] eta 0:01:19 lr 0.000124 wd 0.0500 time 0.3451 (0.4063) data time 0.0007 (0.0040) model time 0.3443 (0.4023) loss 3.1245 (2.7138) grad_norm 2.6106 (2.8178) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][440/625] eta 0:01:15 lr 0.000124 wd 0.0500 time 0.3452 (0.4056) data time 0.0006 (0.0039) model time 0.3446 (0.4017) loss 2.9663 (2.7061) grad_norm 2.5915 (2.7963) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][450/625] eta 0:01:10 lr 0.000124 wd 0.0500 time 0.3476 (0.4039) data time 0.0006 (0.0038) model time 0.3470 (0.4001) loss 3.1140 (2.7081) grad_norm 2.8411 (2.7879) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][460/625] eta 0:01:06 lr 0.000124 wd 0.0500 time 0.3512 (0.4016) data time 0.0006 (0.0036) model time 0.3505 (0.3980) loss 2.9948 (2.6961) grad_norm 2.9777 (2.7750) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][470/625] eta 0:01:01 lr 0.000124 wd 0.0500 time 0.3496 (0.3995) data time 0.0008 (0.0035) model time 0.3488 (0.3960) loss 2.6597 (2.6877) grad_norm 2.2952 (2.7763) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][480/625] eta 0:00:57 lr 0.000124 wd 0.0500 time 0.3446 (0.3975) data time 0.0008 (0.0034) model time 0.3438 (0.3941) loss 2.6040 (2.6797) grad_norm 2.4105 (2.7772) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][490/625] eta 0:00:53 lr 0.000124 wd 0.0500 time 0.3436 (0.3957) data time 0.0009 (0.0033) model time 0.3427 (0.3924) loss 1.8647 (2.6793) grad_norm 2.2278 (2.7696) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][500/625] eta 0:00:49 lr 0.000124 wd 0.0500 time 0.3472 (0.3940) data time 0.0009 (0.0033) model time 0.3463 (0.3907) loss 2.5787 (2.6725) grad_norm 1.8866 (2.7566) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][510/625] eta 0:00:45 lr 0.000124 wd 0.0500 time 0.3454 (0.3924) data time 0.0008 (0.0032) model time 0.3446 (0.3892) loss 2.7321 (2.6652) grad_norm 2.9960 (2.7521) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][520/625] eta 0:00:41 lr 0.000124 wd 0.0500 time 0.3459 (0.3910) data time 0.0009 (0.0031) model time 0.3451 (0.3879) loss 2.9736 (2.6647) grad_norm 2.9381 (2.7619) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][530/625] eta 0:00:37 lr 0.000124 wd 0.0500 time 0.3471 (0.3896) data time 0.0009 (0.0030) model time 0.3461 (0.3866) loss 2.8487 (2.6722) grad_norm 1.8121 (2.8147) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][540/625] eta 0:00:33 lr 0.000124 wd 0.0500 time 0.3437 (0.3883) data time 0.0010 (0.0030) model time 0.3428 (0.3853) loss 2.6226 (2.6708) grad_norm 2.0840 (2.8088) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][550/625] eta 0:00:29 lr 0.000124 wd 0.0500 time 0.3490 (0.3870) data time 0.0007 (0.0029) model time 0.3483 (0.3841) loss 3.0064 (2.6741) grad_norm 2.1158 (2.8073) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][560/625] eta 0:00:25 lr 0.000123 wd 0.0500 time 0.3501 (0.3872) data time 0.0007 (0.0028) model time 0.3494 (0.3843) loss 3.4892 (2.6767) grad_norm 2.0396 (2.8027) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][570/625] eta 0:00:21 lr 0.000123 wd 0.0500 time 0.3474 (0.3861) data time 0.0006 (0.0028) model time 0.3469 (0.3833) loss 3.3789 (2.6757) grad_norm 2.7278 (2.8010) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:12:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][580/625] eta 0:00:17 lr 0.000123 wd 0.0500 time 0.3470 (0.3850) data time 0.0008 (0.0027) model time 0.3462 (0.3822) loss 2.4910 (2.6749) grad_norm 1.7673 (2.7964) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:13:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][590/625] eta 0:00:13 lr 0.000123 wd 0.0500 time 0.3430 (0.3839) data time 0.0006 (0.0027) model time 0.3424 (0.3812) loss 3.3859 (2.6751) grad_norm 2.3067 (2.7954) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:13:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][600/625] eta 0:00:09 lr 0.000123 wd 0.0500 time 0.3480 (0.3830) data time 0.0006 (0.0026) model time 0.3474 (0.3804) loss 2.4943 (2.6689) grad_norm 4.4450 (2.7922) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:13:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][610/625] eta 0:00:05 lr 0.000123 wd 0.0500 time 0.3377 (0.3820) data time 0.0004 (0.0026) model time 0.3373 (0.3794) loss 3.4010 (2.6736) grad_norm 3.5732 (2.8001) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:13:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [258/300][620/625] eta 0:00:01 lr 0.000123 wd 0.0500 time 0.3377 (0.3810) data time 0.0006 (0.0026) model time 0.3372 (0.3785) loss 2.7734 (2.6773) grad_norm 1.8004 (2.7904) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:13:13 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 258 training takes 0:02:38 [2024-09-01 13:13:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:13:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:13:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.439 (0.439) Loss 0.5190 (0.5190) Acc@1 88.574 (88.574) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 13:13:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.8398 (0.6166) Acc@1 80.518 (86.745) Acc@5 95.947 (97.781) Mem 11219MB [2024-09-01 13:13:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.110) Loss 0.9175 (0.7306) Acc@1 79.639 (83.859) Acc@5 95.215 (96.619) Mem 11219MB [2024-09-01 13:13:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.553 Acc@5 96.617 [2024-09-01 13:13:20 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 13:13:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.805 (0.805) Loss 0.4924 (0.4924) Acc@1 89.355 (89.355) Acc@5 98.682 (98.682) Mem 11219MB [2024-09-01 13:13:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.166) Loss 0.8062 (0.5932) Acc@1 81.055 (87.211) Acc@5 95.752 (97.847) Mem 11219MB [2024-09-01 13:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.130) Loss 0.8818 (0.7040) Acc@1 79.541 (84.405) Acc@5 95.215 (96.747) Mem 11219MB [2024-09-01 13:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.111 Acc@5 96.771 [2024-09-01 13:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.1% [2024-09-01 13:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 84.11% [2024-09-01 13:13:23 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 13:13:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 13:13:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][0/625] eta 0:08:03 lr 0.000123 wd 0.0500 time 0.7741 (0.7741) data time 0.4226 (0.4226) model time 0.0000 (0.0000) loss 1.9797 (1.9797) grad_norm 2.6178 (2.6178) loss_scale 128.0000 (128.0000) mem 11227MB [2024-09-01 13:13:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][10/625] eta 0:03:56 lr 0.000123 wd 0.0500 time 0.3450 (0.3852) data time 0.0008 (0.0391) model time 0.0000 (0.0000) loss 2.9609 (2.8342) grad_norm 1.7564 (2.3128) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:13:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][20/625] eta 0:03:42 lr 0.000123 wd 0.0500 time 0.3575 (0.3670) data time 0.0008 (0.0208) model time 0.0000 (0.0000) loss 2.9191 (2.8640) grad_norm 1.7483 (2.5253) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:13:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][30/625] eta 0:03:34 lr 0.000123 wd 0.0500 time 0.3464 (0.3607) data time 0.0008 (0.0144) model time 0.0000 (0.0000) loss 2.5194 (2.8190) grad_norm 1.7177 (2.5053) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:13:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][40/625] eta 0:03:29 lr 0.000123 wd 0.0500 time 0.3397 (0.3576) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 1.7231 (2.7266) grad_norm 2.5852 (2.5208) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:13:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][50/625] eta 0:03:24 lr 0.000123 wd 0.0500 time 0.3420 (0.3554) data time 0.0008 (0.0091) model time 0.0000 (0.0000) loss 2.1803 (2.6537) grad_norm 2.1471 (2.5146) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:13:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][60/625] eta 0:03:19 lr 0.000122 wd 0.0500 time 0.3507 (0.3539) data time 0.0007 (0.0077) model time 0.3501 (0.3451) loss 3.0392 (2.6489) grad_norm 3.7936 (2.6118) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:13:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][70/625] eta 0:03:15 lr 0.000122 wd 0.0500 time 0.3475 (0.3528) data time 0.0007 (0.0068) model time 0.3467 (0.3452) loss 2.5938 (2.6654) grad_norm 3.4937 (2.6541) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:13:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][80/625] eta 0:03:14 lr 0.000122 wd 0.0500 time 0.3483 (0.3567) data time 0.0007 (0.0061) model time 0.3476 (0.3578) loss 2.9246 (2.6522) grad_norm 3.0793 (2.8469) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:13:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][90/625] eta 0:03:10 lr 0.000122 wd 0.0500 time 0.3523 (0.3554) data time 0.0010 (0.0055) model time 0.3514 (0.3545) loss 2.6430 (2.6558) grad_norm 5.0043 (2.8656) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:14:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][100/625] eta 0:03:06 lr 0.000122 wd 0.0500 time 0.3496 (0.3544) data time 0.0008 (0.0050) model time 0.3488 (0.3524) loss 2.9068 (2.6837) grad_norm 3.2396 (2.8577) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:14:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][110/625] eta 0:03:02 lr 0.000122 wd 0.0500 time 0.3424 (0.3536) data time 0.0008 (0.0046) model time 0.3416 (0.3511) loss 2.8713 (2.6550) grad_norm 3.4368 (2.8707) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:14:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][120/625] eta 0:02:58 lr 0.000122 wd 0.0500 time 0.3468 (0.3528) data time 0.0006 (0.0043) model time 0.3462 (0.3501) loss 1.8656 (2.6391) grad_norm 6.8416 (2.8691) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:14:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][130/625] eta 0:02:55 lr 0.000122 wd 0.0500 time 0.5423 (0.3538) data time 0.0007 (0.0041) model time 0.5416 (0.3519) loss 3.1330 (2.6503) grad_norm 2.4742 (2.8396) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:14:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][140/625] eta 0:02:51 lr 0.000122 wd 0.0500 time 0.3456 (0.3530) data time 0.0007 (0.0038) model time 0.3448 (0.3508) loss 1.5452 (2.6654) grad_norm 2.7709 (2.8319) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:14:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][150/625] eta 0:02:47 lr 0.000122 wd 0.0500 time 0.3484 (0.3525) data time 0.0009 (0.0036) model time 0.3476 (0.3501) loss 2.9000 (2.6814) grad_norm 3.2671 (2.8183) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:14:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][160/625] eta 0:02:43 lr 0.000122 wd 0.0500 time 0.3463 (0.3521) data time 0.0006 (0.0035) model time 0.3457 (0.3497) loss 3.4830 (2.6865) grad_norm 3.5101 (2.8228) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 13:14:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:14:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:25:16 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 13:25:18 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 13:25:32 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 13:25:43 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 13:25:43 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 13:25:46 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 13:25:48 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 13:25:48 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 259) [2024-09-01 13:25:48 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 13:26:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][170/625] eta 1:03:59 lr 0.000122 wd 0.0500 time 0.5585 (8.4378) data time 0.0010 (0.3395) model time 0.5576 (8.0982) loss 3.2209 (3.1446) grad_norm 1.8744 (2.8208) loss_scale 128.0000 (128.0000) mem 11217MB [2024-09-01 13:26:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][180/625] eta 0:12:45 lr 0.000122 wd 0.0500 time 0.3910 (1.7192) data time 0.0007 (0.0574) model time 0.3903 (1.6618) loss 2.0717 (2.7746) grad_norm 1.6761 (2.2451) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][190/625] eta 0:08:02 lr 0.000121 wd 0.0500 time 0.3859 (1.1090) data time 0.0009 (0.0317) model time 0.3850 (1.0772) loss 2.8180 (2.8096) grad_norm 2.0905 (2.2872) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][200/625] eta 0:06:16 lr 0.000121 wd 0.0500 time 0.3741 (0.8861) data time 0.0008 (0.0221) model time 0.3733 (0.8640) loss 2.7344 (2.8068) grad_norm 2.1937 (2.2944) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][210/625] eta 0:05:19 lr 0.000121 wd 0.0500 time 0.3689 (0.7703) data time 0.0010 (0.0171) model time 0.3679 (0.7532) loss 2.7747 (2.7669) grad_norm 2.4471 (2.3558) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][220/625] eta 0:04:41 lr 0.000121 wd 0.0500 time 0.3848 (0.6940) data time 0.0007 (0.0140) model time 0.3841 (0.6801) loss 3.0092 (2.7695) grad_norm 2.3446 (2.6416) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][230/625] eta 0:04:13 lr 0.000121 wd 0.0500 time 0.3669 (0.6423) data time 0.0009 (0.0119) model time 0.3660 (0.6304) loss 3.6593 (2.7588) grad_norm 1.8111 (3.0220) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][240/625] eta 0:03:52 lr 0.000121 wd 0.0500 time 0.3720 (0.6050) data time 0.0010 (0.0104) model time 0.3710 (0.5946) loss 2.6177 (2.7272) grad_norm 2.2782 (2.9690) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][250/625] eta 0:03:36 lr 0.000121 wd 0.0500 time 0.3677 (0.5763) data time 0.0010 (0.0093) model time 0.3667 (0.5671) loss 2.8448 (2.7237) grad_norm 2.1334 (2.8681) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][260/625] eta 0:03:22 lr 0.000121 wd 0.0500 time 0.3644 (0.5540) data time 0.0008 (0.0084) model time 0.3636 (0.5456) loss 2.0027 (2.7146) grad_norm 19.0089 (3.0074) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][270/625] eta 0:03:10 lr 0.000121 wd 0.0500 time 0.3630 (0.5361) data time 0.0008 (0.0076) model time 0.3622 (0.5284) loss 3.0450 (2.7373) grad_norm 3.2885 (2.9583) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][280/625] eta 0:02:59 lr 0.000121 wd 0.0500 time 0.3725 (0.5215) data time 0.0010 (0.0070) model time 0.3715 (0.5145) loss 3.1087 (2.7348) grad_norm 1.9777 (2.8686) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][290/625] eta 0:02:50 lr 0.000121 wd 0.0500 time 0.3698 (0.5094) data time 0.0008 (0.0065) model time 0.3690 (0.5028) loss 2.7764 (2.7390) grad_norm 2.2845 (2.8611) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:26:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][300/625] eta 0:02:42 lr 0.000121 wd 0.0500 time 0.3807 (0.4992) data time 0.0010 (0.0061) model time 0.3797 (0.4930) loss 2.7651 (2.7330) grad_norm 2.9623 (2.8290) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][310/625] eta 0:02:34 lr 0.000120 wd 0.0500 time 0.3729 (0.4905) data time 0.0010 (0.0058) model time 0.3719 (0.4847) loss 2.6730 (2.7237) grad_norm 3.3140 (2.8195) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][320/625] eta 0:02:27 lr 0.000120 wd 0.0500 time 0.3717 (0.4828) data time 0.0010 (0.0054) model time 0.3707 (0.4774) loss 2.6576 (2.7196) grad_norm 3.2485 (2.8242) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][330/625] eta 0:02:20 lr 0.000120 wd 0.0500 time 0.3701 (0.4760) data time 0.0010 (0.0052) model time 0.3691 (0.4709) loss 3.0971 (2.7242) grad_norm 2.4862 (2.8074) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][340/625] eta 0:02:13 lr 0.000120 wd 0.0500 time 0.3697 (0.4699) data time 0.0010 (0.0049) model time 0.3687 (0.4650) loss 2.2747 (2.7154) grad_norm 2.7073 (2.7888) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][350/625] eta 0:02:07 lr 0.000120 wd 0.0500 time 0.3691 (0.4646) data time 0.0010 (0.0047) model time 0.3681 (0.4598) loss 2.7981 (2.7046) grad_norm 2.1739 (2.8078) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][360/625] eta 0:02:01 lr 0.000120 wd 0.0500 time 0.3718 (0.4598) data time 0.0011 (0.0045) model time 0.3708 (0.4552) loss 3.0264 (2.7002) grad_norm 1.9709 (2.7806) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][370/625] eta 0:01:56 lr 0.000120 wd 0.0500 time 0.3716 (0.4554) data time 0.0008 (0.0044) model time 0.3707 (0.4510) loss 2.8914 (2.6900) grad_norm 1.8472 (2.7541) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][380/625] eta 0:01:50 lr 0.000120 wd 0.0500 time 0.3695 (0.4515) data time 0.0010 (0.0042) model time 0.3685 (0.4473) loss 2.7977 (2.6840) grad_norm 2.0352 (2.7354) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][390/625] eta 0:01:45 lr 0.000120 wd 0.0500 time 0.3687 (0.4480) data time 0.0009 (0.0041) model time 0.3678 (0.4439) loss 3.4016 (2.6815) grad_norm 2.0355 (2.7333) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][400/625] eta 0:01:40 lr 0.000120 wd 0.0500 time 0.3719 (0.4465) data time 0.0007 (0.0039) model time 0.3712 (0.4426) loss 2.6442 (2.6768) grad_norm 2.3108 (2.7137) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][410/625] eta 0:01:35 lr 0.000120 wd 0.0500 time 0.3698 (0.4441) data time 0.0008 (0.0038) model time 0.3690 (0.4403) loss 3.2014 (2.6800) grad_norm 1.8446 (2.7022) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][420/625] eta 0:01:30 lr 0.000120 wd 0.0500 time 0.3699 (0.4412) data time 0.0007 (0.0037) model time 0.3692 (0.4375) loss 3.0731 (2.6722) grad_norm 2.8993 (2.6961) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][430/625] eta 0:01:25 lr 0.000120 wd 0.0500 time 0.3778 (0.4385) data time 0.0011 (0.0036) model time 0.3768 (0.4349) loss 2.7480 (2.6665) grad_norm 2.3099 (2.7231) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][440/625] eta 0:01:20 lr 0.000119 wd 0.0500 time 0.3811 (0.4361) data time 0.0009 (0.0035) model time 0.3802 (0.4326) loss 2.6864 (2.6590) grad_norm 2.3518 (2.7157) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][450/625] eta 0:01:15 lr 0.000119 wd 0.0500 time 0.3706 (0.4338) data time 0.0010 (0.0034) model time 0.3696 (0.4304) loss 1.7883 (2.6613) grad_norm 1.5097 (2.7159) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:27:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][460/625] eta 0:01:11 lr 0.000119 wd 0.0500 time 0.3693 (0.4316) data time 0.0009 (0.0033) model time 0.3683 (0.4283) loss 2.7319 (2.6601) grad_norm 2.1763 (2.7056) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][470/625] eta 0:01:06 lr 0.000119 wd 0.0500 time 0.3719 (0.4297) data time 0.0010 (0.0032) model time 0.3710 (0.4265) loss 2.7801 (2.6544) grad_norm 1.7468 (2.6953) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][480/625] eta 0:01:02 lr 0.000119 wd 0.0500 time 0.3724 (0.4280) data time 0.0011 (0.0032) model time 0.3714 (0.4249) loss 3.0414 (2.6561) grad_norm 2.2296 (2.7216) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][490/625] eta 0:00:57 lr 0.000119 wd 0.0500 time 0.3857 (0.4264) data time 0.0010 (0.0031) model time 0.3848 (0.4233) loss 3.0113 (2.6648) grad_norm 4.4373 (2.7225) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][500/625] eta 0:00:53 lr 0.000119 wd 0.0500 time 0.3758 (0.4249) data time 0.0009 (0.0030) model time 0.3750 (0.4218) loss 3.1406 (2.6636) grad_norm 3.2723 (2.8115) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][510/625] eta 0:00:48 lr 0.000119 wd 0.0500 time 0.3774 (0.4234) data time 0.0008 (0.0030) model time 0.3767 (0.4205) loss 3.0499 (2.6680) grad_norm 17.5706 (2.8834) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][520/625] eta 0:00:44 lr 0.000119 wd 0.0500 time 0.3629 (0.4232) data time 0.0008 (0.0029) model time 0.3621 (0.4202) loss 2.9117 (2.6688) grad_norm 2.2393 (2.8745) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][530/625] eta 0:00:40 lr 0.000119 wd 0.0500 time 0.3761 (0.4217) data time 0.0007 (0.0029) model time 0.3753 (0.4188) loss 3.2544 (2.6715) grad_norm 2.3832 (2.8604) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][540/625] eta 0:00:35 lr 0.000119 wd 0.0500 time 0.3703 (0.4204) data time 0.0009 (0.0028) model time 0.3693 (0.4176) loss 2.9365 (2.6690) grad_norm 12.9951 (2.9068) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][550/625] eta 0:00:31 lr 0.000119 wd 0.0500 time 0.3684 (0.4193) data time 0.0008 (0.0028) model time 0.3676 (0.4165) loss 3.3834 (2.6704) grad_norm 5.3032 (2.9043) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][560/625] eta 0:00:27 lr 0.000119 wd 0.0500 time 0.3684 (0.4180) data time 0.0007 (0.0027) model time 0.3677 (0.4153) loss 2.3045 (2.6659) grad_norm 2.4925 (2.9018) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][570/625] eta 0:00:22 lr 0.000118 wd 0.0500 time 0.3700 (0.4168) data time 0.0008 (0.0027) model time 0.3693 (0.4141) loss 3.4922 (2.6731) grad_norm 2.1510 (2.8915) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][580/625] eta 0:00:18 lr 0.000118 wd 0.0500 time 0.3727 (0.4157) data time 0.0009 (0.0026) model time 0.3718 (0.4131) loss 2.9601 (2.6761) grad_norm 11.5257 (2.9331) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][590/625] eta 0:00:14 lr 0.000118 wd 0.0500 time 0.3861 (0.4151) data time 0.0008 (0.0026) model time 0.3853 (0.4125) loss 2.7485 (2.6735) grad_norm 2.4635 (2.9250) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][600/625] eta 0:00:10 lr 0.000118 wd 0.0500 time 0.3710 (0.4142) data time 0.0008 (0.0025) model time 0.3703 (0.4117) loss 2.5267 (2.6772) grad_norm 2.2602 (2.9199) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][610/625] eta 0:00:06 lr 0.000118 wd 0.0500 time 0.3677 (0.4133) data time 0.0005 (0.0025) model time 0.3672 (0.4108) loss 2.6417 (2.6787) grad_norm 4.2239 (2.9210) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:28:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [259/300][620/625] eta 0:00:02 lr 0.000118 wd 0.0500 time 0.3700 (0.4123) data time 0.0007 (0.0025) model time 0.3693 (0.4098) loss 2.1256 (2.6760) grad_norm 1.5191 (2.9247) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 13:29:00 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 259 training takes 0:03:07 [2024-09-01 13:29:00 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:29:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:29:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.483 (0.483) Loss 0.5205 (0.5205) Acc@1 88.477 (88.477) Acc@5 98.535 (98.535) Mem 11219MB [2024-09-01 13:29:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.140) Loss 0.8560 (0.6224) Acc@1 80.078 (86.626) Acc@5 95.703 (97.727) Mem 11219MB [2024-09-01 13:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 0.9150 (0.7381) Acc@1 78.662 (83.691) Acc@5 95.215 (96.545) Mem 11219MB [2024-09-01 13:29:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.353 Acc@5 96.513 [2024-09-01 13:29:09 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 13:29:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.829 (0.829) Loss 0.4929 (0.4929) Acc@1 89.209 (89.209) Acc@5 98.682 (98.682) Mem 11219MB [2024-09-01 13:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.174) Loss 0.8062 (0.5934) Acc@1 81.201 (87.216) Acc@5 95.850 (97.874) Mem 11219MB [2024-09-01 13:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.136) Loss 0.8828 (0.7043) Acc@1 79.590 (84.422) Acc@5 95.312 (96.780) Mem 11219MB [2024-09-01 13:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.121 Acc@5 96.795 [2024-09-01 13:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.1% [2024-09-01 13:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 84.12% [2024-09-01 13:29:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 13:29:15 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 13:29:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][0/625] eta 0:08:30 lr 0.000118 wd 0.0500 time 0.8165 (0.8165) data time 0.4309 (0.4309) model time 0.0000 (0.0000) loss 1.8313 (1.8313) grad_norm 2.4990 (2.4990) loss_scale 128.0000 (128.0000) mem 11227MB [2024-09-01 13:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][10/625] eta 0:04:13 lr 0.000118 wd 0.0500 time 0.3712 (0.4127) data time 0.0010 (0.0400) model time 0.0000 (0.0000) loss 2.3670 (2.3745) grad_norm 2.1251 (4.1973) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][20/625] eta 0:03:58 lr 0.000118 wd 0.0500 time 0.3676 (0.3940) data time 0.0008 (0.0214) model time 0.0000 (0.0000) loss 2.8892 (2.4758) grad_norm 2.9727 (3.4591) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:29:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][30/625] eta 0:03:50 lr 0.000118 wd 0.0500 time 0.3727 (0.3878) data time 0.0010 (0.0148) model time 0.0000 (0.0000) loss 2.6817 (2.5595) grad_norm 2.3728 (3.2179) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:29:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][40/625] eta 0:03:44 lr 0.000118 wd 0.0500 time 0.3781 (0.3843) data time 0.0007 (0.0115) model time 0.0000 (0.0000) loss 2.9588 (2.5686) grad_norm 4.8481 (3.1294) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 13:29:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][50/625] eta 0:03:42 lr 0.000118 wd 0.0500 time 0.3937 (0.3870) data time 0.0009 (0.0094) model time 0.0000 (0.0000) loss 2.5037 (2.6051) grad_norm 3.5071 (3.0623) loss_scale 256.0000 (140.5490) mem 11224MB [2024-09-01 13:29:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][60/625] eta 0:03:41 lr 0.000118 wd 0.0500 time 0.3924 (0.3916) data time 0.0010 (0.0080) model time 0.3914 (0.4140) loss 2.7339 (2.6660) grad_norm 2.0756 (3.0566) loss_scale 256.0000 (159.4754) mem 11224MB [2024-09-01 13:29:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][70/625] eta 0:03:35 lr 0.000118 wd 0.0500 time 0.3746 (0.3892) data time 0.0009 (0.0070) model time 0.3737 (0.3938) loss 2.3761 (2.6195) grad_norm 3.5309 (2.9810) loss_scale 256.0000 (173.0704) mem 11224MB [2024-09-01 13:29:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][80/625] eta 0:03:30 lr 0.000117 wd 0.0500 time 0.3719 (0.3871) data time 0.0007 (0.0063) model time 0.3711 (0.3864) loss 1.8266 (2.6006) grad_norm 2.6642 (3.0187) loss_scale 256.0000 (183.3086) mem 11224MB [2024-09-01 13:29:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][90/625] eta 0:03:26 lr 0.000117 wd 0.0500 time 0.3728 (0.3854) data time 0.0008 (0.0057) model time 0.3721 (0.3825) loss 3.0959 (2.6174) grad_norm 2.8138 (2.9808) loss_scale 256.0000 (191.2967) mem 11224MB [2024-09-01 13:29:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][100/625] eta 0:03:21 lr 0.000117 wd 0.0500 time 0.3704 (0.3841) data time 0.0009 (0.0052) model time 0.3695 (0.3803) loss 2.0089 (2.6284) grad_norm 2.0965 (2.9249) loss_scale 256.0000 (197.7030) mem 11224MB [2024-09-01 13:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][110/625] eta 0:03:17 lr 0.000117 wd 0.0500 time 0.3685 (0.3834) data time 0.0009 (0.0049) model time 0.3676 (0.3793) loss 2.9766 (2.6462) grad_norm 3.6501 (2.9119) loss_scale 256.0000 (202.9550) mem 11224MB [2024-09-01 13:30:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][120/625] eta 0:03:13 lr 0.000117 wd 0.0500 time 0.3704 (0.3826) data time 0.0009 (0.0045) model time 0.3695 (0.3784) loss 3.0941 (2.6546) grad_norm 2.0853 (2.9030) loss_scale 256.0000 (207.3388) mem 11224MB [2024-09-01 13:30:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][130/625] eta 0:03:09 lr 0.000117 wd 0.0500 time 0.3732 (0.3819) data time 0.0008 (0.0043) model time 0.3724 (0.3777) loss 3.0428 (2.6658) grad_norm 1.7958 (2.8518) loss_scale 256.0000 (211.0534) mem 11224MB [2024-09-01 13:30:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][140/625] eta 0:03:05 lr 0.000117 wd 0.0500 time 0.3759 (0.3815) data time 0.0008 (0.0040) model time 0.3751 (0.3775) loss 3.1404 (2.6645) grad_norm 2.8421 (2.8655) loss_scale 256.0000 (214.2411) mem 11224MB [2024-09-01 13:30:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][150/625] eta 0:03:01 lr 0.000117 wd 0.0500 time 0.3680 (0.3812) data time 0.0010 (0.0038) model time 0.3671 (0.3772) loss 2.8214 (2.6578) grad_norm 3.2405 (2.8544) loss_scale 256.0000 (217.0066) mem 11224MB [2024-09-01 13:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][160/625] eta 0:02:57 lr 0.000117 wd 0.0500 time 0.3772 (0.3808) data time 0.0010 (0.0037) model time 0.3762 (0.3769) loss 3.2461 (2.6641) grad_norm 2.6002 (2.8478) loss_scale 256.0000 (219.4286) mem 11224MB [2024-09-01 13:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][170/625] eta 0:02:53 lr 0.000117 wd 0.0500 time 0.3716 (0.3804) data time 0.0009 (0.0035) model time 0.3707 (0.3766) loss 2.3739 (2.6682) grad_norm 2.0286 (2.8286) loss_scale 256.0000 (221.5673) mem 11224MB [2024-09-01 13:30:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][180/625] eta 0:02:49 lr 0.000117 wd 0.0500 time 0.3800 (0.3801) data time 0.0008 (0.0034) model time 0.3792 (0.3765) loss 2.5491 (2.6717) grad_norm 2.4080 (2.8416) loss_scale 256.0000 (223.4696) mem 11224MB [2024-09-01 13:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][190/625] eta 0:02:45 lr 0.000117 wd 0.0500 time 0.3655 (0.3798) data time 0.0008 (0.0032) model time 0.3646 (0.3762) loss 2.5831 (2.6586) grad_norm 2.8316 (2.9030) loss_scale 256.0000 (225.1728) mem 11224MB [2024-09-01 13:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][200/625] eta 0:02:41 lr 0.000117 wd 0.0500 time 0.3724 (0.3795) data time 0.0009 (0.0031) model time 0.3714 (0.3760) loss 1.6343 (2.6482) grad_norm 2.8769 (2.8798) loss_scale 256.0000 (226.7065) mem 11224MB [2024-09-01 13:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][210/625] eta 0:02:37 lr 0.000116 wd 0.0500 time 0.3746 (0.3792) data time 0.0010 (0.0030) model time 0.3736 (0.3758) loss 3.1272 (2.6483) grad_norm 1.9485 (2.8589) loss_scale 256.0000 (228.0948) mem 11224MB [2024-09-01 13:30:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][220/625] eta 0:02:33 lr 0.000116 wd 0.0500 time 0.3729 (0.3791) data time 0.0008 (0.0029) model time 0.3721 (0.3758) loss 2.4807 (2.6579) grad_norm 2.4375 (2.8389) loss_scale 256.0000 (229.3575) mem 11224MB [2024-09-01 13:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][230/625] eta 0:02:29 lr 0.000116 wd 0.0500 time 0.3826 (0.3790) data time 0.0008 (0.0028) model time 0.3818 (0.3758) loss 3.0495 (2.6578) grad_norm 14.5838 (2.8928) loss_scale 256.0000 (230.5108) mem 11224MB [2024-09-01 13:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][240/625] eta 0:02:25 lr 0.000116 wd 0.0500 time 0.3717 (0.3787) data time 0.0009 (0.0028) model time 0.3708 (0.3756) loss 1.6968 (2.6478) grad_norm 3.5412 (2.9070) loss_scale 256.0000 (231.5685) mem 11224MB [2024-09-01 13:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][250/625] eta 0:02:21 lr 0.000116 wd 0.0500 time 0.3788 (0.3786) data time 0.0010 (0.0027) model time 0.3777 (0.3755) loss 1.8766 (2.6518) grad_norm 2.3559 (2.8960) loss_scale 256.0000 (232.5418) mem 11224MB [2024-09-01 13:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][260/625] eta 0:02:18 lr 0.000116 wd 0.0500 time 0.3775 (0.3792) data time 0.0009 (0.0026) model time 0.3766 (0.3763) loss 1.8305 (2.6398) grad_norm 3.3740 (2.8852) loss_scale 256.0000 (233.4406) mem 11224MB [2024-09-01 13:30:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][270/625] eta 0:02:15 lr 0.000116 wd 0.0500 time 0.3774 (0.3806) data time 0.0008 (0.0026) model time 0.3766 (0.3781) loss 1.9662 (2.6385) grad_norm 2.9275 (2.8759) loss_scale 256.0000 (234.2731) mem 11224MB [2024-09-01 13:31:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][280/625] eta 0:02:11 lr 0.000116 wd 0.0500 time 0.3772 (0.3804) data time 0.0007 (0.0025) model time 0.3765 (0.3779) loss 3.1562 (2.6515) grad_norm 2.8421 (2.8753) loss_scale 256.0000 (235.0463) mem 11224MB [2024-09-01 13:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][290/625] eta 0:02:07 lr 0.000116 wd 0.0500 time 0.3700 (0.3800) data time 0.0007 (0.0025) model time 0.3693 (0.3775) loss 3.5773 (2.6525) grad_norm 2.9250 (2.8655) loss_scale 256.0000 (235.7663) mem 11224MB [2024-09-01 13:31:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][300/625] eta 0:02:03 lr 0.000116 wd 0.0500 time 0.3646 (0.3798) data time 0.0011 (0.0024) model time 0.3635 (0.3773) loss 2.9314 (2.6466) grad_norm 2.7929 (2.8730) loss_scale 256.0000 (236.4385) mem 11224MB [2024-09-01 13:31:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][310/625] eta 0:01:59 lr 0.000116 wd 0.0500 time 0.3737 (0.3795) data time 0.0009 (0.0024) model time 0.3728 (0.3770) loss 2.1392 (2.6510) grad_norm 2.1318 (2.9130) loss_scale 256.0000 (237.0675) mem 11224MB [2024-09-01 13:31:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][320/625] eta 0:01:55 lr 0.000116 wd 0.0500 time 0.3706 (0.3793) data time 0.0010 (0.0023) model time 0.3696 (0.3768) loss 2.2102 (2.6513) grad_norm 1.9994 (2.8997) loss_scale 256.0000 (237.6573) mem 11224MB [2024-09-01 13:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][330/625] eta 0:01:51 lr 0.000116 wd 0.0500 time 0.3711 (0.3791) data time 0.0007 (0.0023) model time 0.3703 (0.3766) loss 2.7562 (2.6499) grad_norm 2.1422 (2.8792) loss_scale 256.0000 (238.2115) mem 11224MB [2024-09-01 13:31:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][340/625] eta 0:01:47 lr 0.000115 wd 0.0500 time 0.3795 (0.3789) data time 0.0008 (0.0022) model time 0.3787 (0.3765) loss 3.3467 (2.6513) grad_norm 1.8700 (2.8607) loss_scale 256.0000 (238.7331) mem 11224MB [2024-09-01 13:31:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][350/625] eta 0:01:44 lr 0.000115 wd 0.0500 time 0.3708 (0.3788) data time 0.0007 (0.0022) model time 0.3700 (0.3764) loss 1.8584 (2.6446) grad_norm 2.1085 (2.8524) loss_scale 256.0000 (239.2251) mem 11224MB [2024-09-01 13:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][360/625] eta 0:01:40 lr 0.000115 wd 0.0500 time 0.3730 (0.3786) data time 0.0010 (0.0022) model time 0.3720 (0.3762) loss 2.9195 (2.6449) grad_norm 2.6498 (2.8417) loss_scale 256.0000 (239.6898) mem 11224MB [2024-09-01 13:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][370/625] eta 0:01:36 lr 0.000115 wd 0.0500 time 0.3737 (0.3785) data time 0.0007 (0.0021) model time 0.3730 (0.3762) loss 2.1090 (2.6372) grad_norm 2.3951 (2.8652) loss_scale 256.0000 (240.1294) mem 11224MB [2024-09-01 13:31:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][380/625] eta 0:01:32 lr 0.000115 wd 0.0500 time 0.3668 (0.3783) data time 0.0011 (0.0021) model time 0.3657 (0.3760) loss 2.2568 (2.6358) grad_norm 1.8275 (2.8709) loss_scale 256.0000 (240.5459) mem 11224MB [2024-09-01 13:31:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][390/625] eta 0:01:28 lr 0.000115 wd 0.0500 time 0.3837 (0.3782) data time 0.0009 (0.0021) model time 0.3828 (0.3759) loss 2.9101 (2.6338) grad_norm 2.4673 (2.8567) loss_scale 256.0000 (240.9412) mem 11224MB [2024-09-01 13:31:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][400/625] eta 0:01:25 lr 0.000115 wd 0.0500 time 0.3739 (0.3781) data time 0.0010 (0.0021) model time 0.3730 (0.3758) loss 2.4951 (2.6376) grad_norm 3.6489 (2.8807) loss_scale 256.0000 (241.3167) mem 11224MB [2024-09-01 13:31:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][410/625] eta 0:01:21 lr 0.000115 wd 0.0500 time 0.3714 (0.3780) data time 0.0008 (0.0020) model time 0.3706 (0.3758) loss 2.2253 (2.6344) grad_norm 2.1414 (2.8651) loss_scale 256.0000 (241.6740) mem 11224MB [2024-09-01 13:31:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][420/625] eta 0:01:17 lr 0.000115 wd 0.0500 time 0.3658 (0.3779) data time 0.0008 (0.0020) model time 0.3650 (0.3757) loss 2.0237 (2.6342) grad_norm 3.8681 (2.8515) loss_scale 256.0000 (242.0143) mem 11224MB [2024-09-01 13:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][430/625] eta 0:01:13 lr 0.000115 wd 0.0500 time 0.3768 (0.3778) data time 0.0008 (0.0020) model time 0.3760 (0.3756) loss 2.3961 (2.6351) grad_norm 5.9930 (2.8633) loss_scale 256.0000 (242.3387) mem 11224MB [2024-09-01 13:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][440/625] eta 0:01:09 lr 0.000115 wd 0.0500 time 0.3809 (0.3782) data time 0.0008 (0.0020) model time 0.3801 (0.3761) loss 2.4839 (2.6338) grad_norm 2.7455 (2.8665) loss_scale 256.0000 (242.6485) mem 11224MB [2024-09-01 13:32:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][450/625] eta 0:01:06 lr 0.000115 wd 0.0500 time 0.5748 (0.3795) data time 0.0007 (0.0019) model time 0.5741 (0.3775) loss 1.7839 (2.6348) grad_norm 4.2129 (2.8665) loss_scale 256.0000 (242.9446) mem 11224MB [2024-09-01 13:32:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][460/625] eta 0:01:02 lr 0.000115 wd 0.0500 time 0.3688 (0.3793) data time 0.0009 (0.0019) model time 0.3678 (0.3774) loss 2.5847 (2.6366) grad_norm 2.1614 (2.8572) loss_scale 256.0000 (243.2278) mem 11224MB [2024-09-01 13:32:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][470/625] eta 0:00:58 lr 0.000114 wd 0.0500 time 0.3707 (0.3792) data time 0.0010 (0.0019) model time 0.3698 (0.3773) loss 2.8710 (2.6425) grad_norm 1.9965 (2.8448) loss_scale 256.0000 (243.4989) mem 11224MB [2024-09-01 13:32:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][480/625] eta 0:00:54 lr 0.000114 wd 0.0500 time 0.3754 (0.3791) data time 0.0010 (0.0019) model time 0.3744 (0.3772) loss 2.9228 (2.6410) grad_norm 2.7779 (2.8396) loss_scale 256.0000 (243.7588) mem 11224MB [2024-09-01 13:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][490/625] eta 0:00:51 lr 0.000114 wd 0.0500 time 0.3732 (0.3790) data time 0.0007 (0.0018) model time 0.3725 (0.3772) loss 1.9545 (2.6382) grad_norm 1.7523 (2.8460) loss_scale 256.0000 (244.0081) mem 11224MB [2024-09-01 13:32:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][500/625] eta 0:00:47 lr 0.000114 wd 0.0500 time 0.3711 (0.3789) data time 0.0010 (0.0018) model time 0.3701 (0.3770) loss 2.4782 (2.6348) grad_norm 1.9821 (2.8426) loss_scale 256.0000 (244.2475) mem 11224MB [2024-09-01 13:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][510/625] eta 0:00:43 lr 0.000114 wd 0.0500 time 0.3763 (0.3788) data time 0.0009 (0.0018) model time 0.3754 (0.3769) loss 2.9256 (2.6358) grad_norm 1.8479 (2.8378) loss_scale 256.0000 (244.4775) mem 11224MB [2024-09-01 13:32:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][520/625] eta 0:00:39 lr 0.000114 wd 0.0500 time 0.3754 (0.3787) data time 0.0010 (0.0018) model time 0.3744 (0.3768) loss 1.6753 (2.6376) grad_norm 3.3658 (2.8342) loss_scale 256.0000 (244.6987) mem 11224MB [2024-09-01 13:32:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][530/625] eta 0:00:35 lr 0.000114 wd 0.0500 time 0.3715 (0.3786) data time 0.0007 (0.0018) model time 0.3708 (0.3767) loss 2.7517 (2.6387) grad_norm 1.8970 (2.8346) loss_scale 256.0000 (244.9115) mem 11224MB [2024-09-01 13:32:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][540/625] eta 0:00:32 lr 0.000114 wd 0.0500 time 0.3733 (0.3785) data time 0.0007 (0.0018) model time 0.3726 (0.3766) loss 1.8923 (2.6379) grad_norm 2.5143 (2.8287) loss_scale 256.0000 (245.1165) mem 11224MB [2024-09-01 13:32:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][550/625] eta 0:00:28 lr 0.000114 wd 0.0500 time 0.3803 (0.3784) data time 0.0010 (0.0018) model time 0.3793 (0.3766) loss 2.5153 (2.6390) grad_norm 2.1534 (2.8256) loss_scale 256.0000 (245.3140) mem 11224MB [2024-09-01 13:32:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][560/625] eta 0:00:24 lr 0.000114 wd 0.0500 time 0.3795 (0.3787) data time 0.0010 (0.0017) model time 0.3785 (0.3769) loss 1.5190 (2.6412) grad_norm 2.2136 (2.8215) loss_scale 256.0000 (245.5045) mem 11224MB [2024-09-01 13:32:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][570/625] eta 0:00:20 lr 0.000114 wd 0.0500 time 0.3686 (0.3797) data time 0.0010 (0.0017) model time 0.3676 (0.3780) loss 2.7653 (2.6390) grad_norm 1.7943 (2.8158) loss_scale 256.0000 (245.6883) mem 11224MB [2024-09-01 13:32:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][580/625] eta 0:00:17 lr 0.000114 wd 0.0500 time 0.3669 (0.3796) data time 0.0010 (0.0017) model time 0.3659 (0.3779) loss 2.7803 (2.6401) grad_norm 3.0957 (2.8279) loss_scale 256.0000 (245.8657) mem 11224MB [2024-09-01 13:32:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][590/625] eta 0:00:13 lr 0.000114 wd 0.0500 time 0.3864 (0.3795) data time 0.0008 (0.0017) model time 0.3857 (0.3778) loss 2.6990 (2.6410) grad_norm 2.9441 (2.8265) loss_scale 256.0000 (246.0372) mem 11224MB [2024-09-01 13:33:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][600/625] eta 0:00:09 lr 0.000113 wd 0.0500 time 0.3703 (0.3794) data time 0.0010 (0.0017) model time 0.3694 (0.3777) loss 3.0746 (2.6433) grad_norm 2.7095 (2.8442) loss_scale 256.0000 (246.2030) mem 11224MB [2024-09-01 13:33:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][610/625] eta 0:00:05 lr 0.000113 wd 0.0500 time 0.3599 (0.3793) data time 0.0007 (0.0017) model time 0.3592 (0.3776) loss 2.6226 (2.6451) grad_norm 3.3946 (2.9049) loss_scale 256.0000 (246.3633) mem 11224MB [2024-09-01 13:33:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [260/300][620/625] eta 0:00:01 lr 0.000113 wd 0.0500 time 0.3606 (0.3790) data time 0.0007 (0.0017) model time 0.3599 (0.3774) loss 2.7990 (2.6437) grad_norm 3.2656 (2.9099) loss_scale 256.0000 (246.5185) mem 11224MB [2024-09-01 13:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 260 training takes 0:03:56 [2024-09-01 13:33:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:33:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.499 (0.499) Loss 0.5264 (0.5264) Acc@1 88.916 (88.916) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 13:33:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.8433 (0.6148) Acc@1 80.322 (86.794) Acc@5 95.605 (97.772) Mem 11224MB [2024-09-01 13:33:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 0.9077 (0.7312) Acc@1 78.809 (83.819) Acc@5 95.117 (96.615) Mem 11224MB [2024-09-01 13:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.491 Acc@5 96.609 [2024-09-01 13:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 13:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.802 (0.802) Loss 0.4939 (0.4939) Acc@1 89.160 (89.160) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 13:33:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.175) Loss 0.8066 (0.5934) Acc@1 81.250 (87.216) Acc@5 95.752 (97.860) Mem 11224MB [2024-09-01 13:33:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.8838 (0.7046) Acc@1 79.492 (84.426) Acc@5 95.264 (96.763) Mem 11224MB [2024-09-01 13:33:19 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.119 Acc@5 96.779 [2024-09-01 13:33:19 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.1% [2024-09-01 13:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][0/625] eta 0:13:30 lr 0.000113 wd 0.0500 time 1.2961 (1.2961) data time 0.7208 (0.7208) model time 0.0000 (0.0000) loss 3.0183 (3.0183) grad_norm 2.9158 (2.9158) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][10/625] eta 0:04:42 lr 0.000113 wd 0.0500 time 0.3804 (0.4598) data time 0.0008 (0.0663) model time 0.0000 (0.0000) loss 2.2396 (2.5194) grad_norm 4.0165 (3.1020) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][20/625] eta 0:04:13 lr 0.000113 wd 0.0500 time 0.3743 (0.4195) data time 0.0010 (0.0352) model time 0.0000 (0.0000) loss 2.6986 (2.5972) grad_norm 2.4260 (2.8844) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][30/625] eta 0:04:00 lr 0.000113 wd 0.0500 time 0.3715 (0.4042) data time 0.0010 (0.0241) model time 0.0000 (0.0000) loss 2.7954 (2.6417) grad_norm 5.0677 (2.8958) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][40/625] eta 0:03:52 lr 0.000113 wd 0.0500 time 0.3836 (0.3974) data time 0.0008 (0.0185) model time 0.0000 (0.0000) loss 2.4874 (2.6697) grad_norm 2.9854 (3.0038) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][50/625] eta 0:03:46 lr 0.000113 wd 0.0500 time 0.3827 (0.3933) data time 0.0007 (0.0150) model time 0.0000 (0.0000) loss 1.4726 (2.6696) grad_norm 2.1685 (3.3705) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][60/625] eta 0:03:40 lr 0.000113 wd 0.0500 time 0.3753 (0.3903) data time 0.0009 (0.0128) model time 0.3743 (0.3739) loss 2.6693 (2.6739) grad_norm 2.5337 (3.2563) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][70/625] eta 0:03:35 lr 0.000113 wd 0.0500 time 0.3758 (0.3880) data time 0.0010 (0.0112) model time 0.3748 (0.3729) loss 2.9466 (2.6704) grad_norm 2.0507 (3.1424) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][80/625] eta 0:03:30 lr 0.000113 wd 0.0500 time 0.3789 (0.3865) data time 0.0008 (0.0099) model time 0.3780 (0.3737) loss 2.5562 (2.6648) grad_norm 2.1270 (3.0768) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][90/625] eta 0:03:29 lr 0.000113 wd 0.0500 time 0.3731 (0.3915) data time 0.0008 (0.0090) model time 0.3724 (0.3879) loss 2.5385 (2.6668) grad_norm 2.2838 (3.0156) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:33:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][100/625] eta 0:03:24 lr 0.000113 wd 0.0500 time 0.3647 (0.3896) data time 0.0008 (0.0082) model time 0.3639 (0.3846) loss 2.9521 (2.6896) grad_norm 4.4930 (3.0450) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][110/625] eta 0:03:20 lr 0.000112 wd 0.0500 time 0.3700 (0.3897) data time 0.0009 (0.0075) model time 0.3691 (0.3854) loss 2.8139 (2.6855) grad_norm 4.4235 (3.0518) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][120/625] eta 0:03:16 lr 0.000112 wd 0.0500 time 0.3743 (0.3882) data time 0.0010 (0.0070) model time 0.3734 (0.3834) loss 1.5201 (2.6845) grad_norm 2.4316 (2.9734) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][130/625] eta 0:03:11 lr 0.000112 wd 0.0500 time 0.3723 (0.3871) data time 0.0008 (0.0065) model time 0.3715 (0.3820) loss 3.3805 (2.6934) grad_norm 2.7026 (2.9703) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][140/625] eta 0:03:07 lr 0.000112 wd 0.0500 time 0.3770 (0.3863) data time 0.0010 (0.0061) model time 0.3759 (0.3812) loss 2.8954 (2.6840) grad_norm 3.2620 (3.1413) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][150/625] eta 0:03:03 lr 0.000112 wd 0.0500 time 0.3794 (0.3854) data time 0.0008 (0.0058) model time 0.3786 (0.3803) loss 1.8727 (2.6791) grad_norm 2.3233 (3.1105) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][160/625] eta 0:02:58 lr 0.000112 wd 0.0500 time 0.3731 (0.3846) data time 0.0007 (0.0055) model time 0.3724 (0.3795) loss 2.9987 (2.6782) grad_norm 4.9573 (3.0867) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][170/625] eta 0:02:54 lr 0.000112 wd 0.0500 time 0.3795 (0.3841) data time 0.0009 (0.0052) model time 0.3785 (0.3791) loss 2.5278 (2.6673) grad_norm 2.4875 (3.0866) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][180/625] eta 0:02:50 lr 0.000112 wd 0.0500 time 0.3740 (0.3836) data time 0.0007 (0.0050) model time 0.3733 (0.3788) loss 2.6075 (2.6734) grad_norm 6.0677 (3.0729) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][190/625] eta 0:02:46 lr 0.000112 wd 0.0500 time 0.3671 (0.3832) data time 0.0010 (0.0048) model time 0.3661 (0.3785) loss 2.4886 (2.6771) grad_norm 2.6358 (3.0408) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][200/625] eta 0:02:42 lr 0.000112 wd 0.0500 time 0.3890 (0.3829) data time 0.0010 (0.0046) model time 0.3879 (0.3783) loss 2.8153 (2.6882) grad_norm 2.1776 (2.9943) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][210/625] eta 0:02:38 lr 0.000112 wd 0.0500 time 0.3703 (0.3824) data time 0.0010 (0.0044) model time 0.3693 (0.3778) loss 2.3636 (2.6916) grad_norm 2.2912 (2.9794) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][220/625] eta 0:02:34 lr 0.000112 wd 0.0500 time 0.3686 (0.3819) data time 0.0010 (0.0043) model time 0.3676 (0.3775) loss 2.1934 (2.6924) grad_norm 2.8029 (2.9694) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][230/625] eta 0:02:30 lr 0.000112 wd 0.0500 time 0.3764 (0.3816) data time 0.0010 (0.0041) model time 0.3754 (0.3773) loss 2.6527 (2.6878) grad_norm 1.8940 (2.9828) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][240/625] eta 0:02:26 lr 0.000111 wd 0.0500 time 0.3778 (0.3813) data time 0.0007 (0.0040) model time 0.3771 (0.3771) loss 2.6000 (2.6761) grad_norm 2.4423 (2.9680) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][250/625] eta 0:02:22 lr 0.000111 wd 0.0500 time 0.3715 (0.3810) data time 0.0008 (0.0039) model time 0.3707 (0.3768) loss 2.6065 (2.6799) grad_norm 2.8208 (2.9460) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:34:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][260/625] eta 0:02:18 lr 0.000111 wd 0.0500 time 0.3791 (0.3807) data time 0.0010 (0.0038) model time 0.3781 (0.3766) loss 1.9184 (2.6794) grad_norm 2.0762 (2.9287) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][270/625] eta 0:02:15 lr 0.000111 wd 0.0500 time 0.3748 (0.3805) data time 0.0008 (0.0037) model time 0.3741 (0.3765) loss 3.3140 (2.6839) grad_norm 2.4274 (2.9279) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][280/625] eta 0:02:11 lr 0.000111 wd 0.0500 time 0.3738 (0.3803) data time 0.0009 (0.0036) model time 0.3729 (0.3764) loss 3.0044 (2.6911) grad_norm 1.9379 (2.9278) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][290/625] eta 0:02:07 lr 0.000111 wd 0.0500 time 0.3746 (0.3802) data time 0.0009 (0.0035) model time 0.3737 (0.3763) loss 2.9029 (2.6979) grad_norm 4.0528 (2.9213) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][300/625] eta 0:02:04 lr 0.000111 wd 0.0500 time 0.3685 (0.3821) data time 0.0010 (0.0034) model time 0.3676 (0.3788) loss 2.4221 (2.6979) grad_norm 2.7095 (2.9202) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][310/625] eta 0:02:00 lr 0.000111 wd 0.0500 time 0.3741 (0.3819) data time 0.0008 (0.0033) model time 0.3733 (0.3786) loss 1.8570 (2.6956) grad_norm 2.1628 (2.9125) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][320/625] eta 0:01:56 lr 0.000111 wd 0.0500 time 0.3734 (0.3817) data time 0.0007 (0.0032) model time 0.3726 (0.3784) loss 3.2529 (2.6976) grad_norm 2.0259 (2.9038) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][330/625] eta 0:01:52 lr 0.000111 wd 0.0500 time 0.3752 (0.3814) data time 0.0008 (0.0032) model time 0.3744 (0.3782) loss 2.4056 (2.7002) grad_norm 3.9546 (2.9029) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][340/625] eta 0:01:48 lr 0.000111 wd 0.0500 time 0.3736 (0.3813) data time 0.0009 (0.0031) model time 0.3727 (0.3782) loss 3.1559 (2.6978) grad_norm 2.6598 (2.8965) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][350/625] eta 0:01:44 lr 0.000111 wd 0.0500 time 0.3741 (0.3812) data time 0.0008 (0.0030) model time 0.3733 (0.3781) loss 3.1033 (2.6950) grad_norm 3.0216 (2.8871) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][360/625] eta 0:01:40 lr 0.000111 wd 0.0500 time 0.3725 (0.3810) data time 0.0008 (0.0030) model time 0.3717 (0.3780) loss 2.9710 (2.6897) grad_norm 2.1278 (2.8739) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][370/625] eta 0:01:37 lr 0.000111 wd 0.0500 time 0.3764 (0.3809) data time 0.0010 (0.0029) model time 0.3754 (0.3779) loss 2.9189 (2.6894) grad_norm 2.5893 (2.8677) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][380/625] eta 0:01:33 lr 0.000110 wd 0.0500 time 0.3748 (0.3807) data time 0.0008 (0.0029) model time 0.3741 (0.3778) loss 3.0770 (2.6924) grad_norm 3.6140 (2.8551) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][390/625] eta 0:01:29 lr 0.000110 wd 0.0500 time 0.3852 (0.3806) data time 0.0011 (0.0028) model time 0.3840 (0.3777) loss 2.8327 (2.6888) grad_norm 2.3998 (2.8601) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][400/625] eta 0:01:25 lr 0.000110 wd 0.0500 time 0.3770 (0.3804) data time 0.0008 (0.0028) model time 0.3761 (0.3775) loss 2.4208 (2.6835) grad_norm 2.2282 (2.8516) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][410/625] eta 0:01:21 lr 0.000110 wd 0.0500 time 0.3764 (0.3803) data time 0.0008 (0.0027) model time 0.3756 (0.3774) loss 2.7937 (2.6754) grad_norm 2.2527 (2.8765) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:35:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][420/625] eta 0:01:18 lr 0.000110 wd 0.0500 time 0.3762 (0.3806) data time 0.0009 (0.0027) model time 0.3754 (0.3778) loss 2.7650 (2.6742) grad_norm 2.3318 (2.8709) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][430/625] eta 0:01:14 lr 0.000110 wd 0.0500 time 0.3689 (0.3805) data time 0.0008 (0.0027) model time 0.3681 (0.3777) loss 3.1352 (2.6775) grad_norm 1.8783 (2.8804) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][440/625] eta 0:01:10 lr 0.000110 wd 0.0500 time 0.3831 (0.3803) data time 0.0008 (0.0026) model time 0.3824 (0.3776) loss 2.7036 (2.6798) grad_norm 2.1657 (2.8693) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][450/625] eta 0:01:06 lr 0.000110 wd 0.0500 time 0.3764 (0.3802) data time 0.0010 (0.0026) model time 0.3754 (0.3775) loss 3.3176 (2.6797) grad_norm 1.9442 (2.8711) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][460/625] eta 0:01:02 lr 0.000110 wd 0.0500 time 0.3760 (0.3800) data time 0.0010 (0.0026) model time 0.3750 (0.3774) loss 3.1357 (2.6812) grad_norm 1.7311 (2.8693) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][470/625] eta 0:00:58 lr 0.000110 wd 0.0500 time 0.3762 (0.3799) data time 0.0007 (0.0025) model time 0.3755 (0.3773) loss 1.7446 (2.6755) grad_norm 2.6163 (2.8681) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][480/625] eta 0:00:55 lr 0.000110 wd 0.0500 time 0.3732 (0.3802) data time 0.0010 (0.0025) model time 0.3722 (0.3776) loss 2.0181 (2.6747) grad_norm 3.9693 (2.8694) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][490/625] eta 0:00:51 lr 0.000110 wd 0.0500 time 0.3835 (0.3805) data time 0.0007 (0.0025) model time 0.3828 (0.3780) loss 3.3327 (2.6730) grad_norm 2.2681 (2.8871) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][500/625] eta 0:00:47 lr 0.000110 wd 0.0500 time 0.3790 (0.3804) data time 0.0008 (0.0024) model time 0.3783 (0.3779) loss 2.8144 (2.6712) grad_norm 2.1180 (2.8816) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][510/625] eta 0:00:43 lr 0.000109 wd 0.0500 time 0.3751 (0.3803) data time 0.0010 (0.0024) model time 0.3741 (0.3778) loss 3.1234 (2.6760) grad_norm 1.8530 (2.8818) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][520/625] eta 0:00:39 lr 0.000109 wd 0.0500 time 0.3754 (0.3802) data time 0.0010 (0.0024) model time 0.3744 (0.3777) loss 2.7271 (2.6782) grad_norm 3.1819 (2.9648) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][530/625] eta 0:00:36 lr 0.000109 wd 0.0500 time 0.3805 (0.3801) data time 0.0008 (0.0023) model time 0.3797 (0.3777) loss 2.9612 (2.6802) grad_norm 5.1180 (2.9717) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][540/625] eta 0:00:32 lr 0.000109 wd 0.0500 time 0.3757 (0.3800) data time 0.0010 (0.0023) model time 0.3747 (0.3776) loss 2.7341 (2.6803) grad_norm 2.4025 (2.9836) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][550/625] eta 0:00:28 lr 0.000109 wd 0.0500 time 0.3727 (0.3799) data time 0.0010 (0.0023) model time 0.3717 (0.3775) loss 2.7202 (2.6800) grad_norm 3.0579 (3.0053) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][560/625] eta 0:00:24 lr 0.000109 wd 0.0500 time 0.3746 (0.3798) data time 0.0010 (0.0023) model time 0.3736 (0.3774) loss 2.4870 (2.6793) grad_norm 2.4566 (2.9985) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][570/625] eta 0:00:20 lr 0.000109 wd 0.0500 time 0.3831 (0.3797) data time 0.0008 (0.0022) model time 0.3823 (0.3774) loss 1.8665 (2.6703) grad_norm 2.0992 (3.0394) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:36:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][580/625] eta 0:00:17 lr 0.000109 wd 0.0500 time 0.3747 (0.3796) data time 0.0009 (0.0022) model time 0.3737 (0.3773) loss 2.5083 (2.6665) grad_norm 2.0167 (3.0353) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][590/625] eta 0:00:13 lr 0.000109 wd 0.0500 time 0.3830 (0.3795) data time 0.0007 (0.0022) model time 0.3823 (0.3773) loss 1.4026 (2.6663) grad_norm 2.7708 (3.0320) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][600/625] eta 0:00:09 lr 0.000109 wd 0.0500 time 0.5977 (0.3802) data time 0.0010 (0.0022) model time 0.5968 (0.3780) loss 2.2880 (2.6658) grad_norm 11.4798 (3.0406) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][610/625] eta 0:00:05 lr 0.000109 wd 0.0500 time 0.3705 (0.3800) data time 0.0005 (0.0022) model time 0.3700 (0.3778) loss 3.2170 (2.6690) grad_norm 4.1982 (3.0521) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [261/300][620/625] eta 0:00:01 lr 0.000109 wd 0.0500 time 0.3614 (0.3797) data time 0.0007 (0.0021) model time 0.3607 (0.3775) loss 3.1858 (2.6733) grad_norm 2.5604 (3.0561) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:16 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 261 training takes 0:03:57 [2024-09-01 13:37:16 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:37:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:37:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.500 (0.500) Loss 0.5264 (0.5264) Acc@1 88.818 (88.818) Acc@5 98.389 (98.389) Mem 11224MB [2024-09-01 13:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 0.8516 (0.6147) Acc@1 79.834 (86.603) Acc@5 95.801 (97.789) Mem 11224MB [2024-09-01 13:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.095 (0.119) Loss 0.9297 (0.7339) Acc@1 78.320 (83.726) Acc@5 95.215 (96.622) Mem 11224MB [2024-09-01 13:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.407 Acc@5 96.623 [2024-09-01 13:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 13:37:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.809 (0.809) Loss 0.4956 (0.4956) Acc@1 89.160 (89.160) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 13:37:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.174) Loss 0.8071 (0.5936) Acc@1 81.250 (87.180) Acc@5 95.752 (97.860) Mem 11224MB [2024-09-01 13:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.135) Loss 0.8828 (0.7050) Acc@1 79.443 (84.370) Acc@5 95.215 (96.754) Mem 11224MB [2024-09-01 13:37:24 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.073 Acc@5 96.773 [2024-09-01 13:37:24 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.1% [2024-09-01 13:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][0/625] eta 0:12:05 lr 0.000109 wd 0.0500 time 1.1611 (1.1611) data time 0.4789 (0.4789) model time 0.0000 (0.0000) loss 2.1531 (2.1531) grad_norm 2.6795 (2.6795) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][10/625] eta 0:04:34 lr 0.000109 wd 0.0500 time 0.3693 (0.4457) data time 0.0010 (0.0444) model time 0.0000 (0.0000) loss 2.8378 (2.5070) grad_norm 9.3123 (3.9223) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][20/625] eta 0:04:08 lr 0.000108 wd 0.0500 time 0.3747 (0.4109) data time 0.0009 (0.0237) model time 0.0000 (0.0000) loss 2.2059 (2.5375) grad_norm 4.2659 (3.3884) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][30/625] eta 0:03:57 lr 0.000108 wd 0.0500 time 0.3728 (0.3988) data time 0.0008 (0.0164) model time 0.0000 (0.0000) loss 2.5121 (2.5785) grad_norm 2.1161 (3.0275) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][40/625] eta 0:03:49 lr 0.000108 wd 0.0500 time 0.3739 (0.3927) data time 0.0009 (0.0126) model time 0.0000 (0.0000) loss 2.2707 (2.5865) grad_norm 2.2283 (2.8214) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][50/625] eta 0:03:43 lr 0.000108 wd 0.0500 time 0.3757 (0.3891) data time 0.0010 (0.0103) model time 0.0000 (0.0000) loss 2.8655 (2.5693) grad_norm 2.7290 (2.7589) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][60/625] eta 0:03:38 lr 0.000108 wd 0.0500 time 0.3736 (0.3865) data time 0.0007 (0.0088) model time 0.3729 (0.3723) loss 2.2514 (2.5916) grad_norm 1.9678 (2.7673) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][70/625] eta 0:03:33 lr 0.000108 wd 0.0500 time 0.3717 (0.3847) data time 0.0010 (0.0077) model time 0.3708 (0.3726) loss 2.7719 (2.6288) grad_norm 3.8149 (2.8362) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][80/625] eta 0:03:29 lr 0.000108 wd 0.0500 time 0.3767 (0.3835) data time 0.0008 (0.0068) model time 0.3759 (0.3731) loss 2.2571 (2.6173) grad_norm 2.6120 (2.8368) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:37:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][90/625] eta 0:03:24 lr 0.000108 wd 0.0500 time 0.3799 (0.3827) data time 0.0007 (0.0062) model time 0.3791 (0.3735) loss 3.1401 (2.6436) grad_norm 2.7046 (2.7933) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][100/625] eta 0:03:20 lr 0.000108 wd 0.0500 time 0.3782 (0.3820) data time 0.0010 (0.0057) model time 0.3772 (0.3737) loss 2.9911 (2.6494) grad_norm 1.9846 (2.7671) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][110/625] eta 0:03:17 lr 0.000108 wd 0.0500 time 0.3714 (0.3833) data time 0.0007 (0.0053) model time 0.3707 (0.3775) loss 3.0848 (2.6425) grad_norm 3.4328 (2.7595) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][120/625] eta 0:03:13 lr 0.000108 wd 0.0500 time 0.3681 (0.3826) data time 0.0007 (0.0049) model time 0.3673 (0.3770) loss 2.4441 (2.6515) grad_norm 2.8271 (2.7700) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][130/625] eta 0:03:09 lr 0.000108 wd 0.0500 time 0.3723 (0.3821) data time 0.0010 (0.0046) model time 0.3713 (0.3766) loss 2.5396 (2.6580) grad_norm 2.4904 (2.7769) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][140/625] eta 0:03:05 lr 0.000108 wd 0.0500 time 0.3787 (0.3817) data time 0.0008 (0.0044) model time 0.3779 (0.3765) loss 1.7537 (2.6454) grad_norm 3.7774 (2.7662) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][150/625] eta 0:03:01 lr 0.000108 wd 0.0500 time 0.3743 (0.3812) data time 0.0008 (0.0041) model time 0.3735 (0.3762) loss 2.6055 (2.6363) grad_norm 1.8301 (2.7527) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][160/625] eta 0:02:57 lr 0.000107 wd 0.0500 time 0.3711 (0.3819) data time 0.0008 (0.0039) model time 0.3704 (0.3775) loss 1.5534 (2.6292) grad_norm 2.6455 (2.7717) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][170/625] eta 0:02:54 lr 0.000107 wd 0.0500 time 0.3817 (0.3826) data time 0.0010 (0.0037) model time 0.3808 (0.3788) loss 3.5822 (2.6411) grad_norm 2.3138 (2.7442) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][180/625] eta 0:02:50 lr 0.000107 wd 0.0500 time 0.3750 (0.3832) data time 0.0010 (0.0036) model time 0.3740 (0.3799) loss 2.7680 (2.6480) grad_norm 1.8086 (2.7178) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][190/625] eta 0:02:46 lr 0.000107 wd 0.0500 time 0.3668 (0.3827) data time 0.0008 (0.0035) model time 0.3660 (0.3794) loss 2.0638 (2.6409) grad_norm 1.9733 (2.7152) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][200/625] eta 0:02:42 lr 0.000107 wd 0.0500 time 0.3706 (0.3822) data time 0.0009 (0.0033) model time 0.3696 (0.3789) loss 2.1138 (2.6484) grad_norm 2.5882 (2.6986) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][210/625] eta 0:02:38 lr 0.000107 wd 0.0500 time 0.3792 (0.3820) data time 0.0010 (0.0032) model time 0.3782 (0.3788) loss 2.2559 (2.6399) grad_norm 1.9813 (2.6779) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][220/625] eta 0:02:34 lr 0.000107 wd 0.0500 time 0.3739 (0.3818) data time 0.0007 (0.0031) model time 0.3732 (0.3786) loss 2.2645 (2.6273) grad_norm 2.6159 (2.6864) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][230/625] eta 0:02:30 lr 0.000107 wd 0.0500 time 0.3708 (0.3813) data time 0.0008 (0.0030) model time 0.3700 (0.3781) loss 2.4801 (2.6288) grad_norm 2.9648 (2.7259) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][240/625] eta 0:02:26 lr 0.000107 wd 0.0500 time 0.3722 (0.3810) data time 0.0009 (0.0029) model time 0.3713 (0.3778) loss 2.1639 (2.6232) grad_norm 3.0399 (2.7330) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:38:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][250/625] eta 0:02:22 lr 0.000107 wd 0.0500 time 0.3753 (0.3809) data time 0.0009 (0.0029) model time 0.3744 (0.3778) loss 2.7346 (2.6215) grad_norm 1.7088 (2.7298) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][260/625] eta 0:02:18 lr 0.000107 wd 0.0500 time 0.3755 (0.3807) data time 0.0010 (0.0028) model time 0.3745 (0.3777) loss 3.2187 (2.6180) grad_norm 2.6183 (2.7249) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][270/625] eta 0:02:15 lr 0.000107 wd 0.0500 time 0.3796 (0.3805) data time 0.0009 (0.0027) model time 0.3787 (0.3775) loss 3.3495 (2.6204) grad_norm 1.9216 (2.7446) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][280/625] eta 0:02:11 lr 0.000107 wd 0.0500 time 0.3686 (0.3809) data time 0.0009 (0.0026) model time 0.3677 (0.3782) loss 2.1168 (2.6059) grad_norm 1.9061 (2.7356) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][290/625] eta 0:02:07 lr 0.000107 wd 0.0500 time 0.3776 (0.3814) data time 0.0007 (0.0026) model time 0.3769 (0.3789) loss 3.0591 (2.6102) grad_norm 2.3257 (2.7257) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][300/625] eta 0:02:03 lr 0.000106 wd 0.0500 time 0.3742 (0.3812) data time 0.0010 (0.0025) model time 0.3733 (0.3786) loss 2.9827 (2.6178) grad_norm 1.9455 (2.7419) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][310/625] eta 0:01:59 lr 0.000106 wd 0.0500 time 0.3719 (0.3809) data time 0.0010 (0.0025) model time 0.3710 (0.3784) loss 3.0401 (2.6201) grad_norm 3.2457 (2.7377) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][320/625] eta 0:01:56 lr 0.000106 wd 0.0500 time 0.3718 (0.3816) data time 0.0010 (0.0024) model time 0.3708 (0.3793) loss 3.0202 (2.6261) grad_norm 3.2484 (2.7272) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][330/625] eta 0:01:52 lr 0.000106 wd 0.0500 time 0.3653 (0.3814) data time 0.0008 (0.0024) model time 0.3645 (0.3790) loss 3.2307 (2.6301) grad_norm 2.5501 (2.7510) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][340/625] eta 0:01:48 lr 0.000106 wd 0.0500 time 0.3743 (0.3813) data time 0.0012 (0.0024) model time 0.3731 (0.3789) loss 2.5279 (2.6311) grad_norm 2.4533 (2.7529) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][350/625] eta 0:01:44 lr 0.000106 wd 0.0500 time 0.3800 (0.3810) data time 0.0008 (0.0023) model time 0.3793 (0.3787) loss 2.2014 (2.6319) grad_norm 1.7427 (2.7527) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][360/625] eta 0:01:41 lr 0.000106 wd 0.0500 time 0.3724 (0.3813) data time 0.0008 (0.0023) model time 0.3715 (0.3791) loss 2.2544 (2.6302) grad_norm 2.0717 (2.8117) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][370/625] eta 0:01:37 lr 0.000106 wd 0.0500 time 0.3671 (0.3811) data time 0.0007 (0.0023) model time 0.3664 (0.3789) loss 2.6157 (2.6359) grad_norm 2.4111 (2.8063) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][380/625] eta 0:01:33 lr 0.000106 wd 0.0500 time 0.3731 (0.3809) data time 0.0009 (0.0022) model time 0.3722 (0.3787) loss 3.0343 (2.6371) grad_norm 1.9869 (2.8128) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][390/625] eta 0:01:29 lr 0.000106 wd 0.0500 time 0.3701 (0.3807) data time 0.0008 (0.0022) model time 0.3693 (0.3785) loss 1.8794 (2.6374) grad_norm 2.6162 (2.8170) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:39:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][400/625] eta 0:01:25 lr 0.000106 wd 0.0500 time 0.3785 (0.3806) data time 0.0009 (0.0022) model time 0.3776 (0.3784) loss 3.0675 (2.6385) grad_norm 1.9706 (2.8441) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][410/625] eta 0:01:21 lr 0.000106 wd 0.0500 time 0.3780 (0.3804) data time 0.0010 (0.0021) model time 0.3770 (0.3782) loss 2.8014 (2.6395) grad_norm 2.7456 (2.8358) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][420/625] eta 0:01:17 lr 0.000106 wd 0.0500 time 0.3701 (0.3802) data time 0.0009 (0.0021) model time 0.3691 (0.3780) loss 2.5975 (2.6394) grad_norm 3.1164 (2.8302) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][430/625] eta 0:01:14 lr 0.000105 wd 0.0500 time 0.3723 (0.3800) data time 0.0008 (0.0021) model time 0.3715 (0.3779) loss 3.3078 (2.6464) grad_norm 2.7799 (2.8322) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][440/625] eta 0:01:10 lr 0.000105 wd 0.0500 time 0.3728 (0.3799) data time 0.0009 (0.0020) model time 0.3719 (0.3777) loss 2.5490 (2.6480) grad_norm 2.6532 (2.8159) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][450/625] eta 0:01:06 lr 0.000105 wd 0.0500 time 0.3748 (0.3797) data time 0.0008 (0.0020) model time 0.3740 (0.3776) loss 3.2852 (2.6513) grad_norm 2.1321 (2.8158) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][460/625] eta 0:01:02 lr 0.000105 wd 0.0500 time 0.3741 (0.3796) data time 0.0008 (0.0020) model time 0.3733 (0.3774) loss 2.9641 (2.6515) grad_norm 2.7782 (2.8146) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][470/625] eta 0:00:58 lr 0.000105 wd 0.0500 time 0.3707 (0.3794) data time 0.0010 (0.0020) model time 0.3697 (0.3773) loss 2.9004 (2.6485) grad_norm 2.3846 (2.8095) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][480/625] eta 0:00:55 lr 0.000105 wd 0.0500 time 0.3707 (0.3796) data time 0.0009 (0.0020) model time 0.3697 (0.3775) loss 3.1083 (2.6497) grad_norm 2.4180 (2.8287) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][490/625] eta 0:00:51 lr 0.000105 wd 0.0500 time 0.3703 (0.3799) data time 0.0010 (0.0019) model time 0.3693 (0.3779) loss 2.8523 (2.6505) grad_norm 2.6534 (2.8253) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][500/625] eta 0:00:47 lr 0.000105 wd 0.0500 time 0.3707 (0.3805) data time 0.0009 (0.0019) model time 0.3697 (0.3786) loss 2.8282 (2.6473) grad_norm 2.8178 (2.8213) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][510/625] eta 0:00:43 lr 0.000105 wd 0.0500 time 0.3735 (0.3804) data time 0.0010 (0.0019) model time 0.3725 (0.3785) loss 2.4844 (2.6473) grad_norm 3.4950 (2.8187) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][520/625] eta 0:00:39 lr 0.000105 wd 0.0500 time 0.3754 (0.3803) data time 0.0009 (0.0019) model time 0.3744 (0.3783) loss 2.3084 (2.6414) grad_norm 3.8878 (2.8151) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][530/625] eta 0:00:36 lr 0.000105 wd 0.0500 time 0.3816 (0.3801) data time 0.0008 (0.0019) model time 0.3808 (0.3782) loss 2.6914 (2.6403) grad_norm 3.2674 (2.8194) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][540/625] eta 0:00:32 lr 0.000105 wd 0.0500 time 0.3692 (0.3800) data time 0.0008 (0.0018) model time 0.3684 (0.3781) loss 3.4354 (2.6429) grad_norm 2.9002 (2.8175) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][550/625] eta 0:00:28 lr 0.000105 wd 0.0500 time 0.3673 (0.3799) data time 0.0007 (0.0018) model time 0.3665 (0.3780) loss 2.9328 (2.6426) grad_norm 1.8408 (2.8085) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:40:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][560/625] eta 0:00:24 lr 0.000105 wd 0.0500 time 0.3754 (0.3797) data time 0.0009 (0.0018) model time 0.3744 (0.3778) loss 2.6894 (2.6422) grad_norm 2.4228 (2.8111) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][570/625] eta 0:00:20 lr 0.000104 wd 0.0500 time 0.3714 (0.3796) data time 0.0009 (0.0018) model time 0.3705 (0.3777) loss 2.7463 (2.6432) grad_norm 2.2628 (2.8079) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][580/625] eta 0:00:17 lr 0.000104 wd 0.0500 time 0.3742 (0.3795) data time 0.0008 (0.0018) model time 0.3734 (0.3776) loss 2.9535 (2.6416) grad_norm 3.0670 (2.8074) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][590/625] eta 0:00:13 lr 0.000104 wd 0.0500 time 0.3672 (0.3795) data time 0.0008 (0.0018) model time 0.3664 (0.3776) loss 2.9444 (2.6401) grad_norm 2.0338 (2.8015) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][600/625] eta 0:00:09 lr 0.000104 wd 0.0500 time 0.3724 (0.3794) data time 0.0009 (0.0018) model time 0.3715 (0.3775) loss 2.6922 (2.6389) grad_norm 2.1649 (2.7981) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][610/625] eta 0:00:05 lr 0.000104 wd 0.0500 time 0.3578 (0.3792) data time 0.0005 (0.0018) model time 0.3573 (0.3774) loss 2.8384 (2.6404) grad_norm 2.1031 (2.8062) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [262/300][620/625] eta 0:00:01 lr 0.000104 wd 0.0500 time 0.3584 (0.3793) data time 0.0005 (0.0017) model time 0.3578 (0.3774) loss 3.1215 (2.6441) grad_norm 3.4072 (2.8034) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:21 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 262 training takes 0:03:56 [2024-09-01 13:41:21 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:41:22 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.510 (0.510) Loss 0.5229 (0.5229) Acc@1 89.209 (89.209) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 13:41:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 0.8569 (0.6212) Acc@1 79.932 (86.697) Acc@5 95.508 (97.781) Mem 11224MB [2024-09-01 13:41:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.9316 (0.7368) Acc@1 77.930 (83.696) Acc@5 95.020 (96.617) Mem 11224MB [2024-09-01 13:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.445 Acc@5 96.627 [2024-09-01 13:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.4% [2024-09-01 13:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.871 (0.871) Loss 0.4961 (0.4961) Acc@1 89.160 (89.160) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 13:41:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.181) Loss 0.8086 (0.5939) Acc@1 81.201 (87.189) Acc@5 95.703 (97.865) Mem 11224MB [2024-09-01 13:41:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.139) Loss 0.8857 (0.7056) Acc@1 79.297 (84.384) Acc@5 95.361 (96.759) Mem 11224MB [2024-09-01 13:41:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.073 Acc@5 96.775 [2024-09-01 13:41:28 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.1% [2024-09-01 13:41:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][0/625] eta 0:12:13 lr 0.000104 wd 0.0500 time 1.1732 (1.1732) data time 0.5811 (0.5811) model time 0.0000 (0.0000) loss 3.0106 (3.0106) grad_norm 4.8293 (4.8293) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][10/625] eta 0:04:35 lr 0.000104 wd 0.0500 time 0.3741 (0.4483) data time 0.0009 (0.0537) model time 0.0000 (0.0000) loss 2.5843 (2.5759) grad_norm 2.1435 (2.8630) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][20/625] eta 0:04:09 lr 0.000104 wd 0.0500 time 0.3742 (0.4128) data time 0.0008 (0.0291) model time 0.0000 (0.0000) loss 2.6217 (2.6111) grad_norm 2.1153 (2.7377) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][30/625] eta 0:03:58 lr 0.000104 wd 0.0500 time 0.3764 (0.4012) data time 0.0007 (0.0200) model time 0.0000 (0.0000) loss 2.6108 (2.6015) grad_norm 2.3249 (2.6884) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][40/625] eta 0:03:53 lr 0.000104 wd 0.0500 time 0.3751 (0.3987) data time 0.0008 (0.0154) model time 0.0000 (0.0000) loss 2.7156 (2.6061) grad_norm 2.2376 (2.5732) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][50/625] eta 0:03:46 lr 0.000104 wd 0.0500 time 0.3740 (0.3937) data time 0.0010 (0.0125) model time 0.0000 (0.0000) loss 2.8094 (2.6217) grad_norm 3.8035 (2.6454) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][60/625] eta 0:03:45 lr 0.000104 wd 0.0500 time 0.3737 (0.3989) data time 0.0010 (0.0106) model time 0.3726 (0.4244) loss 2.4566 (2.5882) grad_norm 2.2256 (2.9212) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:41:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][70/625] eta 0:03:39 lr 0.000104 wd 0.0500 time 0.3805 (0.3954) data time 0.0010 (0.0093) model time 0.3796 (0.3986) loss 2.9981 (2.6159) grad_norm 2.4028 (2.8743) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][80/625] eta 0:03:33 lr 0.000104 wd 0.0500 time 0.3715 (0.3926) data time 0.0007 (0.0082) model time 0.3708 (0.3898) loss 2.2739 (2.6273) grad_norm 2.6171 (2.8556) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][90/625] eta 0:03:29 lr 0.000103 wd 0.0500 time 0.3699 (0.3908) data time 0.0009 (0.0075) model time 0.3690 (0.3860) loss 2.2059 (2.6050) grad_norm 4.2512 (2.9190) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][100/625] eta 0:03:24 lr 0.000103 wd 0.0500 time 0.3725 (0.3889) data time 0.0010 (0.0068) model time 0.3716 (0.3831) loss 2.4786 (2.6156) grad_norm 1.6732 (2.9344) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][110/625] eta 0:03:19 lr 0.000103 wd 0.0500 time 0.3721 (0.3874) data time 0.0009 (0.0063) model time 0.3712 (0.3811) loss 2.7215 (2.6274) grad_norm 4.0545 (3.0267) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][120/625] eta 0:03:15 lr 0.000103 wd 0.0500 time 0.3742 (0.3863) data time 0.0007 (0.0059) model time 0.3735 (0.3800) loss 3.3211 (2.6490) grad_norm 2.7836 (2.9890) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][130/625] eta 0:03:10 lr 0.000103 wd 0.0500 time 0.3784 (0.3852) data time 0.0009 (0.0055) model time 0.3775 (0.3788) loss 1.6846 (2.6467) grad_norm 2.1044 (2.9363) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][140/625] eta 0:03:06 lr 0.000103 wd 0.0500 time 0.3736 (0.3842) data time 0.0010 (0.0052) model time 0.3726 (0.3779) loss 2.1314 (2.6472) grad_norm 2.5073 (2.8820) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][150/625] eta 0:03:02 lr 0.000103 wd 0.0500 time 0.3755 (0.3837) data time 0.0007 (0.0049) model time 0.3748 (0.3775) loss 2.3600 (2.6488) grad_norm 2.4079 (2.8734) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][160/625] eta 0:02:58 lr 0.000103 wd 0.0500 time 0.3734 (0.3831) data time 0.0009 (0.0046) model time 0.3725 (0.3772) loss 2.3549 (2.6634) grad_norm 2.4290 (2.9209) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][170/625] eta 0:02:54 lr 0.000103 wd 0.0500 time 0.3754 (0.3827) data time 0.0008 (0.0044) model time 0.3746 (0.3770) loss 1.6249 (2.6578) grad_norm 3.5768 (3.0351) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:42:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][180/625] eta 0:02:50 lr 0.000103 wd 0.0500 time 0.3758 (0.3822) data time 0.0008 (0.0042) model time 0.3750 (0.3767) loss 2.6975 (2.6583) grad_norm 4.7141 (3.0315) loss_scale 512.0000 (270.1436) mem 11224MB [2024-09-01 13:42:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][190/625] eta 0:02:46 lr 0.000103 wd 0.0500 time 0.3764 (0.3819) data time 0.0009 (0.0041) model time 0.3755 (0.3766) loss 2.8094 (2.6689) grad_norm 2.0504 (3.0064) loss_scale 512.0000 (282.8063) mem 11224MB [2024-09-01 13:42:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][200/625] eta 0:02:42 lr 0.000103 wd 0.0500 time 0.3662 (0.3815) data time 0.0009 (0.0039) model time 0.3653 (0.3763) loss 2.4761 (2.6727) grad_norm 3.4410 (3.0302) loss_scale 512.0000 (294.2090) mem 11224MB [2024-09-01 13:42:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][210/625] eta 0:02:38 lr 0.000103 wd 0.0500 time 0.3739 (0.3822) data time 0.0007 (0.0038) model time 0.3731 (0.3775) loss 2.4199 (2.6657) grad_norm 2.1986 (3.0321) loss_scale 512.0000 (304.5308) mem 11224MB [2024-09-01 13:42:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][220/625] eta 0:02:34 lr 0.000103 wd 0.0500 time 0.3724 (0.3826) data time 0.0007 (0.0037) model time 0.3717 (0.3782) loss 2.9245 (2.6677) grad_norm 2.0857 (3.0390) loss_scale 512.0000 (313.9186) mem 11224MB [2024-09-01 13:42:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][230/625] eta 0:02:30 lr 0.000102 wd 0.0500 time 0.3908 (0.3822) data time 0.0010 (0.0035) model time 0.3898 (0.3780) loss 2.3672 (2.6762) grad_norm 2.7028 (3.0273) loss_scale 512.0000 (322.4935) mem 11224MB [2024-09-01 13:43:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][240/625] eta 0:02:27 lr 0.000102 wd 0.0500 time 0.5668 (0.3833) data time 0.0010 (0.0034) model time 0.5659 (0.3795) loss 2.1838 (2.6815) grad_norm 3.5730 (3.0273) loss_scale 512.0000 (330.3568) mem 11224MB [2024-09-01 13:43:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][250/625] eta 0:02:23 lr 0.000102 wd 0.0500 time 0.3734 (0.3829) data time 0.0010 (0.0033) model time 0.3724 (0.3792) loss 2.3063 (2.6808) grad_norm 2.4200 (3.0220) loss_scale 512.0000 (337.5936) mem 11224MB [2024-09-01 13:43:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][260/625] eta 0:02:19 lr 0.000102 wd 0.0500 time 0.4049 (0.3828) data time 0.0011 (0.0033) model time 0.4038 (0.3791) loss 2.3147 (2.6745) grad_norm 2.3955 (3.0013) loss_scale 512.0000 (344.2759) mem 11224MB [2024-09-01 13:43:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][270/625] eta 0:02:15 lr 0.000102 wd 0.0500 time 0.3685 (0.3825) data time 0.0010 (0.0032) model time 0.3675 (0.3789) loss 3.0110 (2.6794) grad_norm 2.5899 (2.9911) loss_scale 512.0000 (350.4649) mem 11224MB [2024-09-01 13:43:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][280/625] eta 0:02:11 lr 0.000102 wd 0.0500 time 0.3722 (0.3822) data time 0.0010 (0.0031) model time 0.3713 (0.3786) loss 2.8290 (2.6735) grad_norm 3.0685 (2.9774) loss_scale 512.0000 (356.2135) mem 11224MB [2024-09-01 13:43:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][290/625] eta 0:02:07 lr 0.000102 wd 0.0500 time 0.3728 (0.3818) data time 0.0009 (0.0030) model time 0.3719 (0.3783) loss 2.5065 (2.6728) grad_norm 2.0505 (2.9625) loss_scale 512.0000 (361.5670) mem 11224MB [2024-09-01 13:43:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][300/625] eta 0:02:04 lr 0.000102 wd 0.0500 time 0.3767 (0.3816) data time 0.0010 (0.0029) model time 0.3757 (0.3781) loss 2.7974 (2.6730) grad_norm 1.9651 (2.9818) loss_scale 512.0000 (366.5648) mem 11224MB [2024-09-01 13:43:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][310/625] eta 0:02:00 lr 0.000102 wd 0.0500 time 0.3720 (0.3813) data time 0.0008 (0.0029) model time 0.3712 (0.3779) loss 2.9306 (2.6788) grad_norm 2.8167 (2.9792) loss_scale 512.0000 (371.2412) mem 11224MB [2024-09-01 13:43:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][320/625] eta 0:01:56 lr 0.000102 wd 0.0500 time 0.3877 (0.3811) data time 0.0010 (0.0028) model time 0.3868 (0.3777) loss 1.7914 (2.6749) grad_norm 3.4216 (2.9724) loss_scale 512.0000 (375.6262) mem 11224MB [2024-09-01 13:43:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][330/625] eta 0:01:52 lr 0.000102 wd 0.0500 time 0.3711 (0.3808) data time 0.0008 (0.0028) model time 0.3703 (0.3775) loss 2.3807 (2.6784) grad_norm 2.4559 (2.9666) loss_scale 512.0000 (379.7462) mem 11224MB [2024-09-01 13:43:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][340/625] eta 0:01:48 lr 0.000102 wd 0.0500 time 0.3708 (0.3811) data time 0.0010 (0.0027) model time 0.3698 (0.3779) loss 1.9437 (2.6793) grad_norm 2.3885 (2.9510) loss_scale 512.0000 (383.6246) mem 11224MB [2024-09-01 13:43:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][350/625] eta 0:01:44 lr 0.000102 wd 0.0500 time 0.3695 (0.3809) data time 0.0007 (0.0027) model time 0.3688 (0.3778) loss 2.0181 (2.6841) grad_norm 1.6895 (2.9439) loss_scale 512.0000 (387.2821) mem 11224MB [2024-09-01 13:43:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][360/625] eta 0:01:41 lr 0.000102 wd 0.0500 time 0.5858 (0.3819) data time 0.0007 (0.0026) model time 0.5850 (0.3789) loss 2.4986 (2.6876) grad_norm 2.4469 (2.9316) loss_scale 512.0000 (390.7368) mem 11224MB [2024-09-01 13:43:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][370/625] eta 0:01:37 lr 0.000101 wd 0.0500 time 0.3722 (0.3817) data time 0.0009 (0.0026) model time 0.3713 (0.3788) loss 3.1289 (2.6920) grad_norm 2.0813 (2.9113) loss_scale 512.0000 (394.0054) mem 11224MB [2024-09-01 13:43:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][380/625] eta 0:01:33 lr 0.000101 wd 0.0500 time 0.3742 (0.3815) data time 0.0007 (0.0025) model time 0.3735 (0.3787) loss 2.7533 (2.6973) grad_norm 1.6822 (2.8957) loss_scale 512.0000 (397.1024) mem 11224MB [2024-09-01 13:43:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][390/625] eta 0:01:29 lr 0.000101 wd 0.0500 time 0.3775 (0.3818) data time 0.0009 (0.0025) model time 0.3766 (0.3790) loss 2.2596 (2.6940) grad_norm 3.0182 (2.8882) loss_scale 512.0000 (400.0409) mem 11224MB [2024-09-01 13:44:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][400/625] eta 0:01:25 lr 0.000101 wd 0.0500 time 0.3754 (0.3816) data time 0.0010 (0.0024) model time 0.3744 (0.3788) loss 2.3508 (2.6986) grad_norm 2.4315 (2.8774) loss_scale 512.0000 (402.8329) mem 11224MB [2024-09-01 13:44:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][410/625] eta 0:01:21 lr 0.000101 wd 0.0500 time 0.3744 (0.3814) data time 0.0009 (0.0024) model time 0.3735 (0.3786) loss 2.7782 (2.6913) grad_norm 2.1724 (2.8672) loss_scale 512.0000 (405.4891) mem 11224MB [2024-09-01 13:44:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][420/625] eta 0:01:18 lr 0.000101 wd 0.0500 time 0.3659 (0.3812) data time 0.0007 (0.0024) model time 0.3652 (0.3784) loss 2.5693 (2.6919) grad_norm 2.3998 (2.8747) loss_scale 512.0000 (408.0190) mem 11224MB [2024-09-01 13:44:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][430/625] eta 0:01:14 lr 0.000101 wd 0.0500 time 0.3720 (0.3810) data time 0.0008 (0.0024) model time 0.3712 (0.3783) loss 2.8550 (2.6860) grad_norm 2.7194 (2.8706) loss_scale 512.0000 (410.4316) mem 11224MB [2024-09-01 13:44:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][440/625] eta 0:01:10 lr 0.000101 wd 0.0500 time 0.3727 (0.3808) data time 0.0008 (0.0023) model time 0.3720 (0.3781) loss 2.2878 (2.6905) grad_norm 2.0637 (2.8698) loss_scale 512.0000 (412.7347) mem 11224MB [2024-09-01 13:44:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][450/625] eta 0:01:06 lr 0.000101 wd 0.0500 time 0.3619 (0.3806) data time 0.0011 (0.0023) model time 0.3608 (0.3779) loss 2.3345 (2.6885) grad_norm 1.8495 (2.8688) loss_scale 512.0000 (414.9357) mem 11224MB [2024-09-01 13:44:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][460/625] eta 0:01:02 lr 0.000101 wd 0.0500 time 0.3691 (0.3805) data time 0.0008 (0.0023) model time 0.3683 (0.3778) loss 1.5283 (2.6847) grad_norm 1.8287 (2.8589) loss_scale 512.0000 (417.0412) mem 11224MB [2024-09-01 13:44:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][470/625] eta 0:00:58 lr 0.000101 wd 0.0500 time 0.3739 (0.3804) data time 0.0007 (0.0022) model time 0.3732 (0.3777) loss 1.8420 (2.6830) grad_norm 2.6322 (2.8597) loss_scale 512.0000 (419.0573) mem 11224MB [2024-09-01 13:44:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][480/625] eta 0:00:55 lr 0.000101 wd 0.0500 time 0.3683 (0.3802) data time 0.0008 (0.0022) model time 0.3675 (0.3776) loss 1.6798 (2.6816) grad_norm 2.8809 (2.8533) loss_scale 512.0000 (420.9896) mem 11224MB [2024-09-01 13:44:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][490/625] eta 0:00:51 lr 0.000101 wd 0.0500 time 0.3808 (0.3801) data time 0.0009 (0.0022) model time 0.3799 (0.3775) loss 2.2792 (2.6825) grad_norm 2.8649 (2.8485) loss_scale 512.0000 (422.8432) mem 11224MB [2024-09-01 13:44:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][500/625] eta 0:00:47 lr 0.000101 wd 0.0500 time 0.3755 (0.3799) data time 0.0007 (0.0022) model time 0.3748 (0.3774) loss 2.2401 (2.6798) grad_norm 5.0032 (2.8594) loss_scale 512.0000 (424.6228) mem 11224MB [2024-09-01 13:44:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][510/625] eta 0:00:43 lr 0.000100 wd 0.0500 time 0.3750 (0.3802) data time 0.0008 (0.0021) model time 0.3743 (0.3777) loss 2.6294 (2.6787) grad_norm 2.3798 (2.8685) loss_scale 512.0000 (426.3327) mem 11224MB [2024-09-01 13:44:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][520/625] eta 0:00:39 lr 0.000100 wd 0.0500 time 0.3699 (0.3800) data time 0.0010 (0.0021) model time 0.3689 (0.3775) loss 3.0249 (2.6759) grad_norm 3.1719 (2.8672) loss_scale 512.0000 (427.9770) mem 11224MB [2024-09-01 13:44:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][530/625] eta 0:00:36 lr 0.000100 wd 0.0500 time 0.3706 (0.3799) data time 0.0009 (0.0021) model time 0.3696 (0.3774) loss 2.2315 (2.6667) grad_norm 2.9716 (2.8751) loss_scale 512.0000 (429.5593) mem 11224MB [2024-09-01 13:44:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][540/625] eta 0:00:32 lr 0.000100 wd 0.0500 time 0.3762 (0.3797) data time 0.0012 (0.0021) model time 0.3751 (0.3773) loss 1.7977 (2.6644) grad_norm 4.0020 (2.8709) loss_scale 512.0000 (431.0832) mem 11224MB [2024-09-01 13:44:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][550/625] eta 0:00:28 lr 0.000100 wd 0.0500 time 0.3740 (0.3799) data time 0.0009 (0.0021) model time 0.3731 (0.3775) loss 3.1741 (2.6645) grad_norm 3.0348 (2.8665) loss_scale 512.0000 (432.5517) mem 11224MB [2024-09-01 13:45:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][560/625] eta 0:00:24 lr 0.000100 wd 0.0500 time 0.3719 (0.3798) data time 0.0008 (0.0020) model time 0.3712 (0.3774) loss 2.6763 (2.6610) grad_norm 2.3807 (2.8639) loss_scale 512.0000 (433.9679) mem 11224MB [2024-09-01 13:45:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][570/625] eta 0:00:20 lr 0.000100 wd 0.0500 time 0.3683 (0.3804) data time 0.0011 (0.0020) model time 0.3672 (0.3781) loss 2.7486 (2.6629) grad_norm 1.8258 (2.8638) loss_scale 512.0000 (435.3345) mem 11224MB [2024-09-01 13:45:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][580/625] eta 0:00:17 lr 0.000100 wd 0.0500 time 0.3749 (0.3804) data time 0.0010 (0.0020) model time 0.3739 (0.3781) loss 2.5153 (2.6594) grad_norm 2.2417 (2.8599) loss_scale 512.0000 (436.6540) mem 11224MB [2024-09-01 13:45:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][590/625] eta 0:00:13 lr 0.000100 wd 0.0500 time 0.3757 (0.3803) data time 0.0010 (0.0020) model time 0.3747 (0.3780) loss 3.0824 (2.6615) grad_norm 2.1585 (2.8685) loss_scale 512.0000 (437.9289) mem 11224MB [2024-09-01 13:45:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][600/625] eta 0:00:09 lr 0.000100 wd 0.0500 time 0.3714 (0.3802) data time 0.0010 (0.0020) model time 0.3704 (0.3779) loss 2.5423 (2.6636) grad_norm 1.8144 (2.8592) loss_scale 512.0000 (439.1614) mem 11224MB [2024-09-01 13:45:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][610/625] eta 0:00:05 lr 0.000100 wd 0.0500 time 0.3596 (0.3800) data time 0.0005 (0.0020) model time 0.3591 (0.3778) loss 3.2727 (2.6686) grad_norm 17.5709 (2.8813) loss_scale 512.0000 (440.3535) mem 11224MB [2024-09-01 13:45:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [263/300][620/625] eta 0:00:01 lr 0.000100 wd 0.0500 time 0.3613 (0.3798) data time 0.0007 (0.0019) model time 0.3606 (0.3775) loss 1.5749 (2.6617) grad_norm 2.9896 (2.8773) loss_scale 512.0000 (441.5072) mem 11224MB [2024-09-01 13:45:25 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 263 training takes 0:03:57 [2024-09-01 13:45:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:45:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:45:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.503 (0.503) Loss 0.5210 (0.5210) Acc@1 89.355 (89.355) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 13:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.145) Loss 0.8325 (0.6171) Acc@1 80.859 (86.763) Acc@5 95.264 (97.776) Mem 11224MB [2024-09-01 13:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 0.9209 (0.7312) Acc@1 78.857 (83.817) Acc@5 95.215 (96.608) Mem 11224MB [2024-09-01 13:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.559 Acc@5 96.613 [2024-09-01 13:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 13:45:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.777 (0.777) Loss 0.4976 (0.4976) Acc@1 89.160 (89.160) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 13:45:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.176) Loss 0.8091 (0.5944) Acc@1 81.104 (87.180) Acc@5 95.752 (97.878) Mem 11224MB [2024-09-01 13:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.8862 (0.7062) Acc@1 79.199 (84.361) Acc@5 95.361 (96.770) Mem 11224MB [2024-09-01 13:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.047 Acc@5 96.783 [2024-09-01 13:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 13:45:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][0/625] eta 0:13:20 lr 0.000100 wd 0.0500 time 1.2812 (1.2812) data time 0.5095 (0.5095) model time 0.0000 (0.0000) loss 2.1234 (2.1234) grad_norm 2.9443 (2.9443) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:45:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][10/625] eta 0:04:40 lr 0.000100 wd 0.0500 time 0.3755 (0.4562) data time 0.0010 (0.0472) model time 0.0000 (0.0000) loss 2.7040 (2.6531) grad_norm 2.0241 (2.8199) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:45:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][20/625] eta 0:04:11 lr 0.000100 wd 0.0500 time 0.3641 (0.4162) data time 0.0010 (0.0252) model time 0.0000 (0.0000) loss 2.7271 (2.6954) grad_norm 2.6942 (2.7207) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:45:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][30/625] eta 0:03:59 lr 0.000099 wd 0.0500 time 0.3688 (0.4023) data time 0.0009 (0.0174) model time 0.0000 (0.0000) loss 2.8549 (2.6086) grad_norm 2.9863 (2.6657) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:45:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][40/625] eta 0:03:51 lr 0.000099 wd 0.0500 time 0.3706 (0.3950) data time 0.0008 (0.0134) model time 0.0000 (0.0000) loss 2.8504 (2.6213) grad_norm 2.6143 (2.7007) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:45:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][50/625] eta 0:03:46 lr 0.000099 wd 0.0500 time 0.5179 (0.3931) data time 0.0007 (0.0109) model time 0.0000 (0.0000) loss 2.4213 (2.6238) grad_norm 2.0586 (2.6691) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:45:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][60/625] eta 0:03:40 lr 0.000099 wd 0.0500 time 0.3805 (0.3897) data time 0.0010 (0.0093) model time 0.3795 (0.3717) loss 2.3480 (2.6384) grad_norm 1.7726 (2.8466) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][70/625] eta 0:03:34 lr 0.000099 wd 0.0500 time 0.3743 (0.3873) data time 0.0007 (0.0081) model time 0.3735 (0.3717) loss 2.2909 (2.6251) grad_norm 3.3913 (2.8661) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][80/625] eta 0:03:31 lr 0.000099 wd 0.0500 time 0.3697 (0.3875) data time 0.0009 (0.0072) model time 0.3688 (0.3770) loss 3.0587 (2.6486) grad_norm 1.9420 (2.8285) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][90/625] eta 0:03:26 lr 0.000099 wd 0.0500 time 0.3715 (0.3856) data time 0.0010 (0.0065) model time 0.3705 (0.3752) loss 2.0437 (2.6329) grad_norm 1.9884 (2.8235) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][100/625] eta 0:03:21 lr 0.000099 wd 0.0500 time 0.3637 (0.3842) data time 0.0010 (0.0060) model time 0.3627 (0.3743) loss 2.8424 (2.6426) grad_norm 1.9310 (2.8108) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][110/625] eta 0:03:17 lr 0.000099 wd 0.0500 time 0.3724 (0.3831) data time 0.0008 (0.0055) model time 0.3716 (0.3736) loss 2.2637 (2.6293) grad_norm 2.1351 (2.8593) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][120/625] eta 0:03:12 lr 0.000099 wd 0.0500 time 0.3690 (0.3821) data time 0.0007 (0.0052) model time 0.3683 (0.3732) loss 1.9095 (2.6249) grad_norm 2.2632 (2.8115) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][130/625] eta 0:03:08 lr 0.000099 wd 0.0500 time 0.3661 (0.3813) data time 0.0007 (0.0048) model time 0.3654 (0.3728) loss 2.7824 (2.6199) grad_norm 1.5441 (2.7902) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][140/625] eta 0:03:04 lr 0.000099 wd 0.0500 time 0.3710 (0.3807) data time 0.0010 (0.0046) model time 0.3700 (0.3727) loss 2.8921 (2.6317) grad_norm 2.1414 (2.7724) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][150/625] eta 0:03:01 lr 0.000099 wd 0.0500 time 0.3813 (0.3829) data time 0.0009 (0.0043) model time 0.3804 (0.3767) loss 3.1163 (2.6263) grad_norm 2.5374 (2.7429) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][160/625] eta 0:02:57 lr 0.000099 wd 0.0500 time 0.3711 (0.3823) data time 0.0007 (0.0041) model time 0.3704 (0.3764) loss 2.7017 (2.6260) grad_norm 4.0749 (2.7640) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][170/625] eta 0:02:53 lr 0.000098 wd 0.0500 time 0.3795 (0.3820) data time 0.0010 (0.0039) model time 0.3785 (0.3763) loss 2.6552 (2.6348) grad_norm 5.1609 (2.7701) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][180/625] eta 0:02:49 lr 0.000098 wd 0.0500 time 0.3725 (0.3815) data time 0.0008 (0.0038) model time 0.3718 (0.3760) loss 1.7789 (2.6299) grad_norm 2.9987 (2.7994) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][190/625] eta 0:02:45 lr 0.000098 wd 0.0500 time 0.3719 (0.3812) data time 0.0008 (0.0036) model time 0.3711 (0.3759) loss 2.8277 (2.6146) grad_norm 3.0069 (2.8235) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][200/625] eta 0:02:42 lr 0.000098 wd 0.0500 time 0.3712 (0.3818) data time 0.0008 (0.0035) model time 0.3704 (0.3770) loss 3.0023 (2.6192) grad_norm 2.4156 (2.8014) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][210/625] eta 0:02:38 lr 0.000098 wd 0.0500 time 0.3759 (0.3814) data time 0.0009 (0.0034) model time 0.3750 (0.3767) loss 1.9950 (2.6141) grad_norm 2.0297 (2.7851) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:46:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][220/625] eta 0:02:34 lr 0.000098 wd 0.0500 time 0.3649 (0.3809) data time 0.0010 (0.0033) model time 0.3640 (0.3763) loss 3.0660 (2.6131) grad_norm 2.3649 (2.8763) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:47:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][230/625] eta 0:02:30 lr 0.000098 wd 0.0500 time 0.3699 (0.3805) data time 0.0010 (0.0032) model time 0.3690 (0.3760) loss 1.5341 (2.6068) grad_norm 2.6562 (2.8782) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:47:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][240/625] eta 0:02:26 lr 0.000098 wd 0.0500 time 0.3725 (0.3802) data time 0.0007 (0.0031) model time 0.3718 (0.3758) loss 2.9190 (2.6143) grad_norm 2.6967 (2.8694) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:47:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][250/625] eta 0:02:22 lr 0.000098 wd 0.0500 time 0.3715 (0.3799) data time 0.0009 (0.0030) model time 0.3706 (0.3756) loss 2.8583 (2.6250) grad_norm 3.2084 (2.9566) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:47:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][260/625] eta 0:02:18 lr 0.000098 wd 0.0500 time 0.5756 (0.3803) data time 0.0009 (0.0029) model time 0.5746 (0.3763) loss 2.4557 (2.6248) grad_norm 2.9803 (2.9681) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:47:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][270/625] eta 0:02:14 lr 0.000098 wd 0.0500 time 0.3770 (0.3800) data time 0.0007 (0.0028) model time 0.3762 (0.3761) loss 2.9082 (2.6329) grad_norm 2.0973 (2.9602) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 13:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][280/625] eta 0:02:11 lr 0.000098 wd 0.0500 time 0.3645 (0.3798) data time 0.0012 (0.0028) model time 0.3633 (0.3759) loss 2.8426 (2.6437) grad_norm 2.3742 (inf) loss_scale 256.0000 (505.6228) mem 11224MB [2024-09-01 13:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][290/625] eta 0:02:07 lr 0.000098 wd 0.0500 time 0.3728 (0.3796) data time 0.0007 (0.0027) model time 0.3721 (0.3758) loss 2.8938 (2.6468) grad_norm 2.5687 (inf) loss_scale 256.0000 (497.0447) mem 11224MB [2024-09-01 13:47:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][300/625] eta 0:02:03 lr 0.000098 wd 0.0500 time 0.3796 (0.3794) data time 0.0007 (0.0026) model time 0.3789 (0.3756) loss 2.2759 (2.6486) grad_norm 2.4690 (inf) loss_scale 256.0000 (489.0365) mem 11224MB [2024-09-01 13:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][310/625] eta 0:01:59 lr 0.000098 wd 0.0500 time 0.3711 (0.3792) data time 0.0007 (0.0026) model time 0.3704 (0.3755) loss 2.2951 (2.6480) grad_norm 1.8541 (inf) loss_scale 256.0000 (481.5434) mem 11224MB [2024-09-01 13:47:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][320/625] eta 0:01:55 lr 0.000097 wd 0.0500 time 0.3720 (0.3790) data time 0.0010 (0.0025) model time 0.3710 (0.3754) loss 2.7025 (2.6437) grad_norm 1.8463 (inf) loss_scale 256.0000 (474.5171) mem 11224MB [2024-09-01 13:47:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][330/625] eta 0:01:51 lr 0.000097 wd 0.0500 time 0.3742 (0.3792) data time 0.0008 (0.0025) model time 0.3734 (0.3757) loss 2.3096 (2.6353) grad_norm 2.4711 (inf) loss_scale 256.0000 (467.9154) mem 11224MB [2024-09-01 13:47:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][340/625] eta 0:01:48 lr 0.000097 wd 0.0500 time 0.3780 (0.3802) data time 0.0008 (0.0024) model time 0.3772 (0.3770) loss 2.8804 (2.6442) grad_norm 2.0137 (inf) loss_scale 256.0000 (461.7009) mem 11224MB [2024-09-01 13:47:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][350/625] eta 0:01:44 lr 0.000097 wd 0.0500 time 0.3780 (0.3801) data time 0.0009 (0.0024) model time 0.3771 (0.3769) loss 2.4818 (2.6454) grad_norm 2.3822 (inf) loss_scale 256.0000 (455.8405) mem 11224MB [2024-09-01 13:47:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][360/625] eta 0:01:40 lr 0.000097 wd 0.0500 time 0.3757 (0.3799) data time 0.0008 (0.0024) model time 0.3749 (0.3768) loss 2.8659 (2.6504) grad_norm 2.6185 (inf) loss_scale 256.0000 (450.3047) mem 11224MB [2024-09-01 13:47:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][370/625] eta 0:01:36 lr 0.000097 wd 0.0500 time 0.3725 (0.3797) data time 0.0008 (0.0023) model time 0.3717 (0.3766) loss 1.8839 (2.6452) grad_norm 1.9246 (inf) loss_scale 256.0000 (445.0674) mem 11224MB [2024-09-01 13:47:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][380/625] eta 0:01:32 lr 0.000097 wd 0.0500 time 0.3744 (0.3795) data time 0.0008 (0.0023) model time 0.3736 (0.3765) loss 1.4755 (2.6389) grad_norm 2.4446 (inf) loss_scale 256.0000 (440.1050) mem 11224MB [2024-09-01 13:48:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][390/625] eta 0:01:29 lr 0.000097 wd 0.0500 time 0.3708 (0.3793) data time 0.0008 (0.0023) model time 0.3699 (0.3763) loss 2.7423 (2.6383) grad_norm 2.1856 (inf) loss_scale 256.0000 (435.3964) mem 11224MB [2024-09-01 13:48:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][400/625] eta 0:01:25 lr 0.000097 wd 0.0500 time 0.3784 (0.3792) data time 0.0007 (0.0022) model time 0.3777 (0.3763) loss 2.9197 (2.6367) grad_norm 8.7509 (inf) loss_scale 256.0000 (430.9227) mem 11224MB [2024-09-01 13:48:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][410/625] eta 0:01:21 lr 0.000097 wd 0.0500 time 0.3755 (0.3795) data time 0.0007 (0.0022) model time 0.3747 (0.3766) loss 3.2486 (2.6370) grad_norm 2.4226 (inf) loss_scale 256.0000 (426.6667) mem 11224MB [2024-09-01 13:48:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][420/625] eta 0:01:17 lr 0.000097 wd 0.0500 time 0.3759 (0.3793) data time 0.0008 (0.0022) model time 0.3751 (0.3765) loss 3.3901 (2.6380) grad_norm 3.1890 (inf) loss_scale 256.0000 (422.6128) mem 11224MB [2024-09-01 13:48:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][430/625] eta 0:01:13 lr 0.000097 wd 0.0500 time 0.3698 (0.3792) data time 0.0010 (0.0021) model time 0.3688 (0.3764) loss 2.8494 (2.6404) grad_norm 2.3120 (inf) loss_scale 256.0000 (418.7471) mem 11224MB [2024-09-01 13:48:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][440/625] eta 0:01:10 lr 0.000097 wd 0.0500 time 0.3636 (0.3790) data time 0.0008 (0.0021) model time 0.3628 (0.3762) loss 2.6004 (2.6429) grad_norm 2.3653 (inf) loss_scale 256.0000 (415.0567) mem 11224MB [2024-09-01 13:48:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][450/625] eta 0:01:06 lr 0.000097 wd 0.0500 time 0.5739 (0.3803) data time 0.0007 (0.0021) model time 0.5732 (0.3777) loss 2.9328 (2.6421) grad_norm 4.3835 (inf) loss_scale 256.0000 (411.5299) mem 11224MB [2024-09-01 13:48:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][460/625] eta 0:01:02 lr 0.000096 wd 0.0500 time 0.3772 (0.3805) data time 0.0007 (0.0021) model time 0.3765 (0.3780) loss 3.0777 (2.6428) grad_norm 2.1496 (inf) loss_scale 256.0000 (408.1562) mem 11224MB [2024-09-01 13:48:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][470/625] eta 0:00:58 lr 0.000096 wd 0.0500 time 0.3635 (0.3803) data time 0.0010 (0.0020) model time 0.3625 (0.3778) loss 2.3649 (2.6418) grad_norm 3.8997 (inf) loss_scale 256.0000 (404.9257) mem 11224MB [2024-09-01 13:48:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][480/625] eta 0:00:55 lr 0.000096 wd 0.0500 time 0.3699 (0.3802) data time 0.0009 (0.0020) model time 0.3690 (0.3777) loss 2.1623 (2.6387) grad_norm 2.9072 (inf) loss_scale 256.0000 (401.8295) mem 11224MB [2024-09-01 13:48:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][490/625] eta 0:00:51 lr 0.000096 wd 0.0500 time 0.3689 (0.3801) data time 0.0009 (0.0020) model time 0.3679 (0.3776) loss 2.7466 (2.6399) grad_norm 2.2085 (inf) loss_scale 256.0000 (398.8595) mem 11224MB [2024-09-01 13:48:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][500/625] eta 0:00:47 lr 0.000096 wd 0.0500 time 0.3722 (0.3799) data time 0.0010 (0.0020) model time 0.3712 (0.3775) loss 2.7367 (2.6403) grad_norm 2.7871 (inf) loss_scale 256.0000 (396.0080) mem 11224MB [2024-09-01 13:48:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][510/625] eta 0:00:43 lr 0.000096 wd 0.0500 time 0.3767 (0.3798) data time 0.0009 (0.0020) model time 0.3758 (0.3774) loss 3.2263 (2.6377) grad_norm 2.3171 (inf) loss_scale 256.0000 (393.2681) mem 11224MB [2024-09-01 13:48:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][520/625] eta 0:00:39 lr 0.000096 wd 0.0500 time 0.3749 (0.3797) data time 0.0008 (0.0019) model time 0.3741 (0.3773) loss 1.8422 (2.6350) grad_norm 1.9607 (inf) loss_scale 256.0000 (390.6334) mem 11224MB [2024-09-01 13:48:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][530/625] eta 0:00:36 lr 0.000096 wd 0.0500 time 0.3754 (0.3796) data time 0.0008 (0.0019) model time 0.3746 (0.3772) loss 3.2962 (2.6369) grad_norm 1.9106 (inf) loss_scale 256.0000 (388.0979) mem 11224MB [2024-09-01 13:48:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][540/625] eta 0:00:32 lr 0.000096 wd 0.0500 time 0.3835 (0.3795) data time 0.0010 (0.0019) model time 0.3825 (0.3772) loss 2.1616 (2.6352) grad_norm 3.3727 (inf) loss_scale 256.0000 (385.6562) mem 11224MB [2024-09-01 13:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][550/625] eta 0:00:28 lr 0.000096 wd 0.0500 time 0.3863 (0.3794) data time 0.0010 (0.0019) model time 0.3852 (0.3771) loss 3.0660 (2.6374) grad_norm 3.3951 (inf) loss_scale 256.0000 (383.3031) mem 11224MB [2024-09-01 13:49:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][560/625] eta 0:00:24 lr 0.000096 wd 0.0500 time 0.3728 (0.3794) data time 0.0008 (0.0019) model time 0.3720 (0.3770) loss 1.4123 (2.6327) grad_norm 3.4975 (inf) loss_scale 256.0000 (381.0339) mem 11224MB [2024-09-01 13:49:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][570/625] eta 0:00:20 lr 0.000096 wd 0.0500 time 0.3788 (0.3796) data time 0.0011 (0.0019) model time 0.3777 (0.3774) loss 2.8876 (2.6332) grad_norm 1.9308 (inf) loss_scale 256.0000 (378.8441) mem 11224MB [2024-09-01 13:49:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][580/625] eta 0:00:17 lr 0.000096 wd 0.0500 time 0.3691 (0.3795) data time 0.0007 (0.0018) model time 0.3684 (0.3773) loss 2.8708 (2.6351) grad_norm 2.8907 (inf) loss_scale 256.0000 (376.7298) mem 11224MB [2024-09-01 13:49:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][590/625] eta 0:00:13 lr 0.000096 wd 0.0500 time 0.3732 (0.3797) data time 0.0008 (0.0018) model time 0.3724 (0.3775) loss 2.0211 (2.6298) grad_norm 2.4661 (inf) loss_scale 256.0000 (374.6870) mem 11224MB [2024-09-01 13:49:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][600/625] eta 0:00:09 lr 0.000096 wd 0.0500 time 0.3647 (0.3796) data time 0.0010 (0.0018) model time 0.3637 (0.3774) loss 2.7808 (2.6308) grad_norm 4.0405 (inf) loss_scale 256.0000 (372.7121) mem 11224MB [2024-09-01 13:49:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][610/625] eta 0:00:05 lr 0.000095 wd 0.0500 time 0.3698 (0.3794) data time 0.0007 (0.0018) model time 0.3691 (0.3772) loss 2.8797 (2.6333) grad_norm 2.4358 (inf) loss_scale 256.0000 (370.8020) mem 11224MB [2024-09-01 13:49:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [264/300][620/625] eta 0:00:01 lr 0.000095 wd 0.0500 time 0.3570 (0.3791) data time 0.0005 (0.0018) model time 0.3565 (0.3769) loss 3.0138 (2.6337) grad_norm 2.2091 (inf) loss_scale 256.0000 (368.9533) mem 11224MB [2024-09-01 13:49:29 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 264 training takes 0:03:56 [2024-09-01 13:49:29 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:49:30 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:49:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 0.5146 (0.5146) Acc@1 89.307 (89.307) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 13:49:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.141) Loss 0.8369 (0.6161) Acc@1 81.006 (86.737) Acc@5 95.752 (97.745) Mem 11224MB [2024-09-01 13:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9209 (0.7334) Acc@1 78.467 (83.891) Acc@5 95.215 (96.543) Mem 11224MB [2024-09-01 13:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.603 Acc@5 96.587 [2024-09-01 13:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 13:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.831 (0.831) Loss 0.4980 (0.4980) Acc@1 89.160 (89.160) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 13:49:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.174) Loss 0.8096 (0.5947) Acc@1 81.201 (87.140) Acc@5 95.752 (97.874) Mem 11224MB [2024-09-01 13:49:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.136) Loss 0.8872 (0.7066) Acc@1 79.053 (84.305) Acc@5 95.312 (96.763) Mem 11224MB [2024-09-01 13:49:36 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.991 Acc@5 96.773 [2024-09-01 13:49:36 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 13:49:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][0/625] eta 0:12:32 lr 0.000095 wd 0.0500 time 1.2042 (1.2042) data time 0.6019 (0.6019) model time 0.0000 (0.0000) loss 2.6384 (2.6384) grad_norm 2.1752 (2.1752) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:49:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][10/625] eta 0:04:35 lr 0.000095 wd 0.0500 time 0.3702 (0.4478) data time 0.0008 (0.0556) model time 0.0000 (0.0000) loss 2.3461 (2.7014) grad_norm 2.3805 (2.3153) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:49:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][20/625] eta 0:04:20 lr 0.000095 wd 0.0500 time 0.5588 (0.4309) data time 0.0008 (0.0296) model time 0.0000 (0.0000) loss 2.6298 (2.6310) grad_norm 2.4204 (2.5024) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:49:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][30/625] eta 0:04:09 lr 0.000095 wd 0.0500 time 0.3720 (0.4194) data time 0.0008 (0.0203) model time 0.0000 (0.0000) loss 2.9866 (2.6479) grad_norm 2.7346 (2.5914) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:49:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][40/625] eta 0:04:00 lr 0.000095 wd 0.0500 time 0.3858 (0.4107) data time 0.0008 (0.0157) model time 0.0000 (0.0000) loss 2.9253 (2.6557) grad_norm 2.6914 (2.5416) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:49:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][50/625] eta 0:03:52 lr 0.000095 wd 0.0500 time 0.3697 (0.4037) data time 0.0009 (0.0128) model time 0.0000 (0.0000) loss 2.7219 (2.6746) grad_norm 2.6274 (2.6748) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][60/625] eta 0:03:45 lr 0.000095 wd 0.0500 time 0.3784 (0.3990) data time 0.0009 (0.0108) model time 0.3774 (0.3742) loss 2.5384 (2.6613) grad_norm 2.6388 (2.7314) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][70/625] eta 0:03:40 lr 0.000095 wd 0.0500 time 0.3782 (0.3979) data time 0.0009 (0.0095) model time 0.3773 (0.3821) loss 2.9292 (2.6797) grad_norm 2.8105 (2.7867) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][80/625] eta 0:03:35 lr 0.000095 wd 0.0500 time 0.3755 (0.3949) data time 0.0011 (0.0084) model time 0.3744 (0.3790) loss 2.8483 (2.6651) grad_norm 3.7378 (2.8295) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][90/625] eta 0:03:31 lr 0.000095 wd 0.0500 time 0.3741 (0.3947) data time 0.0009 (0.0076) model time 0.3731 (0.3823) loss 2.6270 (2.6776) grad_norm 1.6806 (2.7998) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][100/625] eta 0:03:26 lr 0.000095 wd 0.0500 time 0.3691 (0.3925) data time 0.0008 (0.0069) model time 0.3683 (0.3801) loss 3.5719 (2.6813) grad_norm 2.6637 (2.8758) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][110/625] eta 0:03:21 lr 0.000095 wd 0.0500 time 0.3708 (0.3909) data time 0.0010 (0.0064) model time 0.3698 (0.3790) loss 2.7543 (2.6849) grad_norm 1.9557 (2.8999) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][120/625] eta 0:03:16 lr 0.000095 wd 0.0500 time 0.3764 (0.3896) data time 0.0009 (0.0060) model time 0.3755 (0.3784) loss 2.5310 (2.6754) grad_norm 2.3980 (2.9156) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][130/625] eta 0:03:12 lr 0.000094 wd 0.0500 time 0.3854 (0.3885) data time 0.0009 (0.0056) model time 0.3845 (0.3778) loss 1.9141 (2.6683) grad_norm 3.4489 (2.9227) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][140/625] eta 0:03:07 lr 0.000094 wd 0.0500 time 0.3799 (0.3875) data time 0.0011 (0.0053) model time 0.3788 (0.3774) loss 2.6103 (2.6802) grad_norm 3.6862 (3.1803) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][150/625] eta 0:03:03 lr 0.000094 wd 0.0500 time 0.3744 (0.3866) data time 0.0007 (0.0050) model time 0.3737 (0.3770) loss 3.0460 (2.6968) grad_norm 2.2038 (3.1670) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][160/625] eta 0:02:59 lr 0.000094 wd 0.0500 time 0.3713 (0.3857) data time 0.0010 (0.0047) model time 0.3703 (0.3764) loss 2.7003 (2.7003) grad_norm 2.2028 (3.2348) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][170/625] eta 0:02:55 lr 0.000094 wd 0.0500 time 0.3767 (0.3849) data time 0.0009 (0.0045) model time 0.3757 (0.3759) loss 2.3203 (2.6851) grad_norm 2.0287 (3.1852) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][180/625] eta 0:02:50 lr 0.000094 wd 0.0500 time 0.3722 (0.3842) data time 0.0007 (0.0043) model time 0.3714 (0.3755) loss 3.3273 (2.6930) grad_norm 2.1215 (3.1412) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][190/625] eta 0:02:46 lr 0.000094 wd 0.0500 time 0.3735 (0.3836) data time 0.0008 (0.0041) model time 0.3728 (0.3753) loss 2.7820 (2.6961) grad_norm 3.7331 (3.1119) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][200/625] eta 0:02:42 lr 0.000094 wd 0.0500 time 0.3755 (0.3831) data time 0.0010 (0.0040) model time 0.3745 (0.3752) loss 2.4666 (2.6851) grad_norm 2.3185 (3.0765) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:50:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][210/625] eta 0:02:38 lr 0.000094 wd 0.0500 time 0.3728 (0.3827) data time 0.0007 (0.0038) model time 0.3720 (0.3751) loss 2.8734 (2.6924) grad_norm 2.4922 (3.1078) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][220/625] eta 0:02:34 lr 0.000094 wd 0.0500 time 0.3703 (0.3822) data time 0.0008 (0.0037) model time 0.3696 (0.3748) loss 2.4462 (2.6928) grad_norm 3.1088 (3.0850) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][230/625] eta 0:02:31 lr 0.000094 wd 0.0500 time 0.5902 (0.3836) data time 0.0010 (0.0036) model time 0.5893 (0.3769) loss 2.8435 (2.6896) grad_norm 3.0579 (3.0677) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][240/625] eta 0:02:27 lr 0.000094 wd 0.0500 time 0.3739 (0.3831) data time 0.0010 (0.0035) model time 0.3730 (0.3766) loss 2.3442 (2.6809) grad_norm 2.1264 (3.0709) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][250/625] eta 0:02:23 lr 0.000094 wd 0.0500 time 0.3686 (0.3827) data time 0.0010 (0.0034) model time 0.3675 (0.3763) loss 1.9275 (2.6803) grad_norm 4.0796 (3.0635) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][260/625] eta 0:02:19 lr 0.000094 wd 0.0500 time 0.3771 (0.3823) data time 0.0009 (0.0033) model time 0.3762 (0.3761) loss 1.6878 (2.6684) grad_norm 2.0958 (3.0499) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][270/625] eta 0:02:15 lr 0.000094 wd 0.0500 time 0.3744 (0.3820) data time 0.0010 (0.0032) model time 0.3734 (0.3760) loss 2.8707 (2.6764) grad_norm 2.1926 (3.0377) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][280/625] eta 0:02:11 lr 0.000093 wd 0.0500 time 0.3754 (0.3823) data time 0.0008 (0.0031) model time 0.3746 (0.3765) loss 2.8973 (2.6683) grad_norm 2.0434 (3.0148) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][290/625] eta 0:02:07 lr 0.000093 wd 0.0500 time 0.3674 (0.3820) data time 0.0010 (0.0031) model time 0.3664 (0.3764) loss 2.5794 (2.6743) grad_norm 2.1773 (3.0104) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][300/625] eta 0:02:04 lr 0.000093 wd 0.0500 time 0.3885 (0.3830) data time 0.0007 (0.0030) model time 0.3877 (0.3777) loss 2.5733 (2.6678) grad_norm 2.2134 (2.9877) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][310/625] eta 0:02:00 lr 0.000093 wd 0.0500 time 0.3808 (0.3827) data time 0.0009 (0.0029) model time 0.3799 (0.3776) loss 1.7832 (2.6610) grad_norm 2.3816 (2.9849) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][320/625] eta 0:01:56 lr 0.000093 wd 0.0500 time 0.3674 (0.3824) data time 0.0009 (0.0029) model time 0.3664 (0.3773) loss 1.9364 (2.6556) grad_norm 3.6275 (3.0125) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][330/625] eta 0:01:52 lr 0.000093 wd 0.0500 time 0.3693 (0.3821) data time 0.0009 (0.0028) model time 0.3684 (0.3772) loss 2.3694 (2.6595) grad_norm 2.2686 (2.9915) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][340/625] eta 0:01:48 lr 0.000093 wd 0.0500 time 0.3662 (0.3819) data time 0.0009 (0.0027) model time 0.3653 (0.3771) loss 2.4013 (2.6610) grad_norm 2.1395 (2.9721) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][350/625] eta 0:01:44 lr 0.000093 wd 0.0500 time 0.3815 (0.3817) data time 0.0009 (0.0027) model time 0.3806 (0.3770) loss 2.7358 (2.6590) grad_norm 2.9997 (2.9610) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][360/625] eta 0:01:41 lr 0.000093 wd 0.0500 time 0.3690 (0.3815) data time 0.0008 (0.0026) model time 0.3682 (0.3769) loss 2.7447 (2.6545) grad_norm 2.4557 (2.9644) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:51:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][370/625] eta 0:01:37 lr 0.000093 wd 0.0500 time 0.3670 (0.3813) data time 0.0008 (0.0026) model time 0.3662 (0.3768) loss 1.9343 (2.6488) grad_norm 2.9806 (2.9671) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][380/625] eta 0:01:33 lr 0.000093 wd 0.0500 time 0.3677 (0.3811) data time 0.0010 (0.0026) model time 0.3667 (0.3766) loss 2.7182 (2.6522) grad_norm 3.3793 (2.9615) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][390/625] eta 0:01:29 lr 0.000093 wd 0.0500 time 0.3742 (0.3809) data time 0.0007 (0.0025) model time 0.3735 (0.3765) loss 2.1544 (2.6575) grad_norm 1.9404 (2.9468) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][400/625] eta 0:01:25 lr 0.000093 wd 0.0500 time 0.3879 (0.3807) data time 0.0011 (0.0025) model time 0.3868 (0.3764) loss 2.5097 (2.6607) grad_norm 2.3545 (2.9371) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][410/625] eta 0:01:21 lr 0.000093 wd 0.0500 time 0.3733 (0.3810) data time 0.0010 (0.0024) model time 0.3722 (0.3768) loss 2.2269 (2.6591) grad_norm 2.8876 (2.9319) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][420/625] eta 0:01:18 lr 0.000093 wd 0.0500 time 0.3723 (0.3813) data time 0.0008 (0.0024) model time 0.3716 (0.3772) loss 3.1885 (2.6597) grad_norm 3.4368 (2.9379) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][430/625] eta 0:01:14 lr 0.000092 wd 0.0500 time 0.3715 (0.3811) data time 0.0010 (0.0024) model time 0.3705 (0.3771) loss 2.0223 (2.6540) grad_norm 2.8870 (2.9286) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][440/625] eta 0:01:10 lr 0.000092 wd 0.0500 time 0.3739 (0.3810) data time 0.0010 (0.0023) model time 0.3730 (0.3770) loss 2.8174 (2.6566) grad_norm 2.1394 (2.9311) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][450/625] eta 0:01:06 lr 0.000092 wd 0.0500 time 0.3841 (0.3808) data time 0.0009 (0.0023) model time 0.3832 (0.3769) loss 2.2207 (2.6542) grad_norm 2.7507 (2.9334) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][460/625] eta 0:01:02 lr 0.000092 wd 0.0500 time 0.3689 (0.3810) data time 0.0009 (0.0023) model time 0.3680 (0.3772) loss 2.5525 (2.6575) grad_norm 2.2561 (2.9248) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][470/625] eta 0:00:59 lr 0.000092 wd 0.0500 time 0.3771 (0.3808) data time 0.0009 (0.0023) model time 0.3762 (0.3771) loss 2.9616 (2.6631) grad_norm 2.2589 (2.9084) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][480/625] eta 0:00:55 lr 0.000092 wd 0.0500 time 0.3723 (0.3810) data time 0.0007 (0.0022) model time 0.3716 (0.3773) loss 2.9117 (2.6659) grad_norm 2.4254 (2.9422) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][490/625] eta 0:00:51 lr 0.000092 wd 0.0500 time 0.3702 (0.3808) data time 0.0008 (0.0022) model time 0.3695 (0.3771) loss 2.8949 (2.6607) grad_norm 2.9471 (2.9406) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][500/625] eta 0:00:47 lr 0.000092 wd 0.0500 time 0.3711 (0.3806) data time 0.0008 (0.0022) model time 0.3703 (0.3770) loss 2.4589 (2.6580) grad_norm 2.8249 (2.9353) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][510/625] eta 0:00:43 lr 0.000092 wd 0.0500 time 0.3752 (0.3804) data time 0.0009 (0.0022) model time 0.3743 (0.3769) loss 3.1306 (2.6560) grad_norm 1.9932 (2.9276) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][520/625] eta 0:00:39 lr 0.000092 wd 0.0500 time 0.3719 (0.3803) data time 0.0007 (0.0021) model time 0.3712 (0.3767) loss 1.7318 (2.6541) grad_norm 2.2686 (2.9388) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:52:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][530/625] eta 0:00:36 lr 0.000092 wd 0.0500 time 0.3687 (0.3805) data time 0.0010 (0.0021) model time 0.3677 (0.3771) loss 2.7206 (2.6521) grad_norm 3.3346 (2.9553) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][540/625] eta 0:00:32 lr 0.000092 wd 0.0500 time 0.3678 (0.3808) data time 0.0007 (0.0021) model time 0.3670 (0.3774) loss 2.9256 (2.6508) grad_norm 2.2605 (2.9527) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][550/625] eta 0:00:28 lr 0.000092 wd 0.0500 time 0.3660 (0.3806) data time 0.0009 (0.0021) model time 0.3651 (0.3773) loss 2.8516 (2.6522) grad_norm 2.0022 (2.9466) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][560/625] eta 0:00:24 lr 0.000092 wd 0.0500 time 0.3719 (0.3804) data time 0.0009 (0.0020) model time 0.3710 (0.3772) loss 2.4203 (2.6530) grad_norm 1.6583 (2.9592) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][570/625] eta 0:00:20 lr 0.000092 wd 0.0500 time 0.3705 (0.3803) data time 0.0007 (0.0020) model time 0.3698 (0.3770) loss 2.6319 (2.6524) grad_norm 2.6191 (2.9604) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][580/625] eta 0:00:17 lr 0.000091 wd 0.0500 time 0.3777 (0.3805) data time 0.0010 (0.0020) model time 0.3768 (0.3773) loss 2.5831 (2.6512) grad_norm 2.4525 (2.9512) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][590/625] eta 0:00:13 lr 0.000091 wd 0.0500 time 0.3696 (0.3804) data time 0.0008 (0.0020) model time 0.3688 (0.3772) loss 3.0203 (2.6557) grad_norm 2.0324 (2.9370) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][600/625] eta 0:00:09 lr 0.000091 wd 0.0500 time 0.3687 (0.3806) data time 0.0010 (0.0020) model time 0.3677 (0.3774) loss 2.7318 (2.6586) grad_norm 2.0355 (2.9583) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][610/625] eta 0:00:05 lr 0.000091 wd 0.0500 time 0.3685 (0.3804) data time 0.0007 (0.0020) model time 0.3678 (0.3773) loss 2.0085 (2.6552) grad_norm 2.4320 (2.9471) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [265/300][620/625] eta 0:00:01 lr 0.000091 wd 0.0500 time 0.3651 (0.3801) data time 0.0007 (0.0019) model time 0.3644 (0.3771) loss 2.9877 (2.6551) grad_norm 3.3704 (2.9547) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 265 training takes 0:03:57 [2024-09-01 13:53:34 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:53:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.5068 (0.5068) Acc@1 88.672 (88.672) Acc@5 98.730 (98.730) Mem 11224MB [2024-09-01 13:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.145) Loss 0.8345 (0.6077) Acc@1 79.980 (86.705) Acc@5 95.801 (97.865) Mem 11224MB [2024-09-01 13:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.122) Loss 0.9258 (0.7281) Acc@1 78.027 (83.784) Acc@5 95.459 (96.617) Mem 11224MB [2024-09-01 13:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.523 Acc@5 96.613 [2024-09-01 13:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 13:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.866 (0.866) Loss 0.4985 (0.4985) Acc@1 89.209 (89.209) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 13:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.177) Loss 0.8091 (0.5949) Acc@1 81.250 (87.145) Acc@5 95.850 (97.887) Mem 11224MB [2024-09-01 13:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8877 (0.7070) Acc@1 79.150 (84.317) Acc@5 95.459 (96.773) Mem 11224MB [2024-09-01 13:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 84.003 Acc@5 96.781 [2024-09-01 13:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 13:53:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][0/625] eta 0:12:04 lr 0.000091 wd 0.0500 time 1.1587 (1.1587) data time 0.6122 (0.6122) model time 0.0000 (0.0000) loss 2.7010 (2.7010) grad_norm 3.2193 (3.2193) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][10/625] eta 0:04:32 lr 0.000091 wd 0.0500 time 0.3760 (0.4436) data time 0.0009 (0.0565) model time 0.0000 (0.0000) loss 2.9899 (2.6823) grad_norm 2.7038 (6.6294) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][20/625] eta 0:04:07 lr 0.000091 wd 0.0500 time 0.3722 (0.4097) data time 0.0010 (0.0301) model time 0.0000 (0.0000) loss 2.5863 (2.6346) grad_norm 3.0334 (4.7433) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][30/625] eta 0:03:56 lr 0.000091 wd 0.0500 time 0.3765 (0.3982) data time 0.0010 (0.0207) model time 0.0000 (0.0000) loss 3.0284 (2.5284) grad_norm 3.3751 (4.5828) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][40/625] eta 0:03:49 lr 0.000091 wd 0.0500 time 0.3713 (0.3931) data time 0.0007 (0.0159) model time 0.0000 (0.0000) loss 3.0318 (2.5576) grad_norm 2.8698 (4.1019) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][50/625] eta 0:03:44 lr 0.000091 wd 0.0500 time 0.3859 (0.3896) data time 0.0010 (0.0129) model time 0.0000 (0.0000) loss 2.8372 (2.6067) grad_norm 2.7905 (4.0022) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][60/625] eta 0:03:38 lr 0.000091 wd 0.0500 time 0.3724 (0.3870) data time 0.0010 (0.0110) model time 0.3714 (0.3727) loss 2.3798 (2.5872) grad_norm 2.4801 (3.7771) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][70/625] eta 0:03:36 lr 0.000091 wd 0.0500 time 0.3691 (0.3900) data time 0.0009 (0.0096) model time 0.3682 (0.3901) loss 1.8627 (2.5846) grad_norm 2.3068 (3.6744) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][80/625] eta 0:03:31 lr 0.000091 wd 0.0500 time 0.3713 (0.3880) data time 0.0007 (0.0085) model time 0.3705 (0.3844) loss 2.7944 (2.5969) grad_norm 5.1687 (3.5725) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][90/625] eta 0:03:26 lr 0.000091 wd 0.0500 time 0.3712 (0.3864) data time 0.0008 (0.0077) model time 0.3704 (0.3813) loss 3.2041 (2.6046) grad_norm 2.5238 (3.4740) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][100/625] eta 0:03:22 lr 0.000091 wd 0.0500 time 0.3698 (0.3849) data time 0.0009 (0.0070) model time 0.3689 (0.3792) loss 2.6250 (2.5921) grad_norm 3.0309 (3.4060) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][110/625] eta 0:03:18 lr 0.000090 wd 0.0500 time 0.3771 (0.3851) data time 0.0009 (0.0065) model time 0.3762 (0.3804) loss 3.0670 (2.6088) grad_norm 2.4725 (3.2947) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][120/625] eta 0:03:14 lr 0.000090 wd 0.0500 time 0.3709 (0.3842) data time 0.0010 (0.0060) model time 0.3699 (0.3793) loss 1.5450 (2.6212) grad_norm 3.4806 (3.2525) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][130/625] eta 0:03:10 lr 0.000090 wd 0.0500 time 0.3695 (0.3847) data time 0.0008 (0.0056) model time 0.3687 (0.3805) loss 1.4503 (2.6149) grad_norm 2.3247 (3.2243) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][140/625] eta 0:03:06 lr 0.000090 wd 0.0500 time 0.3778 (0.3840) data time 0.0010 (0.0053) model time 0.3768 (0.3798) loss 2.9370 (2.6090) grad_norm 1.8921 (3.1584) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][150/625] eta 0:03:02 lr 0.000090 wd 0.0500 time 0.3649 (0.3832) data time 0.0007 (0.0050) model time 0.3642 (0.3789) loss 2.8230 (2.6136) grad_norm 2.0238 (3.1006) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][160/625] eta 0:02:57 lr 0.000090 wd 0.0500 time 0.3729 (0.3825) data time 0.0009 (0.0048) model time 0.3720 (0.3782) loss 2.8053 (2.6074) grad_norm 7.6307 (3.1696) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][170/625] eta 0:02:53 lr 0.000090 wd 0.0500 time 0.3670 (0.3818) data time 0.0007 (0.0045) model time 0.3663 (0.3776) loss 2.6499 (2.6132) grad_norm 3.2322 (3.1442) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][180/625] eta 0:02:49 lr 0.000090 wd 0.0500 time 0.3732 (0.3812) data time 0.0010 (0.0043) model time 0.3722 (0.3770) loss 2.7634 (2.6035) grad_norm 2.8018 (3.1211) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][190/625] eta 0:02:45 lr 0.000090 wd 0.0500 time 0.3750 (0.3808) data time 0.0007 (0.0042) model time 0.3743 (0.3766) loss 2.3577 (2.6105) grad_norm 66.5426 (3.4202) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:54:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][200/625] eta 0:02:41 lr 0.000090 wd 0.0500 time 0.3722 (0.3804) data time 0.0010 (0.0040) model time 0.3712 (0.3763) loss 3.2852 (2.6221) grad_norm 3.1272 (3.4415) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][210/625] eta 0:02:37 lr 0.000090 wd 0.0500 time 0.3745 (0.3801) data time 0.0007 (0.0039) model time 0.3737 (0.3761) loss 3.0401 (2.6175) grad_norm 3.7558 (3.4341) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][220/625] eta 0:02:33 lr 0.000090 wd 0.0500 time 0.3687 (0.3797) data time 0.0007 (0.0037) model time 0.3680 (0.3758) loss 2.0685 (2.6154) grad_norm 2.4447 (3.5001) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][230/625] eta 0:02:29 lr 0.000090 wd 0.0500 time 0.3690 (0.3794) data time 0.0007 (0.0036) model time 0.3683 (0.3755) loss 3.2287 (2.6128) grad_norm 1.9750 (3.4681) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][240/625] eta 0:02:25 lr 0.000090 wd 0.0500 time 0.3680 (0.3791) data time 0.0009 (0.0035) model time 0.3670 (0.3753) loss 2.2124 (2.6182) grad_norm 2.2419 (3.4313) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][250/625] eta 0:02:22 lr 0.000090 wd 0.0500 time 0.3714 (0.3788) data time 0.0009 (0.0034) model time 0.3705 (0.3751) loss 1.9044 (2.6122) grad_norm 2.7355 (3.4301) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][260/625] eta 0:02:18 lr 0.000089 wd 0.0500 time 0.3796 (0.3786) data time 0.0007 (0.0033) model time 0.3789 (0.3750) loss 3.2605 (2.6231) grad_norm 2.3176 (3.3954) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][270/625] eta 0:02:14 lr 0.000089 wd 0.0500 time 0.3712 (0.3785) data time 0.0006 (0.0032) model time 0.3706 (0.3750) loss 3.0204 (2.6295) grad_norm 3.3968 (3.3917) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][280/625] eta 0:02:10 lr 0.000089 wd 0.0500 time 0.3667 (0.3797) data time 0.0007 (0.0031) model time 0.3660 (0.3766) loss 2.0827 (2.6318) grad_norm 2.0828 (3.3717) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][290/625] eta 0:02:07 lr 0.000089 wd 0.0500 time 0.3796 (0.3795) data time 0.0011 (0.0031) model time 0.3785 (0.3765) loss 2.5616 (2.6299) grad_norm 5.0425 (3.3526) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][300/625] eta 0:02:03 lr 0.000089 wd 0.0500 time 0.3709 (0.3793) data time 0.0009 (0.0030) model time 0.3700 (0.3763) loss 3.0113 (2.6307) grad_norm 3.2794 (3.3533) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][310/625] eta 0:01:59 lr 0.000089 wd 0.0500 time 0.6006 (0.3799) data time 0.0010 (0.0029) model time 0.5997 (0.3770) loss 2.7452 (2.6294) grad_norm 1.9420 (3.3186) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][320/625] eta 0:01:55 lr 0.000089 wd 0.0500 time 0.3767 (0.3801) data time 0.0010 (0.0029) model time 0.3758 (0.3774) loss 2.8750 (2.6280) grad_norm 3.6497 (3.2922) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][330/625] eta 0:01:52 lr 0.000089 wd 0.0500 time 0.3753 (0.3799) data time 0.0008 (0.0028) model time 0.3746 (0.3772) loss 2.2452 (2.6305) grad_norm 2.2835 (3.2707) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][340/625] eta 0:01:48 lr 0.000089 wd 0.0500 time 0.3688 (0.3797) data time 0.0008 (0.0028) model time 0.3680 (0.3770) loss 3.2053 (2.6331) grad_norm 2.0599 (3.2391) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][350/625] eta 0:01:44 lr 0.000089 wd 0.0500 time 0.3696 (0.3795) data time 0.0008 (0.0027) model time 0.3689 (0.3768) loss 2.8125 (2.6364) grad_norm 2.6599 (3.2186) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:55:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][360/625] eta 0:01:40 lr 0.000089 wd 0.0500 time 0.3739 (0.3793) data time 0.0009 (0.0027) model time 0.3730 (0.3766) loss 1.3665 (2.6377) grad_norm 2.9165 (3.1933) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][370/625] eta 0:01:36 lr 0.000089 wd 0.0500 time 0.3696 (0.3792) data time 0.0008 (0.0026) model time 0.3688 (0.3765) loss 3.4773 (2.6415) grad_norm 2.2899 (3.1856) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][380/625] eta 0:01:32 lr 0.000089 wd 0.0500 time 0.3698 (0.3790) data time 0.0008 (0.0026) model time 0.3690 (0.3764) loss 2.7681 (2.6470) grad_norm 1.7866 (3.1737) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][390/625] eta 0:01:29 lr 0.000089 wd 0.0500 time 0.3659 (0.3789) data time 0.0008 (0.0025) model time 0.3651 (0.3763) loss 2.7487 (2.6428) grad_norm 2.6132 (3.1616) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][400/625] eta 0:01:25 lr 0.000089 wd 0.0500 time 0.3704 (0.3787) data time 0.0010 (0.0025) model time 0.3695 (0.3761) loss 1.9682 (2.6403) grad_norm 2.1932 (3.1603) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][410/625] eta 0:01:21 lr 0.000088 wd 0.0500 time 0.3700 (0.3785) data time 0.0008 (0.0025) model time 0.3692 (0.3760) loss 3.0215 (2.6395) grad_norm 2.7223 (3.1714) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][420/625] eta 0:01:17 lr 0.000088 wd 0.0500 time 0.3756 (0.3784) data time 0.0010 (0.0024) model time 0.3746 (0.3759) loss 2.8526 (2.6388) grad_norm 2.9231 (3.1775) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][430/625] eta 0:01:13 lr 0.000088 wd 0.0500 time 0.3460 (0.3787) data time 0.0007 (0.0024) model time 0.3452 (0.3763) loss 2.6441 (2.6417) grad_norm 3.1949 (3.1887) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][440/625] eta 0:01:10 lr 0.000088 wd 0.0500 time 0.3686 (0.3786) data time 0.0011 (0.0024) model time 0.3675 (0.3762) loss 2.1473 (2.6445) grad_norm 2.8376 (3.1812) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][450/625] eta 0:01:06 lr 0.000088 wd 0.0500 time 0.3740 (0.3785) data time 0.0009 (0.0023) model time 0.3730 (0.3762) loss 2.7078 (2.6487) grad_norm 2.9907 (3.1746) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][460/625] eta 0:01:02 lr 0.000088 wd 0.0500 time 0.5769 (0.3798) data time 0.0007 (0.0023) model time 0.5762 (0.3776) loss 2.9462 (2.6517) grad_norm 2.1757 (3.1637) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][470/625] eta 0:00:58 lr 0.000088 wd 0.0500 time 0.3716 (0.3796) data time 0.0008 (0.0023) model time 0.3708 (0.3774) loss 3.5397 (2.6508) grad_norm 3.5147 (3.1505) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][480/625] eta 0:00:55 lr 0.000088 wd 0.0500 time 0.3744 (0.3795) data time 0.0009 (0.0022) model time 0.3734 (0.3774) loss 2.9580 (2.6502) grad_norm 5.4018 (3.1529) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][490/625] eta 0:00:51 lr 0.000088 wd 0.0500 time 0.3694 (0.3794) data time 0.0007 (0.0022) model time 0.3687 (0.3773) loss 3.4907 (2.6540) grad_norm 2.2581 (3.1410) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][500/625] eta 0:00:47 lr 0.000088 wd 0.0500 time 0.3789 (0.3797) data time 0.0007 (0.0022) model time 0.3782 (0.3776) loss 3.0213 (2.6591) grad_norm 2.7867 (3.1277) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][510/625] eta 0:00:43 lr 0.000088 wd 0.0500 time 0.3789 (0.3796) data time 0.0009 (0.0022) model time 0.3781 (0.3775) loss 2.3935 (2.6578) grad_norm 2.4047 (3.1146) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:56:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][520/625] eta 0:00:39 lr 0.000088 wd 0.0500 time 0.3664 (0.3795) data time 0.0008 (0.0021) model time 0.3657 (0.3774) loss 3.4455 (2.6597) grad_norm 2.2257 (3.1044) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][530/625] eta 0:00:36 lr 0.000088 wd 0.0500 time 0.3664 (0.3794) data time 0.0007 (0.0021) model time 0.3657 (0.3773) loss 3.0447 (2.6588) grad_norm 2.2881 (3.0895) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][540/625] eta 0:00:32 lr 0.000088 wd 0.0500 time 0.3723 (0.3792) data time 0.0008 (0.0021) model time 0.3716 (0.3772) loss 2.7208 (2.6560) grad_norm 2.5463 (3.0806) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][550/625] eta 0:00:28 lr 0.000088 wd 0.0500 time 0.3872 (0.3791) data time 0.0011 (0.0021) model time 0.3862 (0.3771) loss 3.1038 (2.6556) grad_norm 2.9143 (3.0768) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][560/625] eta 0:00:24 lr 0.000088 wd 0.0500 time 0.3664 (0.3790) data time 0.0007 (0.0021) model time 0.3657 (0.3770) loss 1.8356 (2.6540) grad_norm 3.1629 (3.0713) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][570/625] eta 0:00:20 lr 0.000087 wd 0.0500 time 0.3735 (0.3790) data time 0.0008 (0.0020) model time 0.3727 (0.3769) loss 2.7012 (2.6524) grad_norm 2.2860 (3.0639) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][580/625] eta 0:00:17 lr 0.000087 wd 0.0500 time 0.3754 (0.3799) data time 0.0010 (0.0020) model time 0.3744 (0.3780) loss 2.9788 (2.6510) grad_norm 1.6160 (3.0567) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][590/625] eta 0:00:13 lr 0.000087 wd 0.0500 time 0.3711 (0.3798) data time 0.0007 (0.0020) model time 0.3704 (0.3779) loss 2.8016 (2.6527) grad_norm 2.9542 (3.0555) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][600/625] eta 0:00:09 lr 0.000087 wd 0.0500 time 0.3727 (0.3797) data time 0.0008 (0.0020) model time 0.3719 (0.3778) loss 2.0527 (2.6522) grad_norm 3.6662 (3.0534) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][610/625] eta 0:00:05 lr 0.000087 wd 0.0500 time 0.3585 (0.3795) data time 0.0007 (0.0020) model time 0.3578 (0.3776) loss 2.8368 (2.6538) grad_norm 2.7590 (3.0525) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [266/300][620/625] eta 0:00:01 lr 0.000087 wd 0.0500 time 0.3639 (0.3796) data time 0.0005 (0.0020) model time 0.3634 (0.3777) loss 2.4009 (2.6497) grad_norm 2.5209 (3.0464) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:38 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 266 training takes 0:03:57 [2024-09-01 13:57:38 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 13:57:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 13:57:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.493 (0.493) Loss 0.5054 (0.5054) Acc@1 89.404 (89.404) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 13:57:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.144) Loss 0.8140 (0.6041) Acc@1 80.859 (86.821) Acc@5 95.850 (97.794) Mem 11224MB [2024-09-01 13:57:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9214 (0.7238) Acc@1 78.809 (83.868) Acc@5 95.020 (96.556) Mem 11224MB [2024-09-01 13:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.589 Acc@5 96.575 [2024-09-01 13:57:43 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 13:57:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.817 (0.817) Loss 0.4990 (0.4990) Acc@1 89.258 (89.258) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 13:57:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.174) Loss 0.8091 (0.5951) Acc@1 81.201 (87.136) Acc@5 95.850 (97.856) Mem 11224MB [2024-09-01 13:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.8896 (0.7077) Acc@1 79.102 (84.305) Acc@5 95.459 (96.749) Mem 11224MB [2024-09-01 13:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.993 Acc@5 96.761 [2024-09-01 13:57:46 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 13:57:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][0/625] eta 0:12:38 lr 0.000087 wd 0.0500 time 1.2137 (1.2137) data time 0.7202 (0.7202) model time 0.0000 (0.0000) loss 3.0251 (3.0251) grad_norm 2.2296 (2.2296) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][10/625] eta 0:04:37 lr 0.000087 wd 0.0500 time 0.3730 (0.4508) data time 0.0007 (0.0663) model time 0.0000 (0.0000) loss 3.1887 (2.7070) grad_norm 1.8575 (3.1796) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][20/625] eta 0:04:11 lr 0.000087 wd 0.0500 time 0.3693 (0.4151) data time 0.0007 (0.0353) model time 0.0000 (0.0000) loss 1.9528 (2.6171) grad_norm 2.8823 (3.0140) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][30/625] eta 0:03:58 lr 0.000087 wd 0.0500 time 0.3724 (0.4014) data time 0.0007 (0.0242) model time 0.0000 (0.0000) loss 1.6943 (2.5284) grad_norm 6.2174 (3.0709) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][40/625] eta 0:03:51 lr 0.000087 wd 0.0500 time 0.3740 (0.3950) data time 0.0012 (0.0186) model time 0.0000 (0.0000) loss 1.8586 (2.5030) grad_norm 11.0651 (3.0927) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][50/625] eta 0:03:44 lr 0.000087 wd 0.0500 time 0.3737 (0.3910) data time 0.0011 (0.0151) model time 0.0000 (0.0000) loss 3.1482 (2.5561) grad_norm 2.1936 (3.0758) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][60/625] eta 0:03:39 lr 0.000087 wd 0.0500 time 0.3672 (0.3882) data time 0.0009 (0.0128) model time 0.3663 (0.3728) loss 2.7798 (2.5513) grad_norm 2.0894 (3.0319) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][70/625] eta 0:03:34 lr 0.000087 wd 0.0500 time 0.3680 (0.3861) data time 0.0007 (0.0111) model time 0.3673 (0.3725) loss 3.1683 (2.5445) grad_norm 10.1303 (3.0636) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][80/625] eta 0:03:29 lr 0.000087 wd 0.0500 time 0.3731 (0.3844) data time 0.0011 (0.0099) model time 0.3720 (0.3722) loss 2.6451 (2.5446) grad_norm 2.5554 (2.9986) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][90/625] eta 0:03:26 lr 0.000087 wd 0.0500 time 0.3790 (0.3853) data time 0.0007 (0.0089) model time 0.3783 (0.3772) loss 2.9645 (2.5741) grad_norm 1.7343 (2.9351) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][100/625] eta 0:03:23 lr 0.000086 wd 0.0500 time 0.3706 (0.3879) data time 0.0008 (0.0081) model time 0.3698 (0.3837) loss 3.0453 (2.5841) grad_norm 2.5930 (2.9864) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][110/625] eta 0:03:18 lr 0.000086 wd 0.0500 time 0.3663 (0.3864) data time 0.0010 (0.0075) model time 0.3653 (0.3815) loss 2.7080 (2.6005) grad_norm 2.1837 (2.9589) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][120/625] eta 0:03:14 lr 0.000086 wd 0.0500 time 0.3686 (0.3855) data time 0.0008 (0.0069) model time 0.3678 (0.3805) loss 2.6465 (2.5980) grad_norm 2.6942 (2.8965) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][130/625] eta 0:03:10 lr 0.000086 wd 0.0500 time 0.3717 (0.3846) data time 0.0007 (0.0065) model time 0.3709 (0.3796) loss 1.8053 (2.5942) grad_norm 11.6959 (2.9286) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][140/625] eta 0:03:06 lr 0.000086 wd 0.0500 time 0.3666 (0.3838) data time 0.0008 (0.0061) model time 0.3658 (0.3788) loss 3.0576 (2.5988) grad_norm 2.4276 (2.8828) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][150/625] eta 0:03:02 lr 0.000086 wd 0.0500 time 0.3804 (0.3833) data time 0.0009 (0.0057) model time 0.3794 (0.3785) loss 2.8965 (2.6134) grad_norm 3.2084 (2.8680) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][160/625] eta 0:02:58 lr 0.000086 wd 0.0500 time 0.3887 (0.3828) data time 0.0010 (0.0055) model time 0.3878 (0.3781) loss 2.5954 (2.6176) grad_norm 1.6628 (2.8411) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][170/625] eta 0:02:53 lr 0.000086 wd 0.0500 time 0.3716 (0.3822) data time 0.0009 (0.0052) model time 0.3707 (0.3776) loss 2.4434 (2.6246) grad_norm 2.6162 (2.8345) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][180/625] eta 0:02:50 lr 0.000086 wd 0.0500 time 0.3728 (0.3828) data time 0.0008 (0.0050) model time 0.3720 (0.3786) loss 2.6847 (2.6215) grad_norm 1.9800 (2.8530) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:58:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][190/625] eta 0:02:46 lr 0.000086 wd 0.0500 time 0.3647 (0.3821) data time 0.0010 (0.0047) model time 0.3638 (0.3779) loss 2.1310 (2.6187) grad_norm 4.0978 (2.8511) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][200/625] eta 0:02:42 lr 0.000086 wd 0.0500 time 0.3704 (0.3826) data time 0.0010 (0.0046) model time 0.3694 (0.3788) loss 2.6893 (2.6151) grad_norm 2.7030 (2.8365) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][210/625] eta 0:02:38 lr 0.000086 wd 0.0500 time 0.3697 (0.3821) data time 0.0009 (0.0044) model time 0.3688 (0.3783) loss 2.9521 (2.6174) grad_norm 3.5214 (2.8438) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][220/625] eta 0:02:34 lr 0.000086 wd 0.0500 time 0.3775 (0.3816) data time 0.0007 (0.0042) model time 0.3768 (0.3778) loss 1.9976 (2.6188) grad_norm 1.9858 (2.8131) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][230/625] eta 0:02:30 lr 0.000086 wd 0.0500 time 0.3667 (0.3812) data time 0.0009 (0.0041) model time 0.3657 (0.3774) loss 2.0222 (2.6306) grad_norm 2.6854 (2.8039) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][240/625] eta 0:02:26 lr 0.000086 wd 0.0500 time 0.3633 (0.3808) data time 0.0008 (0.0040) model time 0.3625 (0.3771) loss 3.0151 (2.6245) grad_norm 2.1648 (2.7950) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][250/625] eta 0:02:22 lr 0.000085 wd 0.0500 time 0.3730 (0.3805) data time 0.0010 (0.0039) model time 0.3720 (0.3768) loss 2.8447 (2.6271) grad_norm 2.1369 (2.7896) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][260/625] eta 0:02:18 lr 0.000085 wd 0.0500 time 0.3799 (0.3803) data time 0.0007 (0.0037) model time 0.3792 (0.3767) loss 2.0793 (2.6224) grad_norm 2.6191 (2.7695) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][270/625] eta 0:02:14 lr 0.000085 wd 0.0500 time 0.3726 (0.3801) data time 0.0007 (0.0036) model time 0.3719 (0.3766) loss 3.6764 (2.6213) grad_norm 3.0902 (2.8053) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][280/625] eta 0:02:11 lr 0.000085 wd 0.0500 time 0.3824 (0.3800) data time 0.0008 (0.0035) model time 0.3816 (0.3766) loss 2.7353 (2.6232) grad_norm 2.2167 (2.7998) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][290/625] eta 0:02:07 lr 0.000085 wd 0.0500 time 0.3769 (0.3798) data time 0.0009 (0.0035) model time 0.3760 (0.3765) loss 3.3657 (2.6252) grad_norm 2.7264 (2.7916) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][300/625] eta 0:02:03 lr 0.000085 wd 0.0500 time 0.3719 (0.3802) data time 0.0008 (0.0034) model time 0.3712 (0.3771) loss 2.5865 (2.6282) grad_norm 3.0359 (2.7817) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][310/625] eta 0:01:59 lr 0.000085 wd 0.0500 time 0.3776 (0.3807) data time 0.0009 (0.0033) model time 0.3767 (0.3777) loss 3.1028 (2.6351) grad_norm 2.7186 (2.7697) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][320/625] eta 0:01:56 lr 0.000085 wd 0.0500 time 0.3704 (0.3805) data time 0.0007 (0.0032) model time 0.3697 (0.3775) loss 2.8298 (2.6337) grad_norm 2.4981 (2.7674) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][330/625] eta 0:01:52 lr 0.000085 wd 0.0500 time 0.3748 (0.3803) data time 0.0008 (0.0032) model time 0.3740 (0.3773) loss 2.9594 (2.6349) grad_norm 2.5665 (2.7622) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][340/625] eta 0:01:48 lr 0.000085 wd 0.0500 time 0.3715 (0.3800) data time 0.0010 (0.0031) model time 0.3705 (0.3771) loss 2.7929 (2.6322) grad_norm 3.2699 (2.7604) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 13:59:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][350/625] eta 0:01:44 lr 0.000085 wd 0.0500 time 0.3758 (0.3798) data time 0.0009 (0.0030) model time 0.3749 (0.3770) loss 2.1582 (2.6365) grad_norm 2.9487 (2.7679) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:00:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][360/625] eta 0:01:40 lr 0.000085 wd 0.0500 time 0.3804 (0.3797) data time 0.0012 (0.0030) model time 0.3791 (0.3769) loss 2.9896 (2.6346) grad_norm 2.2291 (2.7673) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][370/625] eta 0:01:36 lr 0.000085 wd 0.0500 time 0.3654 (0.3795) data time 0.0011 (0.0029) model time 0.3643 (0.3767) loss 2.8904 (2.6389) grad_norm 1.9293 (2.7675) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][380/625] eta 0:01:32 lr 0.000085 wd 0.0500 time 0.3736 (0.3793) data time 0.0009 (0.0029) model time 0.3727 (0.3765) loss 1.9312 (2.6420) grad_norm 2.8083 (2.7726) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][390/625] eta 0:01:29 lr 0.000085 wd 0.0500 time 0.3755 (0.3795) data time 0.0010 (0.0028) model time 0.3746 (0.3768) loss 2.9207 (2.6469) grad_norm 2.4206 (2.7691) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][400/625] eta 0:01:25 lr 0.000085 wd 0.0500 time 0.3761 (0.3793) data time 0.0010 (0.0028) model time 0.3751 (0.3767) loss 2.8491 (2.6458) grad_norm 8.2452 (2.8048) loss_scale 512.0000 (257.2768) mem 11224MB [2024-09-01 14:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][410/625] eta 0:01:21 lr 0.000084 wd 0.0500 time 0.3707 (0.3792) data time 0.0011 (0.0027) model time 0.3696 (0.3765) loss 2.5888 (2.6515) grad_norm 2.3255 (2.8029) loss_scale 512.0000 (263.4745) mem 11224MB [2024-09-01 14:00:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][420/625] eta 0:01:17 lr 0.000084 wd 0.0500 time 0.3768 (0.3790) data time 0.0007 (0.0027) model time 0.3761 (0.3764) loss 2.9002 (2.6510) grad_norm 1.9097 (2.7994) loss_scale 512.0000 (269.3777) mem 11224MB [2024-09-01 14:00:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][430/625] eta 0:01:13 lr 0.000084 wd 0.0500 time 0.3730 (0.3789) data time 0.0008 (0.0026) model time 0.3723 (0.3763) loss 3.3687 (2.6514) grad_norm 2.7574 (2.8107) loss_scale 512.0000 (275.0070) mem 11224MB [2024-09-01 14:00:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][440/625] eta 0:01:10 lr 0.000084 wd 0.0500 time 0.3702 (0.3787) data time 0.0009 (0.0026) model time 0.3693 (0.3762) loss 2.4850 (2.6492) grad_norm 3.6370 (2.8163) loss_scale 512.0000 (280.3810) mem 11224MB [2024-09-01 14:00:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][450/625] eta 0:01:06 lr 0.000084 wd 0.0500 time 0.3733 (0.3786) data time 0.0007 (0.0026) model time 0.3725 (0.3760) loss 2.0456 (2.6499) grad_norm 2.2808 (2.8091) loss_scale 512.0000 (285.5166) mem 11224MB [2024-09-01 14:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][460/625] eta 0:01:02 lr 0.000084 wd 0.0500 time 0.3769 (0.3785) data time 0.0009 (0.0025) model time 0.3759 (0.3760) loss 1.7136 (2.6524) grad_norm 3.6607 (2.8178) loss_scale 512.0000 (290.4295) mem 11224MB [2024-09-01 14:00:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][470/625] eta 0:00:58 lr 0.000084 wd 0.0500 time 0.3765 (0.3784) data time 0.0012 (0.0025) model time 0.3753 (0.3759) loss 2.4978 (2.6555) grad_norm 2.2101 (2.8169) loss_scale 512.0000 (295.1338) mem 11224MB [2024-09-01 14:00:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][480/625] eta 0:00:54 lr 0.000084 wd 0.0500 time 0.3753 (0.3783) data time 0.0008 (0.0025) model time 0.3745 (0.3758) loss 1.6805 (2.6566) grad_norm 3.1039 (2.8130) loss_scale 512.0000 (299.6424) mem 11224MB [2024-09-01 14:00:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][490/625] eta 0:00:51 lr 0.000084 wd 0.0500 time 0.3787 (0.3794) data time 0.0007 (0.0024) model time 0.3780 (0.3771) loss 3.2130 (2.6534) grad_norm 3.3235 (2.8176) loss_scale 512.0000 (303.9674) mem 11224MB [2024-09-01 14:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][500/625] eta 0:00:47 lr 0.000084 wd 0.0500 time 0.3802 (0.3794) data time 0.0007 (0.0024) model time 0.3794 (0.3771) loss 2.4416 (2.6522) grad_norm 4.2512 (2.8141) loss_scale 512.0000 (308.1198) mem 11224MB [2024-09-01 14:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][510/625] eta 0:00:43 lr 0.000084 wd 0.0500 time 0.3664 (0.3799) data time 0.0008 (0.0024) model time 0.3657 (0.3777) loss 2.1661 (2.6531) grad_norm 3.1334 (2.8100) loss_scale 512.0000 (312.1096) mem 11224MB [2024-09-01 14:01:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][520/625] eta 0:00:39 lr 0.000084 wd 0.0500 time 0.3719 (0.3798) data time 0.0007 (0.0024) model time 0.3711 (0.3776) loss 1.8090 (2.6550) grad_norm 2.5473 (2.8030) loss_scale 512.0000 (315.9463) mem 11224MB [2024-09-01 14:01:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][530/625] eta 0:00:36 lr 0.000084 wd 0.0500 time 0.3800 (0.3797) data time 0.0009 (0.0023) model time 0.3791 (0.3775) loss 3.2557 (2.6536) grad_norm 2.4266 (2.7978) loss_scale 512.0000 (319.6384) mem 11224MB [2024-09-01 14:01:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][540/625] eta 0:00:32 lr 0.000084 wd 0.0500 time 0.3757 (0.3796) data time 0.0008 (0.0023) model time 0.3750 (0.3774) loss 2.9128 (2.6509) grad_norm 2.6836 (2.8142) loss_scale 512.0000 (323.1941) mem 11224MB [2024-09-01 14:01:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][550/625] eta 0:00:28 lr 0.000084 wd 0.0500 time 0.3740 (0.3795) data time 0.0007 (0.0023) model time 0.3733 (0.3773) loss 1.6362 (2.6483) grad_norm 2.5288 (2.8267) loss_scale 512.0000 (326.6207) mem 11224MB [2024-09-01 14:01:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][560/625] eta 0:00:24 lr 0.000084 wd 0.0500 time 0.3678 (0.3794) data time 0.0009 (0.0023) model time 0.3669 (0.3772) loss 3.1084 (2.6474) grad_norm 2.6670 (2.8415) loss_scale 512.0000 (329.9251) mem 11224MB [2024-09-01 14:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][570/625] eta 0:00:20 lr 0.000083 wd 0.0500 time 0.3743 (0.3793) data time 0.0010 (0.0022) model time 0.3734 (0.3772) loss 2.7265 (2.6501) grad_norm 2.8841 (2.8450) loss_scale 512.0000 (333.1138) mem 11224MB [2024-09-01 14:01:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][580/625] eta 0:00:17 lr 0.000083 wd 0.0500 time 0.3709 (0.3792) data time 0.0008 (0.0022) model time 0.3701 (0.3771) loss 3.4878 (2.6506) grad_norm 2.9653 (2.8463) loss_scale 512.0000 (336.1928) mem 11224MB [2024-09-01 14:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][590/625] eta 0:00:13 lr 0.000083 wd 0.0500 time 0.3709 (0.3791) data time 0.0007 (0.0022) model time 0.3702 (0.3770) loss 1.7070 (2.6489) grad_norm 3.1347 (2.8402) loss_scale 512.0000 (339.1675) mem 11224MB [2024-09-01 14:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][600/625] eta 0:00:09 lr 0.000083 wd 0.0500 time 0.5642 (0.3793) data time 0.0009 (0.0022) model time 0.5633 (0.3773) loss 2.6879 (2.6479) grad_norm 2.2539 (2.8356) loss_scale 512.0000 (342.0433) mem 11224MB [2024-09-01 14:01:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][610/625] eta 0:00:05 lr 0.000083 wd 0.0500 time 0.3580 (0.3799) data time 0.0005 (0.0022) model time 0.3575 (0.3779) loss 3.0071 (2.6479) grad_norm 2.9342 (2.8459) loss_scale 512.0000 (344.8249) mem 11224MB [2024-09-01 14:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [267/300][620/625] eta 0:00:01 lr 0.000083 wd 0.0500 time 0.3678 (0.3796) data time 0.0005 (0.0021) model time 0.3673 (0.3775) loss 3.0117 (2.6499) grad_norm 2.1670 (2.8439) loss_scale 512.0000 (347.5169) mem 11224MB [2024-09-01 14:01:43 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 267 training takes 0:03:57 [2024-09-01 14:01:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:01:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:01:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.505 (0.505) Loss 0.5244 (0.5244) Acc@1 89.062 (89.062) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 14:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.144) Loss 0.8398 (0.6168) Acc@1 80.371 (86.794) Acc@5 95.410 (97.732) Mem 11224MB [2024-09-01 14:01:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9429 (0.7352) Acc@1 77.588 (83.815) Acc@5 94.922 (96.547) Mem 11224MB [2024-09-01 14:01:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.541 Acc@5 96.541 [2024-09-01 14:01:47 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.5% [2024-09-01 14:01:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.825 (0.825) Loss 0.4998 (0.4998) Acc@1 89.258 (89.258) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.174) Loss 0.8096 (0.5954) Acc@1 81.006 (87.083) Acc@5 95.752 (97.843) Mem 11224MB [2024-09-01 14:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.136) Loss 0.8901 (0.7082) Acc@1 79.102 (84.268) Acc@5 95.459 (96.738) Mem 11224MB [2024-09-01 14:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.959 Acc@5 96.753 [2024-09-01 14:01:50 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 14:01:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][0/625] eta 0:12:38 lr 0.000083 wd 0.0500 time 1.2139 (1.2139) data time 0.8021 (0.8021) model time 0.0000 (0.0000) loss 1.8753 (1.8753) grad_norm 2.1454 (2.1454) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:01:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][10/625] eta 0:04:36 lr 0.000083 wd 0.0500 time 0.3707 (0.4489) data time 0.0009 (0.0738) model time 0.0000 (0.0000) loss 2.8699 (2.7229) grad_norm 2.9604 (3.2401) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:01:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][20/625] eta 0:04:09 lr 0.000083 wd 0.0500 time 0.3742 (0.4118) data time 0.0008 (0.0392) model time 0.0000 (0.0000) loss 2.6074 (2.7151) grad_norm 3.5076 (3.2112) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][30/625] eta 0:03:57 lr 0.000083 wd 0.0500 time 0.3645 (0.3988) data time 0.0010 (0.0269) model time 0.0000 (0.0000) loss 2.6412 (2.6315) grad_norm 2.3592 (3.0291) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][40/625] eta 0:03:51 lr 0.000083 wd 0.0500 time 0.3716 (0.3957) data time 0.0010 (0.0206) model time 0.0000 (0.0000) loss 2.5659 (2.6591) grad_norm 3.3110 (2.9387) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][50/625] eta 0:03:47 lr 0.000083 wd 0.0500 time 0.3756 (0.3951) data time 0.0007 (0.0167) model time 0.0000 (0.0000) loss 1.8364 (2.6446) grad_norm 4.7379 (2.9049) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][60/625] eta 0:03:40 lr 0.000083 wd 0.0500 time 0.3716 (0.3911) data time 0.0011 (0.0141) model time 0.3705 (0.3696) loss 2.8351 (2.6265) grad_norm 2.3184 (2.8762) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][70/625] eta 0:03:35 lr 0.000083 wd 0.0500 time 0.3682 (0.3887) data time 0.0008 (0.0123) model time 0.3674 (0.3713) loss 2.7656 (2.6422) grad_norm 3.5350 (3.0852) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][80/625] eta 0:03:30 lr 0.000083 wd 0.0500 time 0.3775 (0.3869) data time 0.0010 (0.0109) model time 0.3765 (0.3721) loss 2.8775 (2.6201) grad_norm 2.2506 (3.0433) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][90/625] eta 0:03:26 lr 0.000083 wd 0.0500 time 0.3761 (0.3857) data time 0.0009 (0.0098) model time 0.3752 (0.3728) loss 2.6859 (2.5953) grad_norm 3.4486 (3.1020) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][100/625] eta 0:03:21 lr 0.000083 wd 0.0500 time 0.3814 (0.3846) data time 0.0009 (0.0089) model time 0.3805 (0.3729) loss 2.9049 (2.5911) grad_norm 2.5340 (3.0869) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][110/625] eta 0:03:17 lr 0.000082 wd 0.0500 time 0.3698 (0.3834) data time 0.0011 (0.0082) model time 0.3687 (0.3725) loss 2.9271 (2.5888) grad_norm 2.5950 (3.0376) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][120/625] eta 0:03:13 lr 0.000082 wd 0.0500 time 0.3756 (0.3827) data time 0.0009 (0.0076) model time 0.3747 (0.3727) loss 2.8978 (2.5806) grad_norm 3.5894 (3.0315) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][130/625] eta 0:03:09 lr 0.000082 wd 0.0500 time 0.3730 (0.3819) data time 0.0007 (0.0071) model time 0.3723 (0.3725) loss 3.2470 (2.5937) grad_norm 4.9578 (3.0229) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][140/625] eta 0:03:04 lr 0.000082 wd 0.0500 time 0.3694 (0.3812) data time 0.0012 (0.0067) model time 0.3682 (0.3724) loss 2.5100 (2.5997) grad_norm 2.5424 (2.9825) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][150/625] eta 0:03:00 lr 0.000082 wd 0.0500 time 0.3709 (0.3806) data time 0.0011 (0.0063) model time 0.3699 (0.3723) loss 2.6238 (2.5981) grad_norm 2.6771 (2.9899) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][160/625] eta 0:02:57 lr 0.000082 wd 0.0500 time 0.5685 (0.3825) data time 0.0009 (0.0060) model time 0.5676 (0.3757) loss 2.1747 (2.5987) grad_norm 3.0084 (2.9918) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:02:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][170/625] eta 0:02:54 lr 0.000082 wd 0.0500 time 0.3721 (0.3842) data time 0.0007 (0.0057) model time 0.3714 (0.3786) loss 3.4286 (2.6103) grad_norm 2.2378 (2.9929) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][180/625] eta 0:02:50 lr 0.000082 wd 0.0500 time 0.3666 (0.3835) data time 0.0008 (0.0054) model time 0.3658 (0.3779) loss 2.1406 (2.6198) grad_norm 4.8353 (3.1267) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][190/625] eta 0:02:46 lr 0.000082 wd 0.0500 time 0.3723 (0.3828) data time 0.0009 (0.0052) model time 0.3714 (0.3774) loss 1.9607 (2.6141) grad_norm 3.5285 (3.1116) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][200/625] eta 0:02:42 lr 0.000082 wd 0.0500 time 0.3685 (0.3823) data time 0.0009 (0.0050) model time 0.3675 (0.3769) loss 2.0075 (2.6136) grad_norm 3.3074 (3.0785) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][210/625] eta 0:02:38 lr 0.000082 wd 0.0500 time 0.3739 (0.3818) data time 0.0007 (0.0048) model time 0.3731 (0.3766) loss 2.4231 (2.6195) grad_norm 20.5739 (3.1409) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][220/625] eta 0:02:34 lr 0.000082 wd 0.0500 time 0.3710 (0.3813) data time 0.0010 (0.0046) model time 0.3700 (0.3762) loss 2.6531 (2.6153) grad_norm 2.8479 (3.1709) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][230/625] eta 0:02:30 lr 0.000082 wd 0.0500 time 0.3681 (0.3809) data time 0.0009 (0.0044) model time 0.3672 (0.3759) loss 2.6955 (2.6181) grad_norm 2.2561 (3.1472) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][240/625] eta 0:02:26 lr 0.000082 wd 0.0500 time 0.3681 (0.3804) data time 0.0010 (0.0043) model time 0.3670 (0.3755) loss 3.2234 (2.6172) grad_norm 3.2965 (3.1308) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][250/625] eta 0:02:22 lr 0.000082 wd 0.0500 time 0.3721 (0.3801) data time 0.0008 (0.0042) model time 0.3713 (0.3753) loss 2.3747 (2.6218) grad_norm 2.3151 (3.1062) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][260/625] eta 0:02:18 lr 0.000082 wd 0.0500 time 0.3708 (0.3806) data time 0.0007 (0.0040) model time 0.3700 (0.3762) loss 2.2545 (2.6155) grad_norm 3.1894 (3.1094) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][270/625] eta 0:02:15 lr 0.000081 wd 0.0500 time 0.3743 (0.3803) data time 0.0009 (0.0039) model time 0.3735 (0.3759) loss 2.9036 (2.6172) grad_norm 2.5828 (3.1087) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][280/625] eta 0:02:11 lr 0.000081 wd 0.0500 time 0.3807 (0.3808) data time 0.0009 (0.0038) model time 0.3798 (0.3766) loss 2.2400 (2.6145) grad_norm 3.0555 (3.0920) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][290/625] eta 0:02:07 lr 0.000081 wd 0.0500 time 0.3756 (0.3819) data time 0.0007 (0.0037) model time 0.3749 (0.3781) loss 2.6036 (2.6159) grad_norm 3.1919 (3.1969) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 14:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][300/625] eta 0:02:03 lr 0.000081 wd 0.0500 time 0.3717 (0.3815) data time 0.0008 (0.0036) model time 0.3709 (0.3778) loss 2.7309 (2.6190) grad_norm 2.6554 (inf) loss_scale 256.0000 (510.2990) mem 11224MB [2024-09-01 14:03:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][310/625] eta 0:02:00 lr 0.000081 wd 0.0500 time 0.3678 (0.3812) data time 0.0008 (0.0035) model time 0.3669 (0.3775) loss 2.2352 (2.6228) grad_norm 2.7297 (inf) loss_scale 256.0000 (502.1222) mem 11224MB [2024-09-01 14:03:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][320/625] eta 0:01:56 lr 0.000081 wd 0.0500 time 0.3718 (0.3809) data time 0.0010 (0.0035) model time 0.3708 (0.3773) loss 2.7834 (2.6215) grad_norm 3.4369 (inf) loss_scale 256.0000 (494.4548) mem 11224MB [2024-09-01 14:03:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][330/625] eta 0:01:52 lr 0.000081 wd 0.0500 time 0.3703 (0.3807) data time 0.0010 (0.0034) model time 0.3693 (0.3771) loss 1.4320 (2.6171) grad_norm 2.4523 (inf) loss_scale 256.0000 (487.2508) mem 11224MB [2024-09-01 14:04:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][340/625] eta 0:01:48 lr 0.000081 wd 0.0500 time 0.3698 (0.3804) data time 0.0008 (0.0033) model time 0.3690 (0.3769) loss 2.5087 (2.6154) grad_norm 2.2816 (inf) loss_scale 256.0000 (480.4692) mem 11224MB [2024-09-01 14:04:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][350/625] eta 0:01:44 lr 0.000081 wd 0.0500 time 0.3720 (0.3802) data time 0.0008 (0.0033) model time 0.3712 (0.3767) loss 3.0152 (2.6222) grad_norm 2.4980 (inf) loss_scale 256.0000 (474.0741) mem 11224MB [2024-09-01 14:04:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][360/625] eta 0:01:40 lr 0.000081 wd 0.0500 time 0.3715 (0.3801) data time 0.0010 (0.0032) model time 0.3706 (0.3766) loss 2.8172 (2.6187) grad_norm 2.1509 (inf) loss_scale 256.0000 (468.0332) mem 11224MB [2024-09-01 14:04:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][370/625] eta 0:01:36 lr 0.000081 wd 0.0500 time 0.3765 (0.3803) data time 0.0009 (0.0031) model time 0.3756 (0.3770) loss 3.0343 (2.6260) grad_norm 2.2300 (inf) loss_scale 256.0000 (462.3181) mem 11224MB [2024-09-01 14:04:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][380/625] eta 0:01:33 lr 0.000081 wd 0.0500 time 0.3921 (0.3802) data time 0.0007 (0.0031) model time 0.3914 (0.3769) loss 2.5658 (2.6282) grad_norm 3.1807 (inf) loss_scale 256.0000 (456.9029) mem 11224MB [2024-09-01 14:04:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][390/625] eta 0:01:29 lr 0.000081 wd 0.0500 time 0.3815 (0.3800) data time 0.0010 (0.0030) model time 0.3806 (0.3767) loss 2.7518 (2.6273) grad_norm 3.0743 (inf) loss_scale 256.0000 (451.7647) mem 11224MB [2024-09-01 14:04:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][400/625] eta 0:01:25 lr 0.000081 wd 0.0500 time 0.3716 (0.3798) data time 0.0011 (0.0030) model time 0.3705 (0.3766) loss 2.3812 (2.6263) grad_norm 2.1540 (inf) loss_scale 256.0000 (446.8828) mem 11224MB [2024-09-01 14:04:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][410/625] eta 0:01:21 lr 0.000081 wd 0.0500 time 0.3713 (0.3796) data time 0.0008 (0.0029) model time 0.3705 (0.3765) loss 3.0680 (2.6335) grad_norm 3.6381 (inf) loss_scale 256.0000 (442.2384) mem 11224MB [2024-09-01 14:04:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][420/625] eta 0:01:17 lr 0.000081 wd 0.0500 time 0.3969 (0.3795) data time 0.0007 (0.0029) model time 0.3963 (0.3764) loss 2.8913 (2.6406) grad_norm 2.8190 (inf) loss_scale 256.0000 (437.8147) mem 11224MB [2024-09-01 14:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][430/625] eta 0:01:13 lr 0.000080 wd 0.0500 time 0.3733 (0.3793) data time 0.0007 (0.0028) model time 0.3726 (0.3763) loss 3.2334 (2.6412) grad_norm 1.9794 (inf) loss_scale 256.0000 (433.5963) mem 11224MB [2024-09-01 14:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][440/625] eta 0:01:10 lr 0.000080 wd 0.0500 time 0.5707 (0.3796) data time 0.0010 (0.0028) model time 0.5698 (0.3766) loss 1.8012 (2.6328) grad_norm 3.5585 (inf) loss_scale 256.0000 (429.5692) mem 11224MB [2024-09-01 14:04:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][450/625] eta 0:01:06 lr 0.000080 wd 0.0500 time 0.3791 (0.3799) data time 0.0007 (0.0027) model time 0.3784 (0.3770) loss 3.3450 (2.6338) grad_norm 3.1929 (inf) loss_scale 256.0000 (425.7206) mem 11224MB [2024-09-01 14:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][460/625] eta 0:01:02 lr 0.000080 wd 0.0500 time 0.3727 (0.3798) data time 0.0007 (0.0027) model time 0.3720 (0.3769) loss 2.0044 (2.6345) grad_norm 6.4436 (inf) loss_scale 256.0000 (422.0390) mem 11224MB [2024-09-01 14:04:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][470/625] eta 0:00:58 lr 0.000080 wd 0.0500 time 0.3738 (0.3797) data time 0.0009 (0.0027) model time 0.3729 (0.3768) loss 1.8922 (2.6346) grad_norm 2.4383 (inf) loss_scale 256.0000 (418.5138) mem 11224MB [2024-09-01 14:04:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][480/625] eta 0:00:55 lr 0.000080 wd 0.0500 time 0.3708 (0.3796) data time 0.0009 (0.0026) model time 0.3699 (0.3768) loss 2.4108 (2.6384) grad_norm 2.6933 (inf) loss_scale 256.0000 (415.1351) mem 11224MB [2024-09-01 14:04:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][490/625] eta 0:00:51 lr 0.000080 wd 0.0500 time 0.3708 (0.3799) data time 0.0009 (0.0026) model time 0.3699 (0.3772) loss 2.8216 (2.6390) grad_norm 3.2304 (inf) loss_scale 256.0000 (411.8941) mem 11224MB [2024-09-01 14:05:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][500/625] eta 0:00:47 lr 0.000080 wd 0.0500 time 0.3729 (0.3806) data time 0.0010 (0.0026) model time 0.3719 (0.3780) loss 2.5375 (2.6389) grad_norm 2.0206 (inf) loss_scale 256.0000 (408.7824) mem 11224MB [2024-09-01 14:05:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][510/625] eta 0:00:43 lr 0.000080 wd 0.0500 time 0.3683 (0.3805) data time 0.0009 (0.0025) model time 0.3674 (0.3779) loss 2.8804 (2.6363) grad_norm 2.7656 (inf) loss_scale 256.0000 (405.7926) mem 11224MB [2024-09-01 14:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][520/625] eta 0:00:39 lr 0.000080 wd 0.0500 time 0.3689 (0.3804) data time 0.0007 (0.0025) model time 0.3681 (0.3778) loss 2.8643 (2.6376) grad_norm 2.8142 (inf) loss_scale 256.0000 (402.9175) mem 11224MB [2024-09-01 14:05:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][530/625] eta 0:00:36 lr 0.000080 wd 0.0500 time 0.3671 (0.3803) data time 0.0009 (0.0025) model time 0.3661 (0.3778) loss 2.8005 (2.6391) grad_norm 2.6105 (inf) loss_scale 256.0000 (400.1507) mem 11224MB [2024-09-01 14:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][540/625] eta 0:00:32 lr 0.000080 wd 0.0500 time 0.3731 (0.3803) data time 0.0007 (0.0024) model time 0.3723 (0.3778) loss 3.0295 (2.6375) grad_norm 4.2263 (inf) loss_scale 256.0000 (397.4861) mem 11224MB [2024-09-01 14:05:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][550/625] eta 0:00:28 lr 0.000080 wd 0.0500 time 0.5872 (0.3805) data time 0.0009 (0.0024) model time 0.5863 (0.3781) loss 2.5758 (2.6367) grad_norm 3.1902 (inf) loss_scale 256.0000 (394.9183) mem 11224MB [2024-09-01 14:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][560/625] eta 0:00:24 lr 0.000080 wd 0.0500 time 0.5635 (0.3810) data time 0.0010 (0.0024) model time 0.5625 (0.3787) loss 2.4340 (2.6391) grad_norm 3.2273 (inf) loss_scale 256.0000 (392.4421) mem 11224MB [2024-09-01 14:05:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][570/625] eta 0:00:20 lr 0.000080 wd 0.0500 time 0.3696 (0.3809) data time 0.0010 (0.0024) model time 0.3686 (0.3785) loss 1.5674 (2.6363) grad_norm 2.8772 (inf) loss_scale 256.0000 (390.0525) mem 11224MB [2024-09-01 14:05:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][580/625] eta 0:00:17 lr 0.000080 wd 0.0500 time 0.3719 (0.3808) data time 0.0007 (0.0023) model time 0.3712 (0.3785) loss 2.0861 (2.6317) grad_norm 2.0827 (inf) loss_scale 256.0000 (387.7453) mem 11224MB [2024-09-01 14:05:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][590/625] eta 0:00:13 lr 0.000079 wd 0.0500 time 0.3754 (0.3807) data time 0.0008 (0.0023) model time 0.3746 (0.3783) loss 2.2321 (2.6311) grad_norm 3.3073 (inf) loss_scale 256.0000 (385.5161) mem 11224MB [2024-09-01 14:05:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][600/625] eta 0:00:09 lr 0.000079 wd 0.0500 time 0.3757 (0.3806) data time 0.0011 (0.0023) model time 0.3747 (0.3782) loss 2.5001 (2.6314) grad_norm 2.6264 (inf) loss_scale 256.0000 (383.3611) mem 11224MB [2024-09-01 14:05:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][610/625] eta 0:00:05 lr 0.000079 wd 0.0500 time 0.3642 (0.3804) data time 0.0007 (0.0023) model time 0.3635 (0.3781) loss 3.0137 (2.6333) grad_norm 2.9439 (inf) loss_scale 256.0000 (381.2766) mem 11224MB [2024-09-01 14:05:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [268/300][620/625] eta 0:00:01 lr 0.000079 wd 0.0500 time 0.3655 (0.3801) data time 0.0007 (0.0023) model time 0.3648 (0.3778) loss 2.8226 (2.6355) grad_norm 3.2294 (inf) loss_scale 256.0000 (379.2593) mem 11224MB [2024-09-01 14:05:48 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 268 training takes 0:03:57 [2024-09-01 14:05:48 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:05:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:05:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.507 (0.507) Loss 0.5205 (0.5205) Acc@1 88.867 (88.867) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.8481 (0.6160) Acc@1 80.420 (86.825) Acc@5 95.410 (97.767) Mem 11224MB [2024-09-01 14:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.9160 (0.7321) Acc@1 78.271 (83.938) Acc@5 95.459 (96.605) Mem 11224MB [2024-09-01 14:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.611 Acc@5 96.601 [2024-09-01 14:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 14:05:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.919 (0.919) Loss 0.4993 (0.4993) Acc@1 89.209 (89.209) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.181) Loss 0.8105 (0.5955) Acc@1 81.104 (87.092) Acc@5 95.654 (97.856) Mem 11224MB [2024-09-01 14:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.139) Loss 0.8926 (0.7085) Acc@1 79.102 (84.266) Acc@5 95.508 (96.731) Mem 11224MB [2024-09-01 14:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.959 Acc@5 96.741 [2024-09-01 14:05:55 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 14:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][0/625] eta 0:12:44 lr 0.000079 wd 0.0500 time 1.2233 (1.2233) data time 0.6410 (0.6410) model time 0.0000 (0.0000) loss 2.8615 (2.8615) grad_norm 2.2279 (2.2279) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][10/625] eta 0:04:37 lr 0.000079 wd 0.0500 time 0.3689 (0.4511) data time 0.0009 (0.0591) model time 0.0000 (0.0000) loss 2.5273 (2.7964) grad_norm 1.8664 (2.4907) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][20/625] eta 0:04:10 lr 0.000079 wd 0.0500 time 0.3766 (0.4135) data time 0.0007 (0.0314) model time 0.0000 (0.0000) loss 2.3050 (2.6640) grad_norm 2.0397 (2.6453) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][30/625] eta 0:04:01 lr 0.000079 wd 0.0500 time 0.3719 (0.4064) data time 0.0010 (0.0216) model time 0.0000 (0.0000) loss 2.3337 (2.6143) grad_norm 3.1788 (2.8098) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][40/625] eta 0:03:52 lr 0.000079 wd 0.0500 time 0.3751 (0.3979) data time 0.0008 (0.0166) model time 0.0000 (0.0000) loss 2.2871 (2.6120) grad_norm 2.2388 (2.7568) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][50/625] eta 0:03:45 lr 0.000079 wd 0.0500 time 0.3689 (0.3927) data time 0.0007 (0.0135) model time 0.0000 (0.0000) loss 3.1180 (2.6298) grad_norm 2.1162 (2.7681) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][60/625] eta 0:03:43 lr 0.000079 wd 0.0500 time 0.3696 (0.3962) data time 0.0012 (0.0114) model time 0.3684 (0.4128) loss 1.9251 (2.5809) grad_norm 4.5283 (2.8283) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][70/625] eta 0:03:39 lr 0.000079 wd 0.0500 time 0.3764 (0.3957) data time 0.0010 (0.0100) model time 0.3754 (0.4023) loss 2.4473 (2.5934) grad_norm 3.5107 (2.8151) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][80/625] eta 0:03:34 lr 0.000079 wd 0.0500 time 0.3817 (0.3928) data time 0.0010 (0.0089) model time 0.3808 (0.3920) loss 2.6965 (2.5947) grad_norm 2.7887 (2.8327) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][90/625] eta 0:03:31 lr 0.000079 wd 0.0500 time 0.3725 (0.3946) data time 0.0007 (0.0080) model time 0.3718 (0.3960) loss 3.1552 (2.6228) grad_norm 12.4125 (2.8982) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][100/625] eta 0:03:25 lr 0.000079 wd 0.0500 time 0.3749 (0.3923) data time 0.0009 (0.0073) model time 0.3740 (0.3909) loss 2.5864 (2.6503) grad_norm 2.2144 (2.8624) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][110/625] eta 0:03:20 lr 0.000079 wd 0.0500 time 0.3698 (0.3903) data time 0.0009 (0.0067) model time 0.3689 (0.3872) loss 2.8502 (2.6522) grad_norm 2.3265 (2.8211) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][120/625] eta 0:03:16 lr 0.000079 wd 0.0500 time 0.3745 (0.3891) data time 0.0008 (0.0063) model time 0.3738 (0.3854) loss 2.6890 (2.6548) grad_norm 2.5098 (2.7813) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][130/625] eta 0:03:11 lr 0.000078 wd 0.0500 time 0.3677 (0.3878) data time 0.0011 (0.0059) model time 0.3666 (0.3837) loss 2.2154 (2.6281) grad_norm 5.7748 (2.9032) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][140/625] eta 0:03:07 lr 0.000078 wd 0.0500 time 0.3752 (0.3868) data time 0.0008 (0.0055) model time 0.3744 (0.3825) loss 2.8841 (2.6291) grad_norm 3.7043 (2.9404) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][150/625] eta 0:03:03 lr 0.000078 wd 0.0500 time 0.3750 (0.3859) data time 0.0007 (0.0052) model time 0.3742 (0.3815) loss 2.2605 (2.6146) grad_norm 1.8907 (2.9332) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:06:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][160/625] eta 0:02:59 lr 0.000078 wd 0.0500 time 0.3732 (0.3852) data time 0.0010 (0.0049) model time 0.3722 (0.3807) loss 2.9997 (2.6309) grad_norm 11.8939 (2.9863) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][170/625] eta 0:02:55 lr 0.000078 wd 0.0500 time 0.3733 (0.3846) data time 0.0011 (0.0047) model time 0.3722 (0.3802) loss 2.9583 (2.6442) grad_norm 4.3618 (2.9646) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][180/625] eta 0:02:50 lr 0.000078 wd 0.0500 time 0.3788 (0.3840) data time 0.0007 (0.0045) model time 0.3781 (0.3797) loss 3.2512 (2.6486) grad_norm 2.8294 (2.9665) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][190/625] eta 0:02:46 lr 0.000078 wd 0.0500 time 0.3768 (0.3834) data time 0.0009 (0.0043) model time 0.3759 (0.3791) loss 2.1894 (2.6541) grad_norm 2.5463 (2.9727) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][200/625] eta 0:02:42 lr 0.000078 wd 0.0500 time 0.3653 (0.3829) data time 0.0010 (0.0042) model time 0.3643 (0.3786) loss 2.0993 (2.6504) grad_norm 3.7282 (2.9998) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][210/625] eta 0:02:38 lr 0.000078 wd 0.0500 time 0.3729 (0.3824) data time 0.0009 (0.0040) model time 0.3719 (0.3781) loss 3.1511 (2.6597) grad_norm 10.2201 (3.0392) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][220/625] eta 0:02:34 lr 0.000078 wd 0.0500 time 0.3722 (0.3819) data time 0.0007 (0.0039) model time 0.3716 (0.3777) loss 2.7977 (2.6737) grad_norm 2.4048 (3.0221) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][230/625] eta 0:02:30 lr 0.000078 wd 0.0500 time 0.3642 (0.3815) data time 0.0008 (0.0037) model time 0.3633 (0.3774) loss 2.2635 (2.6833) grad_norm 2.3875 (2.9957) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][240/625] eta 0:02:27 lr 0.000078 wd 0.0500 time 0.3816 (0.3826) data time 0.0008 (0.0036) model time 0.3808 (0.3789) loss 3.4638 (2.6840) grad_norm 2.9660 (2.9820) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][250/625] eta 0:02:24 lr 0.000078 wd 0.0500 time 0.3740 (0.3846) data time 0.0009 (0.0035) model time 0.3731 (0.3816) loss 2.3838 (2.6801) grad_norm 2.3926 (2.9613) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][260/625] eta 0:02:20 lr 0.000078 wd 0.0500 time 0.3708 (0.3842) data time 0.0010 (0.0034) model time 0.3699 (0.3811) loss 3.0800 (2.6779) grad_norm 3.7534 (2.9656) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][270/625] eta 0:02:16 lr 0.000078 wd 0.0500 time 0.3775 (0.3838) data time 0.0008 (0.0033) model time 0.3767 (0.3808) loss 1.6043 (2.6723) grad_norm 2.5436 (2.9560) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][280/625] eta 0:02:12 lr 0.000078 wd 0.0500 time 0.3870 (0.3836) data time 0.0008 (0.0032) model time 0.3862 (0.3806) loss 2.8792 (2.6759) grad_norm 5.4720 (2.9665) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][290/625] eta 0:02:08 lr 0.000078 wd 0.0500 time 0.3675 (0.3832) data time 0.0008 (0.0032) model time 0.3667 (0.3802) loss 2.7886 (2.6757) grad_norm 4.3042 (2.9841) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][300/625] eta 0:02:04 lr 0.000077 wd 0.0500 time 0.3772 (0.3835) data time 0.0008 (0.0031) model time 0.3764 (0.3807) loss 2.4901 (2.6746) grad_norm 2.2323 (2.9761) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][310/625] eta 0:02:00 lr 0.000077 wd 0.0500 time 0.3750 (0.3832) data time 0.0008 (0.0030) model time 0.3742 (0.3803) loss 2.6974 (2.6744) grad_norm 1.9735 (2.9565) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:07:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][320/625] eta 0:01:56 lr 0.000077 wd 0.0500 time 0.3734 (0.3828) data time 0.0008 (0.0030) model time 0.3726 (0.3800) loss 2.7654 (2.6724) grad_norm 2.9595 (2.9434) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:08:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][330/625] eta 0:01:52 lr 0.000077 wd 0.0500 time 0.3730 (0.3826) data time 0.0008 (0.0029) model time 0.3722 (0.3798) loss 2.9902 (2.6699) grad_norm 1.5133 (2.9341) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 14:08:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][340/625] eta 0:01:48 lr 0.000077 wd 0.0500 time 0.3746 (0.3823) data time 0.0008 (0.0029) model time 0.3738 (0.3795) loss 2.0318 (2.6717) grad_norm 2.3210 (inf) loss_scale 128.0000 (252.2463) mem 11224MB [2024-09-01 14:08:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][350/625] eta 0:01:45 lr 0.000077 wd 0.0500 time 0.3733 (0.3821) data time 0.0009 (0.0028) model time 0.3723 (0.3793) loss 1.9704 (2.6682) grad_norm 1.9744 (inf) loss_scale 128.0000 (248.7066) mem 11224MB [2024-09-01 14:08:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][360/625] eta 0:01:41 lr 0.000077 wd 0.0500 time 0.3744 (0.3824) data time 0.0010 (0.0027) model time 0.3734 (0.3797) loss 2.8004 (2.6707) grad_norm 1.8884 (inf) loss_scale 128.0000 (245.3629) mem 11224MB [2024-09-01 14:08:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][370/625] eta 0:01:37 lr 0.000077 wd 0.0500 time 0.3708 (0.3839) data time 0.0007 (0.0027) model time 0.3700 (0.3815) loss 2.7027 (2.6659) grad_norm 2.1807 (inf) loss_scale 128.0000 (242.1995) mem 11224MB [2024-09-01 14:08:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][380/625] eta 0:01:33 lr 0.000077 wd 0.0500 time 0.3743 (0.3836) data time 0.0009 (0.0027) model time 0.3734 (0.3812) loss 3.1896 (2.6630) grad_norm 2.2456 (inf) loss_scale 128.0000 (239.2021) mem 11224MB [2024-09-01 14:08:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][390/625] eta 0:01:30 lr 0.000077 wd 0.0500 time 0.3689 (0.3833) data time 0.0011 (0.0026) model time 0.3678 (0.3809) loss 1.8278 (2.6577) grad_norm 1.6506 (inf) loss_scale 128.0000 (236.3581) mem 11224MB [2024-09-01 14:08:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][400/625] eta 0:01:26 lr 0.000077 wd 0.0500 time 0.3761 (0.3831) data time 0.0011 (0.0026) model time 0.3749 (0.3807) loss 2.1612 (2.6522) grad_norm 3.3404 (inf) loss_scale 128.0000 (233.6559) mem 11224MB [2024-09-01 14:08:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][410/625] eta 0:01:22 lr 0.000077 wd 0.0500 time 0.3726 (0.3829) data time 0.0010 (0.0025) model time 0.3716 (0.3806) loss 2.7762 (2.6533) grad_norm 2.4098 (inf) loss_scale 128.0000 (231.0852) mem 11224MB [2024-09-01 14:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][420/625] eta 0:01:18 lr 0.000077 wd 0.0500 time 0.5839 (0.3832) data time 0.0009 (0.0025) model time 0.5829 (0.3809) loss 3.0086 (2.6488) grad_norm 2.5766 (inf) loss_scale 128.0000 (228.6366) mem 11224MB [2024-09-01 14:08:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][430/625] eta 0:01:14 lr 0.000077 wd 0.0500 time 0.3706 (0.3829) data time 0.0010 (0.0025) model time 0.3696 (0.3806) loss 1.9574 (2.6504) grad_norm 3.3915 (inf) loss_scale 128.0000 (226.3016) mem 11224MB [2024-09-01 14:08:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][440/625] eta 0:01:10 lr 0.000077 wd 0.0500 time 0.3703 (0.3827) data time 0.0007 (0.0024) model time 0.3695 (0.3804) loss 2.9240 (2.6501) grad_norm 3.3787 (inf) loss_scale 128.0000 (224.0726) mem 11224MB [2024-09-01 14:08:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][450/625] eta 0:01:06 lr 0.000077 wd 0.0500 time 0.3676 (0.3825) data time 0.0007 (0.0024) model time 0.3668 (0.3802) loss 1.8589 (2.6476) grad_norm 1.6835 (inf) loss_scale 128.0000 (221.9424) mem 11224MB [2024-09-01 14:08:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][460/625] eta 0:01:03 lr 0.000077 wd 0.0500 time 0.3760 (0.3823) data time 0.0009 (0.0024) model time 0.3751 (0.3801) loss 3.1267 (2.6525) grad_norm 3.1386 (inf) loss_scale 128.0000 (219.9046) mem 11224MB [2024-09-01 14:08:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][470/625] eta 0:00:59 lr 0.000076 wd 0.0500 time 0.3726 (0.3822) data time 0.0007 (0.0023) model time 0.3719 (0.3799) loss 3.0170 (2.6562) grad_norm 2.3339 (inf) loss_scale 128.0000 (217.9533) mem 11224MB [2024-09-01 14:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][480/625] eta 0:00:55 lr 0.000076 wd 0.0500 time 0.3772 (0.3824) data time 0.0008 (0.0023) model time 0.3764 (0.3802) loss 2.8929 (2.6528) grad_norm 2.3533 (inf) loss_scale 128.0000 (216.0832) mem 11224MB [2024-09-01 14:09:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][490/625] eta 0:00:51 lr 0.000076 wd 0.0500 time 0.3646 (0.3822) data time 0.0008 (0.0023) model time 0.3638 (0.3801) loss 3.0661 (2.6566) grad_norm 2.8745 (inf) loss_scale 128.0000 (214.2892) mem 11224MB [2024-09-01 14:09:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][500/625] eta 0:00:47 lr 0.000076 wd 0.0500 time 0.3733 (0.3821) data time 0.0009 (0.0022) model time 0.3724 (0.3799) loss 3.2389 (2.6523) grad_norm 2.4021 (inf) loss_scale 128.0000 (212.5669) mem 11224MB [2024-09-01 14:09:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][510/625] eta 0:00:43 lr 0.000076 wd 0.0500 time 0.3700 (0.3819) data time 0.0007 (0.0022) model time 0.3693 (0.3797) loss 2.5155 (2.6553) grad_norm 2.1167 (inf) loss_scale 128.0000 (210.9119) mem 11224MB [2024-09-01 14:09:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][520/625] eta 0:00:40 lr 0.000076 wd 0.0500 time 0.3706 (0.3817) data time 0.0011 (0.0022) model time 0.3695 (0.3795) loss 2.7604 (2.6564) grad_norm 2.7360 (inf) loss_scale 128.0000 (209.3205) mem 11224MB [2024-09-01 14:09:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][530/625] eta 0:00:36 lr 0.000076 wd 0.0500 time 0.3890 (0.3815) data time 0.0010 (0.0022) model time 0.3880 (0.3794) loss 2.8436 (2.6523) grad_norm 3.0526 (inf) loss_scale 128.0000 (207.7891) mem 11224MB [2024-09-01 14:09:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][540/625] eta 0:00:32 lr 0.000076 wd 0.0500 time 0.3386 (0.3817) data time 0.0009 (0.0022) model time 0.3376 (0.3796) loss 3.1850 (2.6566) grad_norm 9.3926 (inf) loss_scale 128.0000 (206.3142) mem 11224MB [2024-09-01 14:09:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][550/625] eta 0:00:28 lr 0.000076 wd 0.0500 time 0.3764 (0.3815) data time 0.0009 (0.0021) model time 0.3754 (0.3794) loss 3.0053 (2.6533) grad_norm 1.7748 (inf) loss_scale 128.0000 (204.8929) mem 11224MB [2024-09-01 14:09:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][560/625] eta 0:00:24 lr 0.000076 wd 0.0500 time 0.3768 (0.3814) data time 0.0007 (0.0021) model time 0.3760 (0.3793) loss 2.2333 (2.6531) grad_norm 2.6053 (inf) loss_scale 128.0000 (203.5223) mem 11224MB [2024-09-01 14:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][570/625] eta 0:00:21 lr 0.000076 wd 0.0500 time 0.5909 (0.3819) data time 0.0007 (0.0021) model time 0.5902 (0.3799) loss 3.3383 (2.6558) grad_norm 1.9823 (inf) loss_scale 128.0000 (202.1996) mem 11224MB [2024-09-01 14:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][580/625] eta 0:00:17 lr 0.000076 wd 0.0500 time 0.3727 (0.3825) data time 0.0010 (0.0021) model time 0.3717 (0.3805) loss 3.1475 (2.6559) grad_norm 3.0683 (inf) loss_scale 128.0000 (200.9225) mem 11224MB [2024-09-01 14:09:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][590/625] eta 0:00:13 lr 0.000076 wd 0.0500 time 0.3724 (0.3823) data time 0.0008 (0.0021) model time 0.3717 (0.3803) loss 2.3156 (2.6515) grad_norm 2.1540 (inf) loss_scale 128.0000 (199.6887) mem 11224MB [2024-09-01 14:09:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][600/625] eta 0:00:09 lr 0.000076 wd 0.0500 time 0.3706 (0.3825) data time 0.0010 (0.0020) model time 0.3696 (0.3806) loss 2.9074 (2.6494) grad_norm 4.6170 (inf) loss_scale 128.0000 (198.4958) mem 11224MB [2024-09-01 14:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][610/625] eta 0:00:05 lr 0.000076 wd 0.0500 time 0.3683 (0.3823) data time 0.0007 (0.0020) model time 0.3676 (0.3804) loss 3.0143 (2.6473) grad_norm 2.5664 (inf) loss_scale 128.0000 (197.3421) mem 11224MB [2024-09-01 14:09:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [269/300][620/625] eta 0:00:01 lr 0.000076 wd 0.0500 time 0.3613 (0.3820) data time 0.0007 (0.0020) model time 0.3606 (0.3801) loss 3.0652 (2.6504) grad_norm 5.1054 (inf) loss_scale 128.0000 (196.2254) mem 11224MB [2024-09-01 14:09:54 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 269 training takes 0:03:58 [2024-09-01 14:09:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:09:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.511 (0.511) Loss 0.5283 (0.5283) Acc@1 89.014 (89.014) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 14:09:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.8506 (0.6244) Acc@1 80.273 (86.834) Acc@5 95.801 (97.732) Mem 11224MB [2024-09-01 14:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9404 (0.7405) Acc@1 77.930 (83.908) Acc@5 94.971 (96.508) Mem 11224MB [2024-09-01 14:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.629 Acc@5 96.537 [2024-09-01 14:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 14:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.63% [2024-09-01 14:09:58 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 14:09:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 14:10:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.501 (0.501) Loss 0.4998 (0.4998) Acc@1 89.160 (89.160) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:10:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.141) Loss 0.8110 (0.5957) Acc@1 81.104 (87.061) Acc@5 95.703 (97.869) Mem 11224MB [2024-09-01 14:10:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.8945 (0.7091) Acc@1 79.150 (84.247) Acc@5 95.459 (96.733) Mem 11224MB [2024-09-01 14:10:02 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.939 Acc@5 96.743 [2024-09-01 14:10:02 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 14:10:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][0/625] eta 0:13:01 lr 0.000076 wd 0.0500 time 1.2509 (1.2509) data time 0.5684 (0.5684) model time 0.0000 (0.0000) loss 2.4167 (2.4167) grad_norm 4.3273 (4.3273) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][10/625] eta 0:04:38 lr 0.000075 wd 0.0500 time 0.3727 (0.4528) data time 0.0008 (0.0526) model time 0.0000 (0.0000) loss 2.9722 (2.6856) grad_norm 2.1835 (3.5000) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][20/625] eta 0:04:10 lr 0.000075 wd 0.0500 time 0.3674 (0.4143) data time 0.0011 (0.0281) model time 0.0000 (0.0000) loss 2.3147 (2.7283) grad_norm 1.9119 (2.9773) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][30/625] eta 0:03:58 lr 0.000075 wd 0.0500 time 0.3886 (0.4011) data time 0.0007 (0.0193) model time 0.0000 (0.0000) loss 3.0240 (2.7236) grad_norm 3.0091 (3.0972) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][40/625] eta 0:03:50 lr 0.000075 wd 0.0500 time 0.3752 (0.3945) data time 0.0007 (0.0148) model time 0.0000 (0.0000) loss 2.1747 (2.6430) grad_norm 2.1280 (3.0538) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][50/625] eta 0:03:44 lr 0.000075 wd 0.0500 time 0.3704 (0.3904) data time 0.0009 (0.0121) model time 0.0000 (0.0000) loss 2.2685 (2.6170) grad_norm 3.1526 (3.0133) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][60/625] eta 0:03:40 lr 0.000075 wd 0.0500 time 0.3710 (0.3897) data time 0.0011 (0.0103) model time 0.3700 (0.3852) loss 2.9557 (2.6008) grad_norm 2.4245 (3.0346) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][70/625] eta 0:03:35 lr 0.000075 wd 0.0500 time 0.3748 (0.3876) data time 0.0009 (0.0090) model time 0.3740 (0.3795) loss 2.7878 (2.5988) grad_norm 2.1109 (3.0814) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][80/625] eta 0:03:30 lr 0.000075 wd 0.0500 time 0.3682 (0.3857) data time 0.0008 (0.0080) model time 0.3674 (0.3768) loss 2.7380 (2.6080) grad_norm 1.9221 (3.0216) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][90/625] eta 0:03:25 lr 0.000075 wd 0.0500 time 0.3698 (0.3846) data time 0.0011 (0.0072) model time 0.3688 (0.3761) loss 2.5661 (2.6370) grad_norm 2.3909 (2.9869) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][100/625] eta 0:03:22 lr 0.000075 wd 0.0500 time 0.5642 (0.3852) data time 0.0009 (0.0066) model time 0.5632 (0.3790) loss 2.6707 (2.6208) grad_norm 1.8200 (2.9427) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 14:10:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:10:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:18:46 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 14:18:47 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 14:18:48 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 14:19:11 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 14:19:11 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 14:19:14 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 14:19:16 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 14:19:16 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 270) [2024-09-01 14:19:16 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 14:19:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][110/625] eta 0:43:43 lr 0.000075 wd 0.0500 time 0.3392 (5.0942) data time 0.0007 (0.2774) model time 0.3385 (4.8168) loss 2.1700 (2.8222) grad_norm 2.5278 (2.5894) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:19:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][120/625] eta 0:12:08 lr 0.000075 wd 0.0500 time 0.3498 (1.4421) data time 0.0010 (0.0647) model time 0.3488 (1.3774) loss 2.8100 (2.8294) grad_norm 4.7960 (2.7520) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:19:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][130/625] eta 0:08:02 lr 0.000075 wd 0.0500 time 0.5718 (0.9746) data time 0.0007 (0.0369) model time 0.5712 (0.9376) loss 3.2243 (2.8414) grad_norm 2.2410 (2.6838) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:19:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][140/625] eta 0:06:19 lr 0.000075 wd 0.0500 time 0.3416 (0.7821) data time 0.0008 (0.0260) model time 0.3409 (0.7561) loss 3.0841 (2.8288) grad_norm 7.2536 (2.8096) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:19:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][150/625] eta 0:05:25 lr 0.000075 wd 0.0500 time 0.3416 (0.6862) data time 0.0009 (0.0201) model time 0.3408 (0.6660) loss 2.9034 (2.7705) grad_norm 2.6991 (2.9037) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:19:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][160/625] eta 0:04:48 lr 0.000075 wd 0.0500 time 0.3448 (0.6215) data time 0.0009 (0.0165) model time 0.3439 (0.6050) loss 3.2569 (2.7773) grad_norm 2.8847 (2.8835) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:19:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][170/625] eta 0:04:22 lr 0.000075 wd 0.0500 time 0.3409 (0.5774) data time 0.0009 (0.0141) model time 0.3400 (0.5633) loss 2.3897 (2.7396) grad_norm 2.6610 (3.0207) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][180/625] eta 0:04:02 lr 0.000075 wd 0.0500 time 0.3448 (0.5452) data time 0.0009 (0.0122) model time 0.3440 (0.5329) loss 2.9481 (2.7206) grad_norm 2.2636 (3.0022) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][190/625] eta 0:03:46 lr 0.000074 wd 0.0500 time 0.3402 (0.5210) data time 0.0006 (0.0109) model time 0.3396 (0.5101) loss 1.7247 (2.6990) grad_norm 1.7336 (3.1860) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][200/625] eta 0:03:33 lr 0.000074 wd 0.0500 time 0.3432 (0.5019) data time 0.0007 (0.0098) model time 0.3425 (0.4922) loss 3.1888 (2.6955) grad_norm 2.1112 (3.0970) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][210/625] eta 0:03:21 lr 0.000074 wd 0.0500 time 0.3442 (0.4865) data time 0.0006 (0.0089) model time 0.3436 (0.4776) loss 2.6516 (2.7141) grad_norm 2.6970 (3.1269) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][220/625] eta 0:03:11 lr 0.000074 wd 0.0500 time 0.3445 (0.4738) data time 0.0006 (0.0082) model time 0.3439 (0.4656) loss 2.4484 (2.7085) grad_norm 3.5621 (3.0749) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][230/625] eta 0:03:02 lr 0.000074 wd 0.0500 time 0.3426 (0.4632) data time 0.0009 (0.0076) model time 0.3417 (0.4556) loss 2.6332 (2.7097) grad_norm 3.8226 (3.0629) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][240/625] eta 0:02:54 lr 0.000074 wd 0.0500 time 0.3403 (0.4543) data time 0.0008 (0.0071) model time 0.3394 (0.4472) loss 2.7141 (2.7003) grad_norm 3.8373 (3.0953) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][250/625] eta 0:02:47 lr 0.000074 wd 0.0500 time 0.3393 (0.4466) data time 0.0008 (0.0067) model time 0.3386 (0.4399) loss 2.8295 (2.6940) grad_norm 4.0732 (3.1522) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][260/625] eta 0:02:40 lr 0.000074 wd 0.0500 time 0.3421 (0.4398) data time 0.0007 (0.0063) model time 0.3414 (0.4335) loss 2.7726 (2.6854) grad_norm 3.2125 (3.1627) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][270/625] eta 0:02:34 lr 0.000074 wd 0.0500 time 0.3437 (0.4339) data time 0.0007 (0.0060) model time 0.3430 (0.4280) loss 2.2903 (2.6875) grad_norm 2.2319 (3.3109) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][280/625] eta 0:02:27 lr 0.000074 wd 0.0500 time 0.3458 (0.4287) data time 0.0009 (0.0057) model time 0.3449 (0.4231) loss 2.8398 (2.6872) grad_norm 2.0854 (3.2642) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][290/625] eta 0:02:22 lr 0.000074 wd 0.0500 time 0.3424 (0.4240) data time 0.0008 (0.0054) model time 0.3416 (0.4186) loss 2.9952 (2.6776) grad_norm 2.4456 (3.2336) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][300/625] eta 0:02:16 lr 0.000074 wd 0.0500 time 0.3378 (0.4198) data time 0.0007 (0.0052) model time 0.3371 (0.4146) loss 2.8562 (2.6728) grad_norm 2.8295 (3.2491) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][310/625] eta 0:02:11 lr 0.000074 wd 0.0500 time 0.3411 (0.4160) data time 0.0006 (0.0050) model time 0.3405 (0.4111) loss 1.8898 (2.6571) grad_norm 2.1632 (3.2092) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][320/625] eta 0:02:05 lr 0.000074 wd 0.0500 time 0.3405 (0.4126) data time 0.0008 (0.0048) model time 0.3396 (0.4079) loss 2.0185 (2.6531) grad_norm 2.7422 (3.1972) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][330/625] eta 0:02:00 lr 0.000074 wd 0.0500 time 0.3395 (0.4095) data time 0.0006 (0.0046) model time 0.3389 (0.4049) loss 2.2950 (2.6562) grad_norm 2.4934 (3.1972) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][340/625] eta 0:01:56 lr 0.000074 wd 0.0500 time 0.5211 (0.4092) data time 0.0009 (0.0044) model time 0.5202 (0.4048) loss 2.0394 (2.6501) grad_norm 2.8962 (3.2571) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:20:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][350/625] eta 0:01:51 lr 0.000074 wd 0.0500 time 0.3481 (0.4065) data time 0.0009 (0.0043) model time 0.3472 (0.4022) loss 2.6989 (2.6525) grad_norm 2.5892 (3.2373) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][360/625] eta 0:01:47 lr 0.000073 wd 0.0500 time 0.3433 (0.4040) data time 0.0008 (0.0041) model time 0.3424 (0.3999) loss 2.8018 (2.6438) grad_norm 3.5016 (3.2346) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][370/625] eta 0:01:42 lr 0.000073 wd 0.0500 time 0.3437 (0.4017) data time 0.0007 (0.0040) model time 0.3430 (0.3977) loss 2.2823 (2.6329) grad_norm 2.3441 (3.2186) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][380/625] eta 0:01:37 lr 0.000073 wd 0.0500 time 0.3522 (0.3996) data time 0.0006 (0.0039) model time 0.3516 (0.3957) loss 3.4303 (2.6349) grad_norm 3.6205 (3.2093) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][390/625] eta 0:01:33 lr 0.000073 wd 0.0500 time 0.3372 (0.3976) data time 0.0009 (0.0038) model time 0.3363 (0.3938) loss 2.9364 (2.6369) grad_norm 2.4415 (3.1822) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][400/625] eta 0:01:29 lr 0.000073 wd 0.0500 time 0.3422 (0.3957) data time 0.0010 (0.0037) model time 0.3412 (0.3920) loss 2.5291 (2.6321) grad_norm 2.5770 (3.1673) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][410/625] eta 0:01:24 lr 0.000073 wd 0.0500 time 0.3442 (0.3940) data time 0.0007 (0.0036) model time 0.3435 (0.3904) loss 2.6952 (2.6275) grad_norm 3.6014 (3.1791) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][420/625] eta 0:01:20 lr 0.000073 wd 0.0500 time 0.3422 (0.3923) data time 0.0006 (0.0035) model time 0.3416 (0.3888) loss 3.3129 (2.6298) grad_norm 3.1419 (3.2066) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][430/625] eta 0:01:16 lr 0.000073 wd 0.0500 time 0.3399 (0.3908) data time 0.0009 (0.0034) model time 0.3390 (0.3874) loss 2.6049 (2.6358) grad_norm 2.7736 (3.2045) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][440/625] eta 0:01:12 lr 0.000073 wd 0.0500 time 0.3384 (0.3893) data time 0.0009 (0.0034) model time 0.3375 (0.3860) loss 2.7593 (2.6329) grad_norm 2.1149 (3.1960) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][450/625] eta 0:01:08 lr 0.000073 wd 0.0500 time 0.5495 (0.3886) data time 0.0010 (0.0033) model time 0.5485 (0.3854) loss 3.0197 (2.6364) grad_norm 2.8550 (3.1860) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][460/625] eta 0:01:04 lr 0.000073 wd 0.0500 time 0.3372 (0.3880) data time 0.0008 (0.0032) model time 0.3364 (0.3847) loss 2.8096 (2.6381) grad_norm 2.3647 (3.1644) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][470/625] eta 0:00:59 lr 0.000073 wd 0.0500 time 0.3495 (0.3868) data time 0.0006 (0.0032) model time 0.3489 (0.3836) loss 2.8190 (2.6373) grad_norm 1.6597 (3.1386) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][480/625] eta 0:00:55 lr 0.000073 wd 0.0500 time 0.3409 (0.3856) data time 0.0007 (0.0031) model time 0.3402 (0.3825) loss 2.7330 (2.6342) grad_norm 2.1530 (3.1183) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][490/625] eta 0:00:51 lr 0.000073 wd 0.0500 time 0.3401 (0.3845) data time 0.0008 (0.0030) model time 0.3393 (0.3814) loss 1.8497 (2.6309) grad_norm 1.9123 (3.1016) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][500/625] eta 0:00:47 lr 0.000073 wd 0.0500 time 0.3453 (0.3834) data time 0.0009 (0.0030) model time 0.3444 (0.3805) loss 2.8899 (2.6281) grad_norm 2.4430 (3.0862) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][510/625] eta 0:00:43 lr 0.000073 wd 0.0500 time 0.3472 (0.3825) data time 0.0009 (0.0029) model time 0.3463 (0.3795) loss 2.5434 (2.6334) grad_norm 3.6858 (3.0752) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:21:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][520/625] eta 0:00:40 lr 0.000073 wd 0.0500 time 0.3495 (0.3815) data time 0.0009 (0.0029) model time 0.3486 (0.3786) loss 2.6996 (2.6313) grad_norm 2.8332 (3.0714) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][530/625] eta 0:00:36 lr 0.000072 wd 0.0500 time 0.3396 (0.3810) data time 0.0007 (0.0028) model time 0.3389 (0.3782) loss 2.5169 (2.6311) grad_norm 4.3721 (3.0746) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][540/625] eta 0:00:32 lr 0.000072 wd 0.0500 time 0.3436 (0.3802) data time 0.0007 (0.0028) model time 0.3429 (0.3774) loss 2.9225 (2.6385) grad_norm 1.8274 (3.1436) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][550/625] eta 0:00:28 lr 0.000072 wd 0.0500 time 0.3439 (0.3794) data time 0.0007 (0.0027) model time 0.3432 (0.3767) loss 2.5119 (2.6401) grad_norm 2.1518 (3.1388) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][560/625] eta 0:00:24 lr 0.000072 wd 0.0500 time 0.3409 (0.3786) data time 0.0007 (0.0027) model time 0.3401 (0.3759) loss 1.8015 (2.6359) grad_norm 2.1280 (3.1438) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][570/625] eta 0:00:20 lr 0.000072 wd 0.0500 time 0.3428 (0.3778) data time 0.0006 (0.0027) model time 0.3422 (0.3752) loss 2.2923 (2.6319) grad_norm 2.8559 (3.1392) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][580/625] eta 0:00:16 lr 0.000072 wd 0.0500 time 0.3419 (0.3771) data time 0.0007 (0.0026) model time 0.3412 (0.3745) loss 2.7151 (2.6276) grad_norm 4.2824 (3.1364) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][590/625] eta 0:00:13 lr 0.000072 wd 0.0500 time 0.3418 (0.3764) data time 0.0007 (0.0026) model time 0.3410 (0.3738) loss 3.2893 (2.6297) grad_norm 3.4363 (3.1379) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][600/625] eta 0:00:09 lr 0.000072 wd 0.0500 time 0.3435 (0.3757) data time 0.0007 (0.0025) model time 0.3428 (0.3732) loss 2.0886 (2.6298) grad_norm 2.1672 (3.1524) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][610/625] eta 0:00:05 lr 0.000072 wd 0.0500 time 0.3384 (0.3751) data time 0.0004 (0.0025) model time 0.3380 (0.3725) loss 2.2233 (2.6296) grad_norm 2.4967 (3.1513) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [270/300][620/625] eta 0:00:01 lr 0.000072 wd 0.0500 time 0.3382 (0.3743) data time 0.0004 (0.0025) model time 0.3378 (0.3718) loss 3.0740 (2.6356) grad_norm 1.9219 (3.1440) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 14:22:33 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 270 training takes 0:03:13 [2024-09-01 14:22:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:22:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:22:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.432 (0.432) Loss 0.5190 (0.5190) Acc@1 89.453 (89.453) Acc@5 98.535 (98.535) Mem 11219MB [2024-09-01 14:22:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.128) Loss 0.8232 (0.6134) Acc@1 81.006 (86.919) Acc@5 95.996 (97.834) Mem 11219MB [2024-09-01 14:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.110) Loss 0.9092 (0.7324) Acc@1 79.004 (83.959) Acc@5 95.264 (96.601) Mem 11219MB [2024-09-01 14:22:42 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.689 Acc@5 96.615 [2024-09-01 14:22:42 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 14:22:42 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.69% [2024-09-01 14:22:42 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 14:22:45 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 14:22:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.428 (0.428) Loss 0.5015 (0.5015) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 14:22:46 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.127) Loss 0.8115 (0.5962) Acc@1 81.152 (87.052) Acc@5 95.703 (97.865) Mem 11219MB [2024-09-01 14:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.109) Loss 0.8945 (0.7096) Acc@1 79.053 (84.238) Acc@5 95.459 (96.708) Mem 11219MB [2024-09-01 14:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.935 Acc@5 96.727 [2024-09-01 14:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 14:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.94% [2024-09-01 14:22:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 14:22:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 14:22:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][0/625] eta 0:07:17 lr 0.000072 wd 0.0500 time 0.6995 (0.6995) data time 0.3488 (0.3488) model time 0.0000 (0.0000) loss 2.5460 (2.5460) grad_norm 2.2555 (2.2555) loss_scale 128.0000 (128.0000) mem 11227MB [2024-09-01 14:22:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][10/625] eta 0:03:50 lr 0.000072 wd 0.0500 time 0.3402 (0.3753) data time 0.0007 (0.0324) model time 0.0000 (0.0000) loss 2.1312 (2.3570) grad_norm 3.9977 (2.4303) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:22:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][20/625] eta 0:03:42 lr 0.000072 wd 0.0500 time 0.4879 (0.3670) data time 0.0007 (0.0174) model time 0.0000 (0.0000) loss 2.9820 (2.4955) grad_norm 2.6302 (2.5677) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][30/625] eta 0:03:37 lr 0.000072 wd 0.0500 time 0.3459 (0.3662) data time 0.0007 (0.0120) model time 0.0000 (0.0000) loss 2.9396 (2.5690) grad_norm 2.6624 (2.6409) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][40/625] eta 0:03:31 lr 0.000072 wd 0.0500 time 0.3545 (0.3612) data time 0.0009 (0.0093) model time 0.0000 (0.0000) loss 2.6054 (2.6052) grad_norm 2.7511 (2.6333) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][50/625] eta 0:03:25 lr 0.000072 wd 0.0500 time 0.3472 (0.3578) data time 0.0006 (0.0077) model time 0.0000 (0.0000) loss 2.9175 (2.6274) grad_norm 2.3459 (2.6581) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][60/625] eta 0:03:20 lr 0.000072 wd 0.0500 time 0.3417 (0.3553) data time 0.0008 (0.0065) model time 0.3409 (0.3418) loss 2.9610 (2.6293) grad_norm 2.6480 (2.7571) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][70/625] eta 0:03:16 lr 0.000072 wd 0.0500 time 0.3403 (0.3536) data time 0.0008 (0.0057) model time 0.3395 (0.3422) loss 2.9445 (2.6551) grad_norm 3.1875 (2.9495) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][80/625] eta 0:03:12 lr 0.000071 wd 0.0500 time 0.3443 (0.3523) data time 0.0009 (0.0051) model time 0.3434 (0.3422) loss 2.6805 (2.6487) grad_norm 2.6901 (3.0042) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][90/625] eta 0:03:07 lr 0.000071 wd 0.0500 time 0.3470 (0.3513) data time 0.0009 (0.0047) model time 0.3461 (0.3423) loss 2.5151 (2.6408) grad_norm 2.5855 (3.0823) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][100/625] eta 0:03:04 lr 0.000071 wd 0.0500 time 0.3434 (0.3518) data time 0.0008 (0.0043) model time 0.3426 (0.3449) loss 2.6867 (2.6489) grad_norm 4.9123 (3.1131) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][110/625] eta 0:03:00 lr 0.000071 wd 0.0500 time 0.3452 (0.3512) data time 0.0007 (0.0040) model time 0.3446 (0.3447) loss 2.4416 (2.6559) grad_norm 2.0061 (3.4500) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][120/625] eta 0:02:57 lr 0.000071 wd 0.0500 time 0.3511 (0.3506) data time 0.0008 (0.0037) model time 0.3503 (0.3445) loss 2.7522 (2.6544) grad_norm 2.2227 (3.3931) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][130/625] eta 0:02:53 lr 0.000071 wd 0.0500 time 0.3421 (0.3500) data time 0.0010 (0.0035) model time 0.3411 (0.3441) loss 2.5382 (2.6301) grad_norm 2.3441 (3.3566) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][140/625] eta 0:02:49 lr 0.000071 wd 0.0500 time 0.3420 (0.3495) data time 0.0007 (0.0033) model time 0.3413 (0.3440) loss 1.8681 (2.6231) grad_norm 1.4106 (3.4673) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][150/625] eta 0:02:45 lr 0.000071 wd 0.0500 time 0.3469 (0.3492) data time 0.0007 (0.0031) model time 0.3462 (0.3439) loss 2.8449 (2.6255) grad_norm 2.3418 (3.4301) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][160/625] eta 0:02:42 lr 0.000071 wd 0.0500 time 0.3423 (0.3487) data time 0.0008 (0.0030) model time 0.3415 (0.3437) loss 3.1182 (2.6410) grad_norm 3.4817 (3.3814) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][170/625] eta 0:02:38 lr 0.000071 wd 0.0500 time 0.3433 (0.3485) data time 0.0007 (0.0029) model time 0.3426 (0.3437) loss 2.5450 (2.6432) grad_norm 2.5332 (3.3342) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][180/625] eta 0:02:34 lr 0.000071 wd 0.0500 time 0.3483 (0.3482) data time 0.0007 (0.0028) model time 0.3476 (0.3436) loss 2.7737 (2.6276) grad_norm 2.8457 (3.2985) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][190/625] eta 0:02:31 lr 0.000071 wd 0.0500 time 0.3432 (0.3479) data time 0.0008 (0.0027) model time 0.3424 (0.3435) loss 2.7507 (2.6204) grad_norm 2.8413 (3.3008) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:23:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][200/625] eta 0:02:27 lr 0.000071 wd 0.0500 time 0.3412 (0.3477) data time 0.0007 (0.0026) model time 0.3404 (0.3434) loss 3.2329 (2.6080) grad_norm 2.5809 (3.3182) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][210/625] eta 0:02:24 lr 0.000071 wd 0.0500 time 0.3450 (0.3475) data time 0.0009 (0.0025) model time 0.3441 (0.3434) loss 2.5465 (2.6056) grad_norm 3.6057 (3.3042) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][220/625] eta 0:02:20 lr 0.000071 wd 0.0500 time 0.3401 (0.3475) data time 0.0008 (0.0024) model time 0.3393 (0.3436) loss 2.9306 (2.6209) grad_norm 2.0121 (3.3041) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][230/625] eta 0:02:17 lr 0.000071 wd 0.0500 time 0.5713 (0.3484) data time 0.0007 (0.0024) model time 0.5706 (0.3449) loss 1.6086 (2.6188) grad_norm 2.0730 (3.2762) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][240/625] eta 0:02:14 lr 0.000071 wd 0.0500 time 0.3450 (0.3482) data time 0.0006 (0.0023) model time 0.3444 (0.3448) loss 3.1549 (2.6182) grad_norm 2.5927 (3.2558) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][250/625] eta 0:02:10 lr 0.000071 wd 0.0500 time 0.3406 (0.3480) data time 0.0009 (0.0022) model time 0.3397 (0.3446) loss 2.8171 (2.6271) grad_norm 2.4190 (3.2523) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][260/625] eta 0:02:06 lr 0.000070 wd 0.0500 time 0.3523 (0.3479) data time 0.0007 (0.0022) model time 0.3517 (0.3446) loss 3.1685 (2.6352) grad_norm 2.0403 (3.2285) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][270/625] eta 0:02:03 lr 0.000070 wd 0.0500 time 0.3402 (0.3477) data time 0.0006 (0.0021) model time 0.3396 (0.3445) loss 2.6555 (2.6349) grad_norm 2.4948 (3.2079) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][280/625] eta 0:01:59 lr 0.000070 wd 0.0500 time 0.3456 (0.3476) data time 0.0006 (0.0021) model time 0.3450 (0.3445) loss 2.7811 (2.6350) grad_norm 1.9839 (3.2108) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][290/625] eta 0:01:56 lr 0.000070 wd 0.0500 time 0.3416 (0.3474) data time 0.0008 (0.0020) model time 0.3408 (0.3444) loss 3.0266 (2.6277) grad_norm 2.4497 (3.1928) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][300/625] eta 0:01:52 lr 0.000070 wd 0.0500 time 0.3443 (0.3473) data time 0.0009 (0.0020) model time 0.3434 (0.3443) loss 2.9468 (2.6251) grad_norm 2.4318 (3.2066) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][310/625] eta 0:01:49 lr 0.000070 wd 0.0500 time 0.3447 (0.3478) data time 0.0008 (0.0020) model time 0.3438 (0.3450) loss 2.6251 (2.6164) grad_norm 2.4760 (3.1959) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][320/625] eta 0:01:46 lr 0.000070 wd 0.0500 time 0.3420 (0.3477) data time 0.0007 (0.0019) model time 0.3413 (0.3449) loss 2.8938 (2.6146) grad_norm 2.6117 (3.1707) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][330/625] eta 0:01:42 lr 0.000070 wd 0.0500 time 0.3407 (0.3475) data time 0.0006 (0.0019) model time 0.3401 (0.3448) loss 1.5419 (2.6100) grad_norm 2.1766 (3.1663) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][340/625] eta 0:01:38 lr 0.000070 wd 0.0500 time 0.3389 (0.3474) data time 0.0007 (0.0019) model time 0.3382 (0.3447) loss 2.2354 (2.6132) grad_norm 2.6552 (3.1476) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][350/625] eta 0:01:35 lr 0.000070 wd 0.0500 time 0.3443 (0.3473) data time 0.0006 (0.0018) model time 0.3436 (0.3447) loss 3.2402 (2.6149) grad_norm 1.9536 (3.1389) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][360/625] eta 0:01:32 lr 0.000070 wd 0.0500 time 0.3440 (0.3472) data time 0.0009 (0.0018) model time 0.3432 (0.3446) loss 2.7968 (2.6150) grad_norm 2.8841 (3.1820) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:24:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][370/625] eta 0:01:28 lr 0.000070 wd 0.0500 time 0.3422 (0.3471) data time 0.0008 (0.0018) model time 0.3414 (0.3446) loss 2.5950 (2.6128) grad_norm 2.8962 (3.1796) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:25:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][380/625] eta 0:01:25 lr 0.000070 wd 0.0500 time 0.3448 (0.3471) data time 0.0006 (0.0017) model time 0.3442 (0.3446) loss 1.8940 (2.6074) grad_norm 2.4594 (3.1715) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:25:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][390/625] eta 0:01:21 lr 0.000070 wd 0.0500 time 0.3403 (0.3470) data time 0.0009 (0.0017) model time 0.3394 (0.3445) loss 2.3863 (2.6063) grad_norm 2.0087 (3.1606) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:25:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][400/625] eta 0:01:18 lr 0.000070 wd 0.0500 time 0.3412 (0.3469) data time 0.0009 (0.0017) model time 0.3403 (0.3445) loss 2.2505 (2.6078) grad_norm 2.5339 (3.1530) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:25:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][410/625] eta 0:01:14 lr 0.000070 wd 0.0500 time 0.3413 (0.3468) data time 0.0007 (0.0017) model time 0.3407 (0.3444) loss 2.6495 (2.6150) grad_norm 6.0853 (3.1477) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:25:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][420/625] eta 0:01:11 lr 0.000070 wd 0.0500 time 0.3429 (0.3478) data time 0.0007 (0.0017) model time 0.3422 (0.3455) loss 2.3956 (2.6169) grad_norm 3.3393 (3.1415) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:25:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][430/625] eta 0:01:07 lr 0.000070 wd 0.0500 time 0.3426 (0.3477) data time 0.0006 (0.0016) model time 0.3420 (0.3455) loss 2.7305 (2.6128) grad_norm 3.3049 (3.1292) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:25:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][440/625] eta 0:01:04 lr 0.000069 wd 0.0500 time 0.3479 (0.3476) data time 0.0007 (0.0016) model time 0.3472 (0.3454) loss 3.0369 (2.6114) grad_norm 4.0819 (3.1325) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:25:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][450/625] eta 0:01:00 lr 0.000069 wd 0.0500 time 0.3467 (0.3475) data time 0.0009 (0.0016) model time 0.3459 (0.3453) loss 2.5775 (2.6129) grad_norm 2.2896 (inf) loss_scale 64.0000 (126.8647) mem 11224MB [2024-09-01 14:25:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][460/625] eta 0:00:57 lr 0.000069 wd 0.0500 time 0.3437 (0.3474) data time 0.0010 (0.0016) model time 0.3427 (0.3453) loss 1.9714 (2.6127) grad_norm 2.1530 (inf) loss_scale 64.0000 (125.5011) mem 11224MB [2024-09-01 14:25:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][470/625] eta 0:00:53 lr 0.000069 wd 0.0500 time 0.3464 (0.3473) data time 0.0009 (0.0016) model time 0.3456 (0.3452) loss 1.5587 (2.6116) grad_norm 2.0655 (inf) loss_scale 64.0000 (124.1953) mem 11224MB [2024-09-01 14:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][480/625] eta 0:00:50 lr 0.000069 wd 0.0500 time 0.3460 (0.3473) data time 0.0009 (0.0016) model time 0.3451 (0.3452) loss 2.8567 (2.6109) grad_norm 2.1679 (inf) loss_scale 64.0000 (122.9439) mem 11224MB [2024-09-01 14:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][490/625] eta 0:00:46 lr 0.000069 wd 0.0500 time 0.3403 (0.3475) data time 0.0007 (0.0016) model time 0.3396 (0.3455) loss 3.0910 (2.6123) grad_norm 2.1495 (inf) loss_scale 64.0000 (121.7434) mem 11224MB [2024-09-01 14:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][500/625] eta 0:00:43 lr 0.000069 wd 0.0500 time 0.3409 (0.3474) data time 0.0007 (0.0015) model time 0.3402 (0.3454) loss 2.2761 (2.6120) grad_norm 4.6907 (inf) loss_scale 64.0000 (120.5908) mem 11224MB [2024-09-01 14:25:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][510/625] eta 0:00:39 lr 0.000069 wd 0.0500 time 0.3523 (0.3474) data time 0.0008 (0.0015) model time 0.3515 (0.3454) loss 2.3493 (2.6073) grad_norm 3.2678 (inf) loss_scale 64.0000 (119.4834) mem 11224MB [2024-09-01 14:25:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][520/625] eta 0:00:36 lr 0.000069 wd 0.0500 time 0.3438 (0.3474) data time 0.0006 (0.0015) model time 0.3432 (0.3455) loss 3.3297 (2.6112) grad_norm 3.0784 (inf) loss_scale 64.0000 (118.4184) mem 11224MB [2024-09-01 14:25:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][530/625] eta 0:00:33 lr 0.000069 wd 0.0500 time 0.3453 (0.3474) data time 0.0011 (0.0015) model time 0.3443 (0.3455) loss 2.3437 (2.6111) grad_norm 2.5573 (inf) loss_scale 64.0000 (117.3936) mem 11224MB [2024-09-01 14:25:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][540/625] eta 0:00:29 lr 0.000069 wd 0.0500 time 0.3500 (0.3481) data time 0.0007 (0.0015) model time 0.3494 (0.3463) loss 3.1210 (2.6148) grad_norm 2.2943 (inf) loss_scale 64.0000 (116.4067) mem 11224MB [2024-09-01 14:26:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][550/625] eta 0:00:26 lr 0.000069 wd 0.0500 time 0.3385 (0.3480) data time 0.0009 (0.0015) model time 0.3376 (0.3462) loss 2.3103 (2.6142) grad_norm 4.1149 (inf) loss_scale 64.0000 (115.4555) mem 11224MB [2024-09-01 14:26:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][560/625] eta 0:00:22 lr 0.000069 wd 0.0500 time 0.3439 (0.3480) data time 0.0009 (0.0015) model time 0.3429 (0.3461) loss 2.7545 (2.6110) grad_norm 2.4967 (inf) loss_scale 64.0000 (114.5383) mem 11224MB [2024-09-01 14:26:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][570/625] eta 0:00:19 lr 0.000069 wd 0.0500 time 0.3489 (0.3479) data time 0.0006 (0.0015) model time 0.3483 (0.3460) loss 1.5582 (2.6088) grad_norm 3.1465 (inf) loss_scale 64.0000 (113.6532) mem 11224MB [2024-09-01 14:26:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][580/625] eta 0:00:15 lr 0.000069 wd 0.0500 time 0.3522 (0.3479) data time 0.0007 (0.0014) model time 0.3515 (0.3461) loss 2.3193 (2.6086) grad_norm 2.8012 (inf) loss_scale 64.0000 (112.7986) mem 11224MB [2024-09-01 14:26:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][590/625] eta 0:00:12 lr 0.000069 wd 0.0500 time 0.3469 (0.3479) data time 0.0007 (0.0014) model time 0.3462 (0.3461) loss 2.5007 (2.6107) grad_norm 3.1046 (inf) loss_scale 64.0000 (111.9729) mem 11224MB [2024-09-01 14:26:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][600/625] eta 0:00:08 lr 0.000069 wd 0.0500 time 0.3498 (0.3479) data time 0.0006 (0.0014) model time 0.3492 (0.3461) loss 2.9494 (2.6124) grad_norm 2.4804 (inf) loss_scale 64.0000 (111.1747) mem 11224MB [2024-09-01 14:26:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][610/625] eta 0:00:05 lr 0.000069 wd 0.0500 time 0.3339 (0.3482) data time 0.0005 (0.0014) model time 0.3334 (0.3464) loss 2.6560 (2.6114) grad_norm 2.8195 (inf) loss_scale 64.0000 (110.4026) mem 11224MB [2024-09-01 14:26:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [271/300][620/625] eta 0:00:01 lr 0.000068 wd 0.0500 time 0.3349 (0.3479) data time 0.0005 (0.0014) model time 0.3344 (0.3462) loss 2.7085 (2.6108) grad_norm 2.3192 (inf) loss_scale 64.0000 (109.6554) mem 11224MB [2024-09-01 14:26:27 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 271 training takes 0:03:37 [2024-09-01 14:26:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:26:28 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:26:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.433 (0.433) Loss 0.5278 (0.5278) Acc@1 88.574 (88.574) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 14:26:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.128) Loss 0.8394 (0.6145) Acc@1 80.566 (86.843) Acc@5 95.947 (97.807) Mem 11224MB [2024-09-01 14:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.110) Loss 0.9287 (0.7312) Acc@1 78.369 (83.936) Acc@5 95.264 (96.626) Mem 11224MB [2024-09-01 14:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.711 Acc@5 96.661 [2024-09-01 14:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 14:26:30 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.71% [2024-09-01 14:26:30 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 14:26:31 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 14:26:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.430 (0.430) Loss 0.5015 (0.5015) Acc@1 89.062 (89.062) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:26:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.129) Loss 0.8105 (0.5963) Acc@1 81.055 (87.052) Acc@5 95.801 (97.874) Mem 11224MB [2024-09-01 14:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.089 (0.110) Loss 0.8936 (0.7101) Acc@1 79.004 (84.226) Acc@5 95.410 (96.726) Mem 11224MB [2024-09-01 14:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.923 Acc@5 96.745 [2024-09-01 14:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 14:26:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][0/625] eta 0:12:27 lr 0.000068 wd 0.0500 time 1.1965 (1.1965) data time 0.5578 (0.5578) model time 0.0000 (0.0000) loss 2.9910 (2.9910) grad_norm 11.9014 (11.9014) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:26:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][10/625] eta 0:04:20 lr 0.000068 wd 0.0500 time 0.3401 (0.4235) data time 0.0007 (0.0515) model time 0.0000 (0.0000) loss 2.9743 (2.6049) grad_norm 2.7355 (3.6464) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:26:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][20/625] eta 0:03:54 lr 0.000068 wd 0.0500 time 0.3425 (0.3875) data time 0.0010 (0.0274) model time 0.0000 (0.0000) loss 2.3161 (2.5245) grad_norm 2.9441 (3.1514) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:26:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][30/625] eta 0:03:43 lr 0.000068 wd 0.0500 time 0.3520 (0.3750) data time 0.0007 (0.0188) model time 0.0000 (0.0000) loss 2.9069 (2.5834) grad_norm 2.3573 (3.0030) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:26:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][40/625] eta 0:03:35 lr 0.000068 wd 0.0500 time 0.3476 (0.3681) data time 0.0007 (0.0144) model time 0.0000 (0.0000) loss 3.2024 (2.6549) grad_norm 2.3101 (2.9656) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:26:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][50/625] eta 0:03:29 lr 0.000068 wd 0.0500 time 0.3446 (0.3640) data time 0.0009 (0.0118) model time 0.0000 (0.0000) loss 2.4604 (2.6718) grad_norm 2.9251 (2.8767) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:26:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][60/625] eta 0:03:24 lr 0.000068 wd 0.0500 time 0.3557 (0.3611) data time 0.0007 (0.0100) model time 0.3550 (0.3458) loss 3.0214 (2.6811) grad_norm 3.4993 (2.8267) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][70/625] eta 0:03:22 lr 0.000068 wd 0.0500 time 0.3444 (0.3647) data time 0.0007 (0.0087) model time 0.3437 (0.3658) loss 2.6428 (2.6831) grad_norm 2.5413 (2.9837) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][80/625] eta 0:03:18 lr 0.000068 wd 0.0500 time 0.3459 (0.3645) data time 0.0009 (0.0077) model time 0.3450 (0.3646) loss 2.8605 (2.6666) grad_norm 2.6738 (2.9281) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][90/625] eta 0:03:13 lr 0.000068 wd 0.0500 time 0.3473 (0.3624) data time 0.0006 (0.0070) model time 0.3467 (0.3596) loss 2.7289 (2.6515) grad_norm 3.3229 (3.0559) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][100/625] eta 0:03:09 lr 0.000068 wd 0.0500 time 0.3568 (0.3610) data time 0.0009 (0.0064) model time 0.3559 (0.3571) loss 3.0596 (2.6572) grad_norm 2.1443 (3.0369) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][110/625] eta 0:03:05 lr 0.000068 wd 0.0500 time 0.3528 (0.3600) data time 0.0009 (0.0059) model time 0.3519 (0.3558) loss 2.6477 (2.6298) grad_norm 1.7392 (3.0392) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][120/625] eta 0:03:01 lr 0.000068 wd 0.0500 time 0.3472 (0.3589) data time 0.0008 (0.0055) model time 0.3463 (0.3543) loss 2.8881 (2.6347) grad_norm 3.9178 (3.0918) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][130/625] eta 0:02:57 lr 0.000068 wd 0.0500 time 0.3457 (0.3579) data time 0.0008 (0.0051) model time 0.3449 (0.3532) loss 2.6512 (2.6378) grad_norm 4.5830 (3.0734) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][140/625] eta 0:02:53 lr 0.000068 wd 0.0500 time 0.3430 (0.3571) data time 0.0009 (0.0048) model time 0.3421 (0.3524) loss 2.3947 (2.6510) grad_norm 3.2915 (3.0656) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][150/625] eta 0:02:49 lr 0.000068 wd 0.0500 time 0.3451 (0.3565) data time 0.0009 (0.0045) model time 0.3442 (0.3519) loss 2.7135 (2.6479) grad_norm 4.0963 (3.1132) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][160/625] eta 0:02:45 lr 0.000068 wd 0.0500 time 0.3450 (0.3559) data time 0.0008 (0.0043) model time 0.3442 (0.3513) loss 3.1326 (2.6528) grad_norm 2.4926 (3.2762) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][170/625] eta 0:02:41 lr 0.000068 wd 0.0500 time 0.3476 (0.3554) data time 0.0009 (0.0041) model time 0.3467 (0.3509) loss 2.0346 (2.6403) grad_norm 2.7356 (3.2559) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][180/625] eta 0:02:37 lr 0.000067 wd 0.0500 time 0.3468 (0.3549) data time 0.0008 (0.0039) model time 0.3459 (0.3504) loss 2.6685 (2.6275) grad_norm 2.4380 (3.2229) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][190/625] eta 0:02:34 lr 0.000067 wd 0.0500 time 0.3426 (0.3544) data time 0.0008 (0.0038) model time 0.3418 (0.3500) loss 2.7726 (2.6304) grad_norm 3.9121 (3.2009) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][200/625] eta 0:02:30 lr 0.000067 wd 0.0500 time 0.3448 (0.3539) data time 0.0007 (0.0036) model time 0.3441 (0.3497) loss 2.6384 (2.6277) grad_norm 2.3883 (3.1899) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][210/625] eta 0:02:26 lr 0.000067 wd 0.0500 time 0.3453 (0.3535) data time 0.0006 (0.0035) model time 0.3447 (0.3494) loss 2.7602 (2.6297) grad_norm 2.6458 (3.1424) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][220/625] eta 0:02:23 lr 0.000067 wd 0.0500 time 0.3484 (0.3532) data time 0.0009 (0.0034) model time 0.3475 (0.3492) loss 2.9465 (2.6409) grad_norm 1.9448 (3.1478) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][230/625] eta 0:02:19 lr 0.000067 wd 0.0500 time 0.3521 (0.3530) data time 0.0007 (0.0033) model time 0.3515 (0.3490) loss 3.3920 (2.6521) grad_norm 3.0118 (3.1578) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:27:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][240/625] eta 0:02:15 lr 0.000067 wd 0.0500 time 0.3508 (0.3527) data time 0.0006 (0.0032) model time 0.3502 (0.3488) loss 2.9807 (2.6583) grad_norm 2.0877 (3.1289) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][250/625] eta 0:02:12 lr 0.000067 wd 0.0500 time 0.3463 (0.3524) data time 0.0006 (0.0031) model time 0.3457 (0.3486) loss 2.2734 (2.6512) grad_norm 2.7779 (3.1101) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][260/625] eta 0:02:08 lr 0.000067 wd 0.0500 time 0.3467 (0.3522) data time 0.0007 (0.0030) model time 0.3460 (0.3485) loss 3.3212 (2.6561) grad_norm 2.7929 (3.1101) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][270/625] eta 0:02:04 lr 0.000067 wd 0.0500 time 0.3464 (0.3520) data time 0.0008 (0.0029) model time 0.3456 (0.3483) loss 2.6380 (2.6571) grad_norm 1.9285 (3.1159) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][280/625] eta 0:02:01 lr 0.000067 wd 0.0500 time 0.3466 (0.3525) data time 0.0006 (0.0028) model time 0.3460 (0.3491) loss 2.5344 (2.6534) grad_norm 2.3979 (3.2447) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][290/625] eta 0:01:58 lr 0.000067 wd 0.0500 time 0.3418 (0.3528) data time 0.0008 (0.0028) model time 0.3410 (0.3496) loss 3.0469 (2.6539) grad_norm 2.2584 (3.2209) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][300/625] eta 0:01:54 lr 0.000067 wd 0.0500 time 0.3439 (0.3526) data time 0.0009 (0.0027) model time 0.3430 (0.3494) loss 2.8275 (2.6507) grad_norm 1.6714 (3.2494) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][310/625] eta 0:01:50 lr 0.000067 wd 0.0500 time 0.3457 (0.3523) data time 0.0006 (0.0026) model time 0.3450 (0.3492) loss 2.3420 (2.6496) grad_norm 2.3735 (3.2238) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][320/625] eta 0:01:47 lr 0.000067 wd 0.0500 time 0.3510 (0.3521) data time 0.0007 (0.0026) model time 0.3503 (0.3491) loss 2.3876 (2.6540) grad_norm 3.4016 (3.2141) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][330/625] eta 0:01:43 lr 0.000067 wd 0.0500 time 0.3512 (0.3520) data time 0.0006 (0.0025) model time 0.3505 (0.3490) loss 2.7298 (2.6498) grad_norm 2.0985 (3.1984) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][340/625] eta 0:01:40 lr 0.000067 wd 0.0500 time 0.3555 (0.3518) data time 0.0006 (0.0025) model time 0.3548 (0.3488) loss 1.5891 (2.6393) grad_norm 3.2603 (3.1796) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][350/625] eta 0:01:36 lr 0.000067 wd 0.0500 time 0.3449 (0.3516) data time 0.0008 (0.0024) model time 0.3441 (0.3487) loss 2.2447 (2.6323) grad_norm 3.8598 (3.1838) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][360/625] eta 0:01:33 lr 0.000066 wd 0.0500 time 0.3472 (0.3515) data time 0.0008 (0.0024) model time 0.3464 (0.3486) loss 2.9774 (2.6308) grad_norm 4.1882 (3.1781) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][370/625] eta 0:01:29 lr 0.000066 wd 0.0500 time 0.3442 (0.3514) data time 0.0006 (0.0024) model time 0.3436 (0.3485) loss 2.7867 (2.6347) grad_norm 2.4500 (3.1700) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][380/625] eta 0:01:26 lr 0.000066 wd 0.0500 time 0.3520 (0.3513) data time 0.0008 (0.0023) model time 0.3511 (0.3485) loss 2.7801 (2.6357) grad_norm 1.9016 (3.1565) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][390/625] eta 0:01:22 lr 0.000066 wd 0.0500 time 0.3490 (0.3511) data time 0.0008 (0.0023) model time 0.3482 (0.3483) loss 2.5690 (2.6332) grad_norm 2.2055 (3.1561) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][400/625] eta 0:01:18 lr 0.000066 wd 0.0500 time 0.3444 (0.3510) data time 0.0007 (0.0022) model time 0.3438 (0.3483) loss 2.5623 (2.6336) grad_norm 1.9696 (3.1578) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:28:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][410/625] eta 0:01:15 lr 0.000066 wd 0.0500 time 0.3508 (0.3509) data time 0.0008 (0.0022) model time 0.3499 (0.3482) loss 3.0525 (2.6274) grad_norm 38.1894 (3.2383) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][420/625] eta 0:01:11 lr 0.000066 wd 0.0500 time 0.3441 (0.3508) data time 0.0008 (0.0022) model time 0.3432 (0.3482) loss 1.8668 (2.6244) grad_norm 2.1946 (3.2220) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][430/625] eta 0:01:08 lr 0.000066 wd 0.0500 time 0.3475 (0.3508) data time 0.0007 (0.0021) model time 0.3468 (0.3481) loss 2.9656 (2.6246) grad_norm 3.1765 (3.2380) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][440/625] eta 0:01:04 lr 0.000066 wd 0.0500 time 0.3492 (0.3507) data time 0.0008 (0.0021) model time 0.3484 (0.3481) loss 2.1463 (2.6219) grad_norm 2.7146 (3.2866) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][450/625] eta 0:01:01 lr 0.000066 wd 0.0500 time 0.3444 (0.3506) data time 0.0008 (0.0021) model time 0.3436 (0.3480) loss 2.8749 (2.6246) grad_norm 5.9488 (3.2841) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][460/625] eta 0:00:57 lr 0.000066 wd 0.0500 time 0.5619 (0.3514) data time 0.0008 (0.0021) model time 0.5611 (0.3490) loss 2.5399 (2.6281) grad_norm 2.8284 (3.2780) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][470/625] eta 0:00:54 lr 0.000066 wd 0.0500 time 0.3414 (0.3512) data time 0.0006 (0.0020) model time 0.3408 (0.3488) loss 2.3007 (2.6291) grad_norm 7.1216 (3.2766) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][480/625] eta 0:00:50 lr 0.000066 wd 0.0500 time 0.3557 (0.3515) data time 0.0006 (0.0020) model time 0.3550 (0.3492) loss 2.7765 (2.6292) grad_norm 3.6128 (3.2682) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][490/625] eta 0:00:47 lr 0.000066 wd 0.0500 time 0.3408 (0.3513) data time 0.0008 (0.0020) model time 0.3399 (0.3490) loss 2.9876 (2.6345) grad_norm 3.3587 (3.2536) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][500/625] eta 0:00:43 lr 0.000066 wd 0.0500 time 0.3537 (0.3512) data time 0.0007 (0.0020) model time 0.3531 (0.3489) loss 1.6642 (2.6307) grad_norm 2.0323 (3.2723) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][510/625] eta 0:00:40 lr 0.000066 wd 0.0500 time 0.3472 (0.3511) data time 0.0008 (0.0019) model time 0.3464 (0.3488) loss 2.6711 (2.6246) grad_norm 3.2444 (3.2647) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][520/625] eta 0:00:36 lr 0.000066 wd 0.0500 time 0.3524 (0.3510) data time 0.0007 (0.0019) model time 0.3517 (0.3487) loss 2.6623 (2.6204) grad_norm 2.3198 (3.2498) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][530/625] eta 0:00:33 lr 0.000066 wd 0.0500 time 0.3501 (0.3509) data time 0.0005 (0.0019) model time 0.3496 (0.3486) loss 3.1320 (2.6189) grad_norm 3.1474 (3.2407) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][540/625] eta 0:00:29 lr 0.000066 wd 0.0500 time 0.3522 (0.3508) data time 0.0006 (0.0019) model time 0.3516 (0.3486) loss 2.0804 (2.6173) grad_norm 2.5083 (3.2501) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][550/625] eta 0:00:26 lr 0.000065 wd 0.0500 time 0.3473 (0.3507) data time 0.0009 (0.0019) model time 0.3464 (0.3485) loss 2.4276 (2.6197) grad_norm 3.2535 (3.2395) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][560/625] eta 0:00:22 lr 0.000065 wd 0.0500 time 0.3475 (0.3506) data time 0.0008 (0.0018) model time 0.3468 (0.3484) loss 2.4968 (2.6256) grad_norm 1.8329 (3.2325) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][570/625] eta 0:00:19 lr 0.000065 wd 0.0500 time 0.3484 (0.3506) data time 0.0008 (0.0018) model time 0.3476 (0.3484) loss 3.0930 (2.6254) grad_norm 2.8173 (3.2364) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:29:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][580/625] eta 0:00:15 lr 0.000065 wd 0.0500 time 0.3427 (0.3512) data time 0.0008 (0.0018) model time 0.3419 (0.3491) loss 2.9835 (2.6251) grad_norm 3.6804 (3.2307) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][590/625] eta 0:00:12 lr 0.000065 wd 0.0500 time 0.3224 (0.3514) data time 0.0008 (0.0018) model time 0.3216 (0.3494) loss 2.7074 (2.6273) grad_norm 3.6227 (3.2353) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][600/625] eta 0:00:08 lr 0.000065 wd 0.0500 time 0.3458 (0.3513) data time 0.0007 (0.0018) model time 0.3451 (0.3493) loss 3.0573 (2.6262) grad_norm 2.5149 (3.2258) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][610/625] eta 0:00:05 lr 0.000065 wd 0.0500 time 0.3341 (0.3512) data time 0.0005 (0.0018) model time 0.3337 (0.3491) loss 1.9107 (2.6226) grad_norm 1.5082 (3.2125) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [272/300][620/625] eta 0:00:01 lr 0.000065 wd 0.0500 time 0.3412 (0.3509) data time 0.0004 (0.0017) model time 0.3408 (0.3489) loss 3.4387 (2.6259) grad_norm 3.3394 (3.2030) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:13 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 272 training takes 0:03:39 [2024-09-01 14:30:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:30:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:30:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.442 (0.442) Loss 0.5215 (0.5215) Acc@1 88.965 (88.965) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 14:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.129) Loss 0.8281 (0.6111) Acc@1 81.055 (86.896) Acc@5 95.898 (97.807) Mem 11224MB [2024-09-01 14:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.111) Loss 0.9058 (0.7299) Acc@1 79.004 (84.022) Acc@5 95.166 (96.659) Mem 11224MB [2024-09-01 14:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.763 Acc@5 96.665 [2024-09-01 14:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 14:30:17 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.76% [2024-09-01 14:30:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 14:30:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 14:30:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.439 (0.439) Loss 0.5024 (0.5024) Acc@1 89.062 (89.062) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:30:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.126) Loss 0.8125 (0.5969) Acc@1 80.908 (87.034) Acc@5 95.898 (97.896) Mem 11224MB [2024-09-01 14:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.109) Loss 0.8955 (0.7108) Acc@1 79.102 (84.215) Acc@5 95.410 (96.747) Mem 11224MB [2024-09-01 14:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.909 Acc@5 96.763 [2024-09-01 14:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 14:30:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][0/625] eta 0:12:22 lr 0.000065 wd 0.0500 time 1.1885 (1.1885) data time 0.5527 (0.5527) model time 0.0000 (0.0000) loss 2.6572 (2.6572) grad_norm 2.8819 (2.8819) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][10/625] eta 0:04:18 lr 0.000065 wd 0.0500 time 0.3424 (0.4206) data time 0.0009 (0.0511) model time 0.0000 (0.0000) loss 2.1003 (2.6497) grad_norm 1.8068 (3.5901) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][20/625] eta 0:03:52 lr 0.000065 wd 0.0500 time 0.3424 (0.3842) data time 0.0009 (0.0271) model time 0.0000 (0.0000) loss 2.7174 (2.5857) grad_norm 2.5908 (3.1231) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][30/625] eta 0:03:41 lr 0.000065 wd 0.0500 time 0.3441 (0.3714) data time 0.0007 (0.0186) model time 0.0000 (0.0000) loss 2.6770 (2.6951) grad_norm 5.4730 (3.1171) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][40/625] eta 0:03:33 lr 0.000065 wd 0.0500 time 0.3476 (0.3653) data time 0.0008 (0.0144) model time 0.0000 (0.0000) loss 2.9760 (2.6608) grad_norm 2.8889 (3.0715) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][50/625] eta 0:03:27 lr 0.000065 wd 0.0500 time 0.3394 (0.3612) data time 0.0008 (0.0117) model time 0.0000 (0.0000) loss 2.9592 (2.6621) grad_norm 1.5989 (3.0220) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][60/625] eta 0:03:22 lr 0.000065 wd 0.0500 time 0.3452 (0.3587) data time 0.0008 (0.0099) model time 0.3444 (0.3451) loss 2.9764 (2.6752) grad_norm 2.7592 (2.9797) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][70/625] eta 0:03:18 lr 0.000065 wd 0.0500 time 0.3466 (0.3570) data time 0.0006 (0.0087) model time 0.3460 (0.3454) loss 1.9589 (2.6744) grad_norm 2.7933 (2.9327) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][80/625] eta 0:03:13 lr 0.000065 wd 0.0500 time 0.3531 (0.3557) data time 0.0008 (0.0077) model time 0.3523 (0.3454) loss 2.4848 (2.6430) grad_norm 2.4243 (2.8664) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][90/625] eta 0:03:09 lr 0.000065 wd 0.0500 time 0.3573 (0.3546) data time 0.0008 (0.0069) model time 0.3565 (0.3453) loss 2.8011 (2.6367) grad_norm 2.5436 (3.6506) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:30:55 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 14:30:55 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:30:56 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:37:39 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 14:37:40 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 14:37:53 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 14:38:05 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 14:38:05 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 14:38:08 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 14:38:10 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 14:38:10 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 273) [2024-09-01 14:38:10 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 14:38:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][100/625] eta 0:23:21 lr 0.000065 wd 0.0500 time 0.3448 (2.6703) data time 0.0007 (0.1171) model time 0.3441 (2.5532) loss 2.9420 (2.9234) grad_norm 3.0200 (3.2318) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:38:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][110/625] eta 0:10:27 lr 0.000064 wd 0.0500 time 0.3516 (1.2185) data time 0.0008 (0.0445) model time 0.3508 (1.1741) loss 2.8177 (2.8370) grad_norm 2.0593 (2.8041) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:38:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][120/625] eta 0:07:30 lr 0.000064 wd 0.0500 time 0.3438 (0.8918) data time 0.0007 (0.0277) model time 0.3430 (0.8641) loss 2.5007 (2.8531) grad_norm 2.6733 (5.5162) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:38:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][130/625] eta 0:06:06 lr 0.000064 wd 0.0500 time 0.3427 (0.7403) data time 0.0009 (0.0202) model time 0.3418 (0.7201) loss 2.8212 (2.8245) grad_norm 2.0542 (4.7895) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][140/625] eta 0:05:20 lr 0.000064 wd 0.0500 time 0.3430 (0.6600) data time 0.0008 (0.0160) model time 0.3422 (0.6440) loss 2.7960 (2.7708) grad_norm 2.9271 (4.2856) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:38:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][150/625] eta 0:04:47 lr 0.000064 wd 0.0500 time 0.3560 (0.6044) data time 0.0006 (0.0133) model time 0.3554 (0.5911) loss 3.1351 (2.7661) grad_norm 2.8128 (4.0036) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][160/625] eta 0:04:22 lr 0.000064 wd 0.0500 time 0.3491 (0.5650) data time 0.0007 (0.0114) model time 0.3484 (0.5536) loss 2.1028 (2.7276) grad_norm 3.0827 (3.7873) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:38:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][170/625] eta 0:04:03 lr 0.000064 wd 0.0500 time 0.3483 (0.5361) data time 0.0008 (0.0100) model time 0.3475 (0.5261) loss 3.0072 (2.6979) grad_norm 1.8164 (3.6263) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:38:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][180/625] eta 0:03:48 lr 0.000064 wd 0.0500 time 0.3407 (0.5138) data time 0.0007 (0.0090) model time 0.3400 (0.5048) loss 1.9174 (2.6780) grad_norm 2.2143 (3.5341) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][190/625] eta 0:03:36 lr 0.000064 wd 0.0500 time 0.3640 (0.4966) data time 0.0008 (0.0081) model time 0.3633 (0.4884) loss 2.6048 (2.6753) grad_norm 3.1386 (3.5184) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][200/625] eta 0:03:24 lr 0.000064 wd 0.0500 time 0.3445 (0.4823) data time 0.0009 (0.0074) model time 0.3437 (0.4749) loss 2.8794 (2.6926) grad_norm 1.8246 (3.4757) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][210/625] eta 0:03:15 lr 0.000064 wd 0.0500 time 0.3463 (0.4707) data time 0.0006 (0.0069) model time 0.3457 (0.4638) loss 3.0522 (2.6875) grad_norm 3.3971 (3.4290) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][220/625] eta 0:03:06 lr 0.000064 wd 0.0500 time 0.3427 (0.4608) data time 0.0006 (0.0064) model time 0.3421 (0.4544) loss 1.7283 (2.6804) grad_norm 1.7851 (3.3900) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][230/625] eta 0:02:58 lr 0.000064 wd 0.0500 time 0.3496 (0.4524) data time 0.0007 (0.0060) model time 0.3489 (0.4464) loss 2.0252 (2.6863) grad_norm 1.9477 (3.3391) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][240/625] eta 0:02:51 lr 0.000064 wd 0.0500 time 0.3476 (0.4450) data time 0.0007 (0.0056) model time 0.3469 (0.4394) loss 2.1854 (2.6786) grad_norm 1.8625 (3.2915) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][250/625] eta 0:02:44 lr 0.000064 wd 0.0500 time 0.3440 (0.4386) data time 0.0009 (0.0053) model time 0.3431 (0.4333) loss 3.0323 (2.6845) grad_norm 2.5000 (3.2462) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][260/625] eta 0:02:38 lr 0.000064 wd 0.0500 time 0.3449 (0.4331) data time 0.0008 (0.0051) model time 0.3441 (0.4280) loss 3.1390 (2.6878) grad_norm 3.7426 (3.2228) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][270/625] eta 0:02:32 lr 0.000064 wd 0.0500 time 0.3464 (0.4284) data time 0.0007 (0.0048) model time 0.3457 (0.4235) loss 2.6791 (2.6793) grad_norm 2.7459 (3.2312) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][280/625] eta 0:02:26 lr 0.000064 wd 0.0500 time 0.3429 (0.4240) data time 0.0010 (0.0046) model time 0.3419 (0.4194) loss 2.4241 (2.6756) grad_norm 2.5442 (3.2065) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:39:36 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 14:39:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:39:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:44:23 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 14:44:25 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 14:44:37 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 14:44:47 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 14:44:47 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 14:44:49 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 14:44:51 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 14:44:51 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 273) [2024-09-01 14:44:51 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 14:45:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][290/625] eta 0:43:54 lr 0.000064 wd 0.0500 time 0.5884 (7.8648) data time 0.0010 (0.3034) model time 0.5874 (7.5614) loss 2.9383 (2.9293) grad_norm 2.5026 (2.7349) loss_scale 64.0000 (64.0000) mem 11217MB [2024-09-01 14:45:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][300/625] eta 0:08:39 lr 0.000064 wd 0.0500 time 0.3488 (1.5997) data time 0.0007 (0.0514) model time 0.3481 (1.5484) loss 2.3205 (2.7091) grad_norm 2.5903 (2.6514) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][310/625] eta 0:05:24 lr 0.000063 wd 0.0500 time 0.3496 (1.0298) data time 0.0009 (0.0285) model time 0.3487 (1.0014) loss 2.5028 (2.7242) grad_norm 1.8980 (2.6974) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][320/625] eta 0:04:11 lr 0.000063 wd 0.0500 time 0.3424 (0.8234) data time 0.0009 (0.0198) model time 0.3415 (0.8036) loss 2.8943 (2.7409) grad_norm 2.2612 (2.6063) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][330/625] eta 0:03:31 lr 0.000063 wd 0.0500 time 0.3373 (0.7158) data time 0.0010 (0.0153) model time 0.3362 (0.7004) loss 2.7667 (2.7117) grad_norm 1.9017 (2.6214) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][340/625] eta 0:03:03 lr 0.000063 wd 0.0500 time 0.3449 (0.6441) data time 0.0007 (0.0126) model time 0.3442 (0.6315) loss 2.4938 (2.7213) grad_norm 1.9698 (2.6094) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][350/625] eta 0:02:43 lr 0.000063 wd 0.0500 time 0.3484 (0.5958) data time 0.0006 (0.0107) model time 0.3477 (0.5851) loss 3.2192 (2.7041) grad_norm 2.7368 (2.7084) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][360/625] eta 0:02:28 lr 0.000063 wd 0.0500 time 0.3437 (0.5610) data time 0.0008 (0.0093) model time 0.3429 (0.5517) loss 2.7858 (2.6758) grad_norm 3.0633 (2.6819) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][370/625] eta 0:02:16 lr 0.000063 wd 0.0500 time 0.3483 (0.5350) data time 0.0009 (0.0083) model time 0.3473 (0.5267) loss 2.5308 (2.6625) grad_norm 2.5186 (2.7047) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][380/625] eta 0:02:06 lr 0.000063 wd 0.0500 time 0.3451 (0.5144) data time 0.0007 (0.0075) model time 0.3444 (0.5070) loss 1.8746 (2.6494) grad_norm 2.5669 (2.7214) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][390/625] eta 0:01:57 lr 0.000063 wd 0.0500 time 0.3461 (0.4981) data time 0.0007 (0.0068) model time 0.3454 (0.4912) loss 3.0452 (2.6760) grad_norm 3.0120 (2.7392) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][400/625] eta 0:01:49 lr 0.000063 wd 0.0500 time 0.3474 (0.4845) data time 0.0008 (0.0063) model time 0.3466 (0.4782) loss 2.8072 (2.6782) grad_norm 14.4822 (2.8317) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][410/625] eta 0:01:41 lr 0.000063 wd 0.0500 time 0.3380 (0.4733) data time 0.0008 (0.0058) model time 0.3372 (0.4674) loss 3.0437 (2.6855) grad_norm 2.5579 (2.8465) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:45:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][420/625] eta 0:01:35 lr 0.000063 wd 0.0500 time 0.3502 (0.4637) data time 0.0009 (0.0055) model time 0.3493 (0.4582) loss 2.7277 (2.6787) grad_norm 2.4729 (2.8240) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][430/625] eta 0:01:28 lr 0.000063 wd 0.0500 time 0.3429 (0.4553) data time 0.0009 (0.0051) model time 0.3419 (0.4502) loss 2.6344 (2.6724) grad_norm 3.2159 (2.8429) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][440/625] eta 0:01:22 lr 0.000063 wd 0.0500 time 0.3431 (0.4483) data time 0.0009 (0.0049) model time 0.3422 (0.4434) loss 3.0110 (2.6735) grad_norm 6.3434 (2.9046) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][450/625] eta 0:01:17 lr 0.000063 wd 0.0500 time 0.3427 (0.4420) data time 0.0010 (0.0046) model time 0.3417 (0.4374) loss 2.9170 (2.6715) grad_norm 3.7999 (2.9228) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][460/625] eta 0:01:12 lr 0.000063 wd 0.0500 time 0.3543 (0.4366) data time 0.0008 (0.0044) model time 0.3535 (0.4322) loss 2.6584 (2.6682) grad_norm 2.0458 (2.8841) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][470/625] eta 0:01:06 lr 0.000063 wd 0.0500 time 0.3547 (0.4318) data time 0.0009 (0.0042) model time 0.3538 (0.4276) loss 2.7035 (2.6607) grad_norm 3.7354 (2.8804) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][480/625] eta 0:01:01 lr 0.000063 wd 0.0500 time 0.3471 (0.4273) data time 0.0009 (0.0040) model time 0.3462 (0.4233) loss 2.7777 (2.6529) grad_norm 2.2830 (2.8725) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][490/625] eta 0:00:57 lr 0.000063 wd 0.0500 time 0.3449 (0.4233) data time 0.0007 (0.0039) model time 0.3443 (0.4194) loss 2.9575 (2.6461) grad_norm 2.6728 (2.8471) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][500/625] eta 0:00:52 lr 0.000062 wd 0.0500 time 0.3473 (0.4196) data time 0.0008 (0.0037) model time 0.3464 (0.4159) loss 2.9366 (2.6400) grad_norm 6.0252 (2.8595) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][510/625] eta 0:00:47 lr 0.000062 wd 0.0500 time 0.3496 (0.4164) data time 0.0008 (0.0036) model time 0.3488 (0.4128) loss 2.8458 (2.6394) grad_norm 3.9900 (2.8705) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][520/625] eta 0:00:43 lr 0.000062 wd 0.0500 time 0.3472 (0.4152) data time 0.0007 (0.0035) model time 0.3465 (0.4117) loss 2.6547 (2.6397) grad_norm 2.2081 (2.8695) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][530/625] eta 0:00:39 lr 0.000062 wd 0.0500 time 0.3492 (0.4133) data time 0.0007 (0.0034) model time 0.3485 (0.4099) loss 2.8342 (2.6400) grad_norm 4.7252 (2.8664) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][540/625] eta 0:00:34 lr 0.000062 wd 0.0500 time 0.3429 (0.4107) data time 0.0007 (0.0033) model time 0.3421 (0.4074) loss 3.2106 (2.6316) grad_norm 2.0020 (2.8715) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][550/625] eta 0:00:30 lr 0.000062 wd 0.0500 time 0.3492 (0.4082) data time 0.0009 (0.0032) model time 0.3483 (0.4050) loss 2.9913 (2.6244) grad_norm 1.8883 (2.8548) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][560/625] eta 0:00:26 lr 0.000062 wd 0.0500 time 0.3445 (0.4060) data time 0.0009 (0.0031) model time 0.3436 (0.4029) loss 2.1789 (2.6157) grad_norm 2.1839 (2.8500) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][570/625] eta 0:00:22 lr 0.000062 wd 0.0500 time 0.3534 (0.4040) data time 0.0009 (0.0030) model time 0.3525 (0.4009) loss 2.1385 (2.6173) grad_norm 2.4333 (2.8562) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][580/625] eta 0:00:18 lr 0.000062 wd 0.0500 time 0.3456 (0.4020) data time 0.0010 (0.0030) model time 0.3446 (0.3991) loss 2.6238 (2.6127) grad_norm 3.5069 (2.8852) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:46:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][590/625] eta 0:00:14 lr 0.000062 wd 0.0500 time 0.3417 (0.4002) data time 0.0010 (0.0029) model time 0.3407 (0.3973) loss 2.9881 (2.6064) grad_norm 2.0095 (2.8795) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:47:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][600/625] eta 0:00:09 lr 0.000062 wd 0.0500 time 0.3469 (0.3985) data time 0.0009 (0.0028) model time 0.3460 (0.3957) loss 2.9773 (2.6072) grad_norm 2.3582 (2.9002) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:47:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][610/625] eta 0:00:05 lr 0.000062 wd 0.0500 time 0.3411 (0.3970) data time 0.0006 (0.0028) model time 0.3404 (0.3942) loss 2.9592 (2.6167) grad_norm 2.2659 (2.8940) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:47:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [273/300][620/625] eta 0:00:01 lr 0.000062 wd 0.0500 time 0.3430 (0.3954) data time 0.0006 (0.0027) model time 0.3424 (0.3926) loss 2.9168 (2.6157) grad_norm 2.1488 (2.8962) loss_scale 64.0000 (64.0000) mem 11219MB [2024-09-01 14:47:08 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 273 training takes 0:02:12 [2024-09-01 14:47:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:47:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:47:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.437 (0.437) Loss 0.5122 (0.5122) Acc@1 88.721 (88.721) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 14:47:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.130) Loss 0.8291 (0.6068) Acc@1 80.615 (86.737) Acc@5 95.850 (97.812) Mem 11219MB [2024-09-01 14:47:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 0.9106 (0.7241) Acc@1 79.004 (83.901) Acc@5 95.264 (96.631) Mem 11219MB [2024-09-01 14:47:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.637 Acc@5 96.625 [2024-09-01 14:47:17 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 14:47:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.766 (0.766) Loss 0.5029 (0.5029) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 14:47:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.161) Loss 0.8115 (0.5969) Acc@1 80.957 (87.065) Acc@5 95.801 (97.900) Mem 11219MB [2024-09-01 14:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.128) Loss 0.8945 (0.7111) Acc@1 79.053 (84.242) Acc@5 95.459 (96.749) Mem 11219MB [2024-09-01 14:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.943 Acc@5 96.771 [2024-09-01 14:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 14:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.94% [2024-09-01 14:47:20 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 14:47:22 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 14:47:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][0/625] eta 0:07:32 lr 0.000062 wd 0.0500 time 0.7247 (0.7247) data time 0.3755 (0.3755) model time 0.0000 (0.0000) loss 2.3410 (2.3410) grad_norm 3.3584 (3.3584) loss_scale 64.0000 (64.0000) mem 11227MB [2024-09-01 14:47:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][10/625] eta 0:04:02 lr 0.000062 wd 0.0500 time 0.4874 (0.3945) data time 0.0008 (0.0350) model time 0.0000 (0.0000) loss 2.2611 (2.7291) grad_norm 3.6538 (2.9023) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][20/625] eta 0:03:51 lr 0.000062 wd 0.0500 time 0.3478 (0.3825) data time 0.0010 (0.0187) model time 0.0000 (0.0000) loss 2.1809 (2.6849) grad_norm 5.8422 (3.3033) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][30/625] eta 0:03:44 lr 0.000062 wd 0.0500 time 0.3511 (0.3777) data time 0.0007 (0.0129) model time 0.0000 (0.0000) loss 1.9344 (2.6637) grad_norm 3.7187 (3.2518) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][40/625] eta 0:03:36 lr 0.000062 wd 0.0500 time 0.3495 (0.3708) data time 0.0008 (0.0100) model time 0.0000 (0.0000) loss 2.5473 (2.6250) grad_norm 3.1907 (3.4809) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][50/625] eta 0:03:30 lr 0.000062 wd 0.0500 time 0.3428 (0.3667) data time 0.0007 (0.0082) model time 0.0000 (0.0000) loss 2.9428 (2.6077) grad_norm 4.2296 (3.3901) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][60/625] eta 0:03:25 lr 0.000062 wd 0.0500 time 0.3477 (0.3637) data time 0.0006 (0.0070) model time 0.3471 (0.3480) loss 2.4933 (2.6137) grad_norm 3.2349 (3.3268) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][70/625] eta 0:03:20 lr 0.000061 wd 0.0500 time 0.3476 (0.3615) data time 0.0010 (0.0061) model time 0.3466 (0.3475) loss 2.6738 (2.6493) grad_norm 26.1239 (3.7442) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][80/625] eta 0:03:15 lr 0.000061 wd 0.0500 time 0.3476 (0.3596) data time 0.0007 (0.0055) model time 0.3468 (0.3466) loss 2.0447 (2.6497) grad_norm 1.9973 (3.6811) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][90/625] eta 0:03:11 lr 0.000061 wd 0.0500 time 0.3459 (0.3583) data time 0.0009 (0.0050) model time 0.3449 (0.3469) loss 2.8440 (2.6596) grad_norm 2.8007 (3.6173) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:47:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][100/625] eta 0:03:07 lr 0.000061 wd 0.0500 time 0.3545 (0.3574) data time 0.0009 (0.0046) model time 0.3536 (0.3471) loss 3.0168 (2.6745) grad_norm 4.3505 (3.5278) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][110/625] eta 0:03:03 lr 0.000061 wd 0.0500 time 0.3522 (0.3566) data time 0.0009 (0.0042) model time 0.3513 (0.3472) loss 2.1444 (2.6708) grad_norm 1.7033 (3.4204) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][120/625] eta 0:02:59 lr 0.000061 wd 0.0500 time 0.3517 (0.3559) data time 0.0009 (0.0039) model time 0.3508 (0.3472) loss 1.7317 (2.6475) grad_norm 2.8771 (3.3692) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][130/625] eta 0:02:56 lr 0.000061 wd 0.0500 time 0.5518 (0.3568) data time 0.0009 (0.0037) model time 0.5509 (0.3497) loss 2.2045 (2.6263) grad_norm 4.7643 (3.3277) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][140/625] eta 0:02:52 lr 0.000061 wd 0.0500 time 0.3470 (0.3561) data time 0.0007 (0.0035) model time 0.3463 (0.3493) loss 3.2601 (2.6264) grad_norm 3.0519 (3.4102) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][150/625] eta 0:02:49 lr 0.000061 wd 0.0500 time 0.3457 (0.3569) data time 0.0009 (0.0033) model time 0.3448 (0.3511) loss 2.4654 (2.6351) grad_norm 2.6137 (3.4091) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][160/625] eta 0:02:45 lr 0.000061 wd 0.0500 time 0.3535 (0.3563) data time 0.0007 (0.0032) model time 0.3527 (0.3506) loss 2.7761 (2.6311) grad_norm 6.0677 (3.4679) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][170/625] eta 0:02:41 lr 0.000061 wd 0.0500 time 0.3452 (0.3558) data time 0.0009 (0.0031) model time 0.3442 (0.3503) loss 2.7958 (2.6331) grad_norm 2.3803 (3.4426) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][180/625] eta 0:02:38 lr 0.000061 wd 0.0500 time 0.3467 (0.3553) data time 0.0008 (0.0029) model time 0.3459 (0.3499) loss 2.7894 (2.6422) grad_norm 2.2778 (3.4446) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][190/625] eta 0:02:34 lr 0.000061 wd 0.0500 time 0.3450 (0.3547) data time 0.0010 (0.0028) model time 0.3440 (0.3495) loss 2.7243 (2.6317) grad_norm 3.0167 (3.4166) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][200/625] eta 0:02:30 lr 0.000061 wd 0.0500 time 0.3418 (0.3544) data time 0.0006 (0.0027) model time 0.3411 (0.3493) loss 1.7033 (2.6199) grad_norm 4.6712 (3.4358) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][210/625] eta 0:02:26 lr 0.000061 wd 0.0500 time 0.3468 (0.3540) data time 0.0007 (0.0026) model time 0.3461 (0.3491) loss 3.1531 (2.6258) grad_norm 3.3157 (3.4309) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][220/625] eta 0:02:23 lr 0.000061 wd 0.0500 time 0.3454 (0.3536) data time 0.0006 (0.0026) model time 0.3447 (0.3488) loss 1.8239 (2.6296) grad_norm 3.0354 (3.4200) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][230/625] eta 0:02:19 lr 0.000061 wd 0.0500 time 0.3446 (0.3533) data time 0.0009 (0.0025) model time 0.3437 (0.3487) loss 2.7374 (2.6355) grad_norm 2.2234 (3.3797) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][240/625] eta 0:02:15 lr 0.000061 wd 0.0500 time 0.3484 (0.3531) data time 0.0006 (0.0024) model time 0.3478 (0.3485) loss 3.3067 (2.6409) grad_norm 2.2126 (3.3620) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][250/625] eta 0:02:12 lr 0.000061 wd 0.0500 time 0.3489 (0.3528) data time 0.0007 (0.0024) model time 0.3482 (0.3484) loss 3.0359 (2.6408) grad_norm 2.6155 (3.3372) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][260/625] eta 0:02:08 lr 0.000061 wd 0.0500 time 0.3454 (0.3525) data time 0.0007 (0.0023) model time 0.3448 (0.3482) loss 2.8966 (2.6474) grad_norm 2.2499 (3.3544) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:48:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][270/625] eta 0:02:05 lr 0.000060 wd 0.0500 time 0.3423 (0.3523) data time 0.0010 (0.0022) model time 0.3413 (0.3481) loss 2.7230 (2.6417) grad_norm 2.0056 (3.3242) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][280/625] eta 0:02:01 lr 0.000060 wd 0.0500 time 0.3480 (0.3521) data time 0.0007 (0.0022) model time 0.3473 (0.3479) loss 2.9196 (2.6441) grad_norm 4.0027 (3.3082) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][290/625] eta 0:01:57 lr 0.000060 wd 0.0500 time 0.3468 (0.3519) data time 0.0007 (0.0022) model time 0.3461 (0.3479) loss 2.4538 (2.6462) grad_norm 2.6150 (3.3029) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][300/625] eta 0:01:54 lr 0.000060 wd 0.0500 time 0.3499 (0.3517) data time 0.0006 (0.0021) model time 0.3493 (0.3478) loss 2.4513 (2.6467) grad_norm 2.5224 (3.2959) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][310/625] eta 0:01:50 lr 0.000060 wd 0.0500 time 0.3443 (0.3516) data time 0.0007 (0.0021) model time 0.3436 (0.3477) loss 2.8763 (2.6413) grad_norm 2.6176 (3.2848) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][320/625] eta 0:01:47 lr 0.000060 wd 0.0500 time 0.3476 (0.3514) data time 0.0008 (0.0020) model time 0.3468 (0.3476) loss 2.0137 (2.6389) grad_norm 2.2914 (3.2809) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][330/625] eta 0:01:43 lr 0.000060 wd 0.0500 time 0.3433 (0.3513) data time 0.0009 (0.0020) model time 0.3423 (0.3476) loss 2.9786 (2.6372) grad_norm 2.1632 (3.2712) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][340/625] eta 0:01:40 lr 0.000060 wd 0.0500 time 0.3425 (0.3518) data time 0.0006 (0.0020) model time 0.3419 (0.3483) loss 2.4543 (2.6409) grad_norm 1.8573 (3.2430) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][350/625] eta 0:01:36 lr 0.000060 wd 0.0500 time 0.3504 (0.3517) data time 0.0007 (0.0019) model time 0.3497 (0.3483) loss 3.1002 (2.6434) grad_norm 2.7666 (3.2243) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][360/625] eta 0:01:33 lr 0.000060 wd 0.0500 time 0.3501 (0.3520) data time 0.0008 (0.0019) model time 0.3493 (0.3488) loss 1.5228 (2.6309) grad_norm 2.8156 (3.2138) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][370/625] eta 0:01:29 lr 0.000060 wd 0.0500 time 0.3525 (0.3520) data time 0.0006 (0.0019) model time 0.3519 (0.3487) loss 1.7314 (2.6326) grad_norm 2.6159 (3.2081) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][380/625] eta 0:01:26 lr 0.000060 wd 0.0500 time 0.3454 (0.3518) data time 0.0009 (0.0018) model time 0.3444 (0.3486) loss 1.5407 (2.6238) grad_norm 2.1482 (3.2096) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][390/625] eta 0:01:22 lr 0.000060 wd 0.0500 time 0.3483 (0.3517) data time 0.0007 (0.0018) model time 0.3476 (0.3486) loss 2.0311 (2.6230) grad_norm 2.7666 (3.2027) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][400/625] eta 0:01:19 lr 0.000060 wd 0.0500 time 0.3505 (0.3516) data time 0.0006 (0.0018) model time 0.3499 (0.3485) loss 2.9754 (2.6317) grad_norm 2.0657 (3.1906) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][410/625] eta 0:01:15 lr 0.000060 wd 0.0500 time 0.3473 (0.3515) data time 0.0008 (0.0018) model time 0.3466 (0.3484) loss 2.8462 (2.6305) grad_norm 3.2577 (3.1847) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][420/625] eta 0:01:12 lr 0.000060 wd 0.0500 time 0.3506 (0.3514) data time 0.0009 (0.0018) model time 0.3497 (0.3484) loss 2.7913 (2.6264) grad_norm 2.6231 (3.1728) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][430/625] eta 0:01:08 lr 0.000060 wd 0.0500 time 0.3495 (0.3513) data time 0.0009 (0.0017) model time 0.3486 (0.3483) loss 2.0957 (2.6308) grad_norm 2.7681 (3.1683) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:49:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][440/625] eta 0:01:04 lr 0.000060 wd 0.0500 time 0.3446 (0.3512) data time 0.0008 (0.0017) model time 0.3437 (0.3483) loss 2.0437 (2.6309) grad_norm 2.2917 (3.1592) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][450/625] eta 0:01:01 lr 0.000060 wd 0.0500 time 0.3498 (0.3511) data time 0.0007 (0.0017) model time 0.3492 (0.3482) loss 2.7850 (2.6317) grad_norm 2.4289 (3.2090) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][460/625] eta 0:00:57 lr 0.000060 wd 0.0500 time 0.3467 (0.3510) data time 0.0007 (0.0017) model time 0.3460 (0.3482) loss 2.5549 (2.6326) grad_norm 2.5492 (3.1928) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][470/625] eta 0:00:54 lr 0.000059 wd 0.0500 time 0.3477 (0.3510) data time 0.0008 (0.0017) model time 0.3469 (0.3482) loss 1.7748 (2.6281) grad_norm 2.3136 (3.2059) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][480/625] eta 0:00:50 lr 0.000059 wd 0.0500 time 0.3445 (0.3509) data time 0.0008 (0.0016) model time 0.3437 (0.3482) loss 3.3040 (2.6293) grad_norm 3.8090 (3.2038) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][490/625] eta 0:00:47 lr 0.000059 wd 0.0500 time 0.3453 (0.3508) data time 0.0007 (0.0016) model time 0.3446 (0.3481) loss 2.3141 (2.6231) grad_norm 4.3779 (3.1939) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][500/625] eta 0:00:43 lr 0.000059 wd 0.0500 time 0.3419 (0.3507) data time 0.0009 (0.0016) model time 0.3411 (0.3481) loss 1.3419 (2.6213) grad_norm 3.2784 (3.1901) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][510/625] eta 0:00:40 lr 0.000059 wd 0.0500 time 0.3424 (0.3507) data time 0.0009 (0.0016) model time 0.3415 (0.3480) loss 2.5867 (2.6197) grad_norm 3.3243 (3.1837) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][520/625] eta 0:00:36 lr 0.000059 wd 0.0500 time 0.3426 (0.3506) data time 0.0009 (0.0016) model time 0.3417 (0.3480) loss 2.4886 (2.6209) grad_norm 2.7447 (3.1889) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][530/625] eta 0:00:33 lr 0.000059 wd 0.0500 time 0.3438 (0.3513) data time 0.0007 (0.0016) model time 0.3431 (0.3488) loss 2.1790 (2.6196) grad_norm 1.8252 (3.1860) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][540/625] eta 0:00:29 lr 0.000059 wd 0.0500 time 0.3466 (0.3514) data time 0.0006 (0.0016) model time 0.3459 (0.3490) loss 1.7834 (2.6190) grad_norm 2.4866 (3.1866) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][550/625] eta 0:00:26 lr 0.000059 wd 0.0500 time 0.3502 (0.3513) data time 0.0007 (0.0015) model time 0.3495 (0.3489) loss 2.3748 (2.6179) grad_norm 2.8092 (3.1762) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][560/625] eta 0:00:22 lr 0.000059 wd 0.0500 time 0.3458 (0.3513) data time 0.0007 (0.0015) model time 0.3451 (0.3489) loss 2.5174 (2.6170) grad_norm 1.9744 (3.1671) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:50:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][570/625] eta 0:00:19 lr 0.000059 wd 0.0500 time 0.3478 (0.3512) data time 0.0007 (0.0015) model time 0.3471 (0.3488) loss 1.5599 (2.6167) grad_norm 1.9195 (3.1541) loss_scale 128.0000 (64.3363) mem 11224MB [2024-09-01 14:50:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][580/625] eta 0:00:15 lr 0.000059 wd 0.0500 time 0.3531 (0.3511) data time 0.0009 (0.0015) model time 0.3522 (0.3488) loss 2.2488 (2.6149) grad_norm 2.1552 (3.1455) loss_scale 128.0000 (65.4320) mem 11224MB [2024-09-01 14:50:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][590/625] eta 0:00:12 lr 0.000059 wd 0.0500 time 0.3480 (0.3511) data time 0.0010 (0.0015) model time 0.3470 (0.3487) loss 2.6645 (2.6179) grad_norm 1.8324 (3.1363) loss_scale 128.0000 (66.4907) mem 11224MB [2024-09-01 14:50:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][600/625] eta 0:00:08 lr 0.000059 wd 0.0500 time 0.3481 (0.3510) data time 0.0009 (0.0015) model time 0.3473 (0.3487) loss 3.1110 (2.6186) grad_norm 2.3877 (3.1298) loss_scale 128.0000 (67.5141) mem 11224MB [2024-09-01 14:50:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][610/625] eta 0:00:05 lr 0.000059 wd 0.0500 time 0.3412 (0.3510) data time 0.0004 (0.0015) model time 0.3407 (0.3487) loss 1.9642 (2.6165) grad_norm 2.4014 (3.1196) loss_scale 128.0000 (68.5041) mem 11224MB [2024-09-01 14:51:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [274/300][620/625] eta 0:00:01 lr 0.000059 wd 0.0500 time 0.3376 (0.3508) data time 0.0006 (0.0015) model time 0.3370 (0.3485) loss 2.6967 (2.6127) grad_norm 2.1427 (3.1164) loss_scale 128.0000 (69.4622) mem 11224MB [2024-09-01 14:51:01 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 274 training takes 0:03:39 [2024-09-01 14:51:02 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:51:03 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:51:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.437 (0.437) Loss 0.5200 (0.5200) Acc@1 88.623 (88.623) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 14:51:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.8516 (0.6115) Acc@1 80.322 (86.839) Acc@5 95.557 (97.820) Mem 11224MB [2024-09-01 14:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.9141 (0.7290) Acc@1 78.955 (83.996) Acc@5 95.361 (96.619) Mem 11224MB [2024-09-01 14:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.711 Acc@5 96.637 [2024-09-01 14:51:05 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 14:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.817 (0.817) Loss 0.5039 (0.5039) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:51:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.165) Loss 0.8135 (0.5975) Acc@1 80.859 (87.052) Acc@5 95.801 (97.896) Mem 11224MB [2024-09-01 14:51:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.129) Loss 0.8955 (0.7119) Acc@1 79.053 (84.231) Acc@5 95.410 (96.735) Mem 11224MB [2024-09-01 14:51:08 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.939 Acc@5 96.761 [2024-09-01 14:51:08 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 14:51:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][0/625] eta 0:12:08 lr 0.000059 wd 0.0500 time 1.1661 (1.1661) data time 0.5197 (0.5197) model time 0.0000 (0.0000) loss 2.5849 (2.5849) grad_norm 2.4899 (2.4899) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][10/625] eta 0:04:19 lr 0.000059 wd 0.0500 time 0.3471 (0.4213) data time 0.0007 (0.0481) model time 0.0000 (0.0000) loss 2.5038 (2.6266) grad_norm 4.4453 (2.8314) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][20/625] eta 0:03:58 lr 0.000059 wd 0.0500 time 0.3466 (0.3939) data time 0.0007 (0.0256) model time 0.0000 (0.0000) loss 3.2650 (2.6525) grad_norm 2.3462 (2.7603) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][30/625] eta 0:03:52 lr 0.000059 wd 0.0500 time 0.3514 (0.3915) data time 0.0011 (0.0177) model time 0.0000 (0.0000) loss 2.4867 (2.6048) grad_norm 3.2543 (2.7539) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][40/625] eta 0:03:42 lr 0.000058 wd 0.0500 time 0.3446 (0.3808) data time 0.0008 (0.0136) model time 0.0000 (0.0000) loss 2.6353 (2.5984) grad_norm 2.9710 (2.6976) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][50/625] eta 0:03:35 lr 0.000058 wd 0.0500 time 0.3455 (0.3743) data time 0.0007 (0.0111) model time 0.0000 (0.0000) loss 2.8042 (2.6412) grad_norm 3.7767 (2.8351) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][60/625] eta 0:03:29 lr 0.000058 wd 0.0500 time 0.3503 (0.3699) data time 0.0008 (0.0094) model time 0.3495 (0.3466) loss 2.7175 (2.6345) grad_norm 2.6013 (2.8296) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][70/625] eta 0:03:23 lr 0.000058 wd 0.0500 time 0.3456 (0.3667) data time 0.0006 (0.0082) model time 0.3449 (0.3465) loss 2.6562 (2.5909) grad_norm 2.6057 (2.8690) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][80/625] eta 0:03:18 lr 0.000058 wd 0.0500 time 0.3471 (0.3645) data time 0.0006 (0.0073) model time 0.3465 (0.3469) loss 2.9334 (2.6177) grad_norm 2.6166 (2.8368) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][90/625] eta 0:03:14 lr 0.000058 wd 0.0500 time 0.3508 (0.3629) data time 0.0008 (0.0066) model time 0.3500 (0.3475) loss 2.9273 (2.6148) grad_norm 2.6440 (2.8576) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][100/625] eta 0:03:09 lr 0.000058 wd 0.0500 time 0.3487 (0.3618) data time 0.0006 (0.0060) model time 0.3481 (0.3482) loss 2.9133 (2.6257) grad_norm 4.0925 (2.8860) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][110/625] eta 0:03:05 lr 0.000058 wd 0.0500 time 0.3450 (0.3607) data time 0.0008 (0.0055) model time 0.3442 (0.3483) loss 1.7162 (2.6171) grad_norm 1.9883 (2.9098) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][120/625] eta 0:03:01 lr 0.000058 wd 0.0500 time 0.3444 (0.3597) data time 0.0009 (0.0052) model time 0.3435 (0.3481) loss 2.8650 (2.6241) grad_norm 3.0483 (2.9323) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][130/625] eta 0:02:57 lr 0.000058 wd 0.0500 time 0.3488 (0.3588) data time 0.0006 (0.0048) model time 0.3482 (0.3480) loss 2.2379 (2.6169) grad_norm 3.3525 (2.9446) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:51:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][140/625] eta 0:02:53 lr 0.000058 wd 0.0500 time 0.3476 (0.3582) data time 0.0009 (0.0045) model time 0.3467 (0.3482) loss 2.1368 (2.6051) grad_norm 3.1467 (3.0025) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][150/625] eta 0:02:51 lr 0.000058 wd 0.0500 time 0.3461 (0.3603) data time 0.0009 (0.0043) model time 0.3452 (0.3523) loss 3.1238 (2.6152) grad_norm 2.4184 (2.9935) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][160/625] eta 0:02:47 lr 0.000058 wd 0.0500 time 0.3535 (0.3596) data time 0.0007 (0.0041) model time 0.3528 (0.3519) loss 2.5493 (2.6263) grad_norm 2.6915 (2.9736) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][170/625] eta 0:02:43 lr 0.000058 wd 0.0500 time 0.3445 (0.3589) data time 0.0007 (0.0039) model time 0.3439 (0.3515) loss 2.0081 (2.6241) grad_norm 2.5364 (2.9888) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][180/625] eta 0:02:39 lr 0.000058 wd 0.0500 time 0.3477 (0.3583) data time 0.0009 (0.0037) model time 0.3468 (0.3512) loss 2.9998 (2.6249) grad_norm 2.2455 (2.9810) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][190/625] eta 0:02:35 lr 0.000058 wd 0.0500 time 0.3483 (0.3579) data time 0.0011 (0.0036) model time 0.3472 (0.3511) loss 2.8648 (2.6284) grad_norm 2.3714 (2.9883) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][200/625] eta 0:02:32 lr 0.000058 wd 0.0500 time 0.3489 (0.3577) data time 0.0007 (0.0034) model time 0.3482 (0.3512) loss 2.1212 (2.6261) grad_norm 3.2412 (3.0413) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][210/625] eta 0:02:28 lr 0.000058 wd 0.0500 time 0.3500 (0.3575) data time 0.0009 (0.0033) model time 0.3492 (0.3513) loss 2.5005 (2.6250) grad_norm 1.8996 (3.0391) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][220/625] eta 0:02:24 lr 0.000058 wd 0.0500 time 0.3476 (0.3572) data time 0.0007 (0.0032) model time 0.3469 (0.3512) loss 3.0712 (2.6366) grad_norm 1.9944 (3.0125) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][230/625] eta 0:02:21 lr 0.000058 wd 0.0500 time 0.3506 (0.3577) data time 0.0009 (0.0031) model time 0.3497 (0.3521) loss 2.5141 (2.6414) grad_norm 2.6003 (3.0055) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][240/625] eta 0:02:17 lr 0.000058 wd 0.0500 time 0.3481 (0.3573) data time 0.0008 (0.0030) model time 0.3473 (0.3519) loss 1.5885 (2.6389) grad_norm 3.6449 (2.9915) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][250/625] eta 0:02:13 lr 0.000057 wd 0.0500 time 0.3506 (0.3570) data time 0.0007 (0.0029) model time 0.3500 (0.3518) loss 3.2353 (2.6429) grad_norm 2.9814 (2.9648) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][260/625] eta 0:02:10 lr 0.000057 wd 0.0500 time 0.3463 (0.3567) data time 0.0009 (0.0028) model time 0.3454 (0.3516) loss 2.7331 (2.6372) grad_norm 6.8131 (2.9709) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][270/625] eta 0:02:06 lr 0.000057 wd 0.0500 time 0.3497 (0.3566) data time 0.0007 (0.0028) model time 0.3490 (0.3516) loss 2.4292 (2.6355) grad_norm 3.2420 (2.9594) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][280/625] eta 0:02:02 lr 0.000057 wd 0.0500 time 0.3453 (0.3564) data time 0.0007 (0.0027) model time 0.3446 (0.3515) loss 3.1258 (2.6408) grad_norm 3.3400 (2.9649) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][290/625] eta 0:01:59 lr 0.000057 wd 0.0500 time 0.3529 (0.3562) data time 0.0008 (0.0026) model time 0.3521 (0.3515) loss 2.8927 (2.6335) grad_norm 2.6840 (2.9678) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][300/625] eta 0:01:55 lr 0.000057 wd 0.0500 time 0.3470 (0.3560) data time 0.0006 (0.0026) model time 0.3465 (0.3514) loss 2.5509 (2.6366) grad_norm 2.1809 (2.9654) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:52:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][310/625] eta 0:01:52 lr 0.000057 wd 0.0500 time 0.3452 (0.3558) data time 0.0009 (0.0025) model time 0.3443 (0.3513) loss 2.3080 (2.6385) grad_norm 19.8336 (3.0433) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][320/625] eta 0:01:48 lr 0.000057 wd 0.0500 time 0.3444 (0.3556) data time 0.0009 (0.0025) model time 0.3435 (0.3512) loss 3.0037 (2.6475) grad_norm 3.4776 (3.0372) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][330/625] eta 0:01:44 lr 0.000057 wd 0.0500 time 0.3496 (0.3554) data time 0.0008 (0.0024) model time 0.3488 (0.3510) loss 2.0925 (2.6485) grad_norm 3.2477 (3.0454) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][340/625] eta 0:01:41 lr 0.000057 wd 0.0500 time 0.3448 (0.3551) data time 0.0006 (0.0024) model time 0.3442 (0.3509) loss 2.2979 (2.6506) grad_norm 2.5558 (3.0404) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][350/625] eta 0:01:37 lr 0.000057 wd 0.0500 time 0.3496 (0.3550) data time 0.0008 (0.0023) model time 0.3487 (0.3509) loss 2.6747 (2.6468) grad_norm 1.9828 (3.0268) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][360/625] eta 0:01:34 lr 0.000057 wd 0.0500 time 0.3506 (0.3560) data time 0.0006 (0.0023) model time 0.3500 (0.3521) loss 2.1568 (2.6375) grad_norm 3.3577 (3.0203) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][370/625] eta 0:01:30 lr 0.000057 wd 0.0500 time 0.3466 (0.3557) data time 0.0006 (0.0023) model time 0.3460 (0.3519) loss 2.7007 (2.6388) grad_norm 3.4901 (3.0325) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][380/625] eta 0:01:27 lr 0.000057 wd 0.0500 time 0.3470 (0.3555) data time 0.0008 (0.0022) model time 0.3462 (0.3518) loss 1.7733 (2.6385) grad_norm 1.9511 (3.0383) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][390/625] eta 0:01:23 lr 0.000057 wd 0.0500 time 0.3470 (0.3553) data time 0.0007 (0.0022) model time 0.3464 (0.3516) loss 3.1719 (2.6410) grad_norm 1.7923 (3.0322) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][400/625] eta 0:01:19 lr 0.000057 wd 0.0500 time 0.3470 (0.3552) data time 0.0009 (0.0021) model time 0.3461 (0.3515) loss 3.0446 (2.6439) grad_norm 2.4146 (3.0253) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][410/625] eta 0:01:16 lr 0.000057 wd 0.0500 time 0.3499 (0.3554) data time 0.0009 (0.0021) model time 0.3490 (0.3519) loss 2.7670 (2.6488) grad_norm 2.5473 (3.0211) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][420/625] eta 0:01:12 lr 0.000057 wd 0.0500 time 0.3468 (0.3553) data time 0.0008 (0.0021) model time 0.3459 (0.3518) loss 2.4589 (2.6494) grad_norm 4.4140 (3.0217) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][430/625] eta 0:01:09 lr 0.000057 wd 0.0500 time 0.3420 (0.3552) data time 0.0007 (0.0021) model time 0.3413 (0.3517) loss 1.6861 (2.6489) grad_norm 2.9455 (3.0166) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][440/625] eta 0:01:05 lr 0.000057 wd 0.0500 time 0.3492 (0.3550) data time 0.0006 (0.0020) model time 0.3485 (0.3516) loss 2.7874 (2.6528) grad_norm 2.5991 (3.0167) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][450/625] eta 0:01:02 lr 0.000057 wd 0.0500 time 0.3540 (0.3549) data time 0.0006 (0.0020) model time 0.3534 (0.3516) loss 2.4878 (2.6548) grad_norm 3.1377 (3.0058) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][460/625] eta 0:00:58 lr 0.000056 wd 0.0500 time 0.3473 (0.3548) data time 0.0008 (0.0020) model time 0.3465 (0.3516) loss 3.1029 (2.6555) grad_norm 1.9902 (2.9944) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][470/625] eta 0:00:54 lr 0.000056 wd 0.0500 time 0.3466 (0.3547) data time 0.0007 (0.0020) model time 0.3459 (0.3515) loss 2.9385 (2.6551) grad_norm 2.6071 (2.9874) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:53:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][480/625] eta 0:00:51 lr 0.000056 wd 0.0500 time 0.3537 (0.3547) data time 0.0006 (0.0019) model time 0.3531 (0.3515) loss 2.8436 (2.6543) grad_norm 2.4268 (2.9887) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][490/625] eta 0:00:47 lr 0.000056 wd 0.0500 time 0.3493 (0.3545) data time 0.0008 (0.0019) model time 0.3485 (0.3514) loss 2.6655 (2.6573) grad_norm 4.2946 (2.9893) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][500/625] eta 0:00:44 lr 0.000056 wd 0.0500 time 0.3740 (0.3545) data time 0.0007 (0.0019) model time 0.3733 (0.3514) loss 2.6239 (2.6589) grad_norm 4.8267 (3.0152) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][510/625] eta 0:00:40 lr 0.000056 wd 0.0500 time 0.3434 (0.3544) data time 0.0009 (0.0019) model time 0.3425 (0.3513) loss 2.5609 (2.6562) grad_norm 4.4273 (3.0219) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][520/625] eta 0:00:37 lr 0.000056 wd 0.0500 time 0.3525 (0.3543) data time 0.0009 (0.0018) model time 0.3515 (0.3513) loss 2.2098 (2.6491) grad_norm 2.3907 (3.0270) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][530/625] eta 0:00:33 lr 0.000056 wd 0.0500 time 0.3467 (0.3546) data time 0.0007 (0.0018) model time 0.3459 (0.3516) loss 3.0401 (2.6438) grad_norm 4.5387 (3.0247) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][540/625] eta 0:00:30 lr 0.000056 wd 0.0500 time 0.5527 (0.3556) data time 0.0007 (0.0018) model time 0.5520 (0.3528) loss 2.5975 (2.6432) grad_norm 3.0195 (3.0283) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][550/625] eta 0:00:26 lr 0.000056 wd 0.0500 time 0.3449 (0.3554) data time 0.0007 (0.0018) model time 0.3441 (0.3527) loss 3.1566 (2.6439) grad_norm 1.5811 (3.0192) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][560/625] eta 0:00:23 lr 0.000056 wd 0.0500 time 0.3480 (0.3553) data time 0.0006 (0.0018) model time 0.3474 (0.3526) loss 2.7880 (2.6453) grad_norm 2.6129 (3.0133) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][570/625] eta 0:00:19 lr 0.000056 wd 0.0500 time 0.3481 (0.3552) data time 0.0008 (0.0018) model time 0.3473 (0.3525) loss 3.0009 (2.6461) grad_norm 2.9844 (3.0487) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][580/625] eta 0:00:15 lr 0.000056 wd 0.0500 time 0.3496 (0.3551) data time 0.0009 (0.0017) model time 0.3487 (0.3524) loss 2.7149 (2.6454) grad_norm 2.5387 (3.0493) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][590/625] eta 0:00:12 lr 0.000056 wd 0.0500 time 0.3504 (0.3550) data time 0.0007 (0.0017) model time 0.3497 (0.3523) loss 2.1001 (2.6408) grad_norm 5.1344 (3.0946) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][600/625] eta 0:00:08 lr 0.000056 wd 0.0500 time 0.3482 (0.3550) data time 0.0009 (0.0017) model time 0.3473 (0.3523) loss 1.7309 (2.6390) grad_norm 3.5060 (3.0933) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][610/625] eta 0:00:05 lr 0.000056 wd 0.0500 time 0.3372 (0.3548) data time 0.0005 (0.0017) model time 0.3368 (0.3522) loss 2.9108 (2.6382) grad_norm 2.1893 (3.1061) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 14:54:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [275/300][620/625] eta 0:00:01 lr 0.000056 wd 0.0500 time 0.3386 (0.3546) data time 0.0004 (0.0017) model time 0.3382 (0.3519) loss 3.0998 (2.6380) grad_norm 2.3900 (inf) loss_scale 64.0000 (127.5878) mem 11224MB [2024-09-01 14:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 275 training takes 0:03:41 [2024-09-01 14:54:50 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:54:51 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:54:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.444 (0.444) Loss 0.5210 (0.5210) Acc@1 88.818 (88.818) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 14:54:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.130) Loss 0.8354 (0.6122) Acc@1 80.762 (86.825) Acc@5 95.996 (97.843) Mem 11224MB [2024-09-01 14:54:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.111) Loss 0.9233 (0.7331) Acc@1 78.320 (83.824) Acc@5 95.264 (96.636) Mem 11224MB [2024-09-01 14:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.557 Acc@5 96.641 [2024-09-01 14:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 14:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.842 (0.842) Loss 0.5034 (0.5034) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:54:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.166) Loss 0.8149 (0.5977) Acc@1 80.762 (87.052) Acc@5 95.801 (97.874) Mem 11224MB [2024-09-01 14:54:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.130) Loss 0.8960 (0.7124) Acc@1 78.857 (84.229) Acc@5 95.459 (96.724) Mem 11224MB [2024-09-01 14:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.943 Acc@5 96.747 [2024-09-01 14:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 14:54:57 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.94% [2024-09-01 14:54:57 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 14:54:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 14:54:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][0/625] eta 0:06:59 lr 0.000056 wd 0.0500 time 0.6715 (0.6715) data time 0.3583 (0.3583) model time 0.0000 (0.0000) loss 2.7124 (2.7124) grad_norm 2.9547 (2.9547) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][10/625] eta 0:03:51 lr 0.000056 wd 0.0500 time 0.3445 (0.3767) data time 0.0006 (0.0334) model time 0.0000 (0.0000) loss 2.9394 (2.7857) grad_norm 3.8005 (3.0399) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][20/625] eta 0:03:39 lr 0.000056 wd 0.0500 time 0.3477 (0.3632) data time 0.0007 (0.0179) model time 0.0000 (0.0000) loss 2.8734 (2.7686) grad_norm 4.0532 (3.2517) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][30/625] eta 0:03:33 lr 0.000056 wd 0.0500 time 0.3413 (0.3580) data time 0.0006 (0.0124) model time 0.0000 (0.0000) loss 2.7280 (2.7171) grad_norm 4.0229 (4.5182) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][40/625] eta 0:03:28 lr 0.000055 wd 0.0500 time 0.3421 (0.3569) data time 0.0009 (0.0095) model time 0.0000 (0.0000) loss 2.9953 (2.6708) grad_norm 2.3137 (4.0439) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][50/625] eta 0:03:28 lr 0.000055 wd 0.0500 time 0.3463 (0.3624) data time 0.0009 (0.0078) model time 0.0000 (0.0000) loss 2.9042 (2.6971) grad_norm 3.7306 (3.8084) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][60/625] eta 0:03:25 lr 0.000055 wd 0.0500 time 0.3436 (0.3628) data time 0.0006 (0.0067) model time 0.3430 (0.3644) loss 1.6445 (2.6014) grad_norm 2.2489 (3.5744) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][70/625] eta 0:03:20 lr 0.000055 wd 0.0500 time 0.3501 (0.3604) data time 0.0009 (0.0059) model time 0.3492 (0.3546) loss 2.7121 (2.5695) grad_norm 1.9457 (3.4669) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][80/625] eta 0:03:15 lr 0.000055 wd 0.0500 time 0.3441 (0.3588) data time 0.0007 (0.0053) model time 0.3434 (0.3520) loss 3.0921 (2.5740) grad_norm 3.2666 (3.4416) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][90/625] eta 0:03:11 lr 0.000055 wd 0.0500 time 0.3512 (0.3576) data time 0.0008 (0.0048) model time 0.3505 (0.3507) loss 2.0559 (2.5704) grad_norm 2.5001 (3.8768) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][100/625] eta 0:03:07 lr 0.000055 wd 0.0500 time 0.3498 (0.3568) data time 0.0008 (0.0044) model time 0.3490 (0.3502) loss 2.8020 (2.5780) grad_norm 2.4618 (3.7782) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][110/625] eta 0:03:03 lr 0.000055 wd 0.0500 time 0.3484 (0.3560) data time 0.0008 (0.0041) model time 0.3476 (0.3497) loss 2.3816 (2.5922) grad_norm 3.5839 (3.6902) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][120/625] eta 0:02:59 lr 0.000055 wd 0.0500 time 0.3511 (0.3555) data time 0.0007 (0.0038) model time 0.3505 (0.3496) loss 1.9979 (2.6139) grad_norm 2.3972 (3.6470) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][130/625] eta 0:02:55 lr 0.000055 wd 0.0500 time 0.3492 (0.3549) data time 0.0009 (0.0036) model time 0.3482 (0.3493) loss 3.0865 (2.6125) grad_norm 4.6382 (3.6208) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][140/625] eta 0:02:51 lr 0.000055 wd 0.0500 time 0.3418 (0.3544) data time 0.0010 (0.0034) model time 0.3408 (0.3490) loss 2.2383 (2.6081) grad_norm 2.5511 (3.6252) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][150/625] eta 0:02:48 lr 0.000055 wd 0.0500 time 0.3438 (0.3541) data time 0.0006 (0.0032) model time 0.3432 (0.3490) loss 2.3796 (2.6085) grad_norm 2.7006 (3.8133) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][160/625] eta 0:02:44 lr 0.000055 wd 0.0500 time 0.3528 (0.3538) data time 0.0008 (0.0031) model time 0.3520 (0.3489) loss 2.0675 (2.6004) grad_norm 3.2518 (3.7555) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:55:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][170/625] eta 0:02:41 lr 0.000055 wd 0.0500 time 0.3470 (0.3560) data time 0.0009 (0.0030) model time 0.3461 (0.3524) loss 3.1684 (2.5879) grad_norm 3.1874 (3.6973) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][180/625] eta 0:02:38 lr 0.000055 wd 0.0500 time 0.3474 (0.3567) data time 0.0006 (0.0028) model time 0.3468 (0.3536) loss 2.0560 (2.5996) grad_norm 3.5257 (3.6418) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][190/625] eta 0:02:34 lr 0.000055 wd 0.0500 time 0.3544 (0.3563) data time 0.0008 (0.0027) model time 0.3536 (0.3532) loss 2.7800 (2.6069) grad_norm 2.4190 (3.6020) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][200/625] eta 0:02:31 lr 0.000055 wd 0.0500 time 0.3549 (0.3559) data time 0.0006 (0.0026) model time 0.3542 (0.3528) loss 2.1968 (2.5981) grad_norm 2.3248 (3.5784) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][210/625] eta 0:02:27 lr 0.000055 wd 0.0500 time 0.3460 (0.3556) data time 0.0007 (0.0026) model time 0.3452 (0.3526) loss 2.7498 (2.6099) grad_norm 2.6431 (3.5496) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][220/625] eta 0:02:23 lr 0.000055 wd 0.0500 time 0.3458 (0.3553) data time 0.0009 (0.0025) model time 0.3449 (0.3524) loss 2.8625 (2.6110) grad_norm 3.4165 (3.5183) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][230/625] eta 0:02:20 lr 0.000055 wd 0.0500 time 0.3133 (0.3557) data time 0.0007 (0.0024) model time 0.3126 (0.3529) loss 3.1386 (2.6105) grad_norm 5.1192 (3.5147) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][240/625] eta 0:02:16 lr 0.000055 wd 0.0500 time 0.3578 (0.3555) data time 0.0007 (0.0023) model time 0.3571 (0.3528) loss 2.3547 (2.6133) grad_norm 2.6797 (3.4787) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][250/625] eta 0:02:13 lr 0.000055 wd 0.0500 time 0.3541 (0.3553) data time 0.0009 (0.0023) model time 0.3532 (0.3526) loss 2.4556 (2.6143) grad_norm 3.2679 (3.5095) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][260/625] eta 0:02:09 lr 0.000054 wd 0.0500 time 0.3520 (0.3550) data time 0.0006 (0.0022) model time 0.3513 (0.3523) loss 1.9823 (2.6082) grad_norm 4.3938 (3.4916) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][270/625] eta 0:02:05 lr 0.000054 wd 0.0500 time 0.3538 (0.3547) data time 0.0006 (0.0022) model time 0.3532 (0.3521) loss 2.5099 (2.6022) grad_norm 3.3643 (3.5233) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][280/625] eta 0:02:02 lr 0.000054 wd 0.0500 time 0.3548 (0.3546) data time 0.0007 (0.0021) model time 0.3541 (0.3520) loss 2.1000 (2.6004) grad_norm 4.3229 (3.5116) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][290/625] eta 0:01:58 lr 0.000054 wd 0.0500 time 0.3520 (0.3544) data time 0.0007 (0.0021) model time 0.3513 (0.3518) loss 2.9385 (2.6042) grad_norm 3.5432 (3.4894) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][300/625] eta 0:01:55 lr 0.000054 wd 0.0500 time 0.3441 (0.3542) data time 0.0010 (0.0020) model time 0.3431 (0.3516) loss 2.7228 (2.6083) grad_norm 2.7120 (3.4626) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][310/625] eta 0:01:51 lr 0.000054 wd 0.0500 time 0.3461 (0.3539) data time 0.0007 (0.0020) model time 0.3454 (0.3514) loss 2.1225 (2.6024) grad_norm 2.4587 (3.4258) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][320/625] eta 0:01:47 lr 0.000054 wd 0.0500 time 0.3464 (0.3537) data time 0.0009 (0.0020) model time 0.3455 (0.3513) loss 2.3665 (2.6051) grad_norm 3.6701 (3.4134) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][330/625] eta 0:01:44 lr 0.000054 wd 0.0500 time 0.3491 (0.3536) data time 0.0009 (0.0019) model time 0.3482 (0.3512) loss 2.2192 (2.5988) grad_norm 3.4878 (3.3959) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:56:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][340/625] eta 0:01:40 lr 0.000054 wd 0.0500 time 0.3419 (0.3535) data time 0.0007 (0.0019) model time 0.3412 (0.3510) loss 2.2636 (2.5936) grad_norm 2.6486 (3.3896) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][350/625] eta 0:01:37 lr 0.000054 wd 0.0500 time 0.3087 (0.3539) data time 0.0008 (0.0019) model time 0.3079 (0.3516) loss 3.0938 (2.5985) grad_norm 2.5758 (3.3776) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][360/625] eta 0:01:33 lr 0.000054 wd 0.0500 time 0.3522 (0.3538) data time 0.0008 (0.0018) model time 0.3514 (0.3515) loss 2.2371 (2.5949) grad_norm 2.8026 (3.4985) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][370/625] eta 0:01:30 lr 0.000054 wd 0.0500 time 0.3451 (0.3536) data time 0.0009 (0.0018) model time 0.3442 (0.3514) loss 2.7440 (2.5916) grad_norm 4.6398 (3.4967) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][380/625] eta 0:01:26 lr 0.000054 wd 0.0500 time 0.3441 (0.3546) data time 0.0009 (0.0018) model time 0.3432 (0.3525) loss 2.8161 (2.5854) grad_norm 2.0440 (3.4700) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][390/625] eta 0:01:23 lr 0.000054 wd 0.0500 time 0.3437 (0.3549) data time 0.0009 (0.0018) model time 0.3428 (0.3529) loss 3.0709 (2.5851) grad_norm 2.3849 (3.4491) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][400/625] eta 0:01:19 lr 0.000054 wd 0.0500 time 0.3481 (0.3547) data time 0.0008 (0.0017) model time 0.3473 (0.3527) loss 2.1941 (2.5760) grad_norm 2.5686 (3.4422) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][410/625] eta 0:01:16 lr 0.000054 wd 0.0500 time 0.3446 (0.3545) data time 0.0007 (0.0017) model time 0.3439 (0.3525) loss 3.0369 (2.5785) grad_norm 2.8022 (3.4294) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][420/625] eta 0:01:12 lr 0.000054 wd 0.0500 time 0.3471 (0.3544) data time 0.0009 (0.0017) model time 0.3462 (0.3524) loss 2.7796 (2.5820) grad_norm 2.6219 (3.4083) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][430/625] eta 0:01:09 lr 0.000054 wd 0.0500 time 0.3454 (0.3543) data time 0.0009 (0.0017) model time 0.3445 (0.3523) loss 2.7314 (2.5804) grad_norm 2.7640 (3.4007) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][440/625] eta 0:01:05 lr 0.000054 wd 0.0500 time 0.3475 (0.3541) data time 0.0008 (0.0017) model time 0.3467 (0.3522) loss 3.0822 (2.5835) grad_norm 3.7067 (3.3885) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][450/625] eta 0:01:01 lr 0.000054 wd 0.0500 time 0.3503 (0.3540) data time 0.0006 (0.0017) model time 0.3497 (0.3521) loss 3.1948 (2.5894) grad_norm 3.9408 (3.3765) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][460/625] eta 0:00:58 lr 0.000054 wd 0.0500 time 0.3502 (0.3539) data time 0.0009 (0.0016) model time 0.3493 (0.3520) loss 2.4909 (2.5896) grad_norm 2.5440 (3.3635) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][470/625] eta 0:00:54 lr 0.000053 wd 0.0500 time 0.3480 (0.3538) data time 0.0007 (0.0016) model time 0.3473 (0.3519) loss 2.0978 (2.5919) grad_norm 2.1893 (3.3448) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][480/625] eta 0:00:51 lr 0.000053 wd 0.0500 time 0.3440 (0.3537) data time 0.0007 (0.0016) model time 0.3434 (0.3518) loss 1.7143 (2.5869) grad_norm 2.0916 (3.3261) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][490/625] eta 0:00:47 lr 0.000053 wd 0.0500 time 0.3450 (0.3536) data time 0.0007 (0.0016) model time 0.3443 (0.3517) loss 2.6080 (2.5908) grad_norm 2.6074 (3.3333) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][500/625] eta 0:00:44 lr 0.000053 wd 0.0500 time 0.3497 (0.3535) data time 0.0010 (0.0016) model time 0.3487 (0.3517) loss 2.5971 (2.5923) grad_norm 4.2723 (3.3364) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:57:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][510/625] eta 0:00:40 lr 0.000053 wd 0.0500 time 0.3448 (0.3534) data time 0.0008 (0.0016) model time 0.3440 (0.3516) loss 2.0609 (2.5934) grad_norm 3.6569 (3.3482) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][520/625] eta 0:00:37 lr 0.000053 wd 0.0500 time 0.3442 (0.3533) data time 0.0007 (0.0015) model time 0.3435 (0.3515) loss 3.0468 (2.5941) grad_norm 2.2479 (3.3414) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][530/625] eta 0:00:33 lr 0.000053 wd 0.0500 time 0.3496 (0.3532) data time 0.0009 (0.0015) model time 0.3487 (0.3514) loss 2.8074 (2.5945) grad_norm 3.3322 (3.3483) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][540/625] eta 0:00:30 lr 0.000053 wd 0.0500 time 0.3452 (0.3531) data time 0.0009 (0.0015) model time 0.3443 (0.3513) loss 2.9582 (2.5947) grad_norm 3.2964 (3.3374) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][550/625] eta 0:00:26 lr 0.000053 wd 0.0500 time 0.3428 (0.3530) data time 0.0007 (0.0015) model time 0.3421 (0.3512) loss 3.2091 (2.5992) grad_norm 3.1678 (3.3297) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][560/625] eta 0:00:22 lr 0.000053 wd 0.0500 time 0.3462 (0.3536) data time 0.0008 (0.0015) model time 0.3455 (0.3518) loss 2.8065 (2.6009) grad_norm 2.1982 (3.3190) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][570/625] eta 0:00:19 lr 0.000053 wd 0.0500 time 0.3464 (0.3542) data time 0.0008 (0.0015) model time 0.3455 (0.3525) loss 2.7194 (2.6020) grad_norm 3.4161 (3.3154) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][580/625] eta 0:00:15 lr 0.000053 wd 0.0500 time 0.3470 (0.3540) data time 0.0007 (0.0015) model time 0.3463 (0.3524) loss 3.2164 (2.6027) grad_norm 2.8638 (3.3117) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][590/625] eta 0:00:12 lr 0.000053 wd 0.0500 time 0.3490 (0.3539) data time 0.0009 (0.0015) model time 0.3481 (0.3523) loss 2.8575 (2.6008) grad_norm 3.6212 (3.3104) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][600/625] eta 0:00:08 lr 0.000053 wd 0.0500 time 0.3409 (0.3538) data time 0.0010 (0.0015) model time 0.3400 (0.3522) loss 3.1892 (2.6029) grad_norm 2.2040 (3.2963) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][610/625] eta 0:00:05 lr 0.000053 wd 0.0500 time 0.3416 (0.3537) data time 0.0006 (0.0014) model time 0.3410 (0.3520) loss 2.6070 (2.6031) grad_norm 4.6585 (3.2941) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [276/300][620/625] eta 0:00:01 lr 0.000053 wd 0.0500 time 0.3375 (0.3535) data time 0.0006 (0.0014) model time 0.3369 (0.3518) loss 2.8029 (2.6010) grad_norm 2.5730 (3.2832) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:39 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 276 training takes 0:03:40 [2024-09-01 14:58:39 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 14:58:40 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 14:58:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.433 (0.433) Loss 0.5117 (0.5117) Acc@1 89.014 (89.014) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 14:58:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.092 (0.128) Loss 0.8403 (0.6091) Acc@1 80.469 (86.896) Acc@5 95.752 (97.820) Mem 11224MB [2024-09-01 14:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.110) Loss 0.9121 (0.7286) Acc@1 79.150 (83.910) Acc@5 95.264 (96.594) Mem 11224MB [2024-09-01 14:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.641 Acc@5 96.617 [2024-09-01 14:58:42 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.6% [2024-09-01 14:58:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.782 (0.782) Loss 0.5039 (0.5039) Acc@1 89.258 (89.258) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 14:58:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.164) Loss 0.8154 (0.5978) Acc@1 80.615 (87.056) Acc@5 95.801 (97.887) Mem 11224MB [2024-09-01 14:58:45 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.129) Loss 0.8955 (0.7128) Acc@1 78.662 (84.219) Acc@5 95.508 (96.733) Mem 11224MB [2024-09-01 14:58:45 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.929 Acc@5 96.759 [2024-09-01 14:58:45 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 14:58:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][0/625] eta 0:12:04 lr 0.000053 wd 0.0500 time 1.1584 (1.1584) data time 0.7958 (0.7958) model time 0.0000 (0.0000) loss 3.0583 (3.0583) grad_norm 2.9733 (2.9733) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][10/625] eta 0:04:20 lr 0.000053 wd 0.0500 time 0.3504 (0.4230) data time 0.0008 (0.0733) model time 0.0000 (0.0000) loss 2.5497 (2.5676) grad_norm 2.5046 (2.5276) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][20/625] eta 0:03:55 lr 0.000053 wd 0.0500 time 0.3469 (0.3885) data time 0.0007 (0.0387) model time 0.0000 (0.0000) loss 1.9540 (2.5281) grad_norm 3.1775 (3.8757) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:58:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][30/625] eta 0:03:43 lr 0.000053 wd 0.0500 time 0.3465 (0.3753) data time 0.0008 (0.0265) model time 0.0000 (0.0000) loss 2.8899 (2.5501) grad_norm 2.4596 (3.7053) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][40/625] eta 0:03:35 lr 0.000053 wd 0.0500 time 0.3458 (0.3686) data time 0.0007 (0.0203) model time 0.0000 (0.0000) loss 3.0831 (2.5340) grad_norm 4.6450 (5.2415) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][50/625] eta 0:03:29 lr 0.000053 wd 0.0500 time 0.3540 (0.3648) data time 0.0010 (0.0165) model time 0.0000 (0.0000) loss 2.9834 (2.5436) grad_norm 3.4876 (4.7372) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][60/625] eta 0:03:24 lr 0.000053 wd 0.0500 time 0.3432 (0.3619) data time 0.0007 (0.0139) model time 0.3425 (0.3464) loss 2.9895 (2.5699) grad_norm 3.5403 (4.5425) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][70/625] eta 0:03:19 lr 0.000052 wd 0.0500 time 0.3440 (0.3596) data time 0.0010 (0.0121) model time 0.3430 (0.3455) loss 3.2652 (2.6015) grad_norm 2.0694 (4.3530) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][80/625] eta 0:03:16 lr 0.000052 wd 0.0500 time 0.4983 (0.3598) data time 0.0008 (0.0107) model time 0.4975 (0.3505) loss 2.5923 (2.6027) grad_norm 2.4454 (4.2445) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][90/625] eta 0:03:12 lr 0.000052 wd 0.0500 time 0.3435 (0.3605) data time 0.0006 (0.0096) model time 0.3429 (0.3541) loss 1.8517 (2.5846) grad_norm 3.3423 (4.1880) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][100/625] eta 0:03:10 lr 0.000052 wd 0.0500 time 0.3464 (0.3626) data time 0.0008 (0.0088) model time 0.3455 (0.3595) loss 2.6732 (2.5834) grad_norm 2.5705 (4.0508) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][110/625] eta 0:03:05 lr 0.000052 wd 0.0500 time 0.3390 (0.3611) data time 0.0009 (0.0081) model time 0.3381 (0.3572) loss 2.7316 (2.6018) grad_norm 2.3197 (3.9420) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][120/625] eta 0:03:01 lr 0.000052 wd 0.0500 time 0.3424 (0.3600) data time 0.0008 (0.0075) model time 0.3417 (0.3556) loss 3.2075 (2.6158) grad_norm 2.8915 (3.8353) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][130/625] eta 0:02:57 lr 0.000052 wd 0.0500 time 0.3422 (0.3589) data time 0.0008 (0.0070) model time 0.3415 (0.3543) loss 2.8054 (2.6065) grad_norm 2.9238 (3.7638) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][140/625] eta 0:02:53 lr 0.000052 wd 0.0500 time 0.3485 (0.3580) data time 0.0007 (0.0065) model time 0.3478 (0.3534) loss 2.6457 (2.6084) grad_norm 3.9268 (3.6872) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][150/625] eta 0:02:49 lr 0.000052 wd 0.0500 time 0.3422 (0.3573) data time 0.0006 (0.0061) model time 0.3416 (0.3526) loss 2.5957 (2.6021) grad_norm 1.9929 (3.6357) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][160/625] eta 0:02:45 lr 0.000052 wd 0.0500 time 0.3402 (0.3565) data time 0.0008 (0.0058) model time 0.3394 (0.3518) loss 3.0505 (2.6083) grad_norm 2.6983 (3.6321) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][170/625] eta 0:02:41 lr 0.000052 wd 0.0500 time 0.3479 (0.3559) data time 0.0008 (0.0055) model time 0.3471 (0.3513) loss 2.7235 (2.6168) grad_norm 5.9134 (3.7465) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][180/625] eta 0:02:38 lr 0.000052 wd 0.0500 time 0.3503 (0.3553) data time 0.0009 (0.0053) model time 0.3493 (0.3507) loss 2.3902 (2.6081) grad_norm 2.5371 (3.7101) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][190/625] eta 0:02:34 lr 0.000052 wd 0.0500 time 0.3476 (0.3549) data time 0.0008 (0.0050) model time 0.3468 (0.3504) loss 2.9113 (2.6008) grad_norm 2.5691 (3.6910) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 14:59:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][200/625] eta 0:02:31 lr 0.000052 wd 0.0500 time 0.5397 (0.3554) data time 0.0007 (0.0048) model time 0.5390 (0.3514) loss 2.5107 (2.6064) grad_norm 2.7484 (3.6734) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][210/625] eta 0:02:27 lr 0.000052 wd 0.0500 time 0.3434 (0.3559) data time 0.0006 (0.0046) model time 0.3428 (0.3523) loss 1.9767 (2.5982) grad_norm 2.9126 (3.6298) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][220/625] eta 0:02:24 lr 0.000052 wd 0.0500 time 0.3490 (0.3573) data time 0.0009 (0.0045) model time 0.3481 (0.3542) loss 2.7525 (2.5981) grad_norm 2.0233 (3.5873) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][230/625] eta 0:02:20 lr 0.000052 wd 0.0500 time 0.3478 (0.3569) data time 0.0010 (0.0043) model time 0.3468 (0.3538) loss 2.8339 (2.6026) grad_norm 3.4363 (3.5611) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][240/625] eta 0:02:17 lr 0.000052 wd 0.0500 time 0.3502 (0.3565) data time 0.0007 (0.0042) model time 0.3495 (0.3535) loss 2.9872 (2.6082) grad_norm 4.3776 (3.5412) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][250/625] eta 0:02:13 lr 0.000052 wd 0.0500 time 0.3501 (0.3562) data time 0.0006 (0.0040) model time 0.3494 (0.3531) loss 1.6384 (2.6008) grad_norm 2.9520 (3.5284) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][260/625] eta 0:02:09 lr 0.000052 wd 0.0500 time 0.3465 (0.3558) data time 0.0009 (0.0039) model time 0.3456 (0.3528) loss 2.2650 (2.5986) grad_norm 2.2148 (3.5107) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][270/625] eta 0:02:06 lr 0.000052 wd 0.0500 time 0.3461 (0.3555) data time 0.0007 (0.0038) model time 0.3454 (0.3525) loss 2.4887 (2.5995) grad_norm 2.7238 (3.6871) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][280/625] eta 0:02:02 lr 0.000052 wd 0.0500 time 0.3508 (0.3552) data time 0.0009 (0.0037) model time 0.3499 (0.3523) loss 2.8409 (2.6079) grad_norm 3.5006 (4.0558) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][290/625] eta 0:01:58 lr 0.000051 wd 0.0500 time 0.3437 (0.3549) data time 0.0007 (0.0036) model time 0.3430 (0.3520) loss 1.7425 (2.6022) grad_norm 2.8715 (4.0043) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][300/625] eta 0:01:55 lr 0.000051 wd 0.0500 time 0.3449 (0.3547) data time 0.0007 (0.0035) model time 0.3442 (0.3518) loss 2.8241 (2.6026) grad_norm 3.6449 (3.9697) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][310/625] eta 0:01:51 lr 0.000051 wd 0.0500 time 0.3462 (0.3544) data time 0.0008 (0.0034) model time 0.3454 (0.3515) loss 2.8812 (2.6067) grad_norm 2.5434 (3.9645) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][320/625] eta 0:01:47 lr 0.000051 wd 0.0500 time 0.3465 (0.3541) data time 0.0008 (0.0033) model time 0.3457 (0.3512) loss 2.4375 (2.6113) grad_norm 2.1268 (3.9739) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][330/625] eta 0:01:44 lr 0.000051 wd 0.0500 time 0.3476 (0.3539) data time 0.0007 (0.0033) model time 0.3469 (0.3511) loss 2.2387 (2.6075) grad_norm 2.7167 (3.9400) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][340/625] eta 0:01:40 lr 0.000051 wd 0.0500 time 0.3441 (0.3537) data time 0.0007 (0.0032) model time 0.3435 (0.3509) loss 2.7513 (2.6120) grad_norm 2.0767 (3.9090) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][350/625] eta 0:01:37 lr 0.000051 wd 0.0500 time 0.3465 (0.3535) data time 0.0009 (0.0031) model time 0.3456 (0.3507) loss 2.6115 (2.6246) grad_norm 3.5898 (3.8762) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][360/625] eta 0:01:33 lr 0.000051 wd 0.0500 time 0.3479 (0.3533) data time 0.0009 (0.0031) model time 0.3469 (0.3506) loss 2.1349 (2.6264) grad_norm 1.8270 (3.8497) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:00:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][370/625] eta 0:01:30 lr 0.000051 wd 0.0500 time 0.3452 (0.3531) data time 0.0009 (0.0030) model time 0.3443 (0.3504) loss 2.6371 (2.6277) grad_norm 2.5101 (3.8183) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][380/625] eta 0:01:26 lr 0.000051 wd 0.0500 time 0.3459 (0.3530) data time 0.0009 (0.0029) model time 0.3450 (0.3503) loss 2.2330 (2.6241) grad_norm 2.2668 (3.7872) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][390/625] eta 0:01:22 lr 0.000051 wd 0.0500 time 0.3476 (0.3528) data time 0.0006 (0.0029) model time 0.3470 (0.3502) loss 2.9119 (2.6263) grad_norm 2.1754 (3.7706) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][400/625] eta 0:01:19 lr 0.000051 wd 0.0500 time 0.3417 (0.3527) data time 0.0010 (0.0028) model time 0.3407 (0.3501) loss 2.7112 (2.6236) grad_norm 4.4021 (3.7606) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][410/625] eta 0:01:15 lr 0.000051 wd 0.0500 time 0.3465 (0.3530) data time 0.0009 (0.0028) model time 0.3455 (0.3505) loss 2.8957 (2.6242) grad_norm 2.2742 (3.7634) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][420/625] eta 0:01:12 lr 0.000051 wd 0.0500 time 0.3411 (0.3534) data time 0.0009 (0.0028) model time 0.3402 (0.3509) loss 2.9572 (2.6248) grad_norm 4.3088 (3.7456) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][430/625] eta 0:01:09 lr 0.000051 wd 0.0500 time 0.3431 (0.3541) data time 0.0007 (0.0027) model time 0.3425 (0.3519) loss 2.9496 (2.6309) grad_norm 2.6441 (3.7282) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][440/625] eta 0:01:05 lr 0.000051 wd 0.0500 time 0.3487 (0.3539) data time 0.0009 (0.0027) model time 0.3478 (0.3517) loss 1.8186 (2.6299) grad_norm 1.9092 (3.7002) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][450/625] eta 0:01:01 lr 0.000051 wd 0.0500 time 0.3418 (0.3538) data time 0.0007 (0.0026) model time 0.3411 (0.3515) loss 2.4261 (2.6331) grad_norm 2.0167 (3.7104) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][460/625] eta 0:00:58 lr 0.000051 wd 0.0500 time 0.3473 (0.3536) data time 0.0008 (0.0026) model time 0.3465 (0.3514) loss 1.9113 (2.6340) grad_norm 3.3982 (3.7285) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][470/625] eta 0:00:54 lr 0.000051 wd 0.0500 time 0.3496 (0.3535) data time 0.0007 (0.0026) model time 0.3489 (0.3513) loss 1.5844 (2.6336) grad_norm 3.1830 (3.7030) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][480/625] eta 0:00:51 lr 0.000051 wd 0.0500 time 0.3562 (0.3534) data time 0.0006 (0.0025) model time 0.3556 (0.3511) loss 2.5707 (2.6358) grad_norm 2.7075 (3.6952) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][490/625] eta 0:00:47 lr 0.000051 wd 0.0500 time 0.3474 (0.3532) data time 0.0009 (0.0025) model time 0.3465 (0.3510) loss 2.9236 (2.6368) grad_norm 3.1090 (3.6944) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][500/625] eta 0:00:44 lr 0.000051 wd 0.0500 time 0.3502 (0.3531) data time 0.0006 (0.0024) model time 0.3495 (0.3510) loss 2.7132 (2.6406) grad_norm 2.9738 (3.6745) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][510/625] eta 0:00:40 lr 0.000051 wd 0.0500 time 0.3485 (0.3530) data time 0.0010 (0.0024) model time 0.3475 (0.3509) loss 2.1780 (2.6405) grad_norm 2.4620 (3.7123) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][520/625] eta 0:00:37 lr 0.000050 wd 0.0500 time 0.3473 (0.3529) data time 0.0009 (0.0024) model time 0.3464 (0.3508) loss 2.1110 (2.6437) grad_norm 3.3201 (3.6917) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][530/625] eta 0:00:33 lr 0.000050 wd 0.0500 time 0.3526 (0.3528) data time 0.0008 (0.0024) model time 0.3519 (0.3507) loss 2.8347 (2.6368) grad_norm 3.0592 (3.6749) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:01:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][540/625] eta 0:00:29 lr 0.000050 wd 0.0500 time 0.3491 (0.3527) data time 0.0006 (0.0023) model time 0.3485 (0.3506) loss 2.7100 (2.6391) grad_norm 2.2836 (3.6548) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][550/625] eta 0:00:26 lr 0.000050 wd 0.0500 time 0.3433 (0.3527) data time 0.0007 (0.0023) model time 0.3426 (0.3506) loss 2.9391 (2.6343) grad_norm 2.0523 (3.6437) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][560/625] eta 0:00:22 lr 0.000050 wd 0.0500 time 0.3516 (0.3526) data time 0.0006 (0.0023) model time 0.3509 (0.3505) loss 2.1266 (2.6354) grad_norm 2.2598 (3.6355) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][570/625] eta 0:00:19 lr 0.000050 wd 0.0500 time 0.3446 (0.3525) data time 0.0009 (0.0023) model time 0.3437 (0.3504) loss 2.5520 (2.6335) grad_norm 1.8310 (3.6558) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][580/625] eta 0:00:15 lr 0.000050 wd 0.0500 time 0.3502 (0.3524) data time 0.0007 (0.0022) model time 0.3495 (0.3503) loss 1.7841 (2.6284) grad_norm 1.9060 (3.6407) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][590/625] eta 0:00:12 lr 0.000050 wd 0.0500 time 0.3448 (0.3523) data time 0.0007 (0.0022) model time 0.3441 (0.3502) loss 1.8716 (2.6278) grad_norm 3.3916 (3.6517) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][600/625] eta 0:00:08 lr 0.000050 wd 0.0500 time 0.3459 (0.3531) data time 0.0006 (0.0022) model time 0.3453 (0.3512) loss 1.4558 (2.6270) grad_norm 8.9255 (3.6446) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][610/625] eta 0:00:05 lr 0.000050 wd 0.0500 time 0.3377 (0.3539) data time 0.0004 (0.0022) model time 0.3373 (0.3521) loss 2.0119 (2.6270) grad_norm 3.1867 (3.6294) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [277/300][620/625] eta 0:00:01 lr 0.000050 wd 0.0500 time 0.3454 (0.3537) data time 0.0004 (0.0021) model time 0.3450 (0.3519) loss 1.9474 (2.6265) grad_norm 4.0442 (3.6103) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 277 training takes 0:03:41 [2024-09-01 15:02:26 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:02:27 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:02:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.461 (0.461) Loss 0.5142 (0.5142) Acc@1 88.721 (88.721) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 15:02:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.132) Loss 0.8105 (0.6067) Acc@1 81.104 (86.879) Acc@5 96.191 (97.834) Mem 11224MB [2024-09-01 15:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.113) Loss 0.9004 (0.7262) Acc@1 78.906 (83.980) Acc@5 95.312 (96.612) Mem 11224MB [2024-09-01 15:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.673 Acc@5 96.639 [2024-09-01 15:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 15:02:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.767 (0.767) Loss 0.5039 (0.5039) Acc@1 89.209 (89.209) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:02:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.164) Loss 0.8159 (0.5983) Acc@1 80.762 (87.043) Acc@5 95.801 (97.878) Mem 11224MB [2024-09-01 15:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.129) Loss 0.8965 (0.7134) Acc@1 78.613 (84.198) Acc@5 95.459 (96.731) Mem 11224MB [2024-09-01 15:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.905 Acc@5 96.751 [2024-09-01 15:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:02:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][0/625] eta 0:11:44 lr 0.000050 wd 0.0500 time 1.1279 (1.1279) data time 0.5141 (0.5141) model time 0.0000 (0.0000) loss 2.5918 (2.5918) grad_norm 2.8166 (2.8166) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][10/625] eta 0:04:17 lr 0.000050 wd 0.0500 time 0.3397 (0.4185) data time 0.0007 (0.0475) model time 0.0000 (0.0000) loss 1.9442 (2.5143) grad_norm 3.1830 (2.7550) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][20/625] eta 0:03:52 lr 0.000050 wd 0.0500 time 0.3442 (0.3845) data time 0.0008 (0.0253) model time 0.0000 (0.0000) loss 2.0415 (2.5099) grad_norm 1.9968 (3.0769) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][30/625] eta 0:03:41 lr 0.000050 wd 0.0500 time 0.3552 (0.3723) data time 0.0007 (0.0174) model time 0.0000 (0.0000) loss 1.9327 (2.4559) grad_norm 2.7481 (2.9704) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][40/625] eta 0:03:34 lr 0.000050 wd 0.0500 time 0.3469 (0.3664) data time 0.0009 (0.0134) model time 0.0000 (0.0000) loss 2.8774 (2.5131) grad_norm 2.3666 (2.9211) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][50/625] eta 0:03:28 lr 0.000050 wd 0.0500 time 0.3477 (0.3629) data time 0.0009 (0.0109) model time 0.0000 (0.0000) loss 2.0662 (2.4670) grad_norm 2.6935 (3.0392) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][60/625] eta 0:03:23 lr 0.000050 wd 0.0500 time 0.3452 (0.3604) data time 0.0007 (0.0093) model time 0.3445 (0.3471) loss 3.0998 (2.5225) grad_norm 2.9313 (3.1599) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][70/625] eta 0:03:19 lr 0.000050 wd 0.0500 time 0.3453 (0.3586) data time 0.0006 (0.0081) model time 0.3447 (0.3469) loss 2.6034 (2.5127) grad_norm 2.1420 (4.1185) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][80/625] eta 0:03:14 lr 0.000050 wd 0.0500 time 0.3506 (0.3574) data time 0.0009 (0.0072) model time 0.3497 (0.3473) loss 2.4814 (2.5280) grad_norm 2.3922 (4.0338) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][90/625] eta 0:03:10 lr 0.000050 wd 0.0500 time 0.3583 (0.3565) data time 0.0009 (0.0065) model time 0.3575 (0.3476) loss 2.6401 (2.5176) grad_norm 2.3845 (3.8676) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][100/625] eta 0:03:06 lr 0.000050 wd 0.0500 time 0.3486 (0.3557) data time 0.0009 (0.0059) model time 0.3478 (0.3475) loss 2.9967 (2.5555) grad_norm 2.7660 (3.7861) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][110/625] eta 0:03:02 lr 0.000050 wd 0.0500 time 0.3459 (0.3549) data time 0.0007 (0.0055) model time 0.3452 (0.3474) loss 1.7683 (2.5558) grad_norm 2.8911 (3.6914) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][120/625] eta 0:02:58 lr 0.000049 wd 0.0500 time 0.3445 (0.3542) data time 0.0006 (0.0051) model time 0.3439 (0.3470) loss 2.1669 (2.5468) grad_norm 2.4574 (3.7434) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][130/625] eta 0:02:55 lr 0.000049 wd 0.0500 time 0.5860 (0.3555) data time 0.0009 (0.0048) model time 0.5851 (0.3499) loss 2.9110 (2.5559) grad_norm 1.9149 (3.6866) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][140/625] eta 0:02:53 lr 0.000049 wd 0.0500 time 0.3451 (0.3571) data time 0.0009 (0.0045) model time 0.3442 (0.3529) loss 2.8200 (2.5704) grad_norm 3.4191 (3.6154) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][150/625] eta 0:02:49 lr 0.000049 wd 0.0500 time 0.3496 (0.3575) data time 0.0008 (0.0043) model time 0.3487 (0.3540) loss 2.4286 (2.5610) grad_norm 2.3319 (3.5738) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][160/625] eta 0:02:45 lr 0.000049 wd 0.0500 time 0.3494 (0.3569) data time 0.0007 (0.0040) model time 0.3488 (0.3533) loss 2.9038 (2.5753) grad_norm 1.8148 (3.5277) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][170/625] eta 0:02:42 lr 0.000049 wd 0.0500 time 0.3443 (0.3564) data time 0.0007 (0.0039) model time 0.3435 (0.3528) loss 2.6260 (2.5732) grad_norm 2.8540 (3.4745) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][180/625] eta 0:02:38 lr 0.000049 wd 0.0500 time 0.5634 (0.3571) data time 0.0009 (0.0037) model time 0.5626 (0.3539) loss 2.1693 (2.5763) grad_norm 2.9124 (3.4190) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][190/625] eta 0:02:35 lr 0.000049 wd 0.0500 time 0.3484 (0.3573) data time 0.0006 (0.0035) model time 0.3478 (0.3544) loss 2.4320 (2.5717) grad_norm 2.7650 (3.3790) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][200/625] eta 0:02:31 lr 0.000049 wd 0.0500 time 0.3493 (0.3568) data time 0.0008 (0.0034) model time 0.3484 (0.3539) loss 2.7477 (2.5842) grad_norm 2.4476 (3.3733) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][210/625] eta 0:02:27 lr 0.000049 wd 0.0500 time 0.3399 (0.3563) data time 0.0011 (0.0033) model time 0.3388 (0.3533) loss 1.9843 (2.5792) grad_norm 1.8734 (3.3486) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][220/625] eta 0:02:24 lr 0.000049 wd 0.0500 time 0.3383 (0.3558) data time 0.0008 (0.0032) model time 0.3375 (0.3528) loss 2.8953 (2.5857) grad_norm 3.6967 (3.3262) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][230/625] eta 0:02:20 lr 0.000049 wd 0.0500 time 0.3480 (0.3554) data time 0.0006 (0.0031) model time 0.3474 (0.3525) loss 2.1643 (2.5789) grad_norm 6.3147 (3.3211) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:03:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][240/625] eta 0:02:16 lr 0.000049 wd 0.0500 time 0.3473 (0.3551) data time 0.0007 (0.0030) model time 0.3466 (0.3521) loss 1.9039 (2.5762) grad_norm 4.4905 (3.3133) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][250/625] eta 0:02:13 lr 0.000049 wd 0.0500 time 0.3112 (0.3555) data time 0.0007 (0.0029) model time 0.3104 (0.3528) loss 2.6195 (2.5672) grad_norm 2.0209 (3.2883) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][260/625] eta 0:02:10 lr 0.000049 wd 0.0500 time 0.3437 (0.3565) data time 0.0009 (0.0028) model time 0.3428 (0.3541) loss 2.6705 (2.5776) grad_norm 2.7755 (3.2732) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][270/625] eta 0:02:06 lr 0.000049 wd 0.0500 time 0.3469 (0.3569) data time 0.0010 (0.0028) model time 0.3459 (0.3546) loss 2.8119 (2.5762) grad_norm 3.2327 (3.2585) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][280/625] eta 0:02:03 lr 0.000049 wd 0.0500 time 0.3481 (0.3566) data time 0.0007 (0.0027) model time 0.3474 (0.3543) loss 2.7066 (2.5779) grad_norm 2.6106 (3.2450) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][290/625] eta 0:01:59 lr 0.000049 wd 0.0500 time 0.3487 (0.3562) data time 0.0008 (0.0026) model time 0.3479 (0.3540) loss 2.4803 (2.5783) grad_norm 2.4754 (3.2329) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][300/625] eta 0:01:55 lr 0.000049 wd 0.0500 time 0.3409 (0.3559) data time 0.0007 (0.0026) model time 0.3402 (0.3536) loss 1.6966 (2.5764) grad_norm 2.1085 (3.2057) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][310/625] eta 0:01:52 lr 0.000049 wd 0.0500 time 0.3567 (0.3556) data time 0.0006 (0.0025) model time 0.3560 (0.3534) loss 3.1276 (2.5681) grad_norm 1.9382 (3.1908) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][320/625] eta 0:01:48 lr 0.000049 wd 0.0500 time 0.3483 (0.3554) data time 0.0007 (0.0025) model time 0.3476 (0.3531) loss 2.7550 (2.5664) grad_norm 3.3364 (3.2046) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][330/625] eta 0:01:44 lr 0.000049 wd 0.0500 time 0.3412 (0.3552) data time 0.0007 (0.0024) model time 0.3404 (0.3529) loss 1.8612 (2.5669) grad_norm 4.3665 (3.2057) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][340/625] eta 0:01:41 lr 0.000049 wd 0.0500 time 0.3465 (0.3550) data time 0.0009 (0.0024) model time 0.3456 (0.3528) loss 3.1457 (2.5698) grad_norm 1.8491 (3.1918) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][350/625] eta 0:01:37 lr 0.000049 wd 0.0500 time 0.3471 (0.3548) data time 0.0008 (0.0023) model time 0.3463 (0.3525) loss 1.4972 (2.5672) grad_norm 2.5664 (3.1872) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][360/625] eta 0:01:33 lr 0.000048 wd 0.0500 time 0.3441 (0.3545) data time 0.0008 (0.0023) model time 0.3433 (0.3523) loss 2.7233 (2.5682) grad_norm 2.9595 (3.1830) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][370/625] eta 0:01:30 lr 0.000048 wd 0.0500 time 0.3423 (0.3552) data time 0.0008 (0.0022) model time 0.3414 (0.3532) loss 2.5866 (2.5725) grad_norm 22.4745 (3.2369) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][380/625] eta 0:01:26 lr 0.000048 wd 0.0500 time 0.3452 (0.3551) data time 0.0009 (0.0022) model time 0.3443 (0.3530) loss 2.3889 (2.5715) grad_norm 2.1167 (3.2295) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][390/625] eta 0:01:23 lr 0.000048 wd 0.0500 time 0.3550 (0.3549) data time 0.0007 (0.0022) model time 0.3543 (0.3528) loss 2.9035 (2.5768) grad_norm 3.2124 (3.2476) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][400/625] eta 0:01:19 lr 0.000048 wd 0.0500 time 0.3464 (0.3547) data time 0.0009 (0.0021) model time 0.3455 (0.3527) loss 2.8639 (2.5833) grad_norm 1.9945 (3.2264) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:04:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][410/625] eta 0:01:16 lr 0.000048 wd 0.0500 time 0.3430 (0.3545) data time 0.0008 (0.0021) model time 0.3422 (0.3525) loss 2.3346 (2.5850) grad_norm 2.3898 (3.2363) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][420/625] eta 0:01:12 lr 0.000048 wd 0.0500 time 0.3500 (0.3544) data time 0.0007 (0.0021) model time 0.3493 (0.3524) loss 2.2258 (2.5875) grad_norm 2.1254 (3.2284) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][430/625] eta 0:01:09 lr 0.000048 wd 0.0500 time 0.3519 (0.3543) data time 0.0006 (0.0020) model time 0.3513 (0.3523) loss 2.4263 (2.5893) grad_norm 2.5061 (3.3136) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][440/625] eta 0:01:05 lr 0.000048 wd 0.0500 time 0.3483 (0.3541) data time 0.0009 (0.0020) model time 0.3474 (0.3522) loss 2.8617 (2.5862) grad_norm 2.4788 (3.3045) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][450/625] eta 0:01:01 lr 0.000048 wd 0.0500 time 0.3488 (0.3540) data time 0.0007 (0.0020) model time 0.3482 (0.3520) loss 2.3053 (2.5835) grad_norm 2.2392 (3.2963) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][460/625] eta 0:00:58 lr 0.000048 wd 0.0500 time 0.3452 (0.3543) data time 0.0006 (0.0020) model time 0.3446 (0.3524) loss 2.4243 (2.5890) grad_norm 1.9402 (3.2903) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][470/625] eta 0:00:55 lr 0.000048 wd 0.0500 time 0.3538 (0.3550) data time 0.0006 (0.0019) model time 0.3531 (0.3532) loss 2.7665 (2.5916) grad_norm 2.8901 (3.2745) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][480/625] eta 0:00:51 lr 0.000048 wd 0.0500 time 0.3467 (0.3553) data time 0.0007 (0.0019) model time 0.3460 (0.3535) loss 3.1500 (2.5939) grad_norm 2.3574 (3.2600) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][490/625] eta 0:00:48 lr 0.000048 wd 0.0500 time 0.3456 (0.3559) data time 0.0007 (0.0019) model time 0.3449 (0.3543) loss 1.8690 (2.5883) grad_norm 2.5174 (3.2562) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][500/625] eta 0:00:44 lr 0.000048 wd 0.0500 time 0.3460 (0.3558) data time 0.0006 (0.0019) model time 0.3454 (0.3541) loss 1.4989 (2.5860) grad_norm 3.1669 (3.2409) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][510/625] eta 0:00:40 lr 0.000048 wd 0.0500 time 0.3555 (0.3556) data time 0.0006 (0.0019) model time 0.3548 (0.3540) loss 3.1232 (2.5890) grad_norm 3.2016 (3.2315) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][520/625] eta 0:00:37 lr 0.000048 wd 0.0500 time 0.3535 (0.3555) data time 0.0007 (0.0018) model time 0.3528 (0.3539) loss 3.1956 (2.5890) grad_norm 1.7766 (3.2163) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][530/625] eta 0:00:33 lr 0.000048 wd 0.0500 time 0.3431 (0.3554) data time 0.0006 (0.0018) model time 0.3425 (0.3537) loss 3.0720 (2.5920) grad_norm 4.0299 (3.2197) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][540/625] eta 0:00:30 lr 0.000048 wd 0.0500 time 0.3476 (0.3552) data time 0.0008 (0.0018) model time 0.3469 (0.3536) loss 3.1833 (2.5946) grad_norm 3.2408 (3.2229) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][550/625] eta 0:00:26 lr 0.000048 wd 0.0500 time 0.3488 (0.3551) data time 0.0008 (0.0018) model time 0.3480 (0.3534) loss 3.0718 (2.5995) grad_norm 1.9807 (3.2188) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][560/625] eta 0:00:23 lr 0.000048 wd 0.0500 time 0.3426 (0.3549) data time 0.0007 (0.0018) model time 0.3419 (0.3533) loss 2.0148 (2.5975) grad_norm 2.1685 (3.2093) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][570/625] eta 0:00:19 lr 0.000048 wd 0.0500 time 0.3525 (0.3548) data time 0.0006 (0.0017) model time 0.3519 (0.3531) loss 2.5460 (2.5986) grad_norm 2.3833 (3.2029) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:05:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][580/625] eta 0:00:15 lr 0.000048 wd 0.0500 time 0.3470 (0.3547) data time 0.0006 (0.0017) model time 0.3464 (0.3530) loss 2.8498 (2.5984) grad_norm 2.9106 (3.2099) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][590/625] eta 0:00:12 lr 0.000047 wd 0.0500 time 0.3458 (0.3545) data time 0.0006 (0.0017) model time 0.3452 (0.3529) loss 2.5498 (2.5994) grad_norm 3.1775 (3.2105) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][600/625] eta 0:00:08 lr 0.000047 wd 0.0500 time 0.3429 (0.3544) data time 0.0009 (0.0017) model time 0.3420 (0.3528) loss 2.4216 (2.5973) grad_norm 3.5429 (3.2061) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][610/625] eta 0:00:05 lr 0.000047 wd 0.0500 time 0.3404 (0.3543) data time 0.0006 (0.0017) model time 0.3398 (0.3527) loss 2.5495 (2.5977) grad_norm 1.9735 (3.2725) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [278/300][620/625] eta 0:00:01 lr 0.000047 wd 0.0500 time 0.3389 (0.3541) data time 0.0004 (0.0017) model time 0.3385 (0.3524) loss 3.0052 (2.5984) grad_norm 2.7847 (3.3005) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:14 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 278 training takes 0:03:41 [2024-09-01 15:06:14 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:06:16 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:06:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.451 (0.451) Loss 0.5283 (0.5283) Acc@1 89.062 (89.062) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 15:06:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.132) Loss 0.8394 (0.6135) Acc@1 80.273 (86.861) Acc@5 95.703 (97.807) Mem 11224MB [2024-09-01 15:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.9141 (0.7322) Acc@1 79.053 (83.973) Acc@5 95.312 (96.631) Mem 11224MB [2024-09-01 15:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.727 Acc@5 96.655 [2024-09-01 15:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 15:06:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.837 (0.837) Loss 0.5039 (0.5039) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:06:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.169) Loss 0.8169 (0.5985) Acc@1 80.713 (87.061) Acc@5 95.801 (97.865) Mem 11224MB [2024-09-01 15:06:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.132) Loss 0.8965 (0.7136) Acc@1 78.613 (84.222) Acc@5 95.459 (96.710) Mem 11224MB [2024-09-01 15:06:21 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.929 Acc@5 96.737 [2024-09-01 15:06:21 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:06:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][0/625] eta 0:12:00 lr 0.000047 wd 0.0500 time 1.1524 (1.1524) data time 0.5970 (0.5970) model time 0.0000 (0.0000) loss 2.4541 (2.4541) grad_norm 3.3321 (3.3321) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][10/625] eta 0:04:18 lr 0.000047 wd 0.0500 time 0.3451 (0.4199) data time 0.0008 (0.0551) model time 0.0000 (0.0000) loss 2.2760 (2.3781) grad_norm 2.4924 (2.9061) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][20/625] eta 0:03:52 lr 0.000047 wd 0.0500 time 0.3455 (0.3847) data time 0.0007 (0.0292) model time 0.0000 (0.0000) loss 2.5976 (2.4653) grad_norm 3.0996 (3.6802) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][30/625] eta 0:03:48 lr 0.000047 wd 0.0500 time 0.4989 (0.3835) data time 0.0007 (0.0200) model time 0.0000 (0.0000) loss 3.2251 (2.5377) grad_norm 3.3195 (3.5148) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][40/625] eta 0:03:47 lr 0.000047 wd 0.0500 time 0.5282 (0.3895) data time 0.0009 (0.0154) model time 0.0000 (0.0000) loss 2.8369 (2.5371) grad_norm 2.0534 (3.5574) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][50/625] eta 0:03:38 lr 0.000047 wd 0.0500 time 0.3468 (0.3808) data time 0.0009 (0.0125) model time 0.0000 (0.0000) loss 2.8552 (2.5776) grad_norm 2.5174 (3.6087) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][60/625] eta 0:03:31 lr 0.000047 wd 0.0500 time 0.3472 (0.3752) data time 0.0009 (0.0106) model time 0.3463 (0.3458) loss 2.2921 (2.5495) grad_norm 5.4507 (3.6115) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][70/625] eta 0:03:25 lr 0.000047 wd 0.0500 time 0.3443 (0.3711) data time 0.0008 (0.0092) model time 0.3435 (0.3454) loss 2.8962 (2.5773) grad_norm 2.8126 (3.6754) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][80/625] eta 0:03:20 lr 0.000047 wd 0.0500 time 0.3438 (0.3680) data time 0.0009 (0.0082) model time 0.3429 (0.3452) loss 2.4439 (2.5776) grad_norm 7.1019 (3.6135) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][90/625] eta 0:03:15 lr 0.000047 wd 0.0500 time 0.3471 (0.3655) data time 0.0010 (0.0074) model time 0.3461 (0.3451) loss 2.9840 (2.5688) grad_norm 2.6983 (3.5302) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:06:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][100/625] eta 0:03:10 lr 0.000047 wd 0.0500 time 0.3450 (0.3634) data time 0.0009 (0.0067) model time 0.3442 (0.3448) loss 2.6929 (2.5892) grad_norm 1.5749 (3.4857) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:07:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][110/625] eta 0:03:06 lr 0.000047 wd 0.0500 time 0.3499 (0.3620) data time 0.0008 (0.0062) model time 0.3491 (0.3451) loss 2.6439 (2.5933) grad_norm 2.3973 (3.4346) loss_scale 64.0000 (64.0000) mem 11224MB [2024-09-01 15:07:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][120/625] eta 0:03:02 lr 0.000047 wd 0.0500 time 0.3400 (0.3620) data time 0.0009 (0.0058) model time 0.3392 (0.3473) loss 2.3228 (2.6051) grad_norm 3.4399 (3.4329) loss_scale 128.0000 (66.1157) mem 11224MB [2024-09-01 15:07:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][130/625] eta 0:02:58 lr 0.000047 wd 0.0500 time 0.3434 (0.3607) data time 0.0007 (0.0054) model time 0.3427 (0.3470) loss 3.0120 (2.6094) grad_norm 2.3747 (3.3494) loss_scale 128.0000 (70.8397) mem 11224MB [2024-09-01 15:07:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][140/625] eta 0:02:54 lr 0.000047 wd 0.0500 time 0.3413 (0.3597) data time 0.0007 (0.0051) model time 0.3406 (0.3469) loss 2.3731 (2.6052) grad_norm 2.9499 (3.3085) loss_scale 128.0000 (74.8936) mem 11224MB [2024-09-01 15:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][150/625] eta 0:02:50 lr 0.000047 wd 0.0500 time 0.3497 (0.3590) data time 0.0007 (0.0048) model time 0.3490 (0.3470) loss 3.5428 (2.6112) grad_norm 2.3704 (3.2678) loss_scale 128.0000 (78.4106) mem 11224MB [2024-09-01 15:07:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][160/625] eta 0:02:46 lr 0.000047 wd 0.0500 time 0.3490 (0.3583) data time 0.0006 (0.0045) model time 0.3484 (0.3470) loss 2.8525 (2.6161) grad_norm 2.7957 (3.2475) loss_scale 128.0000 (81.4907) mem 11224MB [2024-09-01 15:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][170/625] eta 0:02:42 lr 0.000047 wd 0.0500 time 0.3489 (0.3578) data time 0.0009 (0.0043) model time 0.3479 (0.3471) loss 2.8226 (2.6254) grad_norm 3.0104 (3.4954) loss_scale 128.0000 (84.2105) mem 11224MB [2024-09-01 15:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][180/625] eta 0:02:38 lr 0.000047 wd 0.0500 time 0.3484 (0.3572) data time 0.0010 (0.0041) model time 0.3474 (0.3471) loss 2.7372 (2.6229) grad_norm 2.1436 (3.4767) loss_scale 128.0000 (86.6298) mem 11224MB [2024-09-01 15:07:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][190/625] eta 0:02:35 lr 0.000047 wd 0.0500 time 0.3524 (0.3568) data time 0.0009 (0.0040) model time 0.3515 (0.3471) loss 3.1629 (2.6353) grad_norm 2.7260 (3.4562) loss_scale 128.0000 (88.7958) mem 11224MB [2024-09-01 15:07:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][200/625] eta 0:02:31 lr 0.000047 wd 0.0500 time 0.3474 (0.3564) data time 0.0009 (0.0038) model time 0.3465 (0.3472) loss 3.0301 (2.6341) grad_norm 2.2527 (3.4130) loss_scale 128.0000 (90.7463) mem 11224MB [2024-09-01 15:07:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][210/625] eta 0:02:28 lr 0.000046 wd 0.0500 time 0.3474 (0.3566) data time 0.0008 (0.0037) model time 0.3466 (0.3481) loss 3.0550 (2.6376) grad_norm 2.8061 (3.3900) loss_scale 128.0000 (92.5118) mem 11224MB [2024-09-01 15:07:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][220/625] eta 0:02:25 lr 0.000046 wd 0.0500 time 0.3467 (0.3587) data time 0.0006 (0.0036) model time 0.3460 (0.3511) loss 2.7837 (2.6370) grad_norm 2.2296 (3.3634) loss_scale 128.0000 (94.1176) mem 11224MB [2024-09-01 15:07:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][230/625] eta 0:02:21 lr 0.000046 wd 0.0500 time 0.3499 (0.3582) data time 0.0008 (0.0034) model time 0.3492 (0.3509) loss 2.9851 (2.6402) grad_norm 4.1509 (3.3526) loss_scale 128.0000 (95.5844) mem 11224MB [2024-09-01 15:07:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][240/625] eta 0:02:17 lr 0.000046 wd 0.0500 time 0.3490 (0.3576) data time 0.0009 (0.0033) model time 0.3481 (0.3506) loss 3.0614 (2.6373) grad_norm 2.4560 (3.3444) loss_scale 128.0000 (96.9295) mem 11224MB [2024-09-01 15:07:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][250/625] eta 0:02:14 lr 0.000046 wd 0.0500 time 0.5511 (0.3589) data time 0.0008 (0.0032) model time 0.5503 (0.3524) loss 1.6747 (2.6320) grad_norm 2.2308 (3.3531) loss_scale 128.0000 (98.1673) mem 11224MB [2024-09-01 15:07:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][260/625] eta 0:02:10 lr 0.000046 wd 0.0500 time 0.3540 (0.3584) data time 0.0009 (0.0031) model time 0.3531 (0.3521) loss 2.8184 (2.6407) grad_norm 3.1995 (3.3440) loss_scale 128.0000 (99.3103) mem 11224MB [2024-09-01 15:07:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][270/625] eta 0:02:07 lr 0.000046 wd 0.0500 time 0.3512 (0.3580) data time 0.0007 (0.0031) model time 0.3505 (0.3518) loss 3.1709 (2.6462) grad_norm 3.4957 (3.3174) loss_scale 128.0000 (100.3690) mem 11224MB [2024-09-01 15:08:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][280/625] eta 0:02:03 lr 0.000046 wd 0.0500 time 0.3467 (0.3576) data time 0.0009 (0.0030) model time 0.3458 (0.3515) loss 2.8409 (2.6458) grad_norm 2.5253 (3.3080) loss_scale 128.0000 (101.3523) mem 11224MB [2024-09-01 15:08:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][290/625] eta 0:01:59 lr 0.000046 wd 0.0500 time 0.3444 (0.3572) data time 0.0009 (0.0029) model time 0.3435 (0.3513) loss 2.6806 (2.6404) grad_norm 3.2627 (3.2894) loss_scale 128.0000 (102.2680) mem 11224MB [2024-09-01 15:08:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][300/625] eta 0:01:55 lr 0.000046 wd 0.0500 time 0.3495 (0.3568) data time 0.0007 (0.0028) model time 0.3487 (0.3511) loss 2.8646 (2.6449) grad_norm 2.5164 (3.3490) loss_scale 128.0000 (103.1229) mem 11224MB [2024-09-01 15:08:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][310/625] eta 0:01:52 lr 0.000046 wd 0.0500 time 0.3475 (0.3565) data time 0.0007 (0.0028) model time 0.3468 (0.3508) loss 2.5051 (2.6443) grad_norm 2.5445 (3.3427) loss_scale 128.0000 (103.9228) mem 11224MB [2024-09-01 15:08:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][320/625] eta 0:01:48 lr 0.000046 wd 0.0500 time 0.3508 (0.3562) data time 0.0010 (0.0027) model time 0.3498 (0.3507) loss 2.5298 (2.6410) grad_norm 2.6666 (3.3180) loss_scale 128.0000 (104.6729) mem 11224MB [2024-09-01 15:08:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][330/625] eta 0:01:45 lr 0.000046 wd 0.0500 time 0.3484 (0.3570) data time 0.0007 (0.0027) model time 0.3477 (0.3518) loss 2.8409 (2.6449) grad_norm 3.0242 (3.3136) loss_scale 128.0000 (105.3776) mem 11224MB [2024-09-01 15:08:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][340/625] eta 0:01:42 lr 0.000046 wd 0.0500 time 0.3488 (0.3583) data time 0.0007 (0.0026) model time 0.3481 (0.3535) loss 2.2501 (2.6442) grad_norm 2.7393 (3.3036) loss_scale 128.0000 (106.0411) mem 11224MB [2024-09-01 15:08:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][350/625] eta 0:01:38 lr 0.000046 wd 0.0500 time 0.3453 (0.3579) data time 0.0008 (0.0026) model time 0.3445 (0.3532) loss 2.3060 (2.6428) grad_norm 4.7374 (3.2991) loss_scale 128.0000 (106.6667) mem 11224MB [2024-09-01 15:08:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][360/625] eta 0:01:34 lr 0.000046 wd 0.0500 time 0.3452 (0.3576) data time 0.0009 (0.0025) model time 0.3443 (0.3529) loss 2.1901 (2.6378) grad_norm 2.2105 (3.3032) loss_scale 128.0000 (107.2576) mem 11224MB [2024-09-01 15:08:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][370/625] eta 0:01:31 lr 0.000046 wd 0.0500 time 0.3398 (0.3573) data time 0.0008 (0.0025) model time 0.3390 (0.3527) loss 1.8607 (2.6405) grad_norm 2.8439 (3.2811) loss_scale 128.0000 (107.8167) mem 11224MB [2024-09-01 15:08:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][380/625] eta 0:01:27 lr 0.000046 wd 0.0500 time 0.3486 (0.3570) data time 0.0007 (0.0024) model time 0.3479 (0.3525) loss 1.7235 (2.6334) grad_norm 2.7686 (3.2762) loss_scale 128.0000 (108.3465) mem 11224MB [2024-09-01 15:08:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][390/625] eta 0:01:23 lr 0.000046 wd 0.0500 time 0.3456 (0.3567) data time 0.0010 (0.0024) model time 0.3447 (0.3523) loss 2.7317 (2.6384) grad_norm 2.8770 (3.2789) loss_scale 128.0000 (108.8491) mem 11224MB [2024-09-01 15:08:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][400/625] eta 0:01:20 lr 0.000046 wd 0.0500 time 0.3428 (0.3565) data time 0.0007 (0.0023) model time 0.3421 (0.3520) loss 2.8264 (2.6310) grad_norm 2.3979 (3.2602) loss_scale 128.0000 (109.3267) mem 11224MB [2024-09-01 15:08:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][410/625] eta 0:01:16 lr 0.000046 wd 0.0500 time 0.3434 (0.3563) data time 0.0010 (0.0023) model time 0.3424 (0.3519) loss 2.6735 (2.6326) grad_norm 3.0919 (3.2971) loss_scale 128.0000 (109.7810) mem 11224MB [2024-09-01 15:08:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][420/625] eta 0:01:12 lr 0.000046 wd 0.0500 time 0.3477 (0.3560) data time 0.0006 (0.0023) model time 0.3470 (0.3518) loss 2.6421 (2.6302) grad_norm 2.1029 (3.2885) loss_scale 128.0000 (110.2138) mem 11224MB [2024-09-01 15:08:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][430/625] eta 0:01:09 lr 0.000046 wd 0.0500 time 0.3435 (0.3558) data time 0.0008 (0.0022) model time 0.3427 (0.3516) loss 1.9926 (2.6285) grad_norm 1.9462 (3.2863) loss_scale 128.0000 (110.6265) mem 11224MB [2024-09-01 15:08:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][440/625] eta 0:01:06 lr 0.000046 wd 0.0500 time 0.3440 (0.3569) data time 0.0007 (0.0022) model time 0.3433 (0.3528) loss 2.0573 (2.6233) grad_norm 4.1945 (3.2784) loss_scale 128.0000 (111.0204) mem 11224MB [2024-09-01 15:09:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][450/625] eta 0:01:02 lr 0.000046 wd 0.0500 time 0.3437 (0.3566) data time 0.0009 (0.0022) model time 0.3428 (0.3526) loss 2.2548 (2.6263) grad_norm 4.1955 (3.2685) loss_scale 128.0000 (111.3969) mem 11224MB [2024-09-01 15:09:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][460/625] eta 0:00:58 lr 0.000045 wd 0.0500 time 0.3467 (0.3564) data time 0.0009 (0.0022) model time 0.3458 (0.3524) loss 1.9223 (2.6264) grad_norm 2.0192 (3.2610) loss_scale 128.0000 (111.7570) mem 11224MB [2024-09-01 15:09:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][470/625] eta 0:00:55 lr 0.000045 wd 0.0500 time 0.3466 (0.3562) data time 0.0009 (0.0021) model time 0.3457 (0.3523) loss 2.6335 (2.6230) grad_norm 2.1172 (3.2490) loss_scale 128.0000 (112.1019) mem 11224MB [2024-09-01 15:09:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][480/625] eta 0:00:51 lr 0.000045 wd 0.0500 time 0.3445 (0.3560) data time 0.0006 (0.0021) model time 0.3439 (0.3522) loss 2.5970 (2.6208) grad_norm 3.6482 (3.2411) loss_scale 128.0000 (112.4324) mem 11224MB [2024-09-01 15:09:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][490/625] eta 0:00:48 lr 0.000045 wd 0.0500 time 0.3467 (0.3558) data time 0.0006 (0.0021) model time 0.3460 (0.3520) loss 1.7010 (2.6163) grad_norm 3.7305 (3.2350) loss_scale 128.0000 (112.7495) mem 11224MB [2024-09-01 15:09:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][500/625] eta 0:00:44 lr 0.000045 wd 0.0500 time 0.3470 (0.3557) data time 0.0009 (0.0021) model time 0.3461 (0.3519) loss 2.6809 (2.6184) grad_norm 2.9355 (3.2294) loss_scale 128.0000 (113.0539) mem 11224MB [2024-09-01 15:09:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][510/625] eta 0:00:40 lr 0.000045 wd 0.0500 time 0.3505 (0.3558) data time 0.0006 (0.0020) model time 0.3499 (0.3522) loss 2.4600 (2.6228) grad_norm 3.7792 (3.2229) loss_scale 128.0000 (113.3464) mem 11224MB [2024-09-01 15:09:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][520/625] eta 0:00:37 lr 0.000045 wd 0.0500 time 0.3553 (0.3557) data time 0.0008 (0.0020) model time 0.3544 (0.3521) loss 2.6108 (2.6240) grad_norm 4.2276 (3.2163) loss_scale 128.0000 (113.6276) mem 11224MB [2024-09-01 15:09:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][530/625] eta 0:00:33 lr 0.000045 wd 0.0500 time 0.3528 (0.3555) data time 0.0009 (0.0020) model time 0.3519 (0.3519) loss 2.2975 (2.6264) grad_norm 2.3138 (3.2051) loss_scale 128.0000 (113.8983) mem 11224MB [2024-09-01 15:09:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][540/625] eta 0:00:30 lr 0.000045 wd 0.0500 time 0.3488 (0.3556) data time 0.0007 (0.0020) model time 0.3481 (0.3521) loss 3.2388 (2.6265) grad_norm 3.2954 (3.2102) loss_scale 128.0000 (114.1590) mem 11224MB [2024-09-01 15:09:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][550/625] eta 0:00:26 lr 0.000045 wd 0.0500 time 0.3425 (0.3565) data time 0.0009 (0.0020) model time 0.3416 (0.3531) loss 1.9103 (2.6233) grad_norm 3.3225 (3.2322) loss_scale 128.0000 (114.4102) mem 11224MB [2024-09-01 15:09:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][560/625] eta 0:00:23 lr 0.000045 wd 0.0500 time 0.3437 (0.3570) data time 0.0009 (0.0019) model time 0.3428 (0.3538) loss 2.4686 (2.6238) grad_norm 2.8104 (3.2328) loss_scale 128.0000 (114.6524) mem 11224MB [2024-09-01 15:09:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][570/625] eta 0:00:19 lr 0.000045 wd 0.0500 time 0.3460 (0.3568) data time 0.0009 (0.0019) model time 0.3452 (0.3536) loss 2.2884 (2.6243) grad_norm 3.8446 (3.2442) loss_scale 128.0000 (114.8862) mem 11224MB [2024-09-01 15:09:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][580/625] eta 0:00:16 lr 0.000045 wd 0.0500 time 0.3485 (0.3567) data time 0.0010 (0.0019) model time 0.3475 (0.3535) loss 2.7587 (2.6259) grad_norm 6.6296 (3.2415) loss_scale 128.0000 (115.1119) mem 11224MB [2024-09-01 15:09:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][590/625] eta 0:00:12 lr 0.000045 wd 0.0500 time 0.3438 (0.3565) data time 0.0009 (0.0019) model time 0.3429 (0.3533) loss 2.7581 (2.6310) grad_norm 3.3017 (3.2351) loss_scale 128.0000 (115.3299) mem 11224MB [2024-09-01 15:09:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][600/625] eta 0:00:08 lr 0.000045 wd 0.0500 time 0.3510 (0.3564) data time 0.0007 (0.0019) model time 0.3503 (0.3532) loss 2.8266 (2.6324) grad_norm 2.6303 (3.2333) loss_scale 128.0000 (115.5408) mem 11224MB [2024-09-01 15:09:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][610/625] eta 0:00:05 lr 0.000045 wd 0.0500 time 0.3366 (0.3562) data time 0.0004 (0.0019) model time 0.3362 (0.3530) loss 2.7787 (2.6265) grad_norm 3.6144 (3.2283) loss_scale 128.0000 (115.7447) mem 11224MB [2024-09-01 15:10:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [279/300][620/625] eta 0:00:01 lr 0.000045 wd 0.0500 time 0.3372 (0.3559) data time 0.0005 (0.0018) model time 0.3367 (0.3528) loss 2.2433 (2.6239) grad_norm 2.8867 (3.2230) loss_scale 128.0000 (115.9420) mem 11224MB [2024-09-01 15:10:04 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 279 training takes 0:03:42 [2024-09-01 15:10:04 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:10:05 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:10:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.449 (0.449) Loss 0.5186 (0.5186) Acc@1 89.014 (89.014) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:10:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.130) Loss 0.8501 (0.6149) Acc@1 80.664 (86.954) Acc@5 95.850 (97.772) Mem 11224MB [2024-09-01 15:10:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.111) Loss 0.9209 (0.7319) Acc@1 78.906 (84.049) Acc@5 95.117 (96.552) Mem 11224MB [2024-09-01 15:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.741 Acc@5 96.561 [2024-09-01 15:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 15:10:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.757 (0.757) Loss 0.5049 (0.5049) Acc@1 89.062 (89.062) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:10:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.164) Loss 0.8179 (0.5986) Acc@1 80.713 (87.065) Acc@5 95.703 (97.869) Mem 11224MB [2024-09-01 15:10:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.129) Loss 0.8979 (0.7141) Acc@1 78.760 (84.219) Acc@5 95.410 (96.715) Mem 11224MB [2024-09-01 15:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.935 Acc@5 96.739 [2024-09-01 15:10:11 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][0/625] eta 0:12:23 lr 0.000045 wd 0.0500 time 1.1903 (1.1903) data time 0.5336 (0.5336) model time 0.0000 (0.0000) loss 2.6689 (2.6689) grad_norm 2.0365 (2.0365) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][10/625] eta 0:04:20 lr 0.000045 wd 0.0500 time 0.3513 (0.4240) data time 0.0006 (0.0493) model time 0.0000 (0.0000) loss 2.2816 (2.6828) grad_norm 1.9825 (3.3631) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][20/625] eta 0:03:54 lr 0.000045 wd 0.0500 time 0.3459 (0.3871) data time 0.0009 (0.0262) model time 0.0000 (0.0000) loss 2.8810 (2.4965) grad_norm 2.4685 (2.9796) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][30/625] eta 0:03:42 lr 0.000045 wd 0.0500 time 0.3463 (0.3744) data time 0.0008 (0.0180) model time 0.0000 (0.0000) loss 2.7699 (2.5504) grad_norm 2.0499 (2.8366) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][40/625] eta 0:03:35 lr 0.000045 wd 0.0500 time 0.3482 (0.3682) data time 0.0007 (0.0138) model time 0.0000 (0.0000) loss 3.3310 (2.5747) grad_norm 2.0849 (2.8297) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][50/625] eta 0:03:29 lr 0.000045 wd 0.0500 time 0.3519 (0.3644) data time 0.0009 (0.0112) model time 0.0000 (0.0000) loss 2.9440 (2.5711) grad_norm 1.6823 (2.8969) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][60/625] eta 0:03:24 lr 0.000045 wd 0.0500 time 0.3444 (0.3614) data time 0.0006 (0.0095) model time 0.3438 (0.3455) loss 2.6620 (2.5621) grad_norm 2.8973 (3.1110) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][70/625] eta 0:03:19 lr 0.000045 wd 0.0500 time 0.3476 (0.3595) data time 0.0009 (0.0083) model time 0.3467 (0.3462) loss 2.7912 (2.5773) grad_norm 3.2340 (3.0807) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][80/625] eta 0:03:17 lr 0.000045 wd 0.0500 time 0.3490 (0.3624) data time 0.0008 (0.0074) model time 0.3481 (0.3582) loss 2.8563 (2.5803) grad_norm 3.8462 (3.0618) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][90/625] eta 0:03:13 lr 0.000044 wd 0.0500 time 0.3464 (0.3609) data time 0.0007 (0.0067) model time 0.3457 (0.3555) loss 2.9562 (2.5911) grad_norm 2.8831 (3.0138) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][100/625] eta 0:03:08 lr 0.000044 wd 0.0500 time 0.3472 (0.3597) data time 0.0006 (0.0061) model time 0.3466 (0.3541) loss 2.0226 (2.5852) grad_norm 3.7056 (3.0106) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][110/625] eta 0:03:04 lr 0.000044 wd 0.0500 time 0.3535 (0.3588) data time 0.0007 (0.0056) model time 0.3528 (0.3532) loss 3.1601 (2.5823) grad_norm 3.7462 (3.0094) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][120/625] eta 0:03:02 lr 0.000044 wd 0.0500 time 0.5515 (0.3622) data time 0.0009 (0.0052) model time 0.5506 (0.3597) loss 2.3158 (2.5756) grad_norm 3.5072 (3.1506) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:10:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][130/625] eta 0:02:59 lr 0.000044 wd 0.0500 time 0.3462 (0.3626) data time 0.0009 (0.0049) model time 0.3453 (0.3606) loss 2.9512 (2.5898) grad_norm 2.5342 (3.1233) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][140/625] eta 0:02:55 lr 0.000044 wd 0.0500 time 0.3419 (0.3614) data time 0.0009 (0.0046) model time 0.3410 (0.3589) loss 2.8621 (2.5864) grad_norm 19.2520 (3.2122) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][150/625] eta 0:02:51 lr 0.000044 wd 0.0500 time 0.3468 (0.3605) data time 0.0008 (0.0044) model time 0.3460 (0.3576) loss 2.7485 (2.5717) grad_norm 2.7691 (3.1930) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][160/625] eta 0:02:47 lr 0.000044 wd 0.0500 time 0.3477 (0.3597) data time 0.0006 (0.0041) model time 0.3470 (0.3567) loss 3.0673 (2.5716) grad_norm 2.5271 (3.1795) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][170/625] eta 0:02:43 lr 0.000044 wd 0.0500 time 0.3395 (0.3589) data time 0.0009 (0.0040) model time 0.3386 (0.3558) loss 3.0541 (2.5833) grad_norm 2.2903 (3.1379) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][180/625] eta 0:02:39 lr 0.000044 wd 0.0500 time 0.3531 (0.3583) data time 0.0009 (0.0038) model time 0.3522 (0.3551) loss 2.2698 (2.5767) grad_norm 2.0334 (3.1481) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][190/625] eta 0:02:36 lr 0.000044 wd 0.0500 time 0.3507 (0.3589) data time 0.0009 (0.0036) model time 0.3498 (0.3561) loss 1.8181 (2.5713) grad_norm 2.5548 (3.1170) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][200/625] eta 0:02:32 lr 0.000044 wd 0.0500 time 0.3464 (0.3583) data time 0.0007 (0.0035) model time 0.3458 (0.3554) loss 2.9947 (2.5762) grad_norm 3.5018 (3.1073) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][210/625] eta 0:02:28 lr 0.000044 wd 0.0500 time 0.3503 (0.3579) data time 0.0007 (0.0034) model time 0.3496 (0.3549) loss 2.7065 (2.5751) grad_norm 11.6585 (3.1129) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][220/625] eta 0:02:24 lr 0.000044 wd 0.0500 time 0.3481 (0.3574) data time 0.0008 (0.0033) model time 0.3473 (0.3544) loss 2.7440 (2.5670) grad_norm 4.3967 (3.1243) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][230/625] eta 0:02:21 lr 0.000044 wd 0.0500 time 0.3523 (0.3570) data time 0.0008 (0.0032) model time 0.3515 (0.3541) loss 3.1252 (2.5758) grad_norm 3.0287 (3.1230) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][240/625] eta 0:02:17 lr 0.000044 wd 0.0500 time 0.3462 (0.3566) data time 0.0006 (0.0031) model time 0.3456 (0.3537) loss 1.5046 (2.5794) grad_norm 3.4434 (3.1258) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][250/625] eta 0:02:13 lr 0.000044 wd 0.0500 time 0.3492 (0.3562) data time 0.0007 (0.0030) model time 0.3486 (0.3533) loss 1.5110 (2.5716) grad_norm 2.9254 (3.1421) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][260/625] eta 0:02:09 lr 0.000044 wd 0.0500 time 0.3514 (0.3560) data time 0.0009 (0.0029) model time 0.3504 (0.3530) loss 2.8744 (2.5683) grad_norm 4.9099 (3.1633) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][270/625] eta 0:02:06 lr 0.000044 wd 0.0500 time 0.3458 (0.3556) data time 0.0007 (0.0028) model time 0.3451 (0.3527) loss 3.2659 (2.5713) grad_norm 6.4656 (3.1781) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][280/625] eta 0:02:02 lr 0.000044 wd 0.0500 time 0.3488 (0.3554) data time 0.0008 (0.0027) model time 0.3480 (0.3525) loss 2.8852 (2.5710) grad_norm 11.5312 (3.2051) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][290/625] eta 0:01:59 lr 0.000044 wd 0.0500 time 0.3511 (0.3558) data time 0.0007 (0.0027) model time 0.3504 (0.3531) loss 2.0269 (2.5747) grad_norm 1.8948 (3.2055) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:11:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][300/625] eta 0:01:55 lr 0.000044 wd 0.0500 time 0.3445 (0.3564) data time 0.0009 (0.0026) model time 0.3436 (0.3539) loss 2.4806 (2.5666) grad_norm 2.3666 (3.2156) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:12:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][310/625] eta 0:01:52 lr 0.000044 wd 0.0500 time 0.3412 (0.3574) data time 0.0006 (0.0026) model time 0.3406 (0.3552) loss 2.6231 (2.5700) grad_norm 2.5040 (3.2045) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:12:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][320/625] eta 0:01:48 lr 0.000044 wd 0.0500 time 0.3487 (0.3571) data time 0.0009 (0.0025) model time 0.3478 (0.3548) loss 2.6821 (2.5724) grad_norm 3.2027 (3.2372) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:12:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][330/625] eta 0:01:45 lr 0.000044 wd 0.0500 time 0.3435 (0.3567) data time 0.0007 (0.0025) model time 0.3428 (0.3545) loss 3.2011 (2.5743) grad_norm 2.0361 (3.2403) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:12:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][340/625] eta 0:01:41 lr 0.000043 wd 0.0500 time 0.3444 (0.3564) data time 0.0007 (0.0024) model time 0.3437 (0.3541) loss 2.0456 (2.5709) grad_norm 4.4471 (3.2303) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:12:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][350/625] eta 0:01:37 lr 0.000043 wd 0.0500 time 0.3450 (0.3561) data time 0.0007 (0.0024) model time 0.3443 (0.3539) loss 2.9882 (2.5738) grad_norm 3.3848 (3.2450) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:12:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][360/625] eta 0:01:34 lr 0.000043 wd 0.0500 time 0.3519 (0.3559) data time 0.0008 (0.0023) model time 0.3511 (0.3536) loss 2.3036 (2.5774) grad_norm 2.3135 (3.2264) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:12:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][370/625] eta 0:01:30 lr 0.000043 wd 0.0500 time 0.3476 (0.3561) data time 0.0009 (0.0023) model time 0.3467 (0.3540) loss 1.8846 (2.5704) grad_norm 1.6956 (3.2122) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 15:12:25 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:12:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 15:14:09 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 15:14:23 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 15:14:35 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 15:14:35 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 15:14:37 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 15:14:39 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 15:14:39 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 280) [2024-09-01 15:14:39 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 15:14:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][380/625] eta 0:21:21 lr 0.000043 wd 0.0500 time 0.3686 (5.2303) data time 0.0008 (0.2000) model time 0.3677 (5.0303) loss 2.1816 (2.6577) grad_norm 2.0941 (2.4882) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][390/625] eta 0:05:51 lr 0.000043 wd 0.0500 time 0.3698 (1.4952) data time 0.0012 (0.0470) model time 0.3687 (1.4483) loss 2.7527 (2.6916) grad_norm 2.9838 (3.5926) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][400/625] eta 0:03:49 lr 0.000043 wd 0.0500 time 0.6443 (1.0189) data time 0.0008 (0.0270) model time 0.6435 (0.9919) loss 3.2244 (2.8082) grad_norm 3.4597 (3.5791) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][410/625] eta 0:02:56 lr 0.000043 wd 0.0500 time 0.3714 (0.8219) data time 0.0009 (0.0192) model time 0.3705 (0.8027) loss 2.9887 (2.8188) grad_norm 3.6302 (3.4235) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][420/625] eta 0:02:28 lr 0.000043 wd 0.0500 time 0.3749 (0.7235) data time 0.0011 (0.0150) model time 0.3737 (0.7085) loss 2.7043 (2.7444) grad_norm 3.0941 (3.4377) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][430/625] eta 0:02:08 lr 0.000043 wd 0.0500 time 0.3728 (0.6572) data time 0.0010 (0.0123) model time 0.3717 (0.6448) loss 2.7832 (2.7372) grad_norm 2.3582 (3.2516) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][440/625] eta 0:01:53 lr 0.000043 wd 0.0500 time 0.3823 (0.6126) data time 0.0011 (0.0105) model time 0.3811 (0.6021) loss 2.2602 (2.6957) grad_norm 2.3304 (3.2544) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][450/625] eta 0:01:41 lr 0.000043 wd 0.0500 time 0.3678 (0.5796) data time 0.0011 (0.0092) model time 0.3668 (0.5704) loss 3.0742 (2.6726) grad_norm 2.6616 (3.7464) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][460/625] eta 0:01:31 lr 0.000043 wd 0.0500 time 0.3717 (0.5547) data time 0.0008 (0.0082) model time 0.3709 (0.5464) loss 1.7120 (2.6412) grad_norm 3.4133 (3.8024) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][470/625] eta 0:01:22 lr 0.000043 wd 0.0500 time 0.3641 (0.5351) data time 0.0007 (0.0075) model time 0.3633 (0.5276) loss 3.2309 (2.6473) grad_norm 1.7993 (3.6807) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][480/625] eta 0:01:15 lr 0.000043 wd 0.0500 time 0.3812 (0.5195) data time 0.0008 (0.0068) model time 0.3804 (0.5126) loss 3.0054 (2.6763) grad_norm 2.6049 (3.6092) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][490/625] eta 0:01:08 lr 0.000043 wd 0.0500 time 0.3696 (0.5067) data time 0.0009 (0.0063) model time 0.3687 (0.5004) loss 2.6819 (2.6718) grad_norm 7.0017 (3.6606) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][500/625] eta 0:01:02 lr 0.000043 wd 0.0500 time 0.3772 (0.4961) data time 0.0011 (0.0059) model time 0.3761 (0.4902) loss 2.9295 (2.6694) grad_norm 2.6495 (3.7074) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][510/625] eta 0:00:55 lr 0.000043 wd 0.0500 time 0.3767 (0.4868) data time 0.0010 (0.0055) model time 0.3756 (0.4813) loss 2.7935 (2.6703) grad_norm 2.8913 (3.6640) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][520/625] eta 0:00:50 lr 0.000043 wd 0.0500 time 0.3771 (0.4790) data time 0.0011 (0.0052) model time 0.3760 (0.4738) loss 3.1619 (2.6667) grad_norm 2.7470 (3.6425) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][530/625] eta 0:00:44 lr 0.000043 wd 0.0500 time 0.3892 (0.4722) data time 0.0008 (0.0050) model time 0.3884 (0.4672) loss 2.9070 (2.6594) grad_norm 3.5241 (3.6506) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:15:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][540/625] eta 0:00:39 lr 0.000043 wd 0.0500 time 0.3758 (0.4661) data time 0.0010 (0.0047) model time 0.3747 (0.4614) loss 1.8245 (2.6564) grad_norm 2.0960 (3.6424) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][550/625] eta 0:00:34 lr 0.000043 wd 0.0500 time 0.3726 (0.4607) data time 0.0011 (0.0045) model time 0.3715 (0.4562) loss 2.9328 (2.6536) grad_norm 2.7551 (3.5875) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][560/625] eta 0:00:29 lr 0.000043 wd 0.0500 time 0.3794 (0.4560) data time 0.0010 (0.0043) model time 0.3783 (0.4517) loss 2.8105 (2.6495) grad_norm 3.2728 (3.5322) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][570/625] eta 0:00:24 lr 0.000043 wd 0.0500 time 0.3804 (0.4519) data time 0.0008 (0.0042) model time 0.3796 (0.4477) loss 2.8757 (2.6528) grad_norm 2.0722 (3.4999) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][580/625] eta 0:00:20 lr 0.000043 wd 0.0500 time 0.3721 (0.4479) data time 0.0009 (0.0040) model time 0.3713 (0.4439) loss 1.9590 (2.6433) grad_norm 2.5370 (3.4797) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][590/625] eta 0:00:15 lr 0.000043 wd 0.0500 time 0.3859 (0.4445) data time 0.0011 (0.0039) model time 0.3848 (0.4406) loss 1.6605 (2.6397) grad_norm 3.4683 (3.4736) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][600/625] eta 0:00:11 lr 0.000043 wd 0.0500 time 0.3780 (0.4414) data time 0.0009 (0.0037) model time 0.3771 (0.4376) loss 1.9709 (2.6423) grad_norm 2.7833 (3.4473) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][610/625] eta 0:00:06 lr 0.000042 wd 0.0500 time 0.5490 (0.4410) data time 0.0008 (0.0036) model time 0.5483 (0.4373) loss 2.2345 (2.6390) grad_norm 1.8504 (3.4305) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [280/300][620/625] eta 0:00:02 lr 0.000042 wd 0.0500 time 0.3644 (0.4378) data time 0.0008 (0.0035) model time 0.3637 (0.4343) loss 2.9225 (2.6419) grad_norm 4.9999 (3.4034) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:16:31 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 280 training takes 0:01:47 [2024-09-01 15:16:31 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:16:36 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:16:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.530 (0.530) Loss 0.5107 (0.5107) Acc@1 88.672 (88.672) Acc@5 98.633 (98.633) Mem 11219MB [2024-09-01 15:16:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.143) Loss 0.8477 (0.6093) Acc@1 80.078 (86.825) Acc@5 95.459 (97.785) Mem 11219MB [2024-09-01 15:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9048 (0.7256) Acc@1 78.955 (84.005) Acc@5 95.068 (96.608) Mem 11219MB [2024-09-01 15:16:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.705 Acc@5 96.627 [2024-09-01 15:16:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 15:16:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.828 (0.828) Loss 0.5054 (0.5054) Acc@1 89.014 (89.014) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 15:16:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.177) Loss 0.8193 (0.5990) Acc@1 80.762 (87.078) Acc@5 95.703 (97.865) Mem 11219MB [2024-09-01 15:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8984 (0.7144) Acc@1 78.711 (84.231) Acc@5 95.361 (96.715) Mem 11219MB [2024-09-01 15:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.941 Acc@5 96.733 [2024-09-01 15:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:16:44 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.94% [2024-09-01 15:16:44 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 15:16:49 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 15:16:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][0/625] eta 0:08:52 lr 0.000042 wd 0.0500 time 0.8527 (0.8527) data time 0.4749 (0.4749) model time 0.0000 (0.0000) loss 1.9668 (1.9668) grad_norm 2.0625 (2.0625) loss_scale 128.0000 (128.0000) mem 11228MB [2024-09-01 15:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][10/625] eta 0:04:16 lr 0.000042 wd 0.0500 time 0.3679 (0.4173) data time 0.0010 (0.0445) model time 0.0000 (0.0000) loss 2.8131 (2.2664) grad_norm 2.5107 (2.5965) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:16:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][20/625] eta 0:04:00 lr 0.000042 wd 0.0500 time 0.3763 (0.3975) data time 0.0008 (0.0238) model time 0.0000 (0.0000) loss 2.9958 (2.3994) grad_norm 2.0327 (2.8893) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][30/625] eta 0:03:52 lr 0.000042 wd 0.0500 time 0.3715 (0.3902) data time 0.0011 (0.0165) model time 0.0000 (0.0000) loss 1.8668 (2.4612) grad_norm 3.4703 (2.9206) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][40/625] eta 0:03:46 lr 0.000042 wd 0.0500 time 0.3656 (0.3864) data time 0.0008 (0.0127) model time 0.0000 (0.0000) loss 2.9925 (2.5159) grad_norm 2.7476 (2.8802) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][50/625] eta 0:03:40 lr 0.000042 wd 0.0500 time 0.3747 (0.3842) data time 0.0012 (0.0105) model time 0.0000 (0.0000) loss 2.5351 (2.4668) grad_norm 2.2559 (2.8725) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][60/625] eta 0:03:35 lr 0.000042 wd 0.0500 time 0.3699 (0.3822) data time 0.0009 (0.0089) model time 0.3691 (0.3705) loss 2.0608 (2.4863) grad_norm 2.3035 (2.8717) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][70/625] eta 0:03:31 lr 0.000042 wd 0.0500 time 0.3687 (0.3807) data time 0.0011 (0.0078) model time 0.3677 (0.3706) loss 2.6765 (2.5435) grad_norm 4.0218 (2.9376) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][80/625] eta 0:03:27 lr 0.000042 wd 0.0500 time 0.3691 (0.3815) data time 0.0007 (0.0070) model time 0.3683 (0.3758) loss 2.8491 (2.5699) grad_norm 2.3031 (3.0076) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][90/625] eta 0:03:23 lr 0.000042 wd 0.0500 time 0.3719 (0.3802) data time 0.0014 (0.0063) model time 0.3705 (0.3740) loss 2.5554 (2.5652) grad_norm 2.4697 (2.9668) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][100/625] eta 0:03:19 lr 0.000042 wd 0.0500 time 0.3641 (0.3794) data time 0.0009 (0.0058) model time 0.3632 (0.3734) loss 2.2108 (2.5658) grad_norm 2.7502 (2.9369) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][110/625] eta 0:03:15 lr 0.000042 wd 0.0500 time 0.3821 (0.3788) data time 0.0012 (0.0054) model time 0.3810 (0.3731) loss 2.9885 (2.5686) grad_norm 4.3732 (2.9692) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][120/625] eta 0:03:10 lr 0.000042 wd 0.0500 time 0.3679 (0.3782) data time 0.0009 (0.0050) model time 0.3670 (0.3728) loss 2.0309 (2.5643) grad_norm 2.5061 (2.9450) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][130/625] eta 0:03:07 lr 0.000042 wd 0.0500 time 0.3762 (0.3779) data time 0.0011 (0.0047) model time 0.3752 (0.3728) loss 2.8329 (2.5639) grad_norm 6.0289 (3.0364) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][140/625] eta 0:03:03 lr 0.000042 wd 0.0500 time 0.3702 (0.3778) data time 0.0011 (0.0045) model time 0.3691 (0.3731) loss 2.0626 (2.5501) grad_norm 9.6777 (3.1158) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][150/625] eta 0:02:59 lr 0.000042 wd 0.0500 time 0.3695 (0.3777) data time 0.0009 (0.0042) model time 0.3687 (0.3734) loss 2.4457 (2.5645) grad_norm 5.8931 (3.1557) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][160/625] eta 0:02:55 lr 0.000042 wd 0.0500 time 0.3746 (0.3776) data time 0.0010 (0.0040) model time 0.3736 (0.3735) loss 3.0520 (2.5881) grad_norm 2.5958 (3.1142) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][170/625] eta 0:02:52 lr 0.000042 wd 0.0500 time 0.3732 (0.3797) data time 0.0009 (0.0039) model time 0.3723 (0.3767) loss 3.2107 (2.5899) grad_norm 4.6942 (3.0922) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:17:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][180/625] eta 0:02:48 lr 0.000042 wd 0.0500 time 0.3741 (0.3794) data time 0.0010 (0.0037) model time 0.3732 (0.3765) loss 3.1432 (2.5983) grad_norm 2.5352 (3.0570) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][190/625] eta 0:02:44 lr 0.000042 wd 0.0500 time 0.3812 (0.3793) data time 0.0011 (0.0036) model time 0.3801 (0.3763) loss 2.5444 (2.6083) grad_norm 3.4570 (3.0700) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][200/625] eta 0:02:41 lr 0.000042 wd 0.0500 time 0.3754 (0.3789) data time 0.0010 (0.0035) model time 0.3743 (0.3760) loss 2.3663 (2.6100) grad_norm 2.4387 (3.0866) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][210/625] eta 0:02:37 lr 0.000042 wd 0.0500 time 0.3747 (0.3787) data time 0.0009 (0.0034) model time 0.3738 (0.3759) loss 2.4231 (2.6026) grad_norm 4.2932 (3.0993) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][220/625] eta 0:02:33 lr 0.000042 wd 0.0500 time 0.3882 (0.3785) data time 0.0010 (0.0033) model time 0.3872 (0.3757) loss 2.3231 (2.5912) grad_norm 3.4998 (3.1090) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][230/625] eta 0:02:29 lr 0.000042 wd 0.0500 time 0.3669 (0.3782) data time 0.0011 (0.0032) model time 0.3658 (0.3755) loss 1.7317 (2.5814) grad_norm 2.4515 (3.0795) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][240/625] eta 0:02:25 lr 0.000042 wd 0.0500 time 0.3680 (0.3781) data time 0.0010 (0.0031) model time 0.3669 (0.3754) loss 2.9763 (2.5936) grad_norm 1.8969 (3.0548) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][250/625] eta 0:02:21 lr 0.000041 wd 0.0500 time 0.3791 (0.3780) data time 0.0008 (0.0030) model time 0.3783 (0.3754) loss 2.8126 (2.5883) grad_norm 4.2236 (3.0538) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][260/625] eta 0:02:17 lr 0.000041 wd 0.0500 time 0.3695 (0.3778) data time 0.0008 (0.0029) model time 0.3687 (0.3751) loss 3.1117 (2.5902) grad_norm 4.0479 (3.0534) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][270/625] eta 0:02:14 lr 0.000041 wd 0.0500 time 0.3700 (0.3776) data time 0.0010 (0.0028) model time 0.3690 (0.3750) loss 3.1032 (2.5992) grad_norm 2.4633 (3.0692) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][280/625] eta 0:02:10 lr 0.000041 wd 0.0500 time 0.3731 (0.3775) data time 0.0011 (0.0028) model time 0.3719 (0.3749) loss 1.4080 (2.5861) grad_norm 2.3927 (3.1006) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][290/625] eta 0:02:06 lr 0.000041 wd 0.0500 time 0.3761 (0.3780) data time 0.0008 (0.0027) model time 0.3753 (0.3756) loss 2.9925 (2.5848) grad_norm 3.1527 (3.1495) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][300/625] eta 0:02:02 lr 0.000041 wd 0.0500 time 0.3706 (0.3778) data time 0.0008 (0.0027) model time 0.3697 (0.3755) loss 2.9181 (2.5869) grad_norm 4.5948 (3.1596) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][310/625] eta 0:01:58 lr 0.000041 wd 0.0500 time 0.3693 (0.3777) data time 0.0012 (0.0026) model time 0.3682 (0.3753) loss 2.5832 (2.5910) grad_norm 2.3188 (3.1401) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][320/625] eta 0:01:55 lr 0.000041 wd 0.0500 time 0.3867 (0.3776) data time 0.0009 (0.0026) model time 0.3858 (0.3753) loss 2.9318 (2.5945) grad_norm 3.3747 (3.1615) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][330/625] eta 0:01:51 lr 0.000041 wd 0.0500 time 0.3786 (0.3776) data time 0.0010 (0.0025) model time 0.3776 (0.3753) loss 3.2678 (2.5985) grad_norm 3.5637 (3.2003) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:18:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][340/625] eta 0:01:47 lr 0.000041 wd 0.0500 time 0.3830 (0.3776) data time 0.0012 (0.0025) model time 0.3819 (0.3753) loss 2.8199 (2.6013) grad_norm 2.6508 (3.1885) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][350/625] eta 0:01:43 lr 0.000041 wd 0.0500 time 0.3733 (0.3775) data time 0.0010 (0.0024) model time 0.3723 (0.3753) loss 2.3524 (2.6002) grad_norm 2.4091 (3.2068) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][360/625] eta 0:01:40 lr 0.000041 wd 0.0500 time 0.3727 (0.3790) data time 0.0011 (0.0024) model time 0.3716 (0.3771) loss 2.3682 (2.6003) grad_norm 2.2770 (3.2075) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][370/625] eta 0:01:36 lr 0.000041 wd 0.0500 time 0.3784 (0.3789) data time 0.0010 (0.0024) model time 0.3773 (0.3770) loss 2.6863 (2.6044) grad_norm 3.1472 (3.2148) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][380/625] eta 0:01:32 lr 0.000041 wd 0.0500 time 0.3786 (0.3788) data time 0.0008 (0.0023) model time 0.3777 (0.3769) loss 2.2595 (2.6076) grad_norm 2.2159 (3.1998) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][390/625] eta 0:01:28 lr 0.000041 wd 0.0500 time 0.3724 (0.3786) data time 0.0011 (0.0023) model time 0.3713 (0.3767) loss 2.8702 (2.6104) grad_norm 2.9543 (3.2807) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][400/625] eta 0:01:25 lr 0.000041 wd 0.0500 time 0.3626 (0.3786) data time 0.0010 (0.0023) model time 0.3616 (0.3767) loss 2.5479 (2.6031) grad_norm 2.1720 (3.3025) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][410/625] eta 0:01:21 lr 0.000041 wd 0.0500 time 0.3739 (0.3784) data time 0.0010 (0.0022) model time 0.3729 (0.3766) loss 2.2312 (2.5998) grad_norm 2.4485 (3.2985) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][420/625] eta 0:01:17 lr 0.000041 wd 0.0500 time 0.3742 (0.3783) data time 0.0008 (0.0022) model time 0.3735 (0.3764) loss 3.1696 (2.6012) grad_norm 2.7788 (3.2832) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][430/625] eta 0:01:13 lr 0.000041 wd 0.0500 time 0.3708 (0.3782) data time 0.0012 (0.0022) model time 0.3696 (0.3763) loss 2.8712 (2.6055) grad_norm 2.7821 (3.2692) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][440/625] eta 0:01:09 lr 0.000041 wd 0.0500 time 0.3707 (0.3781) data time 0.0008 (0.0021) model time 0.3699 (0.3762) loss 2.3278 (2.6058) grad_norm 3.0595 (3.2637) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][450/625] eta 0:01:06 lr 0.000041 wd 0.0500 time 0.3732 (0.3781) data time 0.0008 (0.0021) model time 0.3724 (0.3762) loss 2.7640 (2.6006) grad_norm 1.9827 (3.2535) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][460/625] eta 0:01:02 lr 0.000041 wd 0.0500 time 0.3801 (0.3780) data time 0.0010 (0.0021) model time 0.3790 (0.3762) loss 2.3435 (2.5990) grad_norm 3.1247 (3.2423) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][470/625] eta 0:00:58 lr 0.000041 wd 0.0500 time 0.3721 (0.3783) data time 0.0008 (0.0021) model time 0.3712 (0.3765) loss 3.3229 (2.5931) grad_norm 1.7807 (3.2310) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][480/625] eta 0:00:54 lr 0.000041 wd 0.0500 time 0.3684 (0.3790) data time 0.0010 (0.0021) model time 0.3674 (0.3774) loss 2.7292 (2.5920) grad_norm 6.0487 (3.2528) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][490/625] eta 0:00:51 lr 0.000041 wd 0.0500 time 0.3796 (0.3790) data time 0.0010 (0.0020) model time 0.3785 (0.3773) loss 2.8086 (2.5978) grad_norm 1.6865 (3.2606) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:19:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][500/625] eta 0:00:47 lr 0.000041 wd 0.0500 time 0.3702 (0.3789) data time 0.0008 (0.0020) model time 0.3694 (0.3772) loss 1.7185 (2.5993) grad_norm 3.3758 (3.2747) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][510/625] eta 0:00:43 lr 0.000041 wd 0.0500 time 0.3674 (0.3788) data time 0.0008 (0.0020) model time 0.3666 (0.3771) loss 3.1526 (2.5998) grad_norm 3.8460 (3.2974) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][520/625] eta 0:00:39 lr 0.000040 wd 0.0500 time 0.3622 (0.3787) data time 0.0010 (0.0020) model time 0.3612 (0.3770) loss 2.5286 (2.6033) grad_norm 2.8427 (3.3571) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][530/625] eta 0:00:35 lr 0.000040 wd 0.0500 time 0.3687 (0.3785) data time 0.0008 (0.0020) model time 0.3679 (0.3769) loss 3.1840 (2.6052) grad_norm 3.3688 (3.3504) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][540/625] eta 0:00:32 lr 0.000040 wd 0.0500 time 0.3665 (0.3785) data time 0.0008 (0.0019) model time 0.3657 (0.3768) loss 2.7629 (2.6053) grad_norm 2.2015 (3.3584) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][550/625] eta 0:00:28 lr 0.000040 wd 0.0500 time 0.3775 (0.3784) data time 0.0009 (0.0019) model time 0.3766 (0.3767) loss 2.8211 (2.6061) grad_norm 1.8254 (3.3436) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][560/625] eta 0:00:24 lr 0.000040 wd 0.0500 time 0.3812 (0.3784) data time 0.0010 (0.0019) model time 0.3802 (0.3767) loss 3.1418 (2.6026) grad_norm 2.0857 (3.3417) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][570/625] eta 0:00:20 lr 0.000040 wd 0.0500 time 0.3731 (0.3783) data time 0.0011 (0.0019) model time 0.3721 (0.3767) loss 2.7627 (2.5998) grad_norm 3.5839 (3.3366) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][580/625] eta 0:00:17 lr 0.000040 wd 0.0500 time 0.3702 (0.3782) data time 0.0010 (0.0019) model time 0.3692 (0.3766) loss 2.4933 (2.5955) grad_norm 4.0769 (3.3503) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][590/625] eta 0:00:13 lr 0.000040 wd 0.0500 time 0.3760 (0.3785) data time 0.0008 (0.0019) model time 0.3752 (0.3769) loss 3.1106 (2.5938) grad_norm 2.7489 (3.3384) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][600/625] eta 0:00:09 lr 0.000040 wd 0.0500 time 0.3723 (0.3784) data time 0.0008 (0.0019) model time 0.3716 (0.3768) loss 1.4853 (2.5899) grad_norm 2.8604 (3.3351) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][610/625] eta 0:00:05 lr 0.000040 wd 0.0500 time 0.3643 (0.3783) data time 0.0006 (0.0018) model time 0.3637 (0.3767) loss 2.3282 (2.5922) grad_norm 2.3294 (3.3250) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [281/300][620/625] eta 0:00:01 lr 0.000040 wd 0.0500 time 0.3623 (0.3780) data time 0.0005 (0.0018) model time 0.3617 (0.3764) loss 2.7036 (2.5927) grad_norm 2.4447 (3.3103) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:20:45 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 281 training takes 0:03:56 [2024-09-01 15:20:45 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:20:47 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.508 (0.508) Loss 0.5137 (0.5137) Acc@1 89.160 (89.160) Acc@5 98.438 (98.438) Mem 11224MB [2024-09-01 15:20:48 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.146) Loss 0.8188 (0.6062) Acc@1 81.152 (87.043) Acc@5 95.850 (97.820) Mem 11224MB [2024-09-01 15:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.121) Loss 0.9077 (0.7266) Acc@1 79.053 (84.115) Acc@5 95.166 (96.610) Mem 11224MB [2024-09-01 15:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.819 Acc@5 96.623 [2024-09-01 15:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 15:20:49 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.82% [2024-09-01 15:20:49 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 15:20:54 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 15:20:55 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.483 (0.483) Loss 0.5054 (0.5054) Acc@1 88.965 (88.965) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:20:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.140) Loss 0.8203 (0.5992) Acc@1 80.811 (87.087) Acc@5 95.801 (97.878) Mem 11224MB [2024-09-01 15:20:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.118) Loss 0.8994 (0.7149) Acc@1 78.613 (84.222) Acc@5 95.312 (96.712) Mem 11224MB [2024-09-01 15:20:57 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.931 Acc@5 96.731 [2024-09-01 15:20:57 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:20:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][0/625] eta 0:13:04 lr 0.000040 wd 0.0500 time 1.2545 (1.2545) data time 0.5225 (0.5225) model time 0.0000 (0.0000) loss 2.9177 (2.9177) grad_norm 6.2740 (6.2740) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][10/625] eta 0:04:40 lr 0.000040 wd 0.0500 time 0.3645 (0.4563) data time 0.0011 (0.0484) model time 0.0000 (0.0000) loss 2.9393 (2.4470) grad_norm 2.7912 (2.9374) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][20/625] eta 0:04:11 lr 0.000040 wd 0.0500 time 0.3667 (0.4155) data time 0.0010 (0.0258) model time 0.0000 (0.0000) loss 2.2507 (2.4971) grad_norm 3.4586 (2.8085) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][30/625] eta 0:04:03 lr 0.000040 wd 0.0500 time 0.5847 (0.4089) data time 0.0010 (0.0178) model time 0.0000 (0.0000) loss 2.9948 (2.5375) grad_norm 2.1836 (2.9259) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][40/625] eta 0:03:59 lr 0.000040 wd 0.0500 time 0.3652 (0.4099) data time 0.0008 (0.0137) model time 0.0000 (0.0000) loss 2.7663 (2.5173) grad_norm 2.4340 (2.8575) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][50/625] eta 0:03:51 lr 0.000040 wd 0.0500 time 0.3954 (0.4031) data time 0.0010 (0.0113) model time 0.0000 (0.0000) loss 2.4987 (2.5469) grad_norm 2.7826 (2.8355) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][60/625] eta 0:03:45 lr 0.000040 wd 0.0500 time 0.3765 (0.3984) data time 0.0008 (0.0096) model time 0.3757 (0.3732) loss 2.6866 (2.5903) grad_norm 2.4183 (2.9097) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][70/625] eta 0:03:39 lr 0.000040 wd 0.0500 time 0.3688 (0.3951) data time 0.0011 (0.0084) model time 0.3677 (0.3736) loss 2.2589 (2.5555) grad_norm 1.9971 (2.8288) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][80/625] eta 0:03:34 lr 0.000040 wd 0.0500 time 0.3740 (0.3927) data time 0.0008 (0.0075) model time 0.3733 (0.3739) loss 2.4755 (2.5299) grad_norm 14.4789 (3.0248) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][90/625] eta 0:03:29 lr 0.000040 wd 0.0500 time 0.3768 (0.3911) data time 0.0010 (0.0068) model time 0.3758 (0.3747) loss 2.4554 (2.5289) grad_norm 2.6397 (3.0980) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][100/625] eta 0:03:24 lr 0.000040 wd 0.0500 time 0.3699 (0.3894) data time 0.0008 (0.0062) model time 0.3691 (0.3743) loss 2.7319 (2.5442) grad_norm 3.0593 (3.0612) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][110/625] eta 0:03:20 lr 0.000040 wd 0.0500 time 0.3720 (0.3892) data time 0.0009 (0.0058) model time 0.3711 (0.3763) loss 1.5652 (2.5340) grad_norm 8.6127 (3.0622) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][120/625] eta 0:03:16 lr 0.000040 wd 0.0500 time 0.3809 (0.3882) data time 0.0008 (0.0054) model time 0.3801 (0.3762) loss 2.3690 (2.5416) grad_norm 2.3899 (3.0576) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][130/625] eta 0:03:11 lr 0.000040 wd 0.0500 time 0.3767 (0.3871) data time 0.0008 (0.0051) model time 0.3759 (0.3758) loss 2.5859 (2.5443) grad_norm 3.4598 (3.0446) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][140/625] eta 0:03:07 lr 0.000040 wd 0.0500 time 0.3702 (0.3864) data time 0.0011 (0.0048) model time 0.3691 (0.3758) loss 2.7501 (2.5525) grad_norm 2.0728 (3.0367) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][150/625] eta 0:03:03 lr 0.000040 wd 0.0500 time 0.3922 (0.3859) data time 0.0008 (0.0045) model time 0.3914 (0.3761) loss 1.3830 (2.5426) grad_norm 2.6813 (3.0863) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:21:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][160/625] eta 0:02:59 lr 0.000040 wd 0.0500 time 0.3769 (0.3853) data time 0.0008 (0.0043) model time 0.3761 (0.3760) loss 2.0465 (2.5338) grad_norm 2.8857 (3.5211) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][170/625] eta 0:02:55 lr 0.000040 wd 0.0500 time 0.3799 (0.3849) data time 0.0007 (0.0041) model time 0.3792 (0.3762) loss 2.7670 (2.5559) grad_norm 2.8549 (3.4759) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][180/625] eta 0:02:51 lr 0.000039 wd 0.0500 time 0.3728 (0.3844) data time 0.0012 (0.0039) model time 0.3716 (0.3761) loss 2.7968 (2.5571) grad_norm 4.2661 (3.4629) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][190/625] eta 0:02:47 lr 0.000039 wd 0.0500 time 0.3656 (0.3839) data time 0.0008 (0.0038) model time 0.3648 (0.3759) loss 2.5745 (2.5668) grad_norm 3.9777 (3.4800) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][200/625] eta 0:02:43 lr 0.000039 wd 0.0500 time 0.3805 (0.3836) data time 0.0010 (0.0036) model time 0.3795 (0.3759) loss 2.5034 (2.5572) grad_norm 3.0024 (3.4493) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][210/625] eta 0:02:39 lr 0.000039 wd 0.0500 time 0.3832 (0.3832) data time 0.0012 (0.0035) model time 0.3819 (0.3758) loss 3.1002 (2.5656) grad_norm 7.8343 (3.4416) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][220/625] eta 0:02:35 lr 0.000039 wd 0.0500 time 0.3672 (0.3828) data time 0.0010 (0.0034) model time 0.3662 (0.3757) loss 2.9449 (2.5612) grad_norm 3.2513 (3.3984) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][230/625] eta 0:02:31 lr 0.000039 wd 0.0500 time 0.3783 (0.3826) data time 0.0011 (0.0033) model time 0.3772 (0.3758) loss 2.7175 (2.5556) grad_norm 6.7342 (3.3945) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][240/625] eta 0:02:27 lr 0.000039 wd 0.0500 time 0.3771 (0.3831) data time 0.0009 (0.0032) model time 0.3762 (0.3767) loss 2.2741 (2.5612) grad_norm 4.7747 (3.4010) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:22:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][250/625] eta 0:02:24 lr 0.000039 wd 0.0500 time 0.3791 (0.3846) data time 0.0010 (0.0032) model time 0.3781 (0.3787) loss 2.5340 (2.5720) grad_norm 3.2491 (3.4145) loss_scale 256.0000 (132.5896) mem 11224MB [2024-09-01 15:22:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][260/625] eta 0:02:20 lr 0.000039 wd 0.0500 time 0.3685 (0.3841) data time 0.0011 (0.0031) model time 0.3674 (0.3784) loss 2.0695 (2.5728) grad_norm 3.6372 (3.4117) loss_scale 256.0000 (137.3180) mem 11224MB [2024-09-01 15:22:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][270/625] eta 0:02:16 lr 0.000039 wd 0.0500 time 0.3652 (0.3836) data time 0.0009 (0.0030) model time 0.3643 (0.3780) loss 3.1618 (2.5766) grad_norm 3.2625 (inf) loss_scale 128.0000 (137.4465) mem 11224MB [2024-09-01 15:22:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][280/625] eta 0:02:12 lr 0.000039 wd 0.0500 time 0.3741 (0.3833) data time 0.0009 (0.0029) model time 0.3732 (0.3779) loss 3.1628 (2.5781) grad_norm 2.6053 (inf) loss_scale 128.0000 (137.1103) mem 11224MB [2024-09-01 15:22:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][290/625] eta 0:02:08 lr 0.000039 wd 0.0500 time 0.3349 (0.3836) data time 0.0011 (0.0029) model time 0.3338 (0.3783) loss 2.3066 (2.5750) grad_norm 2.2271 (inf) loss_scale 128.0000 (136.7973) mem 11224MB [2024-09-01 15:22:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][300/625] eta 0:02:04 lr 0.000039 wd 0.0500 time 0.3754 (0.3834) data time 0.0008 (0.0028) model time 0.3745 (0.3783) loss 2.8283 (2.5758) grad_norm 2.2085 (inf) loss_scale 128.0000 (136.5050) mem 11224MB [2024-09-01 15:22:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][310/625] eta 0:02:00 lr 0.000039 wd 0.0500 time 0.3730 (0.3832) data time 0.0008 (0.0028) model time 0.3723 (0.3782) loss 2.7148 (2.5834) grad_norm 4.0570 (inf) loss_scale 128.0000 (136.2315) mem 11224MB [2024-09-01 15:23:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][320/625] eta 0:01:56 lr 0.000039 wd 0.0500 time 0.3711 (0.3830) data time 0.0010 (0.0027) model time 0.3701 (0.3781) loss 2.5313 (2.5840) grad_norm 2.5462 (inf) loss_scale 128.0000 (135.9751) mem 11224MB [2024-09-01 15:23:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][330/625] eta 0:01:52 lr 0.000039 wd 0.0500 time 0.3731 (0.3828) data time 0.0008 (0.0027) model time 0.3723 (0.3780) loss 2.8910 (2.5831) grad_norm 3.6577 (inf) loss_scale 128.0000 (135.7341) mem 11224MB [2024-09-01 15:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][340/625] eta 0:01:49 lr 0.000039 wd 0.0500 time 0.3742 (0.3825) data time 0.0008 (0.0026) model time 0.3734 (0.3779) loss 2.5165 (2.5847) grad_norm 4.3558 (inf) loss_scale 128.0000 (135.5073) mem 11224MB [2024-09-01 15:23:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][350/625] eta 0:01:45 lr 0.000039 wd 0.0500 time 0.3818 (0.3824) data time 0.0011 (0.0026) model time 0.3808 (0.3778) loss 2.8409 (2.5807) grad_norm 1.8701 (inf) loss_scale 128.0000 (135.2934) mem 11224MB [2024-09-01 15:23:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][360/625] eta 0:01:41 lr 0.000039 wd 0.0500 time 0.3818 (0.3822) data time 0.0008 (0.0025) model time 0.3810 (0.3777) loss 3.1886 (2.5815) grad_norm 3.6025 (inf) loss_scale 128.0000 (135.0914) mem 11224MB [2024-09-01 15:23:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][370/625] eta 0:01:37 lr 0.000039 wd 0.0500 time 0.3724 (0.3821) data time 0.0014 (0.0025) model time 0.3710 (0.3776) loss 3.0531 (2.5863) grad_norm 3.3169 (inf) loss_scale 128.0000 (134.9003) mem 11224MB [2024-09-01 15:23:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][380/625] eta 0:01:33 lr 0.000039 wd 0.0500 time 0.3772 (0.3820) data time 0.0011 (0.0024) model time 0.3761 (0.3776) loss 2.9710 (2.5794) grad_norm 3.0411 (inf) loss_scale 128.0000 (134.7192) mem 11224MB [2024-09-01 15:23:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][390/625] eta 0:01:29 lr 0.000039 wd 0.0500 time 0.3785 (0.3819) data time 0.0012 (0.0024) model time 0.3773 (0.3776) loss 2.7411 (2.5809) grad_norm 2.8241 (inf) loss_scale 128.0000 (134.5473) mem 11224MB [2024-09-01 15:23:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][400/625] eta 0:01:25 lr 0.000039 wd 0.0500 time 0.3687 (0.3818) data time 0.0010 (0.0024) model time 0.3677 (0.3776) loss 2.8281 (2.5839) grad_norm 3.3462 (inf) loss_scale 128.0000 (134.3840) mem 11224MB [2024-09-01 15:23:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][410/625] eta 0:01:22 lr 0.000039 wd 0.0500 time 0.3330 (0.3821) data time 0.0012 (0.0023) model time 0.3318 (0.3780) loss 2.9728 (2.5925) grad_norm 2.2110 (inf) loss_scale 128.0000 (134.2287) mem 11224MB [2024-09-01 15:23:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][420/625] eta 0:01:18 lr 0.000039 wd 0.0500 time 0.3716 (0.3820) data time 0.0011 (0.0023) model time 0.3705 (0.3780) loss 3.0014 (2.5943) grad_norm 3.6143 (inf) loss_scale 128.0000 (134.0808) mem 11224MB [2024-09-01 15:23:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][430/625] eta 0:01:14 lr 0.000039 wd 0.0500 time 0.3729 (0.3838) data time 0.0010 (0.0023) model time 0.3719 (0.3801) loss 2.7873 (2.5967) grad_norm 2.3897 (inf) loss_scale 128.0000 (133.9397) mem 11224MB [2024-09-01 15:23:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][440/625] eta 0:01:10 lr 0.000039 wd 0.0500 time 0.3899 (0.3836) data time 0.0010 (0.0023) model time 0.3890 (0.3800) loss 1.8836 (2.5922) grad_norm 4.0835 (inf) loss_scale 128.0000 (133.8050) mem 11224MB [2024-09-01 15:23:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][450/625] eta 0:01:07 lr 0.000039 wd 0.0500 time 0.3794 (0.3835) data time 0.0010 (0.0022) model time 0.3784 (0.3799) loss 2.6646 (2.5875) grad_norm 3.7093 (inf) loss_scale 128.0000 (133.6763) mem 11224MB [2024-09-01 15:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][460/625] eta 0:01:03 lr 0.000039 wd 0.0500 time 0.3739 (0.3833) data time 0.0010 (0.0022) model time 0.3730 (0.3797) loss 2.5499 (2.5905) grad_norm 4.8857 (inf) loss_scale 128.0000 (133.5531) mem 11224MB [2024-09-01 15:23:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][470/625] eta 0:00:59 lr 0.000038 wd 0.0500 time 0.3734 (0.3830) data time 0.0010 (0.0022) model time 0.3723 (0.3795) loss 2.1670 (2.5911) grad_norm 1.8376 (inf) loss_scale 128.0000 (133.4352) mem 11224MB [2024-09-01 15:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][480/625] eta 0:00:55 lr 0.000038 wd 0.0500 time 0.3732 (0.3829) data time 0.0007 (0.0022) model time 0.3724 (0.3795) loss 2.5892 (2.5974) grad_norm 3.5257 (inf) loss_scale 128.0000 (133.3222) mem 11224MB [2024-09-01 15:24:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][490/625] eta 0:00:51 lr 0.000038 wd 0.0500 time 0.3797 (0.3828) data time 0.0010 (0.0021) model time 0.3787 (0.3794) loss 2.8719 (2.5991) grad_norm 2.5491 (inf) loss_scale 128.0000 (133.2138) mem 11224MB [2024-09-01 15:24:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][500/625] eta 0:00:47 lr 0.000038 wd 0.0500 time 0.3726 (0.3827) data time 0.0009 (0.0021) model time 0.3717 (0.3793) loss 2.8142 (2.6013) grad_norm 2.7391 (inf) loss_scale 128.0000 (133.1098) mem 11224MB [2024-09-01 15:24:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][510/625] eta 0:00:43 lr 0.000038 wd 0.0500 time 0.3822 (0.3826) data time 0.0008 (0.0021) model time 0.3814 (0.3792) loss 3.0490 (2.6028) grad_norm 3.2731 (inf) loss_scale 128.0000 (133.0098) mem 11224MB [2024-09-01 15:24:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][520/625] eta 0:00:40 lr 0.000038 wd 0.0500 time 0.3839 (0.3825) data time 0.0008 (0.0021) model time 0.3831 (0.3792) loss 2.6199 (2.5994) grad_norm 7.4418 (inf) loss_scale 128.0000 (132.9136) mem 11224MB [2024-09-01 15:24:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][530/625] eta 0:00:36 lr 0.000038 wd 0.0500 time 0.3841 (0.3824) data time 0.0008 (0.0020) model time 0.3834 (0.3792) loss 3.1599 (2.6028) grad_norm 2.6956 (inf) loss_scale 128.0000 (132.8211) mem 11224MB [2024-09-01 15:24:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][540/625] eta 0:00:32 lr 0.000038 wd 0.0500 time 0.3787 (0.3824) data time 0.0010 (0.0020) model time 0.3777 (0.3792) loss 2.7825 (2.6043) grad_norm 4.5325 (inf) loss_scale 128.0000 (132.7320) mem 11224MB [2024-09-01 15:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][550/625] eta 0:00:28 lr 0.000038 wd 0.0500 time 0.3763 (0.3838) data time 0.0008 (0.0020) model time 0.3755 (0.3807) loss 3.0147 (2.6032) grad_norm 3.5316 (inf) loss_scale 128.0000 (132.6461) mem 11224MB [2024-09-01 15:24:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][560/625] eta 0:00:24 lr 0.000038 wd 0.0500 time 0.3878 (0.3836) data time 0.0010 (0.0020) model time 0.3868 (0.3806) loss 2.8696 (2.6031) grad_norm 1.8350 (inf) loss_scale 128.0000 (132.5633) mem 11224MB [2024-09-01 15:24:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][570/625] eta 0:00:21 lr 0.000038 wd 0.0500 time 0.3724 (0.3835) data time 0.0010 (0.0020) model time 0.3713 (0.3805) loss 2.9475 (2.6020) grad_norm 2.3260 (inf) loss_scale 128.0000 (132.4834) mem 11224MB [2024-09-01 15:24:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][580/625] eta 0:00:17 lr 0.000038 wd 0.0500 time 0.3771 (0.3834) data time 0.0009 (0.0020) model time 0.3762 (0.3804) loss 2.2782 (2.5997) grad_norm 3.7875 (inf) loss_scale 128.0000 (132.4062) mem 11224MB [2024-09-01 15:24:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][590/625] eta 0:00:13 lr 0.000038 wd 0.0500 time 0.3858 (0.3833) data time 0.0008 (0.0019) model time 0.3850 (0.3804) loss 2.3542 (2.6025) grad_norm 3.1653 (inf) loss_scale 128.0000 (132.3316) mem 11224MB [2024-09-01 15:24:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][600/625] eta 0:00:09 lr 0.000038 wd 0.0500 time 0.3800 (0.3832) data time 0.0007 (0.0019) model time 0.3792 (0.3803) loss 2.7264 (2.6028) grad_norm 2.1673 (inf) loss_scale 128.0000 (132.2596) mem 11224MB [2024-09-01 15:24:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][610/625] eta 0:00:05 lr 0.000038 wd 0.0500 time 0.3635 (0.3831) data time 0.0005 (0.0019) model time 0.3629 (0.3802) loss 1.4830 (2.5964) grad_norm 3.4348 (inf) loss_scale 128.0000 (132.1899) mem 11224MB [2024-09-01 15:24:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [282/300][620/625] eta 0:00:01 lr 0.000038 wd 0.0500 time 0.3700 (0.3831) data time 0.0008 (0.0019) model time 0.3692 (0.3803) loss 2.0599 (2.5933) grad_norm 3.3769 (inf) loss_scale 128.0000 (132.1224) mem 11224MB [2024-09-01 15:24:56 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 282 training takes 0:03:59 [2024-09-01 15:24:56 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:24:58 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:24:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.506 (0.506) Loss 0.5117 (0.5117) Acc@1 88.867 (88.867) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 15:24:59 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.8320 (0.6064) Acc@1 80.811 (86.998) Acc@5 95.605 (97.807) Mem 11224MB [2024-09-01 15:25:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9106 (0.7270) Acc@1 79.102 (84.070) Acc@5 95.410 (96.617) Mem 11224MB [2024-09-01 15:25:00 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.839 Acc@5 96.627 [2024-09-01 15:25:00 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 15:25:00 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.84% [2024-09-01 15:25:00 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 15:25:02 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 15:25:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 0.5059 (0.5059) Acc@1 88.965 (88.965) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:25:03 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.152) Loss 0.8203 (0.5997) Acc@1 80.762 (87.056) Acc@5 95.850 (97.892) Mem 11224MB [2024-09-01 15:25:04 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.124) Loss 0.8979 (0.7156) Acc@1 78.662 (84.205) Acc@5 95.312 (96.726) Mem 11224MB [2024-09-01 15:25:05 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.913 Acc@5 96.741 [2024-09-01 15:25:05 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:25:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][0/625] eta 0:12:34 lr 0.000038 wd 0.0500 time 1.2069 (1.2069) data time 0.5582 (0.5582) model time 0.0000 (0.0000) loss 2.4504 (2.4504) grad_norm 2.8375 (2.8375) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][10/625] eta 0:04:36 lr 0.000038 wd 0.0500 time 0.3742 (0.4503) data time 0.0010 (0.0517) model time 0.0000 (0.0000) loss 2.8807 (2.7975) grad_norm 2.6609 (3.3145) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][20/625] eta 0:04:09 lr 0.000038 wd 0.0500 time 0.3681 (0.4131) data time 0.0010 (0.0276) model time 0.0000 (0.0000) loss 2.7072 (2.6737) grad_norm 3.0832 (3.9484) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][30/625] eta 0:03:58 lr 0.000038 wd 0.0500 time 0.3732 (0.4005) data time 0.0007 (0.0190) model time 0.0000 (0.0000) loss 1.8816 (2.6524) grad_norm 5.0466 (3.8489) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][40/625] eta 0:03:50 lr 0.000038 wd 0.0500 time 0.3712 (0.3947) data time 0.0012 (0.0147) model time 0.0000 (0.0000) loss 2.8266 (2.6322) grad_norm 2.3092 (3.7277) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][50/625] eta 0:03:44 lr 0.000038 wd 0.0500 time 0.3691 (0.3903) data time 0.0010 (0.0121) model time 0.0000 (0.0000) loss 3.0404 (2.6080) grad_norm 2.0554 (3.6356) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][60/625] eta 0:03:38 lr 0.000038 wd 0.0500 time 0.3758 (0.3874) data time 0.0009 (0.0103) model time 0.3749 (0.3718) loss 2.0845 (2.5851) grad_norm 1.6341 (3.4536) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][70/625] eta 0:03:34 lr 0.000038 wd 0.0500 time 0.3808 (0.3860) data time 0.0008 (0.0090) model time 0.3801 (0.3741) loss 1.7593 (2.5420) grad_norm 4.0080 (3.5262) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][80/625] eta 0:03:35 lr 0.000038 wd 0.0500 time 0.5955 (0.3953) data time 0.0011 (0.0080) model time 0.5944 (0.4029) loss 2.9058 (2.5551) grad_norm 5.4022 (3.5560) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][90/625] eta 0:03:29 lr 0.000038 wd 0.0500 time 0.3735 (0.3923) data time 0.0008 (0.0072) model time 0.3727 (0.3939) loss 3.1166 (2.5690) grad_norm 2.8817 (3.5082) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][100/625] eta 0:03:24 lr 0.000038 wd 0.0500 time 0.3726 (0.3903) data time 0.0010 (0.0066) model time 0.3715 (0.3894) loss 2.9751 (2.5725) grad_norm 2.4971 (3.4379) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][110/625] eta 0:03:20 lr 0.000038 wd 0.0500 time 0.3715 (0.3891) data time 0.0009 (0.0061) model time 0.3705 (0.3870) loss 2.1197 (2.5907) grad_norm 2.7626 (3.3593) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][120/625] eta 0:03:15 lr 0.000038 wd 0.0500 time 0.3775 (0.3880) data time 0.0008 (0.0057) model time 0.3767 (0.3852) loss 2.8759 (2.5893) grad_norm 2.6976 (3.3646) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][130/625] eta 0:03:11 lr 0.000038 wd 0.0500 time 0.3772 (0.3872) data time 0.0008 (0.0053) model time 0.3764 (0.3842) loss 2.1384 (2.5752) grad_norm 2.2383 (3.3245) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][140/625] eta 0:03:07 lr 0.000037 wd 0.0500 time 0.3788 (0.3865) data time 0.0008 (0.0050) model time 0.3780 (0.3833) loss 2.3255 (2.5909) grad_norm 2.6336 (3.2944) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][150/625] eta 0:03:03 lr 0.000037 wd 0.0500 time 0.3807 (0.3858) data time 0.0010 (0.0048) model time 0.3797 (0.3824) loss 2.0236 (2.5618) grad_norm 3.4649 (3.2974) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][160/625] eta 0:02:59 lr 0.000037 wd 0.0500 time 0.3721 (0.3852) data time 0.0008 (0.0045) model time 0.3713 (0.3818) loss 1.6808 (2.5541) grad_norm 3.0563 (3.2700) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][170/625] eta 0:02:54 lr 0.000037 wd 0.0500 time 0.3773 (0.3845) data time 0.0008 (0.0043) model time 0.3765 (0.3811) loss 2.0074 (2.5569) grad_norm 2.9805 (3.2869) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][180/625] eta 0:02:50 lr 0.000037 wd 0.0500 time 0.3850 (0.3840) data time 0.0010 (0.0041) model time 0.3840 (0.3806) loss 2.6866 (2.5552) grad_norm 4.5031 (3.3005) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][190/625] eta 0:02:46 lr 0.000037 wd 0.0500 time 0.3755 (0.3834) data time 0.0008 (0.0040) model time 0.3747 (0.3799) loss 2.9497 (2.5600) grad_norm 3.4887 (3.4119) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][200/625] eta 0:02:42 lr 0.000037 wd 0.0500 time 0.3751 (0.3831) data time 0.0010 (0.0038) model time 0.3741 (0.3797) loss 2.8068 (2.5661) grad_norm 2.5906 (3.3765) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][210/625] eta 0:02:38 lr 0.000037 wd 0.0500 time 0.3755 (0.3828) data time 0.0008 (0.0037) model time 0.3747 (0.3794) loss 3.1709 (2.5748) grad_norm 2.8531 (3.3880) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][220/625] eta 0:02:34 lr 0.000037 wd 0.0500 time 0.3806 (0.3824) data time 0.0008 (0.0036) model time 0.3798 (0.3790) loss 1.8629 (2.5688) grad_norm 2.3014 (3.4868) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][230/625] eta 0:02:30 lr 0.000037 wd 0.0500 time 0.3776 (0.3821) data time 0.0010 (0.0035) model time 0.3766 (0.3788) loss 2.8413 (2.5690) grad_norm 3.5934 (3.4787) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][240/625] eta 0:02:27 lr 0.000037 wd 0.0500 time 0.3809 (0.3819) data time 0.0010 (0.0033) model time 0.3798 (0.3786) loss 2.9084 (2.5720) grad_norm 2.8279 (3.4636) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][250/625] eta 0:02:23 lr 0.000037 wd 0.0500 time 0.3806 (0.3818) data time 0.0009 (0.0033) model time 0.3797 (0.3786) loss 3.1129 (2.5713) grad_norm 2.3737 (3.6116) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][260/625] eta 0:02:19 lr 0.000037 wd 0.0500 time 0.3794 (0.3816) data time 0.0008 (0.0032) model time 0.3786 (0.3784) loss 1.8642 (2.5589) grad_norm 4.1246 (3.6262) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][270/625] eta 0:02:15 lr 0.000037 wd 0.0500 time 0.3748 (0.3813) data time 0.0010 (0.0031) model time 0.3739 (0.3782) loss 2.7866 (2.5638) grad_norm 1.9314 (3.7678) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][280/625] eta 0:02:11 lr 0.000037 wd 0.0500 time 0.3770 (0.3812) data time 0.0011 (0.0030) model time 0.3759 (0.3781) loss 1.9447 (2.5659) grad_norm 2.5541 (3.7410) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:26:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][290/625] eta 0:02:08 lr 0.000037 wd 0.0500 time 0.5575 (0.3838) data time 0.0012 (0.0030) model time 0.5563 (0.3813) loss 2.6368 (2.5622) grad_norm 2.1651 (3.7421) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][300/625] eta 0:02:04 lr 0.000037 wd 0.0500 time 0.3743 (0.3835) data time 0.0008 (0.0029) model time 0.3735 (0.3811) loss 2.8005 (2.5731) grad_norm 3.0094 (3.7130) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][310/625] eta 0:02:00 lr 0.000037 wd 0.0500 time 0.3721 (0.3831) data time 0.0011 (0.0028) model time 0.3710 (0.3807) loss 2.4777 (2.5714) grad_norm 2.4504 (3.7033) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][320/625] eta 0:01:56 lr 0.000037 wd 0.0500 time 0.3795 (0.3829) data time 0.0010 (0.0028) model time 0.3785 (0.3804) loss 2.8946 (2.5706) grad_norm 2.3732 (3.6696) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][330/625] eta 0:01:52 lr 0.000037 wd 0.0500 time 0.3868 (0.3827) data time 0.0010 (0.0027) model time 0.3858 (0.3803) loss 2.7679 (2.5733) grad_norm 3.3631 (3.6345) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][340/625] eta 0:01:49 lr 0.000037 wd 0.0500 time 0.3836 (0.3826) data time 0.0009 (0.0027) model time 0.3827 (0.3803) loss 2.2402 (2.5758) grad_norm 4.4734 (3.6242) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][350/625] eta 0:01:45 lr 0.000037 wd 0.0500 time 0.3779 (0.3825) data time 0.0010 (0.0026) model time 0.3769 (0.3801) loss 2.7031 (2.5752) grad_norm 3.4542 (3.6130) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][360/625] eta 0:01:41 lr 0.000037 wd 0.0500 time 0.3726 (0.3823) data time 0.0010 (0.0026) model time 0.3715 (0.3800) loss 2.8905 (2.5746) grad_norm 4.7847 (3.6016) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][370/625] eta 0:01:37 lr 0.000037 wd 0.0500 time 0.3731 (0.3821) data time 0.0011 (0.0025) model time 0.3720 (0.3798) loss 2.5812 (2.5728) grad_norm 3.6284 (3.5934) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][380/625] eta 0:01:33 lr 0.000037 wd 0.0500 time 0.3782 (0.3820) data time 0.0012 (0.0025) model time 0.3769 (0.3797) loss 2.4847 (2.5766) grad_norm 2.5306 (3.5764) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][390/625] eta 0:01:29 lr 0.000037 wd 0.0500 time 0.3800 (0.3819) data time 0.0008 (0.0025) model time 0.3792 (0.3796) loss 2.9572 (2.5800) grad_norm 3.0965 (3.5635) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][400/625] eta 0:01:25 lr 0.000037 wd 0.0500 time 0.3799 (0.3817) data time 0.0011 (0.0024) model time 0.3787 (0.3795) loss 2.1940 (2.5729) grad_norm 2.2647 (3.5449) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][410/625] eta 0:01:22 lr 0.000037 wd 0.0500 time 0.3803 (0.3817) data time 0.0011 (0.0024) model time 0.3792 (0.3794) loss 2.5427 (2.5766) grad_norm 2.8806 (3.5513) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][420/625] eta 0:01:18 lr 0.000037 wd 0.0500 time 0.3773 (0.3816) data time 0.0008 (0.0024) model time 0.3764 (0.3794) loss 1.6791 (2.5692) grad_norm 2.0932 (3.6026) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][430/625] eta 0:01:14 lr 0.000037 wd 0.0500 time 0.3748 (0.3815) data time 0.0010 (0.0023) model time 0.3737 (0.3792) loss 2.9115 (2.5660) grad_norm 3.1813 (3.6061) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][440/625] eta 0:01:10 lr 0.000037 wd 0.0500 time 0.3943 (0.3814) data time 0.0008 (0.0023) model time 0.3936 (0.3792) loss 2.4815 (2.5673) grad_norm 2.3678 (3.6189) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:27:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][450/625] eta 0:01:06 lr 0.000036 wd 0.0500 time 0.3743 (0.3812) data time 0.0009 (0.0023) model time 0.3734 (0.3791) loss 1.6722 (2.5638) grad_norm 2.2467 (3.5979) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][460/625] eta 0:01:02 lr 0.000036 wd 0.0500 time 0.3769 (0.3811) data time 0.0010 (0.0023) model time 0.3759 (0.3790) loss 2.8525 (2.5640) grad_norm 2.4371 (3.5739) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][470/625] eta 0:00:59 lr 0.000036 wd 0.0500 time 0.3805 (0.3824) data time 0.0008 (0.0022) model time 0.3797 (0.3804) loss 2.2141 (2.5575) grad_norm 3.2784 (3.5651) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][480/625] eta 0:00:55 lr 0.000036 wd 0.0500 time 0.3754 (0.3826) data time 0.0008 (0.0022) model time 0.3746 (0.3807) loss 3.2736 (2.5622) grad_norm 5.1563 (3.5507) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][490/625] eta 0:00:51 lr 0.000036 wd 0.0500 time 0.3779 (0.3825) data time 0.0008 (0.0022) model time 0.3770 (0.3805) loss 3.0321 (2.5574) grad_norm 2.6308 (3.5350) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][500/625] eta 0:00:47 lr 0.000036 wd 0.0500 time 0.3785 (0.3823) data time 0.0008 (0.0022) model time 0.3777 (0.3804) loss 1.6288 (2.5565) grad_norm 2.0976 (3.6280) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][510/625] eta 0:00:43 lr 0.000036 wd 0.0500 time 0.3779 (0.3822) data time 0.0010 (0.0021) model time 0.3768 (0.3803) loss 2.7214 (2.5623) grad_norm 2.9845 (3.6072) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][520/625] eta 0:00:40 lr 0.000036 wd 0.0500 time 0.3780 (0.3821) data time 0.0010 (0.0021) model time 0.3770 (0.3802) loss 2.9579 (2.5627) grad_norm 2.6263 (3.5907) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][530/625] eta 0:00:36 lr 0.000036 wd 0.0500 time 0.3711 (0.3820) data time 0.0010 (0.0021) model time 0.3701 (0.3801) loss 2.4042 (2.5644) grad_norm 3.3982 (3.5831) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][540/625] eta 0:00:32 lr 0.000036 wd 0.0500 time 0.3813 (0.3819) data time 0.0010 (0.0021) model time 0.3804 (0.3800) loss 2.5906 (2.5669) grad_norm 2.9550 (3.5812) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][550/625] eta 0:00:28 lr 0.000036 wd 0.0500 time 0.3709 (0.3818) data time 0.0009 (0.0021) model time 0.3700 (0.3799) loss 2.9416 (2.5684) grad_norm 5.0573 (3.5896) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][560/625] eta 0:00:24 lr 0.000036 wd 0.0500 time 0.3759 (0.3817) data time 0.0012 (0.0020) model time 0.3746 (0.3798) loss 2.1921 (2.5683) grad_norm 6.2767 (3.5976) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][570/625] eta 0:00:20 lr 0.000036 wd 0.0500 time 0.3807 (0.3816) data time 0.0010 (0.0020) model time 0.3796 (0.3797) loss 2.8727 (2.5678) grad_norm 4.2273 (3.5915) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][580/625] eta 0:00:17 lr 0.000036 wd 0.0500 time 0.3702 (0.3814) data time 0.0011 (0.0020) model time 0.3692 (0.3795) loss 2.7921 (2.5706) grad_norm 3.6327 (3.5803) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][590/625] eta 0:00:13 lr 0.000036 wd 0.0500 time 0.3809 (0.3824) data time 0.0013 (0.0020) model time 0.3796 (0.3806) loss 2.7156 (2.5708) grad_norm 3.3576 (3.5753) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][600/625] eta 0:00:09 lr 0.000036 wd 0.0500 time 0.3765 (0.3826) data time 0.0009 (0.0020) model time 0.3755 (0.3808) loss 2.3107 (2.5696) grad_norm 39.5993 (3.6263) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:28:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][610/625] eta 0:00:05 lr 0.000036 wd 0.0500 time 0.3670 (0.3824) data time 0.0005 (0.0020) model time 0.3665 (0.3807) loss 2.0978 (2.5695) grad_norm 3.0449 (3.6175) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [283/300][620/625] eta 0:00:01 lr 0.000036 wd 0.0500 time 0.3586 (0.3821) data time 0.0005 (0.0019) model time 0.3580 (0.3803) loss 2.5966 (2.5709) grad_norm 2.8382 (3.6059) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:03 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 283 training takes 0:03:58 [2024-09-01 15:29:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:29:04 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:29:05 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.509 (0.509) Loss 0.5142 (0.5142) Acc@1 89.014 (89.014) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 15:29:06 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.142) Loss 0.8379 (0.6079) Acc@1 80.518 (86.963) Acc@5 95.703 (97.812) Mem 11224MB [2024-09-01 15:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9248 (0.7280) Acc@1 78.613 (84.036) Acc@5 95.068 (96.638) Mem 11224MB [2024-09-01 15:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.735 Acc@5 96.645 [2024-09-01 15:29:07 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 15:29:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.834 (0.834) Loss 0.5068 (0.5068) Acc@1 89.014 (89.014) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:29:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.177) Loss 0.8208 (0.6001) Acc@1 80.908 (87.078) Acc@5 95.850 (97.892) Mem 11224MB [2024-09-01 15:29:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.8994 (0.7161) Acc@1 78.760 (84.217) Acc@5 95.312 (96.724) Mem 11224MB [2024-09-01 15:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.925 Acc@5 96.737 [2024-09-01 15:29:11 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:29:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][0/625] eta 0:12:44 lr 0.000036 wd 0.0500 time 1.2239 (1.2239) data time 0.5858 (0.5858) model time 0.0000 (0.0000) loss 2.7695 (2.7695) grad_norm 3.1039 (3.1039) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][10/625] eta 0:04:39 lr 0.000036 wd 0.0500 time 0.3800 (0.4549) data time 0.0011 (0.0542) model time 0.0000 (0.0000) loss 2.8489 (2.6365) grad_norm 3.4062 (3.2135) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][20/625] eta 0:04:12 lr 0.000036 wd 0.0500 time 0.3661 (0.4166) data time 0.0008 (0.0289) model time 0.0000 (0.0000) loss 2.0421 (2.6581) grad_norm 2.1644 (3.4982) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][30/625] eta 0:03:59 lr 0.000036 wd 0.0500 time 0.3620 (0.4026) data time 0.0011 (0.0199) model time 0.0000 (0.0000) loss 2.6865 (2.6705) grad_norm 2.3579 (4.3470) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][40/625] eta 0:03:51 lr 0.000036 wd 0.0500 time 0.3815 (0.3959) data time 0.0008 (0.0153) model time 0.0000 (0.0000) loss 2.9594 (2.6637) grad_norm 3.2150 (3.9927) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][50/625] eta 0:03:45 lr 0.000036 wd 0.0500 time 0.3700 (0.3914) data time 0.0008 (0.0125) model time 0.0000 (0.0000) loss 1.9575 (2.6502) grad_norm 2.4674 (3.7072) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][60/625] eta 0:03:39 lr 0.000036 wd 0.0500 time 0.3844 (0.3889) data time 0.0011 (0.0106) model time 0.3834 (0.3751) loss 2.2990 (2.6489) grad_norm 3.6812 (3.7686) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][70/625] eta 0:03:34 lr 0.000036 wd 0.0500 time 0.3742 (0.3868) data time 0.0009 (0.0093) model time 0.3733 (0.3740) loss 2.8684 (2.6489) grad_norm 2.4978 (3.7621) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][80/625] eta 0:03:29 lr 0.000036 wd 0.0500 time 0.3746 (0.3849) data time 0.0011 (0.0083) model time 0.3736 (0.3729) loss 2.5962 (2.6131) grad_norm 4.3356 (3.6666) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][90/625] eta 0:03:25 lr 0.000036 wd 0.0500 time 0.3817 (0.3835) data time 0.0012 (0.0075) model time 0.3806 (0.3722) loss 2.3577 (2.6127) grad_norm 2.6522 (3.5870) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][100/625] eta 0:03:23 lr 0.000036 wd 0.0500 time 0.3787 (0.3885) data time 0.0011 (0.0069) model time 0.3776 (0.3844) loss 2.2978 (2.5968) grad_norm 3.0562 (3.5148) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][110/625] eta 0:03:19 lr 0.000036 wd 0.0500 time 0.3692 (0.3871) data time 0.0008 (0.0063) model time 0.3684 (0.3823) loss 2.5722 (2.5864) grad_norm 2.0705 (3.5175) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][120/625] eta 0:03:14 lr 0.000036 wd 0.0500 time 0.3731 (0.3861) data time 0.0010 (0.0059) model time 0.3722 (0.3811) loss 2.4450 (2.5877) grad_norm 3.7295 (3.4966) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][130/625] eta 0:03:10 lr 0.000035 wd 0.0500 time 0.3807 (0.3851) data time 0.0011 (0.0055) model time 0.3797 (0.3800) loss 2.1040 (2.5788) grad_norm 2.8078 (3.4905) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][140/625] eta 0:03:06 lr 0.000035 wd 0.0500 time 0.3780 (0.3843) data time 0.0009 (0.0052) model time 0.3771 (0.3793) loss 2.4777 (2.5721) grad_norm 2.5102 (3.4554) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][150/625] eta 0:03:02 lr 0.000035 wd 0.0500 time 0.3670 (0.3848) data time 0.0009 (0.0049) model time 0.3661 (0.3803) loss 2.9769 (2.5824) grad_norm 2.8321 (3.4186) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][160/625] eta 0:02:58 lr 0.000035 wd 0.0500 time 0.3759 (0.3841) data time 0.0008 (0.0047) model time 0.3752 (0.3797) loss 2.2099 (2.5887) grad_norm 3.6850 (3.3857) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][170/625] eta 0:02:54 lr 0.000035 wd 0.0500 time 0.3723 (0.3833) data time 0.0008 (0.0045) model time 0.3715 (0.3789) loss 1.5424 (2.5818) grad_norm 3.3476 (3.4061) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][180/625] eta 0:02:50 lr 0.000035 wd 0.0500 time 0.3698 (0.3828) data time 0.0009 (0.0043) model time 0.3689 (0.3783) loss 2.9167 (2.5822) grad_norm 4.6196 (3.3753) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][190/625] eta 0:02:46 lr 0.000035 wd 0.0500 time 0.3750 (0.3823) data time 0.0010 (0.0041) model time 0.3741 (0.3780) loss 2.8390 (2.5791) grad_norm 3.7818 (3.3662) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][200/625] eta 0:02:42 lr 0.000035 wd 0.0500 time 0.3766 (0.3820) data time 0.0011 (0.0040) model time 0.3755 (0.3777) loss 2.8609 (2.5861) grad_norm 5.5049 (3.3460) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][210/625] eta 0:02:38 lr 0.000035 wd 0.0500 time 0.3763 (0.3815) data time 0.0010 (0.0038) model time 0.3753 (0.3773) loss 2.0781 (2.5914) grad_norm 1.9724 (3.3424) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][220/625] eta 0:02:34 lr 0.000035 wd 0.0500 time 0.3801 (0.3811) data time 0.0011 (0.0037) model time 0.3791 (0.3770) loss 2.6702 (2.5945) grad_norm 2.5398 (3.3311) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][230/625] eta 0:02:30 lr 0.000035 wd 0.0500 time 0.3749 (0.3808) data time 0.0010 (0.0036) model time 0.3738 (0.3767) loss 3.0260 (2.5961) grad_norm 1.8299 (3.2978) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][240/625] eta 0:02:26 lr 0.000035 wd 0.0500 time 0.3757 (0.3805) data time 0.0010 (0.0035) model time 0.3746 (0.3765) loss 2.4973 (2.5859) grad_norm 5.2323 (3.2929) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][250/625] eta 0:02:22 lr 0.000035 wd 0.0500 time 0.3759 (0.3802) data time 0.0011 (0.0034) model time 0.3748 (0.3763) loss 3.0020 (2.5869) grad_norm 4.1438 (3.2926) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][260/625] eta 0:02:18 lr 0.000035 wd 0.0500 time 0.3692 (0.3800) data time 0.0011 (0.0033) model time 0.3681 (0.3761) loss 2.5883 (2.5896) grad_norm 4.4350 (3.3097) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][270/625] eta 0:02:14 lr 0.000035 wd 0.0500 time 0.3798 (0.3800) data time 0.0010 (0.0032) model time 0.3788 (0.3763) loss 2.9363 (2.5836) grad_norm 3.4384 (3.3002) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][280/625] eta 0:02:11 lr 0.000035 wd 0.0500 time 0.3745 (0.3798) data time 0.0012 (0.0031) model time 0.3734 (0.3762) loss 2.9968 (2.5860) grad_norm 2.2102 (3.3281) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][290/625] eta 0:02:07 lr 0.000035 wd 0.0500 time 0.3775 (0.3796) data time 0.0008 (0.0031) model time 0.3767 (0.3760) loss 3.1176 (2.5894) grad_norm 2.5133 (3.3050) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][300/625] eta 0:02:03 lr 0.000035 wd 0.0500 time 0.3754 (0.3793) data time 0.0011 (0.0030) model time 0.3743 (0.3758) loss 2.4887 (2.5789) grad_norm 2.5407 (3.3036) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][310/625] eta 0:01:59 lr 0.000035 wd 0.0500 time 0.3743 (0.3806) data time 0.0010 (0.0029) model time 0.3732 (0.3774) loss 2.6947 (2.5824) grad_norm 2.5028 (3.3344) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][320/625] eta 0:01:56 lr 0.000035 wd 0.0500 time 0.3740 (0.3804) data time 0.0010 (0.0029) model time 0.3730 (0.3772) loss 2.7684 (2.5865) grad_norm 2.7998 (3.3247) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][330/625] eta 0:01:52 lr 0.000035 wd 0.0500 time 0.3677 (0.3807) data time 0.0008 (0.0028) model time 0.3669 (0.3777) loss 2.9156 (2.5954) grad_norm 2.1524 (3.3056) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][340/625] eta 0:01:48 lr 0.000035 wd 0.0500 time 0.3766 (0.3805) data time 0.0008 (0.0028) model time 0.3758 (0.3775) loss 2.2931 (2.5907) grad_norm 3.2499 (3.2915) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][350/625] eta 0:01:44 lr 0.000035 wd 0.0500 time 0.3739 (0.3803) data time 0.0011 (0.0027) model time 0.3729 (0.3774) loss 2.3317 (2.5916) grad_norm 2.1108 (3.2792) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][360/625] eta 0:01:40 lr 0.000035 wd 0.0500 time 0.3683 (0.3802) data time 0.0012 (0.0027) model time 0.3670 (0.3773) loss 2.7652 (2.5903) grad_norm 3.2507 (3.2779) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][370/625] eta 0:01:36 lr 0.000035 wd 0.0500 time 0.3684 (0.3800) data time 0.0010 (0.0026) model time 0.3674 (0.3771) loss 2.0160 (2.5940) grad_norm 2.7635 (3.2777) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][380/625] eta 0:01:33 lr 0.000035 wd 0.0500 time 0.3699 (0.3798) data time 0.0008 (0.0026) model time 0.3691 (0.3770) loss 2.7356 (2.5905) grad_norm 1.9787 (3.2586) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][390/625] eta 0:01:29 lr 0.000035 wd 0.0500 time 0.3685 (0.3797) data time 0.0010 (0.0025) model time 0.3675 (0.3769) loss 2.1942 (2.5895) grad_norm 9.3995 (3.2589) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][400/625] eta 0:01:25 lr 0.000035 wd 0.0500 time 0.3659 (0.3796) data time 0.0008 (0.0025) model time 0.3651 (0.3768) loss 1.7716 (2.5894) grad_norm 4.4202 (3.2832) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][410/625] eta 0:01:21 lr 0.000035 wd 0.0500 time 0.3728 (0.3794) data time 0.0008 (0.0025) model time 0.3720 (0.3766) loss 2.0018 (2.5918) grad_norm 2.7783 (3.2811) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][420/625] eta 0:01:17 lr 0.000035 wd 0.0500 time 0.3709 (0.3792) data time 0.0008 (0.0024) model time 0.3701 (0.3765) loss 3.2307 (2.5938) grad_norm 2.3645 (3.2758) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][430/625] eta 0:01:13 lr 0.000035 wd 0.0500 time 0.3731 (0.3791) data time 0.0011 (0.0024) model time 0.3720 (0.3764) loss 2.7778 (2.5953) grad_norm 3.2489 (3.2663) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:31:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][440/625] eta 0:01:10 lr 0.000035 wd 0.0500 time 0.3714 (0.3790) data time 0.0010 (0.0024) model time 0.3704 (0.3763) loss 2.5591 (2.6016) grad_norm 2.6533 (3.2891) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][450/625] eta 0:01:06 lr 0.000035 wd 0.0500 time 0.3818 (0.3793) data time 0.0008 (0.0023) model time 0.3810 (0.3768) loss 2.8366 (2.6046) grad_norm 3.2774 (3.2876) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][460/625] eta 0:01:02 lr 0.000034 wd 0.0500 time 0.3763 (0.3792) data time 0.0012 (0.0023) model time 0.3751 (0.3766) loss 2.3354 (2.6061) grad_norm 1.9657 (3.2935) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][470/625] eta 0:00:58 lr 0.000034 wd 0.0500 time 0.3706 (0.3791) data time 0.0008 (0.0023) model time 0.3698 (0.3765) loss 3.3083 (2.6044) grad_norm 2.3270 (3.2821) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][480/625] eta 0:00:54 lr 0.000034 wd 0.0500 time 0.3701 (0.3790) data time 0.0007 (0.0023) model time 0.3694 (0.3765) loss 2.7278 (2.6067) grad_norm 3.6855 (3.2941) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][490/625] eta 0:00:51 lr 0.000034 wd 0.0500 time 0.3841 (0.3797) data time 0.0010 (0.0022) model time 0.3831 (0.3772) loss 1.9992 (2.6019) grad_norm 3.4604 (3.2891) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][500/625] eta 0:00:47 lr 0.000034 wd 0.0500 time 0.3794 (0.3800) data time 0.0010 (0.0022) model time 0.3783 (0.3777) loss 2.9821 (2.6054) grad_norm 2.5468 (3.2763) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][510/625] eta 0:00:43 lr 0.000034 wd 0.0500 time 0.3656 (0.3799) data time 0.0008 (0.0022) model time 0.3648 (0.3776) loss 3.2041 (2.6059) grad_norm 3.9431 (3.2853) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][520/625] eta 0:00:39 lr 0.000034 wd 0.0500 time 0.3713 (0.3798) data time 0.0010 (0.0022) model time 0.3702 (0.3775) loss 2.6577 (2.6096) grad_norm 3.1486 (3.2824) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][530/625] eta 0:00:36 lr 0.000034 wd 0.0500 time 0.3779 (0.3797) data time 0.0008 (0.0021) model time 0.3771 (0.3774) loss 1.7783 (2.6065) grad_norm 7.8497 (3.2821) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][540/625] eta 0:00:32 lr 0.000034 wd 0.0500 time 0.3720 (0.3796) data time 0.0009 (0.0021) model time 0.3712 (0.3773) loss 2.6985 (2.6073) grad_norm 2.8538 (3.2736) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][550/625] eta 0:00:28 lr 0.000034 wd 0.0500 time 0.3733 (0.3795) data time 0.0011 (0.0021) model time 0.3722 (0.3772) loss 2.6113 (2.6082) grad_norm 2.3328 (3.2714) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][560/625] eta 0:00:24 lr 0.000034 wd 0.0500 time 0.3716 (0.3794) data time 0.0011 (0.0021) model time 0.3705 (0.3772) loss 3.0204 (2.6107) grad_norm 2.4938 (3.2645) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][570/625] eta 0:00:20 lr 0.000034 wd 0.0500 time 0.3792 (0.3793) data time 0.0008 (0.0021) model time 0.3784 (0.3771) loss 2.0477 (2.6135) grad_norm 3.3209 (3.2771) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][580/625] eta 0:00:17 lr 0.000034 wd 0.0500 time 0.3696 (0.3792) data time 0.0011 (0.0021) model time 0.3685 (0.3770) loss 2.4115 (2.6162) grad_norm 4.2691 (3.2892) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][590/625] eta 0:00:13 lr 0.000034 wd 0.0500 time 0.3774 (0.3792) data time 0.0008 (0.0020) model time 0.3766 (0.3770) loss 2.6244 (2.6203) grad_norm 2.0447 (3.2769) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:32:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][600/625] eta 0:00:09 lr 0.000034 wd 0.0500 time 0.3720 (0.3791) data time 0.0013 (0.0020) model time 0.3707 (0.3769) loss 2.8235 (2.6192) grad_norm 2.0626 (3.2805) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][610/625] eta 0:00:05 lr 0.000034 wd 0.0500 time 0.5895 (0.3800) data time 0.0005 (0.0020) model time 0.5890 (0.3779) loss 2.8462 (2.6224) grad_norm 2.5116 (3.2936) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [284/300][620/625] eta 0:00:01 lr 0.000034 wd 0.0500 time 0.3588 (0.3797) data time 0.0007 (0.0020) model time 0.3581 (0.3776) loss 3.1737 (2.6182) grad_norm 2.0914 (3.2858) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:08 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 284 training takes 0:03:57 [2024-09-01 15:33:08 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:33:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:33:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.510 (0.510) Loss 0.5098 (0.5098) Acc@1 88.965 (88.965) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 15:33:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.147) Loss 0.8257 (0.6048) Acc@1 81.299 (87.034) Acc@5 95.850 (97.825) Mem 11224MB [2024-09-01 15:33:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.122) Loss 0.9126 (0.7234) Acc@1 79.199 (84.170) Acc@5 95.508 (96.663) Mem 11224MB [2024-09-01 15:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.885 Acc@5 96.673 [2024-09-01 15:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:33:12 msvmambav3_small_224] (main_hfai_mnodes.py 269): INFO New max accuracy: 83.89% [2024-09-01 15:33:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saving...... [2024-09-01 15:33:13 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt.pth saved !!! [2024-09-01 15:33:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 0.5073 (0.5073) Acc@1 88.916 (88.916) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:33:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.143) Loss 0.8208 (0.6003) Acc@1 80.908 (87.074) Acc@5 95.703 (97.883) Mem 11224MB [2024-09-01 15:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.120) Loss 0.9004 (0.7165) Acc@1 78.809 (84.208) Acc@5 95.361 (96.710) Mem 11224MB [2024-09-01 15:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.915 Acc@5 96.725 [2024-09-01 15:33:16 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][0/625] eta 0:12:27 lr 0.000034 wd 0.0500 time 1.1966 (1.1966) data time 0.5738 (0.5738) model time 0.0000 (0.0000) loss 2.8725 (2.8725) grad_norm 3.0302 (3.0302) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][10/625] eta 0:04:35 lr 0.000034 wd 0.0500 time 0.3726 (0.4483) data time 0.0010 (0.0531) model time 0.0000 (0.0000) loss 2.4804 (2.5436) grad_norm 4.0592 (2.9589) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][20/625] eta 0:04:08 lr 0.000034 wd 0.0500 time 0.3694 (0.4108) data time 0.0011 (0.0283) model time 0.0000 (0.0000) loss 3.0976 (2.5993) grad_norm 2.6885 (3.0051) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][30/625] eta 0:03:56 lr 0.000034 wd 0.0500 time 0.3715 (0.3982) data time 0.0008 (0.0195) model time 0.0000 (0.0000) loss 2.6580 (2.6350) grad_norm 4.2099 (3.2231) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][40/625] eta 0:03:49 lr 0.000034 wd 0.0500 time 0.3820 (0.3917) data time 0.0008 (0.0150) model time 0.0000 (0.0000) loss 3.1821 (2.5929) grad_norm 2.3952 (3.2705) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][50/625] eta 0:03:43 lr 0.000034 wd 0.0500 time 0.3837 (0.3887) data time 0.0008 (0.0123) model time 0.0000 (0.0000) loss 2.9553 (2.5738) grad_norm 4.4491 (3.2606) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][60/625] eta 0:03:38 lr 0.000034 wd 0.0500 time 0.3769 (0.3866) data time 0.0011 (0.0104) model time 0.3759 (0.3749) loss 2.7814 (2.5860) grad_norm 3.5249 (3.2193) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][70/625] eta 0:03:33 lr 0.000034 wd 0.0500 time 0.3783 (0.3853) data time 0.0010 (0.0091) model time 0.3773 (0.3756) loss 2.0501 (2.5939) grad_norm 3.2079 (3.1637) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][80/625] eta 0:03:29 lr 0.000034 wd 0.0500 time 0.3770 (0.3843) data time 0.0009 (0.0081) model time 0.3761 (0.3757) loss 2.9601 (2.5873) grad_norm 2.5156 (3.1221) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][90/625] eta 0:03:24 lr 0.000034 wd 0.0500 time 0.3639 (0.3829) data time 0.0011 (0.0073) model time 0.3628 (0.3743) loss 2.3068 (2.5908) grad_norm 3.0488 (3.2622) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][100/625] eta 0:03:20 lr 0.000034 wd 0.0500 time 0.3742 (0.3822) data time 0.0011 (0.0067) model time 0.3732 (0.3745) loss 2.3089 (2.5750) grad_norm 5.8864 (3.2447) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:33:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][110/625] eta 0:03:16 lr 0.000034 wd 0.0500 time 0.3715 (0.3815) data time 0.0011 (0.0062) model time 0.3704 (0.3743) loss 2.0557 (2.5602) grad_norm 6.4380 (3.2704) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][120/625] eta 0:03:12 lr 0.000034 wd 0.0500 time 0.3850 (0.3809) data time 0.0008 (0.0058) model time 0.3842 (0.3742) loss 2.0028 (2.5392) grad_norm 1.9804 (3.2598) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][130/625] eta 0:03:08 lr 0.000034 wd 0.0500 time 0.3723 (0.3804) data time 0.0007 (0.0054) model time 0.3716 (0.3740) loss 2.1832 (2.5531) grad_norm 4.5376 (3.2919) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][140/625] eta 0:03:04 lr 0.000034 wd 0.0500 time 0.3737 (0.3800) data time 0.0012 (0.0051) model time 0.3724 (0.3741) loss 1.9221 (2.5371) grad_norm 3.7178 (3.3070) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][150/625] eta 0:03:00 lr 0.000034 wd 0.0500 time 0.3660 (0.3797) data time 0.0008 (0.0048) model time 0.3652 (0.3741) loss 3.0406 (2.5575) grad_norm 2.8965 (3.6370) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][160/625] eta 0:02:56 lr 0.000034 wd 0.0500 time 0.3652 (0.3793) data time 0.0010 (0.0046) model time 0.3642 (0.3739) loss 2.9426 (2.5576) grad_norm 3.2795 (3.6192) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][170/625] eta 0:02:53 lr 0.000033 wd 0.0500 time 0.3843 (0.3812) data time 0.0008 (0.0044) model time 0.3835 (0.3769) loss 2.5857 (2.5588) grad_norm 2.4997 (3.5605) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][180/625] eta 0:02:49 lr 0.000033 wd 0.0500 time 0.3730 (0.3819) data time 0.0010 (0.0042) model time 0.3720 (0.3782) loss 3.0615 (2.5618) grad_norm 3.3570 (3.5133) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][190/625] eta 0:02:46 lr 0.000033 wd 0.0500 time 0.3708 (0.3822) data time 0.0011 (0.0040) model time 0.3698 (0.3789) loss 2.6224 (2.5731) grad_norm 1.8382 (3.4725) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][200/625] eta 0:02:42 lr 0.000033 wd 0.0500 time 0.3770 (0.3818) data time 0.0011 (0.0039) model time 0.3759 (0.3784) loss 2.4456 (2.5688) grad_norm 2.9161 (3.4915) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][210/625] eta 0:02:38 lr 0.000033 wd 0.0500 time 0.3807 (0.3815) data time 0.0008 (0.0038) model time 0.3799 (0.3781) loss 2.9098 (2.5680) grad_norm 3.2437 (3.4589) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][220/625] eta 0:02:34 lr 0.000033 wd 0.0500 time 0.3640 (0.3810) data time 0.0011 (0.0036) model time 0.3629 (0.3776) loss 2.7650 (2.5700) grad_norm 2.4129 (3.4464) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][230/625] eta 0:02:30 lr 0.000033 wd 0.0500 time 0.3773 (0.3807) data time 0.0010 (0.0035) model time 0.3763 (0.3774) loss 3.0046 (2.5783) grad_norm 2.4799 (3.4060) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][240/625] eta 0:02:26 lr 0.000033 wd 0.0500 time 0.3724 (0.3805) data time 0.0010 (0.0034) model time 0.3714 (0.3772) loss 2.7259 (2.5678) grad_norm 2.4800 (3.4040) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][250/625] eta 0:02:22 lr 0.000033 wd 0.0500 time 0.3797 (0.3803) data time 0.0010 (0.0033) model time 0.3787 (0.3771) loss 2.6841 (2.5715) grad_norm 3.1177 (3.4460) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][260/625] eta 0:02:18 lr 0.000033 wd 0.0500 time 0.3753 (0.3800) data time 0.0007 (0.0032) model time 0.3746 (0.3769) loss 1.9072 (2.5650) grad_norm 4.2171 (3.4446) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][270/625] eta 0:02:14 lr 0.000033 wd 0.0500 time 0.3699 (0.3798) data time 0.0008 (0.0032) model time 0.3691 (0.3767) loss 2.7784 (2.5702) grad_norm 5.5640 (3.4547) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][280/625] eta 0:02:10 lr 0.000033 wd 0.0500 time 0.3627 (0.3795) data time 0.0011 (0.0031) model time 0.3616 (0.3764) loss 2.5753 (2.5679) grad_norm 2.6653 (3.4414) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][290/625] eta 0:02:07 lr 0.000033 wd 0.0500 time 0.3738 (0.3807) data time 0.0011 (0.0030) model time 0.3727 (0.3780) loss 3.2527 (2.5742) grad_norm 3.9114 (3.4420) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][300/625] eta 0:02:03 lr 0.000033 wd 0.0500 time 0.3741 (0.3813) data time 0.0010 (0.0029) model time 0.3731 (0.3788) loss 2.8121 (2.5730) grad_norm 2.5350 (3.4243) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][310/625] eta 0:02:00 lr 0.000033 wd 0.0500 time 0.3656 (0.3811) data time 0.0008 (0.0029) model time 0.3648 (0.3786) loss 3.0778 (2.5796) grad_norm 3.4144 (3.4072) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][320/625] eta 0:01:56 lr 0.000033 wd 0.0500 time 0.3684 (0.3810) data time 0.0011 (0.0028) model time 0.3674 (0.3784) loss 2.5195 (2.5712) grad_norm 3.1166 (3.4218) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][330/625] eta 0:01:52 lr 0.000033 wd 0.0500 time 0.3801 (0.3808) data time 0.0010 (0.0028) model time 0.3791 (0.3783) loss 2.5761 (2.5696) grad_norm 9.2145 (3.4326) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][340/625] eta 0:01:48 lr 0.000033 wd 0.0500 time 0.3691 (0.3805) data time 0.0011 (0.0027) model time 0.3680 (0.3780) loss 2.7795 (2.5688) grad_norm 2.5111 (3.4460) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][350/625] eta 0:01:44 lr 0.000033 wd 0.0500 time 0.3781 (0.3804) data time 0.0010 (0.0027) model time 0.3771 (0.3779) loss 2.5823 (2.5753) grad_norm 3.2089 (3.4550) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][360/625] eta 0:01:40 lr 0.000033 wd 0.0500 time 0.3727 (0.3802) data time 0.0010 (0.0026) model time 0.3717 (0.3777) loss 3.1220 (2.5755) grad_norm 2.3434 (3.4743) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][370/625] eta 0:01:36 lr 0.000033 wd 0.0500 time 0.3641 (0.3803) data time 0.0010 (0.0026) model time 0.3631 (0.3779) loss 2.5151 (2.5750) grad_norm 2.8904 (3.4642) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][380/625] eta 0:01:33 lr 0.000033 wd 0.0500 time 0.3721 (0.3802) data time 0.0008 (0.0025) model time 0.3713 (0.3778) loss 2.5516 (2.5774) grad_norm 2.1954 (3.4801) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:35:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][390/625] eta 0:01:29 lr 0.000033 wd 0.0500 time 0.3669 (0.3800) data time 0.0007 (0.0025) model time 0.3662 (0.3776) loss 1.9942 (2.5754) grad_norm 3.9543 (3.4871) loss_scale 256.0000 (129.3095) mem 11224MB [2024-09-01 15:35:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][400/625] eta 0:01:25 lr 0.000033 wd 0.0500 time 0.3755 (0.3799) data time 0.0010 (0.0025) model time 0.3745 (0.3776) loss 2.8685 (2.5732) grad_norm 2.5236 (3.4722) loss_scale 256.0000 (132.4688) mem 11224MB [2024-09-01 15:35:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][410/625] eta 0:01:21 lr 0.000033 wd 0.0500 time 0.3729 (0.3798) data time 0.0010 (0.0024) model time 0.3719 (0.3775) loss 1.8221 (2.5728) grad_norm 3.0074 (3.4648) loss_scale 256.0000 (135.4745) mem 11224MB [2024-09-01 15:35:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][420/625] eta 0:01:17 lr 0.000033 wd 0.0500 time 0.3821 (0.3797) data time 0.0008 (0.0024) model time 0.3813 (0.3775) loss 3.0532 (2.5743) grad_norm 2.4736 (3.4610) loss_scale 256.0000 (138.3373) mem 11224MB [2024-09-01 15:36:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][430/625] eta 0:01:14 lr 0.000033 wd 0.0500 time 0.3721 (0.3796) data time 0.0011 (0.0024) model time 0.3710 (0.3773) loss 2.8735 (2.5761) grad_norm 2.4370 (3.4543) loss_scale 256.0000 (141.0673) mem 11224MB [2024-09-01 15:36:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][440/625] eta 0:01:10 lr 0.000033 wd 0.0500 time 0.3734 (0.3795) data time 0.0011 (0.0023) model time 0.3723 (0.3773) loss 2.8511 (2.5754) grad_norm 3.0778 (3.4369) loss_scale 256.0000 (143.6735) mem 11224MB [2024-09-01 15:36:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][450/625] eta 0:01:06 lr 0.000033 wd 0.0500 time 0.3755 (0.3794) data time 0.0010 (0.0023) model time 0.3745 (0.3772) loss 2.6999 (2.5771) grad_norm 3.6418 (3.4290) loss_scale 256.0000 (146.1641) mem 11224MB [2024-09-01 15:36:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][460/625] eta 0:01:02 lr 0.000033 wd 0.0500 time 0.3757 (0.3793) data time 0.0010 (0.0023) model time 0.3747 (0.3771) loss 1.7640 (2.5827) grad_norm 2.7621 (3.4210) loss_scale 256.0000 (148.5466) mem 11224MB [2024-09-01 15:36:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][470/625] eta 0:00:58 lr 0.000033 wd 0.0500 time 0.3758 (0.3792) data time 0.0011 (0.0023) model time 0.3747 (0.3770) loss 2.5018 (2.5827) grad_norm 3.1418 (3.4887) loss_scale 256.0000 (150.8280) mem 11224MB [2024-09-01 15:36:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][480/625] eta 0:00:54 lr 0.000033 wd 0.0500 time 0.3709 (0.3792) data time 0.0009 (0.0022) model time 0.3700 (0.3770) loss 2.1238 (2.5857) grad_norm 3.2916 (3.4922) loss_scale 256.0000 (153.0146) mem 11224MB [2024-09-01 15:36:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][490/625] eta 0:00:51 lr 0.000033 wd 0.0500 time 0.3753 (0.3795) data time 0.0010 (0.0022) model time 0.3742 (0.3774) loss 3.0591 (2.5928) grad_norm 3.6230 (3.4876) loss_scale 256.0000 (155.1120) mem 11224MB [2024-09-01 15:36:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][500/625] eta 0:00:47 lr 0.000033 wd 0.0500 time 0.3764 (0.3804) data time 0.0009 (0.0022) model time 0.3755 (0.3785) loss 2.6062 (2.5933) grad_norm 2.2003 (3.4766) loss_scale 256.0000 (157.1257) mem 11224MB [2024-09-01 15:36:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][510/625] eta 0:00:43 lr 0.000033 wd 0.0500 time 0.3738 (0.3808) data time 0.0008 (0.0022) model time 0.3730 (0.3788) loss 2.7889 (2.5927) grad_norm 4.5359 (3.5017) loss_scale 256.0000 (159.0607) mem 11224MB [2024-09-01 15:36:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][520/625] eta 0:00:39 lr 0.000032 wd 0.0500 time 0.3736 (0.3807) data time 0.0008 (0.0021) model time 0.3728 (0.3788) loss 2.0690 (2.5948) grad_norm 3.3487 (3.4869) loss_scale 256.0000 (160.9213) mem 11224MB [2024-09-01 15:36:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][530/625] eta 0:00:36 lr 0.000032 wd 0.0500 time 0.3820 (0.3806) data time 0.0008 (0.0021) model time 0.3812 (0.3787) loss 2.2804 (2.5941) grad_norm 3.0720 (3.5685) loss_scale 256.0000 (162.7119) mem 11224MB [2024-09-01 15:36:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][540/625] eta 0:00:32 lr 0.000032 wd 0.0500 time 0.3804 (0.3805) data time 0.0009 (0.0021) model time 0.3795 (0.3786) loss 3.2112 (2.5936) grad_norm 5.8073 (3.5664) loss_scale 256.0000 (164.4362) mem 11224MB [2024-09-01 15:36:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][550/625] eta 0:00:28 lr 0.000032 wd 0.0500 time 0.3770 (0.3804) data time 0.0010 (0.0021) model time 0.3759 (0.3786) loss 2.6950 (2.5953) grad_norm 2.2987 (3.5624) loss_scale 256.0000 (166.0980) mem 11224MB [2024-09-01 15:36:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][560/625] eta 0:00:24 lr 0.000032 wd 0.0500 time 0.3719 (0.3803) data time 0.0008 (0.0021) model time 0.3711 (0.3785) loss 2.5869 (2.5962) grad_norm 3.4998 (3.5695) loss_scale 256.0000 (167.7005) mem 11224MB [2024-09-01 15:36:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][570/625] eta 0:00:20 lr 0.000032 wd 0.0500 time 0.3748 (0.3802) data time 0.0009 (0.0020) model time 0.3739 (0.3784) loss 2.7751 (2.5991) grad_norm 2.6176 (3.5532) loss_scale 256.0000 (169.2469) mem 11224MB [2024-09-01 15:36:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][580/625] eta 0:00:17 lr 0.000032 wd 0.0500 time 0.3922 (0.3801) data time 0.0008 (0.0020) model time 0.3914 (0.3783) loss 3.2350 (2.5954) grad_norm 5.6118 (3.5471) loss_scale 256.0000 (170.7401) mem 11224MB [2024-09-01 15:37:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][590/625] eta 0:00:13 lr 0.000032 wd 0.0500 time 0.3774 (0.3800) data time 0.0009 (0.0020) model time 0.3765 (0.3782) loss 2.8122 (2.5950) grad_norm 2.5461 (3.5422) loss_scale 256.0000 (172.1827) mem 11224MB [2024-09-01 15:37:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][600/625] eta 0:00:09 lr 0.000032 wd 0.0500 time 0.3774 (0.3799) data time 0.0010 (0.0020) model time 0.3764 (0.3781) loss 2.0995 (2.5941) grad_norm 3.0941 (3.5443) loss_scale 256.0000 (173.5774) mem 11224MB [2024-09-01 15:37:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][610/625] eta 0:00:05 lr 0.000032 wd 0.0500 time 0.3606 (0.3798) data time 0.0005 (0.0020) model time 0.3601 (0.3779) loss 2.3913 (2.5938) grad_norm 2.8175 (3.5376) loss_scale 256.0000 (174.9264) mem 11224MB [2024-09-01 15:37:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [285/300][620/625] eta 0:00:01 lr 0.000032 wd 0.0500 time 0.3619 (0.3795) data time 0.0007 (0.0020) model time 0.3612 (0.3776) loss 2.5947 (2.5929) grad_norm 3.7407 (3.5326) loss_scale 256.0000 (176.2319) mem 11224MB [2024-09-01 15:37:13 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 285 training takes 0:03:57 [2024-09-01 15:37:13 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:37:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:37:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.535 (0.535) Loss 0.5205 (0.5205) Acc@1 88.916 (88.916) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:37:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.144) Loss 0.8330 (0.6076) Acc@1 80.420 (86.941) Acc@5 96.045 (97.834) Mem 11224MB [2024-09-01 15:37:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.120) Loss 0.9136 (0.7282) Acc@1 78.711 (84.017) Acc@5 95.312 (96.633) Mem 11224MB [2024-09-01 15:37:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.723 Acc@5 96.643 [2024-09-01 15:37:17 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 15:37:18 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.817 (0.817) Loss 0.5073 (0.5073) Acc@1 88.916 (88.916) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:37:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.177) Loss 0.8218 (0.6005) Acc@1 80.713 (87.061) Acc@5 95.752 (97.874) Mem 11224MB [2024-09-01 15:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.9014 (0.7170) Acc@1 78.760 (84.212) Acc@5 95.361 (96.701) Mem 11224MB [2024-09-01 15:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.917 Acc@5 96.717 [2024-09-01 15:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:37:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][0/625] eta 0:13:11 lr 0.000032 wd 0.0500 time 1.2658 (1.2658) data time 0.5793 (0.5793) model time 0.0000 (0.0000) loss 2.2846 (2.2846) grad_norm 2.1709 (2.1709) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][10/625] eta 0:04:42 lr 0.000032 wd 0.0500 time 0.3691 (0.4591) data time 0.0008 (0.0535) model time 0.0000 (0.0000) loss 3.1710 (2.6080) grad_norm 2.1991 (3.2205) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][20/625] eta 0:04:13 lr 0.000032 wd 0.0500 time 0.3737 (0.4189) data time 0.0009 (0.0285) model time 0.0000 (0.0000) loss 2.5445 (2.6714) grad_norm 2.8149 (3.1405) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][30/625] eta 0:04:01 lr 0.000032 wd 0.0500 time 0.3786 (0.4053) data time 0.0010 (0.0197) model time 0.0000 (0.0000) loss 2.9316 (2.6007) grad_norm 4.0641 (3.1323) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][40/625] eta 0:03:55 lr 0.000032 wd 0.0500 time 0.5501 (0.4028) data time 0.0008 (0.0151) model time 0.0000 (0.0000) loss 2.1575 (2.5822) grad_norm 2.6843 (3.1125) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][50/625] eta 0:03:48 lr 0.000032 wd 0.0500 time 0.3695 (0.3974) data time 0.0011 (0.0124) model time 0.0000 (0.0000) loss 2.9963 (2.6018) grad_norm 2.8496 (3.1018) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][60/625] eta 0:03:42 lr 0.000032 wd 0.0500 time 0.3748 (0.3935) data time 0.0011 (0.0105) model time 0.3737 (0.3729) loss 2.7606 (2.5634) grad_norm 4.2926 (3.1270) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][70/625] eta 0:03:42 lr 0.000032 wd 0.0500 time 0.5780 (0.4016) data time 0.0012 (0.0092) model time 0.5768 (0.4112) loss 2.8150 (2.5523) grad_norm 2.0517 (3.0982) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][80/625] eta 0:03:37 lr 0.000032 wd 0.0500 time 0.3874 (0.3984) data time 0.0008 (0.0082) model time 0.3867 (0.3990) loss 3.0842 (2.5557) grad_norm 2.7738 (3.1547) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:37:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][90/625] eta 0:03:31 lr 0.000032 wd 0.0500 time 0.3709 (0.3958) data time 0.0008 (0.0074) model time 0.3701 (0.3928) loss 3.2368 (2.5710) grad_norm 3.8853 (3.5750) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][100/625] eta 0:03:26 lr 0.000032 wd 0.0500 time 0.3704 (0.3939) data time 0.0010 (0.0068) model time 0.3694 (0.3894) loss 2.5007 (2.5511) grad_norm 3.9001 (3.5960) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][110/625] eta 0:03:22 lr 0.000032 wd 0.0500 time 0.3752 (0.3924) data time 0.0008 (0.0062) model time 0.3744 (0.3872) loss 1.9021 (2.5605) grad_norm 2.6135 (3.5683) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][120/625] eta 0:03:17 lr 0.000032 wd 0.0500 time 0.3820 (0.3910) data time 0.0008 (0.0058) model time 0.3812 (0.3853) loss 2.6917 (2.5669) grad_norm 3.5957 (3.6598) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][130/625] eta 0:03:12 lr 0.000032 wd 0.0500 time 0.3664 (0.3898) data time 0.0008 (0.0054) model time 0.3657 (0.3840) loss 3.3833 (2.5634) grad_norm 3.0340 (3.5753) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][140/625] eta 0:03:08 lr 0.000032 wd 0.0500 time 0.3743 (0.3887) data time 0.0009 (0.0051) model time 0.3735 (0.3827) loss 3.3913 (2.5736) grad_norm 2.7267 (3.5526) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][150/625] eta 0:03:04 lr 0.000032 wd 0.0500 time 0.3795 (0.3878) data time 0.0008 (0.0049) model time 0.3787 (0.3819) loss 2.9970 (2.5598) grad_norm 3.0865 (3.6849) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][160/625] eta 0:02:59 lr 0.000032 wd 0.0500 time 0.3734 (0.3869) data time 0.0008 (0.0046) model time 0.3727 (0.3811) loss 3.0836 (2.5652) grad_norm 2.7898 (3.6643) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][170/625] eta 0:02:55 lr 0.000032 wd 0.0500 time 0.3639 (0.3861) data time 0.0009 (0.0044) model time 0.3630 (0.3803) loss 2.0756 (2.5590) grad_norm 2.6006 (3.6083) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][180/625] eta 0:02:51 lr 0.000032 wd 0.0500 time 0.3736 (0.3854) data time 0.0008 (0.0042) model time 0.3728 (0.3797) loss 3.1792 (2.5592) grad_norm 3.6240 (3.7219) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][190/625] eta 0:02:47 lr 0.000032 wd 0.0500 time 0.3760 (0.3848) data time 0.0008 (0.0041) model time 0.3751 (0.3792) loss 2.8426 (2.5749) grad_norm 2.1126 (3.6636) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][200/625] eta 0:02:43 lr 0.000032 wd 0.0500 time 0.3852 (0.3843) data time 0.0010 (0.0039) model time 0.3842 (0.3788) loss 2.9377 (2.5797) grad_norm 2.4352 (3.6087) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][210/625] eta 0:02:39 lr 0.000032 wd 0.0500 time 0.3810 (0.3839) data time 0.0007 (0.0038) model time 0.3802 (0.3786) loss 2.4616 (2.5843) grad_norm 3.1985 (3.5796) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][220/625] eta 0:02:35 lr 0.000032 wd 0.0500 time 0.3800 (0.3836) data time 0.0008 (0.0036) model time 0.3792 (0.3784) loss 2.0620 (2.5859) grad_norm 3.2275 (3.5638) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][230/625] eta 0:02:31 lr 0.000032 wd 0.0500 time 0.3767 (0.3839) data time 0.0008 (0.0035) model time 0.3759 (0.3791) loss 1.6008 (2.5810) grad_norm 3.3559 (3.5476) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][240/625] eta 0:02:27 lr 0.000032 wd 0.0500 time 0.3718 (0.3835) data time 0.0011 (0.0034) model time 0.3707 (0.3788) loss 2.6235 (2.5853) grad_norm 2.3323 (3.5253) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:38:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][250/625] eta 0:02:24 lr 0.000032 wd 0.0500 time 0.5523 (0.3853) data time 0.0008 (0.0033) model time 0.5515 (0.3812) loss 2.2408 (2.5849) grad_norm 2.9448 (3.5519) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][260/625] eta 0:02:20 lr 0.000031 wd 0.0500 time 0.3859 (0.3856) data time 0.0010 (0.0032) model time 0.3849 (0.3817) loss 2.5848 (2.5933) grad_norm 3.3580 (3.6307) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][270/625] eta 0:02:16 lr 0.000031 wd 0.0500 time 0.3857 (0.3852) data time 0.0008 (0.0032) model time 0.3849 (0.3814) loss 3.2022 (2.5907) grad_norm 2.5577 (3.5932) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][280/625] eta 0:02:12 lr 0.000031 wd 0.0500 time 0.3712 (0.3848) data time 0.0011 (0.0031) model time 0.3701 (0.3809) loss 2.8921 (2.5951) grad_norm 2.2687 (3.5580) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][290/625] eta 0:02:08 lr 0.000031 wd 0.0500 time 0.3771 (0.3845) data time 0.0008 (0.0030) model time 0.3763 (0.3807) loss 3.2545 (2.5950) grad_norm 2.8502 (3.5329) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][300/625] eta 0:02:04 lr 0.000031 wd 0.0500 time 0.3662 (0.3841) data time 0.0010 (0.0030) model time 0.3652 (0.3804) loss 2.5301 (2.5995) grad_norm 2.9866 (3.5241) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][310/625] eta 0:02:00 lr 0.000031 wd 0.0500 time 0.3608 (0.3837) data time 0.0010 (0.0029) model time 0.3598 (0.3800) loss 2.8736 (2.6034) grad_norm 2.3720 (3.5295) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][320/625] eta 0:01:56 lr 0.000031 wd 0.0500 time 0.3705 (0.3834) data time 0.0010 (0.0028) model time 0.3694 (0.3797) loss 2.3135 (2.6039) grad_norm 3.1070 (3.5678) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][330/625] eta 0:01:53 lr 0.000031 wd 0.0500 time 0.3697 (0.3831) data time 0.0008 (0.0028) model time 0.3690 (0.3795) loss 2.3453 (2.6034) grad_norm 3.7167 (3.5547) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][340/625] eta 0:01:49 lr 0.000031 wd 0.0500 time 0.3761 (0.3829) data time 0.0008 (0.0027) model time 0.3753 (0.3793) loss 2.9464 (2.5996) grad_norm 2.9434 (3.6531) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][350/625] eta 0:01:45 lr 0.000031 wd 0.0500 time 0.3721 (0.3831) data time 0.0012 (0.0027) model time 0.3709 (0.3797) loss 2.8035 (2.6034) grad_norm 2.1285 (3.6290) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][360/625] eta 0:01:41 lr 0.000031 wd 0.0500 time 0.3738 (0.3829) data time 0.0011 (0.0026) model time 0.3727 (0.3795) loss 2.5115 (2.6068) grad_norm 3.0404 (3.6449) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][370/625] eta 0:01:37 lr 0.000031 wd 0.0500 time 0.5744 (0.3838) data time 0.0010 (0.0026) model time 0.5734 (0.3806) loss 2.8589 (2.6121) grad_norm 4.6037 (3.6696) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][380/625] eta 0:01:34 lr 0.000031 wd 0.0500 time 0.3784 (0.3841) data time 0.0010 (0.0026) model time 0.3774 (0.3810) loss 2.5984 (2.6089) grad_norm 5.0993 (3.6687) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][390/625] eta 0:01:30 lr 0.000031 wd 0.0500 time 0.3742 (0.3838) data time 0.0011 (0.0025) model time 0.3732 (0.3808) loss 2.9413 (2.6128) grad_norm 2.8597 (3.6428) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][400/625] eta 0:01:26 lr 0.000031 wd 0.0500 time 0.3730 (0.3837) data time 0.0008 (0.0025) model time 0.3722 (0.3806) loss 1.8554 (2.6114) grad_norm 4.2282 (3.6381) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:39:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][410/625] eta 0:01:22 lr 0.000031 wd 0.0500 time 0.3796 (0.3834) data time 0.0012 (0.0024) model time 0.3785 (0.3804) loss 2.4621 (2.6089) grad_norm 2.5354 (3.6239) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][420/625] eta 0:01:18 lr 0.000031 wd 0.0500 time 0.3780 (0.3832) data time 0.0011 (0.0024) model time 0.3769 (0.3802) loss 2.2493 (2.6112) grad_norm 2.5219 (3.6134) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][430/625] eta 0:01:14 lr 0.000031 wd 0.0500 time 0.3784 (0.3830) data time 0.0010 (0.0024) model time 0.3773 (0.3801) loss 2.8455 (2.6120) grad_norm 6.7575 (3.6127) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][440/625] eta 0:01:10 lr 0.000031 wd 0.0500 time 0.3846 (0.3829) data time 0.0010 (0.0024) model time 0.3836 (0.3799) loss 1.9104 (2.6089) grad_norm 2.7892 (3.5944) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][450/625] eta 0:01:06 lr 0.000031 wd 0.0500 time 0.3744 (0.3827) data time 0.0010 (0.0023) model time 0.3734 (0.3798) loss 2.7360 (2.6068) grad_norm 2.8953 (3.5814) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][460/625] eta 0:01:03 lr 0.000031 wd 0.0500 time 0.3691 (0.3825) data time 0.0010 (0.0023) model time 0.3680 (0.3796) loss 2.9857 (2.6077) grad_norm 2.2415 (3.5768) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][470/625] eta 0:00:59 lr 0.000031 wd 0.0500 time 0.3753 (0.3823) data time 0.0008 (0.0023) model time 0.3745 (0.3795) loss 3.0111 (2.6026) grad_norm 2.9437 (3.5712) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][480/625] eta 0:00:55 lr 0.000031 wd 0.0500 time 0.3751 (0.3822) data time 0.0010 (0.0022) model time 0.3741 (0.3794) loss 2.8191 (2.6068) grad_norm 2.5715 (3.6790) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][490/625] eta 0:00:51 lr 0.000031 wd 0.0500 time 0.3663 (0.3820) data time 0.0010 (0.0022) model time 0.3653 (0.3792) loss 2.8997 (2.6019) grad_norm 2.7737 (3.6671) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][500/625] eta 0:00:47 lr 0.000031 wd 0.0500 time 0.3711 (0.3819) data time 0.0008 (0.0022) model time 0.3703 (0.3791) loss 2.0350 (2.5992) grad_norm 2.1481 (3.6420) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][510/625] eta 0:00:43 lr 0.000031 wd 0.0500 time 0.3667 (0.3817) data time 0.0008 (0.0022) model time 0.3659 (0.3789) loss 2.0373 (2.5969) grad_norm 3.2054 (3.6282) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][520/625] eta 0:00:40 lr 0.000031 wd 0.0500 time 0.3764 (0.3816) data time 0.0009 (0.0021) model time 0.3755 (0.3789) loss 2.9609 (2.5967) grad_norm 3.0896 (3.6449) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][530/625] eta 0:00:36 lr 0.000031 wd 0.0500 time 0.3711 (0.3815) data time 0.0008 (0.0021) model time 0.3703 (0.3787) loss 3.2483 (2.5947) grad_norm 2.7526 (3.6309) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][540/625] eta 0:00:32 lr 0.000031 wd 0.0500 time 0.3688 (0.3814) data time 0.0008 (0.0021) model time 0.3680 (0.3786) loss 2.9213 (2.5955) grad_norm 3.8067 (3.6189) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][550/625] eta 0:00:28 lr 0.000031 wd 0.0500 time 0.3694 (0.3812) data time 0.0009 (0.0021) model time 0.3685 (0.3785) loss 2.3226 (2.5942) grad_norm 3.0624 (3.6774) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][560/625] eta 0:00:24 lr 0.000031 wd 0.0500 time 0.3834 (0.3814) data time 0.0008 (0.0021) model time 0.3826 (0.3788) loss 2.8222 (2.5935) grad_norm 3.3811 (3.6790) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:40:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][570/625] eta 0:00:20 lr 0.000031 wd 0.0500 time 0.3641 (0.3813) data time 0.0010 (0.0020) model time 0.3631 (0.3787) loss 2.8128 (2.5898) grad_norm 2.9580 (3.6821) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][580/625] eta 0:00:17 lr 0.000031 wd 0.0500 time 0.3793 (0.3819) data time 0.0008 (0.0020) model time 0.3785 (0.3794) loss 2.8946 (2.5863) grad_norm 3.0073 (3.6694) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][590/625] eta 0:00:13 lr 0.000031 wd 0.0500 time 0.3651 (0.3821) data time 0.0011 (0.0020) model time 0.3641 (0.3797) loss 2.5201 (2.5879) grad_norm 2.4688 (3.6517) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][600/625] eta 0:00:09 lr 0.000031 wd 0.0500 time 0.3701 (0.3820) data time 0.0008 (0.0020) model time 0.3694 (0.3795) loss 3.0075 (2.5927) grad_norm 6.2973 (3.6612) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][610/625] eta 0:00:05 lr 0.000031 wd 0.0500 time 0.3581 (0.3819) data time 0.0008 (0.0020) model time 0.3574 (0.3794) loss 2.6558 (2.5918) grad_norm 2.6198 (3.6798) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [286/300][620/625] eta 0:00:01 lr 0.000031 wd 0.0500 time 0.3611 (0.3815) data time 0.0007 (0.0020) model time 0.3603 (0.3791) loss 3.0370 (2.5915) grad_norm 1.7970 (3.6698) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:19 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 286 training takes 0:03:58 [2024-09-01 15:41:19 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:41:20 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:41:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.539 (0.539) Loss 0.5181 (0.5181) Acc@1 88.818 (88.818) Acc@5 98.730 (98.730) Mem 11224MB [2024-09-01 15:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.146) Loss 0.8252 (0.6085) Acc@1 80.908 (87.078) Acc@5 95.947 (97.909) Mem 11224MB [2024-09-01 15:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.121) Loss 0.9238 (0.7298) Acc@1 78.809 (84.122) Acc@5 95.020 (96.640) Mem 11224MB [2024-09-01 15:41:23 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.809 Acc@5 96.655 [2024-09-01 15:41:23 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 15:41:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.795 (0.795) Loss 0.5078 (0.5078) Acc@1 88.916 (88.916) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.174) Loss 0.8203 (0.6004) Acc@1 80.615 (87.069) Acc@5 95.752 (97.883) Mem 11224MB [2024-09-01 15:41:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.136) Loss 0.9023 (0.7173) Acc@1 78.857 (84.224) Acc@5 95.312 (96.703) Mem 11224MB [2024-09-01 15:41:26 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.927 Acc@5 96.723 [2024-09-01 15:41:26 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:41:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][0/625] eta 0:12:31 lr 0.000031 wd 0.0500 time 1.2028 (1.2028) data time 0.6108 (0.6108) model time 0.0000 (0.0000) loss 1.5709 (1.5709) grad_norm 3.0181 (3.0181) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][10/625] eta 0:04:35 lr 0.000030 wd 0.0500 time 0.3754 (0.4481) data time 0.0008 (0.0565) model time 0.0000 (0.0000) loss 2.3911 (2.4335) grad_norm 5.0186 (3.6401) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][20/625] eta 0:04:09 lr 0.000030 wd 0.0500 time 0.3753 (0.4129) data time 0.0010 (0.0301) model time 0.0000 (0.0000) loss 2.6707 (2.5492) grad_norm 3.5201 (3.3059) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][30/625] eta 0:03:58 lr 0.000030 wd 0.0500 time 0.3786 (0.4007) data time 0.0010 (0.0207) model time 0.0000 (0.0000) loss 2.7087 (2.5612) grad_norm 3.8163 (3.4314) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][40/625] eta 0:03:50 lr 0.000030 wd 0.0500 time 0.3721 (0.3943) data time 0.0009 (0.0159) model time 0.0000 (0.0000) loss 2.9060 (2.5798) grad_norm 4.3146 (3.3906) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][50/625] eta 0:03:44 lr 0.000030 wd 0.0500 time 0.3719 (0.3904) data time 0.0008 (0.0130) model time 0.0000 (0.0000) loss 2.8709 (2.6210) grad_norm 3.4426 (3.2264) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][60/625] eta 0:03:38 lr 0.000030 wd 0.0500 time 0.3654 (0.3875) data time 0.0011 (0.0110) model time 0.3643 (0.3719) loss 1.9581 (2.6219) grad_norm 2.7824 (3.2167) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][70/625] eta 0:03:34 lr 0.000030 wd 0.0500 time 0.3831 (0.3857) data time 0.0010 (0.0096) model time 0.3821 (0.3728) loss 2.7201 (2.5811) grad_norm 1.9266 (3.2173) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:41:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][80/625] eta 0:03:29 lr 0.000030 wd 0.0500 time 0.3678 (0.3841) data time 0.0010 (0.0086) model time 0.3667 (0.3725) loss 2.0592 (2.5712) grad_norm 1.9103 (3.1554) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][90/625] eta 0:03:25 lr 0.000030 wd 0.0500 time 0.3781 (0.3843) data time 0.0008 (0.0077) model time 0.3773 (0.3756) loss 2.8433 (2.5445) grad_norm 4.1466 (3.1250) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][100/625] eta 0:03:21 lr 0.000030 wd 0.0500 time 0.3680 (0.3833) data time 0.0009 (0.0071) model time 0.3671 (0.3750) loss 2.6478 (2.5425) grad_norm 1.8482 (3.0641) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][110/625] eta 0:03:17 lr 0.000030 wd 0.0500 time 0.3697 (0.3827) data time 0.0010 (0.0065) model time 0.3686 (0.3751) loss 2.7751 (2.5269) grad_norm 2.3516 (3.0454) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][120/625] eta 0:03:12 lr 0.000030 wd 0.0500 time 0.3759 (0.3821) data time 0.0008 (0.0061) model time 0.3751 (0.3751) loss 2.6160 (2.5408) grad_norm 3.6911 (3.0359) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][130/625] eta 0:03:08 lr 0.000030 wd 0.0500 time 0.3705 (0.3814) data time 0.0011 (0.0057) model time 0.3694 (0.3747) loss 2.5287 (2.5409) grad_norm 1.9663 (3.0332) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][140/625] eta 0:03:04 lr 0.000030 wd 0.0500 time 0.3693 (0.3810) data time 0.0013 (0.0054) model time 0.3680 (0.3746) loss 2.9614 (2.5375) grad_norm 3.4007 (3.0827) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][150/625] eta 0:03:00 lr 0.000030 wd 0.0500 time 0.3664 (0.3807) data time 0.0008 (0.0051) model time 0.3657 (0.3748) loss 2.7539 (2.5337) grad_norm 4.0548 (3.0872) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][160/625] eta 0:02:59 lr 0.000030 wd 0.0500 time 0.3809 (0.3850) data time 0.0008 (0.0048) model time 0.3801 (0.3814) loss 2.2633 (2.5443) grad_norm 3.0746 (3.1022) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][170/625] eta 0:02:55 lr 0.000030 wd 0.0500 time 0.3710 (0.3855) data time 0.0008 (0.0046) model time 0.3702 (0.3824) loss 1.9883 (2.5445) grad_norm 5.3914 (3.1180) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][180/625] eta 0:02:51 lr 0.000030 wd 0.0500 time 0.3819 (0.3849) data time 0.0010 (0.0044) model time 0.3809 (0.3817) loss 2.6854 (2.5555) grad_norm 2.6252 (3.1096) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][190/625] eta 0:02:47 lr 0.000030 wd 0.0500 time 0.3701 (0.3843) data time 0.0010 (0.0042) model time 0.3691 (0.3811) loss 2.6529 (2.5539) grad_norm 2.8190 (3.1007) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][200/625] eta 0:02:43 lr 0.000030 wd 0.0500 time 0.3869 (0.3839) data time 0.0008 (0.0041) model time 0.3861 (0.3806) loss 1.6941 (2.5502) grad_norm 3.6425 (3.1269) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][210/625] eta 0:02:39 lr 0.000030 wd 0.0500 time 0.3644 (0.3843) data time 0.0009 (0.0039) model time 0.3636 (0.3814) loss 2.0400 (2.5548) grad_norm 2.6900 (3.1411) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][220/625] eta 0:02:35 lr 0.000030 wd 0.0500 time 0.3720 (0.3840) data time 0.0008 (0.0038) model time 0.3712 (0.3810) loss 1.7369 (2.5604) grad_norm 2.3497 (3.1363) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][230/625] eta 0:02:31 lr 0.000030 wd 0.0500 time 0.3746 (0.3835) data time 0.0008 (0.0037) model time 0.3738 (0.3806) loss 1.4920 (2.5559) grad_norm 2.0752 (3.1064) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:42:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][240/625] eta 0:02:27 lr 0.000030 wd 0.0500 time 0.3753 (0.3832) data time 0.0011 (0.0036) model time 0.3741 (0.3803) loss 2.4017 (2.5511) grad_norm 2.5176 (3.1195) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][250/625] eta 0:02:23 lr 0.000030 wd 0.0500 time 0.3755 (0.3830) data time 0.0010 (0.0035) model time 0.3745 (0.3801) loss 2.6510 (2.5554) grad_norm 3.7535 (3.1283) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][260/625] eta 0:02:19 lr 0.000030 wd 0.0500 time 0.4167 (0.3828) data time 0.0010 (0.0034) model time 0.4157 (0.3800) loss 3.0273 (2.5621) grad_norm 2.5552 (3.1611) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][270/625] eta 0:02:15 lr 0.000030 wd 0.0500 time 0.3784 (0.3826) data time 0.0010 (0.0033) model time 0.3774 (0.3797) loss 2.5963 (2.5590) grad_norm 3.2319 (3.1812) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][280/625] eta 0:02:11 lr 0.000030 wd 0.0500 time 0.3730 (0.3823) data time 0.0008 (0.0032) model time 0.3722 (0.3795) loss 2.4556 (2.5588) grad_norm 2.3991 (3.1773) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][290/625] eta 0:02:07 lr 0.000030 wd 0.0500 time 0.3740 (0.3821) data time 0.0010 (0.0031) model time 0.3730 (0.3793) loss 2.0171 (2.5594) grad_norm 4.3428 (3.2040) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][300/625] eta 0:02:04 lr 0.000030 wd 0.0500 time 0.3691 (0.3818) data time 0.0010 (0.0031) model time 0.3681 (0.3790) loss 2.5405 (2.5617) grad_norm 2.3634 (3.2055) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][310/625] eta 0:02:00 lr 0.000030 wd 0.0500 time 0.3715 (0.3816) data time 0.0010 (0.0030) model time 0.3705 (0.3788) loss 2.7467 (2.5546) grad_norm 3.7715 (3.2096) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][320/625] eta 0:01:56 lr 0.000030 wd 0.0500 time 0.3767 (0.3813) data time 0.0011 (0.0029) model time 0.3756 (0.3786) loss 2.7180 (2.5592) grad_norm 2.3155 (3.2034) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][330/625] eta 0:01:52 lr 0.000030 wd 0.0500 time 0.3744 (0.3811) data time 0.0010 (0.0029) model time 0.3734 (0.3784) loss 2.2975 (2.5596) grad_norm 2.1078 (3.1863) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][340/625] eta 0:01:48 lr 0.000030 wd 0.0500 time 0.5606 (0.3820) data time 0.0009 (0.0028) model time 0.5598 (0.3795) loss 1.9137 (2.5565) grad_norm 2.8572 (3.2619) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][350/625] eta 0:01:45 lr 0.000030 wd 0.0500 time 0.3736 (0.3834) data time 0.0008 (0.0028) model time 0.3728 (0.3812) loss 1.7312 (2.5520) grad_norm 2.8449 (3.2536) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][360/625] eta 0:01:41 lr 0.000030 wd 0.0500 time 0.3666 (0.3832) data time 0.0011 (0.0027) model time 0.3655 (0.3809) loss 2.4545 (2.5557) grad_norm 5.2087 (3.2558) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][370/625] eta 0:01:37 lr 0.000030 wd 0.0500 time 0.3636 (0.3829) data time 0.0008 (0.0027) model time 0.3627 (0.3807) loss 2.1721 (2.5525) grad_norm 3.0963 (3.2578) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][380/625] eta 0:01:33 lr 0.000030 wd 0.0500 time 0.3625 (0.3826) data time 0.0011 (0.0026) model time 0.3614 (0.3804) loss 1.9644 (2.5535) grad_norm 2.2945 (3.2527) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][390/625] eta 0:01:29 lr 0.000030 wd 0.0500 time 0.3704 (0.3824) data time 0.0008 (0.0026) model time 0.3695 (0.3802) loss 2.2667 (2.5504) grad_norm 2.4649 (3.2740) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:43:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][400/625] eta 0:01:26 lr 0.000030 wd 0.0500 time 0.3879 (0.3822) data time 0.0011 (0.0026) model time 0.3868 (0.3800) loss 3.0927 (2.5559) grad_norm 2.5187 (3.2674) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][410/625] eta 0:01:22 lr 0.000029 wd 0.0500 time 0.3635 (0.3821) data time 0.0011 (0.0025) model time 0.3624 (0.3799) loss 2.6922 (2.5603) grad_norm 2.8453 (3.2638) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][420/625] eta 0:01:18 lr 0.000029 wd 0.0500 time 0.3676 (0.3823) data time 0.0010 (0.0025) model time 0.3666 (0.3802) loss 1.3559 (2.5573) grad_norm 3.1075 (3.2461) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][430/625] eta 0:01:14 lr 0.000029 wd 0.0500 time 0.3702 (0.3821) data time 0.0008 (0.0025) model time 0.3694 (0.3799) loss 1.4338 (2.5543) grad_norm 3.2075 (3.2409) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][440/625] eta 0:01:10 lr 0.000029 wd 0.0500 time 0.3763 (0.3819) data time 0.0010 (0.0024) model time 0.3752 (0.3797) loss 2.9370 (2.5568) grad_norm 3.2467 (3.2299) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][450/625] eta 0:01:06 lr 0.000029 wd 0.0500 time 0.3766 (0.3817) data time 0.0010 (0.0024) model time 0.3756 (0.3796) loss 2.9966 (2.5620) grad_norm 2.2934 (3.2168) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][460/625] eta 0:01:03 lr 0.000029 wd 0.0500 time 0.5740 (0.3824) data time 0.0010 (0.0024) model time 0.5730 (0.3804) loss 2.7793 (2.5607) grad_norm 2.5412 (3.2098) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][470/625] eta 0:00:59 lr 0.000029 wd 0.0500 time 0.3675 (0.3830) data time 0.0010 (0.0023) model time 0.3665 (0.3810) loss 2.6904 (2.5631) grad_norm 2.3889 (3.2151) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][480/625] eta 0:00:55 lr 0.000029 wd 0.0500 time 0.3743 (0.3828) data time 0.0010 (0.0023) model time 0.3733 (0.3809) loss 2.7755 (2.5650) grad_norm 4.5219 (3.2109) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][490/625] eta 0:00:51 lr 0.000029 wd 0.0500 time 0.3807 (0.3826) data time 0.0008 (0.0023) model time 0.3800 (0.3807) loss 2.2087 (2.5659) grad_norm 2.9137 (3.2010) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][500/625] eta 0:00:47 lr 0.000029 wd 0.0500 time 0.3698 (0.3825) data time 0.0008 (0.0023) model time 0.3690 (0.3806) loss 2.7371 (2.5656) grad_norm 2.7377 (3.1950) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][510/625] eta 0:00:43 lr 0.000029 wd 0.0500 time 0.3737 (0.3823) data time 0.0008 (0.0022) model time 0.3729 (0.3804) loss 3.0348 (2.5646) grad_norm 3.0657 (3.1945) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][520/625] eta 0:00:40 lr 0.000029 wd 0.0500 time 0.3671 (0.3821) data time 0.0008 (0.0022) model time 0.3663 (0.3802) loss 3.0109 (2.5674) grad_norm 3.5412 (3.1843) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][530/625] eta 0:00:36 lr 0.000029 wd 0.0500 time 0.3772 (0.3820) data time 0.0011 (0.0022) model time 0.3761 (0.3800) loss 3.1806 (2.5685) grad_norm 2.5400 (3.1813) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][540/625] eta 0:00:32 lr 0.000029 wd 0.0500 time 0.3741 (0.3819) data time 0.0010 (0.0022) model time 0.3732 (0.3799) loss 2.6769 (2.5695) grad_norm 3.1333 (3.1959) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:44:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][550/625] eta 0:00:28 lr 0.000029 wd 0.0500 time 0.3728 (0.3817) data time 0.0010 (0.0021) model time 0.3718 (0.3798) loss 2.1338 (2.5690) grad_norm 1.9222 (3.1871) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][560/625] eta 0:00:24 lr 0.000029 wd 0.0500 time 0.3894 (0.3817) data time 0.0010 (0.0021) model time 0.3884 (0.3797) loss 2.8587 (2.5723) grad_norm 3.7614 (3.1896) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][570/625] eta 0:00:20 lr 0.000029 wd 0.0500 time 0.3783 (0.3815) data time 0.0010 (0.0021) model time 0.3773 (0.3796) loss 2.9695 (2.5728) grad_norm 2.9602 (3.1905) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][580/625] eta 0:00:17 lr 0.000029 wd 0.0500 time 0.3743 (0.3814) data time 0.0010 (0.0021) model time 0.3733 (0.3795) loss 2.7481 (2.5730) grad_norm 2.7798 (3.1855) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][590/625] eta 0:00:13 lr 0.000029 wd 0.0500 time 0.3815 (0.3813) data time 0.0007 (0.0021) model time 0.3808 (0.3794) loss 2.0005 (2.5765) grad_norm 1.7937 (3.1725) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][600/625] eta 0:00:09 lr 0.000029 wd 0.0500 time 0.3754 (0.3815) data time 0.0010 (0.0021) model time 0.3745 (0.3796) loss 2.5833 (2.5763) grad_norm 3.1907 (3.1702) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][610/625] eta 0:00:05 lr 0.000029 wd 0.0500 time 0.3662 (0.3813) data time 0.0007 (0.0020) model time 0.3654 (0.3795) loss 1.9709 (2.5728) grad_norm 2.7440 (3.1652) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [287/300][620/625] eta 0:00:01 lr 0.000029 wd 0.0500 time 0.3621 (0.3811) data time 0.0005 (0.0020) model time 0.3616 (0.3792) loss 2.9945 (2.5760) grad_norm 1.6600 (3.2443) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:24 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 287 training takes 0:03:58 [2024-09-01 15:45:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:45:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:45:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.512 (0.512) Loss 0.5068 (0.5068) Acc@1 89.209 (89.209) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:45:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.142) Loss 0.8267 (0.6065) Acc@1 80.859 (86.976) Acc@5 96.094 (97.852) Mem 11224MB [2024-09-01 15:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.119) Loss 0.9326 (0.7265) Acc@1 78.320 (84.031) Acc@5 95.312 (96.675) Mem 11224MB [2024-09-01 15:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.717 Acc@5 96.683 [2024-09-01 15:45:28 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.7% [2024-09-01 15:45:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.851 (0.851) Loss 0.5078 (0.5078) Acc@1 88.867 (88.867) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:45:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.096 (0.177) Loss 0.8208 (0.6006) Acc@1 80.615 (87.087) Acc@5 95.752 (97.878) Mem 11224MB [2024-09-01 15:45:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.137) Loss 0.9038 (0.7177) Acc@1 78.906 (84.252) Acc@5 95.410 (96.705) Mem 11224MB [2024-09-01 15:45:31 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.941 Acc@5 96.723 [2024-09-01 15:45:31 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:45:31 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.94% [2024-09-01 15:45:31 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 15:45:33 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 15:45:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][0/625] eta 0:08:19 lr 0.000029 wd 0.0500 time 0.7989 (0.7989) data time 0.4662 (0.4662) model time 0.0000 (0.0000) loss 2.7682 (2.7682) grad_norm 2.6892 (2.6892) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][10/625] eta 0:04:13 lr 0.000029 wd 0.0500 time 0.3774 (0.4128) data time 0.0011 (0.0433) model time 0.0000 (0.0000) loss 2.7533 (2.6440) grad_norm 2.7081 (3.1962) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][20/625] eta 0:03:57 lr 0.000029 wd 0.0500 time 0.3698 (0.3933) data time 0.0008 (0.0232) model time 0.0000 (0.0000) loss 2.7902 (2.7228) grad_norm 2.5374 (4.0534) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][30/625] eta 0:04:05 lr 0.000029 wd 0.0500 time 0.3718 (0.4124) data time 0.0010 (0.0160) model time 0.0000 (0.0000) loss 2.3235 (2.7037) grad_norm 2.1132 (4.2358) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][40/625] eta 0:03:55 lr 0.000029 wd 0.0500 time 0.3702 (0.4033) data time 0.0008 (0.0124) model time 0.0000 (0.0000) loss 2.6939 (2.7101) grad_norm 1.8210 (4.0831) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][50/625] eta 0:03:48 lr 0.000029 wd 0.0500 time 0.3760 (0.3976) data time 0.0009 (0.0101) model time 0.0000 (0.0000) loss 2.2291 (2.6705) grad_norm 4.7638 (3.9055) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:45:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][60/625] eta 0:03:42 lr 0.000029 wd 0.0500 time 0.3773 (0.3939) data time 0.0011 (0.0087) model time 0.3762 (0.3737) loss 2.9597 (2.6639) grad_norm 4.1937 (3.8068) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][70/625] eta 0:03:37 lr 0.000029 wd 0.0500 time 0.3824 (0.3916) data time 0.0012 (0.0076) model time 0.3812 (0.3752) loss 2.7752 (2.6815) grad_norm 12.6966 (3.8127) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][80/625] eta 0:03:32 lr 0.000029 wd 0.0500 time 0.3725 (0.3896) data time 0.0010 (0.0068) model time 0.3715 (0.3748) loss 2.2748 (2.6990) grad_norm 3.3233 (3.7109) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][90/625] eta 0:03:27 lr 0.000029 wd 0.0500 time 0.3797 (0.3883) data time 0.0012 (0.0061) model time 0.3785 (0.3752) loss 2.4535 (2.6809) grad_norm 2.3229 (3.6449) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][100/625] eta 0:03:23 lr 0.000029 wd 0.0500 time 0.3850 (0.3881) data time 0.0010 (0.0056) model time 0.3840 (0.3773) loss 2.7517 (2.6837) grad_norm 2.5024 (3.8244) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][110/625] eta 0:03:19 lr 0.000029 wd 0.0500 time 0.3918 (0.3870) data time 0.0010 (0.0052) model time 0.3908 (0.3768) loss 2.8601 (2.6690) grad_norm 4.5093 (3.7692) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][120/625] eta 0:03:14 lr 0.000029 wd 0.0500 time 0.3694 (0.3859) data time 0.0008 (0.0049) model time 0.3686 (0.3763) loss 2.5386 (2.6734) grad_norm 2.5519 (3.7557) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][130/625] eta 0:03:10 lr 0.000029 wd 0.0500 time 0.3767 (0.3849) data time 0.0011 (0.0046) model time 0.3756 (0.3758) loss 2.5770 (2.6570) grad_norm 14.4972 (3.7969) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][140/625] eta 0:03:06 lr 0.000029 wd 0.0500 time 0.3768 (0.3841) data time 0.0011 (0.0043) model time 0.3758 (0.3753) loss 2.5172 (2.6558) grad_norm 1.9935 (3.7429) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][150/625] eta 0:03:02 lr 0.000029 wd 0.0500 time 0.3729 (0.3834) data time 0.0010 (0.0041) model time 0.3719 (0.3751) loss 1.9343 (2.6427) grad_norm 3.5016 (3.6945) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][160/625] eta 0:02:58 lr 0.000029 wd 0.0500 time 0.3702 (0.3830) data time 0.0010 (0.0039) model time 0.3692 (0.3752) loss 1.8009 (2.6314) grad_norm 2.8728 (3.6816) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][170/625] eta 0:02:54 lr 0.000029 wd 0.0500 time 0.3721 (0.3825) data time 0.0010 (0.0038) model time 0.3711 (0.3751) loss 3.1025 (2.6265) grad_norm 2.9729 (3.6334) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][180/625] eta 0:02:50 lr 0.000029 wd 0.0500 time 0.3766 (0.3820) data time 0.0008 (0.0036) model time 0.3758 (0.3749) loss 2.6978 (2.6284) grad_norm 2.5472 (3.5985) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][190/625] eta 0:02:46 lr 0.000029 wd 0.0500 time 0.3737 (0.3818) data time 0.0011 (0.0035) model time 0.3726 (0.3750) loss 2.9060 (2.6330) grad_norm 2.9204 (3.5614) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][200/625] eta 0:02:42 lr 0.000028 wd 0.0500 time 0.3740 (0.3813) data time 0.0008 (0.0033) model time 0.3732 (0.3748) loss 2.1883 (2.6269) grad_norm 4.0892 (3.7784) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][210/625] eta 0:02:38 lr 0.000028 wd 0.0500 time 0.3752 (0.3810) data time 0.0008 (0.0032) model time 0.3744 (0.3747) loss 2.7689 (2.6271) grad_norm 2.7270 (3.7553) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:46:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][220/625] eta 0:02:34 lr 0.000028 wd 0.0500 time 0.3729 (0.3816) data time 0.0008 (0.0031) model time 0.3721 (0.3758) loss 2.8458 (2.6268) grad_norm 2.2708 (3.7421) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][230/625] eta 0:02:30 lr 0.000028 wd 0.0500 time 0.3787 (0.3815) data time 0.0010 (0.0030) model time 0.3776 (0.3759) loss 2.9019 (2.6212) grad_norm 4.0208 (3.7166) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][240/625] eta 0:02:27 lr 0.000028 wd 0.0500 time 0.3819 (0.3839) data time 0.0008 (0.0030) model time 0.3812 (0.3792) loss 2.4120 (2.6120) grad_norm 2.2183 (3.6721) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][250/625] eta 0:02:23 lr 0.000028 wd 0.0500 time 0.3728 (0.3835) data time 0.0011 (0.0029) model time 0.3717 (0.3789) loss 2.3378 (2.5996) grad_norm 27.3412 (3.7465) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][260/625] eta 0:02:19 lr 0.000028 wd 0.0500 time 0.3795 (0.3832) data time 0.0010 (0.0028) model time 0.3785 (0.3787) loss 3.1886 (2.6038) grad_norm 2.3520 (3.7184) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][270/625] eta 0:02:15 lr 0.000028 wd 0.0500 time 0.3747 (0.3830) data time 0.0010 (0.0027) model time 0.3737 (0.3786) loss 2.8827 (2.6008) grad_norm 2.9073 (3.6874) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][280/625] eta 0:02:12 lr 0.000028 wd 0.0500 time 0.3757 (0.3828) data time 0.0010 (0.0027) model time 0.3747 (0.3785) loss 3.1268 (2.5953) grad_norm 2.6574 (3.8629) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][290/625] eta 0:02:08 lr 0.000028 wd 0.0500 time 0.3779 (0.3826) data time 0.0010 (0.0026) model time 0.3768 (0.3783) loss 2.5672 (2.5905) grad_norm 8.5240 (3.8457) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][300/625] eta 0:02:04 lr 0.000028 wd 0.0500 time 0.3772 (0.3825) data time 0.0010 (0.0026) model time 0.3762 (0.3784) loss 2.3615 (2.5919) grad_norm 3.3263 (3.8647) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 15:47:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][310/625] eta 0:02:00 lr 0.000028 wd 0.0500 time 0.3595 (0.3823) data time 0.0008 (0.0025) model time 0.3587 (0.3783) loss 3.1390 (2.5924) grad_norm inf (inf) loss_scale 128.0000 (255.5884) mem 11224MB [2024-09-01 15:47:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][320/625] eta 0:01:56 lr 0.000028 wd 0.0500 time 0.3733 (0.3820) data time 0.0008 (0.0025) model time 0.3725 (0.3780) loss 2.7870 (2.5966) grad_norm 2.1020 (inf) loss_scale 128.0000 (251.6137) mem 11224MB [2024-09-01 15:47:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][330/625] eta 0:01:52 lr 0.000028 wd 0.0500 time 0.3752 (0.3818) data time 0.0008 (0.0024) model time 0.3744 (0.3779) loss 3.0808 (2.5964) grad_norm 7.4283 (inf) loss_scale 128.0000 (247.8792) mem 11224MB [2024-09-01 15:47:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][340/625] eta 0:01:48 lr 0.000028 wd 0.0500 time 0.3756 (0.3815) data time 0.0008 (0.0024) model time 0.3748 (0.3777) loss 2.2571 (2.5919) grad_norm 5.3142 (inf) loss_scale 128.0000 (244.3636) mem 11224MB [2024-09-01 15:47:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][350/625] eta 0:01:44 lr 0.000028 wd 0.0500 time 0.3685 (0.3814) data time 0.0010 (0.0023) model time 0.3674 (0.3776) loss 2.7371 (2.5919) grad_norm 2.3876 (inf) loss_scale 128.0000 (241.0484) mem 11224MB [2024-09-01 15:47:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][360/625] eta 0:01:41 lr 0.000028 wd 0.0500 time 0.3708 (0.3811) data time 0.0009 (0.0023) model time 0.3699 (0.3774) loss 3.0684 (2.5991) grad_norm 2.4753 (inf) loss_scale 128.0000 (237.9169) mem 11224MB [2024-09-01 15:47:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][370/625] eta 0:01:37 lr 0.000028 wd 0.0500 time 0.3717 (0.3809) data time 0.0010 (0.0023) model time 0.3707 (0.3773) loss 2.5581 (2.5994) grad_norm 2.9004 (inf) loss_scale 128.0000 (234.9542) mem 11224MB [2024-09-01 15:47:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][380/625] eta 0:01:33 lr 0.000028 wd 0.0500 time 0.3717 (0.3807) data time 0.0008 (0.0023) model time 0.3709 (0.3771) loss 2.1225 (2.6007) grad_norm 1.9657 (inf) loss_scale 128.0000 (232.1470) mem 11224MB [2024-09-01 15:48:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][390/625] eta 0:01:29 lr 0.000028 wd 0.0500 time 0.3796 (0.3806) data time 0.0008 (0.0022) model time 0.3788 (0.3770) loss 3.1758 (2.6040) grad_norm 2.5340 (inf) loss_scale 128.0000 (229.4834) mem 11224MB [2024-09-01 15:48:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][400/625] eta 0:01:25 lr 0.000028 wd 0.0500 time 0.3670 (0.3805) data time 0.0010 (0.0022) model time 0.3660 (0.3769) loss 2.1951 (2.6015) grad_norm 2.5269 (inf) loss_scale 128.0000 (226.9526) mem 11224MB [2024-09-01 15:48:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][410/625] eta 0:01:21 lr 0.000028 wd 0.0500 time 0.3767 (0.3803) data time 0.0008 (0.0022) model time 0.3758 (0.3768) loss 2.3084 (2.6001) grad_norm 6.3990 (inf) loss_scale 128.0000 (224.5450) mem 11224MB [2024-09-01 15:48:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][420/625] eta 0:01:18 lr 0.000028 wd 0.0500 time 0.6183 (0.3818) data time 0.0009 (0.0022) model time 0.6173 (0.3786) loss 2.1218 (2.5994) grad_norm 2.2787 (inf) loss_scale 128.0000 (222.2518) mem 11224MB [2024-09-01 15:48:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][430/625] eta 0:01:14 lr 0.000028 wd 0.0500 time 0.3734 (0.3823) data time 0.0008 (0.0021) model time 0.3726 (0.3792) loss 2.4043 (2.5960) grad_norm 3.9684 (inf) loss_scale 128.0000 (220.0650) mem 11224MB [2024-09-01 15:48:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][440/625] eta 0:01:10 lr 0.000028 wd 0.0500 time 0.3717 (0.3821) data time 0.0008 (0.0021) model time 0.3709 (0.3790) loss 2.2851 (2.5980) grad_norm 3.9859 (inf) loss_scale 128.0000 (217.9773) mem 11224MB [2024-09-01 15:48:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][450/625] eta 0:01:06 lr 0.000028 wd 0.0500 time 0.3769 (0.3819) data time 0.0010 (0.0021) model time 0.3758 (0.3789) loss 1.9822 (2.5940) grad_norm 2.2737 (inf) loss_scale 128.0000 (215.9823) mem 11224MB [2024-09-01 15:48:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][460/625] eta 0:01:02 lr 0.000028 wd 0.0500 time 0.3754 (0.3818) data time 0.0008 (0.0021) model time 0.3746 (0.3787) loss 2.8902 (2.5930) grad_norm 1.9468 (inf) loss_scale 128.0000 (214.0738) mem 11224MB [2024-09-01 15:48:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][470/625] eta 0:00:59 lr 0.000028 wd 0.0500 time 0.3721 (0.3816) data time 0.0010 (0.0020) model time 0.3711 (0.3786) loss 2.2920 (2.5872) grad_norm 2.1107 (inf) loss_scale 128.0000 (212.2463) mem 11224MB [2024-09-01 15:48:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][480/625] eta 0:00:55 lr 0.000028 wd 0.0500 time 0.3692 (0.3815) data time 0.0010 (0.0020) model time 0.3682 (0.3785) loss 2.6653 (2.5862) grad_norm 2.3640 (inf) loss_scale 128.0000 (210.4948) mem 11224MB [2024-09-01 15:48:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][490/625] eta 0:00:51 lr 0.000028 wd 0.0500 time 0.3734 (0.3813) data time 0.0010 (0.0020) model time 0.3724 (0.3784) loss 2.6955 (2.5874) grad_norm 2.8929 (inf) loss_scale 128.0000 (208.8147) mem 11224MB [2024-09-01 15:48:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][500/625] eta 0:00:47 lr 0.000028 wd 0.0500 time 0.3720 (0.3812) data time 0.0011 (0.0020) model time 0.3709 (0.3783) loss 2.4681 (2.5871) grad_norm 2.9383 (inf) loss_scale 128.0000 (207.2016) mem 11224MB [2024-09-01 15:48:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][510/625] eta 0:00:43 lr 0.000028 wd 0.0500 time 0.3696 (0.3810) data time 0.0011 (0.0020) model time 0.3686 (0.3781) loss 2.7630 (2.5897) grad_norm 3.3801 (inf) loss_scale 128.0000 (205.6517) mem 11224MB [2024-09-01 15:48:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][520/625] eta 0:00:39 lr 0.000028 wd 0.0500 time 0.3498 (0.3809) data time 0.0013 (0.0019) model time 0.3485 (0.3781) loss 2.6298 (2.5921) grad_norm 2.2239 (inf) loss_scale 128.0000 (204.1612) mem 11224MB [2024-09-01 15:48:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][530/625] eta 0:00:36 lr 0.000028 wd 0.0500 time 0.3802 (0.3808) data time 0.0011 (0.0019) model time 0.3791 (0.3780) loss 2.3723 (2.5897) grad_norm 4.5699 (inf) loss_scale 128.0000 (202.7269) mem 11224MB [2024-09-01 15:48:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][540/625] eta 0:00:32 lr 0.000028 wd 0.0500 time 0.3733 (0.3819) data time 0.0008 (0.0019) model time 0.3725 (0.3792) loss 2.9616 (2.5931) grad_norm 3.5640 (inf) loss_scale 128.0000 (201.3457) mem 11224MB [2024-09-01 15:49:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][550/625] eta 0:00:28 lr 0.000028 wd 0.0500 time 0.3727 (0.3820) data time 0.0008 (0.0019) model time 0.3718 (0.3794) loss 2.9167 (2.5929) grad_norm 2.4898 (inf) loss_scale 128.0000 (200.0145) mem 11224MB [2024-09-01 15:49:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][560/625] eta 0:00:24 lr 0.000028 wd 0.0500 time 0.3729 (0.3819) data time 0.0010 (0.0019) model time 0.3719 (0.3792) loss 2.7354 (2.5964) grad_norm 30.3187 (inf) loss_scale 128.0000 (198.7308) mem 11224MB [2024-09-01 15:49:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][570/625] eta 0:00:20 lr 0.000028 wd 0.0500 time 0.3742 (0.3818) data time 0.0008 (0.0019) model time 0.3734 (0.3792) loss 2.3247 (2.5933) grad_norm 5.1611 (inf) loss_scale 128.0000 (197.4921) mem 11224MB [2024-09-01 15:49:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][580/625] eta 0:00:17 lr 0.000028 wd 0.0500 time 0.3703 (0.3816) data time 0.0009 (0.0018) model time 0.3694 (0.3791) loss 3.0392 (2.5973) grad_norm 1.9759 (inf) loss_scale 128.0000 (196.2960) mem 11224MB [2024-09-01 15:49:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][590/625] eta 0:00:13 lr 0.000028 wd 0.0500 time 0.3631 (0.3815) data time 0.0011 (0.0018) model time 0.3620 (0.3789) loss 2.3953 (2.5968) grad_norm 2.2502 (inf) loss_scale 128.0000 (195.1404) mem 11224MB [2024-09-01 15:49:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][600/625] eta 0:00:09 lr 0.000028 wd 0.0500 time 0.3656 (0.3814) data time 0.0010 (0.0018) model time 0.3646 (0.3788) loss 2.8245 (2.6002) grad_norm 2.4913 (inf) loss_scale 128.0000 (194.0233) mem 11224MB [2024-09-01 15:49:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][610/625] eta 0:00:05 lr 0.000028 wd 0.0500 time 0.3652 (0.3815) data time 0.0008 (0.0018) model time 0.3644 (0.3790) loss 2.6583 (2.5984) grad_norm 3.0815 (inf) loss_scale 128.0000 (192.9427) mem 11224MB [2024-09-01 15:49:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [288/300][620/625] eta 0:00:01 lr 0.000028 wd 0.0500 time 0.3649 (0.3812) data time 0.0005 (0.0018) model time 0.3644 (0.3787) loss 2.5009 (2.6008) grad_norm 3.3884 (inf) loss_scale 128.0000 (191.8969) mem 11224MB [2024-09-01 15:49:31 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 288 training takes 0:03:58 [2024-09-01 15:49:31 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:49:32 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:49:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.516 (0.516) Loss 0.5132 (0.5132) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:49:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.095 (0.143) Loss 0.8350 (0.6053) Acc@1 80.615 (87.087) Acc@5 96.094 (97.896) Mem 11224MB [2024-09-01 15:49:34 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.119) Loss 0.9136 (0.7239) Acc@1 78.857 (84.117) Acc@5 95.312 (96.656) Mem 11224MB [2024-09-01 15:49:35 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.813 Acc@5 96.665 [2024-09-01 15:49:35 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 15:49:35 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.794 (0.794) Loss 0.5083 (0.5083) Acc@1 88.818 (88.818) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:49:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.097 (0.173) Loss 0.8223 (0.6012) Acc@1 80.762 (87.074) Acc@5 95.801 (97.883) Mem 11224MB [2024-09-01 15:49:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.136) Loss 0.9053 (0.7181) Acc@1 78.760 (84.219) Acc@5 95.410 (96.705) Mem 11224MB [2024-09-01 15:49:38 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.915 Acc@5 96.723 [2024-09-01 15:49:38 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:49:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][0/625] eta 0:12:58 lr 0.000028 wd 0.0500 time 1.2456 (1.2456) data time 0.8355 (0.8355) model time 0.0000 (0.0000) loss 2.0647 (2.0647) grad_norm 4.1519 (4.1519) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:49:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][10/625] eta 0:04:38 lr 0.000028 wd 0.0500 time 0.3692 (0.4527) data time 0.0010 (0.0769) model time 0.0000 (0.0000) loss 2.8688 (2.6337) grad_norm 2.4211 (2.8803) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:49:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][20/625] eta 0:04:11 lr 0.000027 wd 0.0500 time 0.3722 (0.4156) data time 0.0010 (0.0408) model time 0.0000 (0.0000) loss 1.8440 (2.6029) grad_norm 4.7601 (3.0584) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:49:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][30/625] eta 0:03:59 lr 0.000027 wd 0.0500 time 0.3763 (0.4027) data time 0.0010 (0.0280) model time 0.0000 (0.0000) loss 2.6648 (2.6004) grad_norm 2.5903 (3.1945) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:49:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][40/625] eta 0:03:51 lr 0.000027 wd 0.0500 time 0.3686 (0.3960) data time 0.0009 (0.0214) model time 0.0000 (0.0000) loss 2.5591 (2.6566) grad_norm 5.0365 (3.4730) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:49:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][50/625] eta 0:03:45 lr 0.000027 wd 0.0500 time 0.3740 (0.3916) data time 0.0007 (0.0174) model time 0.0000 (0.0000) loss 2.7834 (2.6515) grad_norm 2.7978 (3.3204) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][60/625] eta 0:03:39 lr 0.000027 wd 0.0500 time 0.3719 (0.3880) data time 0.0011 (0.0147) model time 0.3708 (0.3687) loss 2.9367 (2.6502) grad_norm 2.7617 (3.3641) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][70/625] eta 0:03:39 lr 0.000027 wd 0.0500 time 0.5862 (0.3950) data time 0.0010 (0.0128) model time 0.5851 (0.4026) loss 2.6440 (2.6267) grad_norm 3.1424 (3.2737) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][80/625] eta 0:03:34 lr 0.000027 wd 0.0500 time 0.3804 (0.3939) data time 0.0009 (0.0114) model time 0.3795 (0.3967) loss 2.9704 (2.6302) grad_norm 3.9289 (3.2421) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][90/625] eta 0:03:29 lr 0.000027 wd 0.0500 time 0.3769 (0.3919) data time 0.0010 (0.0102) model time 0.3759 (0.3913) loss 2.6155 (2.6057) grad_norm 2.7473 (3.1990) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][100/625] eta 0:03:24 lr 0.000027 wd 0.0500 time 0.3724 (0.3901) data time 0.0011 (0.0093) model time 0.3713 (0.3875) loss 2.8281 (2.6115) grad_norm 4.1962 (3.1960) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][110/625] eta 0:03:20 lr 0.000027 wd 0.0500 time 0.3669 (0.3888) data time 0.0008 (0.0086) model time 0.3661 (0.3854) loss 1.8509 (2.5943) grad_norm 3.2481 (3.2324) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][120/625] eta 0:03:15 lr 0.000027 wd 0.0500 time 0.3746 (0.3879) data time 0.0009 (0.0080) model time 0.3738 (0.3842) loss 2.3322 (2.5851) grad_norm 2.6418 (3.2219) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][130/625] eta 0:03:12 lr 0.000027 wd 0.0500 time 0.3813 (0.3881) data time 0.0008 (0.0074) model time 0.3805 (0.3847) loss 2.4568 (2.5903) grad_norm 2.7194 (3.3572) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][140/625] eta 0:03:07 lr 0.000027 wd 0.0500 time 0.3688 (0.3872) data time 0.0011 (0.0070) model time 0.3678 (0.3837) loss 2.6185 (2.5762) grad_norm 2.9901 (3.4032) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][150/625] eta 0:03:03 lr 0.000027 wd 0.0500 time 0.3757 (0.3863) data time 0.0010 (0.0066) model time 0.3747 (0.3826) loss 2.5885 (2.5867) grad_norm 2.4020 (3.3998) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][160/625] eta 0:02:59 lr 0.000027 wd 0.0500 time 0.3677 (0.3855) data time 0.0011 (0.0062) model time 0.3666 (0.3817) loss 2.9735 (2.5892) grad_norm 3.2833 (3.4148) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][170/625] eta 0:02:55 lr 0.000027 wd 0.0500 time 0.3879 (0.3848) data time 0.0010 (0.0059) model time 0.3869 (0.3809) loss 3.1396 (2.5975) grad_norm 3.8416 (3.4033) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][180/625] eta 0:02:51 lr 0.000027 wd 0.0500 time 0.3790 (0.3843) data time 0.0008 (0.0057) model time 0.3783 (0.3804) loss 2.9964 (2.5940) grad_norm 1.8320 (3.3728) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][190/625] eta 0:02:46 lr 0.000027 wd 0.0500 time 0.3749 (0.3839) data time 0.0011 (0.0054) model time 0.3739 (0.3800) loss 2.5716 (2.5926) grad_norm 3.8064 (3.3861) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][200/625] eta 0:02:42 lr 0.000027 wd 0.0500 time 0.3720 (0.3835) data time 0.0008 (0.0052) model time 0.3712 (0.3797) loss 2.9623 (2.5929) grad_norm 2.6192 (3.3752) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:50:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][210/625] eta 0:02:39 lr 0.000027 wd 0.0500 time 0.3806 (0.3832) data time 0.0008 (0.0050) model time 0.3798 (0.3795) loss 3.0118 (2.5956) grad_norm 2.4316 (3.4415) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][220/625] eta 0:02:35 lr 0.000027 wd 0.0500 time 0.3801 (0.3828) data time 0.0011 (0.0048) model time 0.3791 (0.3791) loss 2.2574 (2.5990) grad_norm 4.8526 (3.5339) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][230/625] eta 0:02:31 lr 0.000027 wd 0.0500 time 0.3752 (0.3825) data time 0.0007 (0.0047) model time 0.3745 (0.3788) loss 2.5422 (2.5982) grad_norm 3.0436 (3.5326) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][240/625] eta 0:02:27 lr 0.000027 wd 0.0500 time 0.3705 (0.3821) data time 0.0013 (0.0045) model time 0.3691 (0.3785) loss 2.6877 (2.5913) grad_norm 5.6098 (3.5244) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][250/625] eta 0:02:23 lr 0.000027 wd 0.0500 time 0.3839 (0.3826) data time 0.0008 (0.0044) model time 0.3831 (0.3792) loss 2.3564 (2.5947) grad_norm 3.1044 (3.4947) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][260/625] eta 0:02:19 lr 0.000027 wd 0.0500 time 0.3736 (0.3823) data time 0.0009 (0.0042) model time 0.3727 (0.3790) loss 2.7619 (2.5951) grad_norm 5.5546 (3.4830) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][270/625] eta 0:02:15 lr 0.000027 wd 0.0500 time 0.3720 (0.3820) data time 0.0011 (0.0041) model time 0.3709 (0.3787) loss 2.6268 (2.5877) grad_norm 2.8841 (3.4821) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][280/625] eta 0:02:12 lr 0.000027 wd 0.0500 time 0.3773 (0.3831) data time 0.0011 (0.0040) model time 0.3762 (0.3801) loss 2.0235 (2.5892) grad_norm 2.6230 (3.4819) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][290/625] eta 0:02:08 lr 0.000027 wd 0.0500 time 0.3723 (0.3841) data time 0.0010 (0.0039) model time 0.3713 (0.3814) loss 2.5750 (2.5856) grad_norm 5.7974 (3.4785) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][300/625] eta 0:02:04 lr 0.000027 wd 0.0500 time 0.3698 (0.3839) data time 0.0010 (0.0038) model time 0.3688 (0.3813) loss 2.9267 (2.5816) grad_norm 2.5516 (3.4996) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][310/625] eta 0:02:00 lr 0.000027 wd 0.0500 time 0.3791 (0.3836) data time 0.0008 (0.0037) model time 0.3784 (0.3809) loss 2.7577 (2.5821) grad_norm 3.4287 (3.4764) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][320/625] eta 0:01:56 lr 0.000027 wd 0.0500 time 0.3695 (0.3832) data time 0.0010 (0.0036) model time 0.3685 (0.3806) loss 2.5137 (2.5841) grad_norm 2.7698 (3.4976) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][330/625] eta 0:01:52 lr 0.000027 wd 0.0500 time 0.3937 (0.3830) data time 0.0010 (0.0036) model time 0.3927 (0.3804) loss 3.3279 (2.5854) grad_norm 2.9659 (3.4789) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][340/625] eta 0:01:49 lr 0.000027 wd 0.0500 time 0.3717 (0.3828) data time 0.0013 (0.0035) model time 0.3705 (0.3802) loss 2.9026 (2.5876) grad_norm 10.9141 (3.4799) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][350/625] eta 0:01:45 lr 0.000027 wd 0.0500 time 0.3684 (0.3826) data time 0.0009 (0.0034) model time 0.3675 (0.3800) loss 2.8827 (2.5870) grad_norm 4.4643 (3.4732) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:51:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][360/625] eta 0:01:41 lr 0.000027 wd 0.0500 time 0.3733 (0.3823) data time 0.0008 (0.0034) model time 0.3725 (0.3798) loss 3.2279 (2.5858) grad_norm 2.7906 (3.4526) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][370/625] eta 0:01:37 lr 0.000027 wd 0.0500 time 0.3754 (0.3821) data time 0.0011 (0.0033) model time 0.3743 (0.3796) loss 2.9119 (2.5817) grad_norm 3.7589 (3.4471) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][380/625] eta 0:01:33 lr 0.000027 wd 0.0500 time 0.3683 (0.3820) data time 0.0008 (0.0032) model time 0.3675 (0.3795) loss 3.1247 (2.5847) grad_norm 2.4659 (3.4279) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][390/625] eta 0:01:29 lr 0.000027 wd 0.0500 time 0.3818 (0.3819) data time 0.0011 (0.0032) model time 0.3807 (0.3795) loss 2.0401 (2.5872) grad_norm 4.0364 (3.4181) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][400/625] eta 0:01:25 lr 0.000027 wd 0.0500 time 0.3704 (0.3818) data time 0.0010 (0.0031) model time 0.3694 (0.3794) loss 1.8539 (2.5824) grad_norm 4.3301 (3.4096) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][410/625] eta 0:01:22 lr 0.000027 wd 0.0500 time 0.3737 (0.3816) data time 0.0009 (0.0031) model time 0.3728 (0.3792) loss 3.2440 (2.5818) grad_norm 2.4298 (3.4235) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][420/625] eta 0:01:18 lr 0.000027 wd 0.0500 time 0.3699 (0.3815) data time 0.0008 (0.0030) model time 0.3691 (0.3790) loss 2.1299 (2.5812) grad_norm 1.7647 (3.4353) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][430/625] eta 0:01:14 lr 0.000027 wd 0.0500 time 0.3735 (0.3814) data time 0.0010 (0.0030) model time 0.3725 (0.3790) loss 2.8909 (2.5830) grad_norm 2.0984 (3.4392) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][440/625] eta 0:01:10 lr 0.000027 wd 0.0500 time 0.3761 (0.3813) data time 0.0008 (0.0029) model time 0.3753 (0.3789) loss 2.9585 (2.5841) grad_norm 4.6701 (3.4337) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][450/625] eta 0:01:06 lr 0.000027 wd 0.0500 time 0.3769 (0.3812) data time 0.0010 (0.0029) model time 0.3759 (0.3789) loss 2.8943 (2.5829) grad_norm 1.8990 (3.4165) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][460/625] eta 0:01:03 lr 0.000027 wd 0.0500 time 0.5606 (0.3820) data time 0.0008 (0.0028) model time 0.5599 (0.3797) loss 2.7619 (2.5823) grad_norm 2.7823 (3.4183) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][470/625] eta 0:00:59 lr 0.000027 wd 0.0500 time 0.3795 (0.3837) data time 0.0008 (0.0028) model time 0.3787 (0.3817) loss 2.1670 (2.5788) grad_norm 3.7930 (3.4149) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][480/625] eta 0:00:55 lr 0.000027 wd 0.0500 time 0.3708 (0.3835) data time 0.0008 (0.0028) model time 0.3700 (0.3815) loss 2.2548 (2.5786) grad_norm 2.8685 (3.4030) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][490/625] eta 0:00:51 lr 0.000026 wd 0.0500 time 0.3789 (0.3833) data time 0.0010 (0.0027) model time 0.3779 (0.3813) loss 2.8210 (2.5770) grad_norm 2.4046 (3.3906) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][500/625] eta 0:00:47 lr 0.000026 wd 0.0500 time 0.3653 (0.3831) data time 0.0011 (0.0027) model time 0.3642 (0.3812) loss 2.8787 (2.5763) grad_norm 2.1043 (3.3820) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][510/625] eta 0:00:44 lr 0.000026 wd 0.0500 time 0.3692 (0.3830) data time 0.0008 (0.0027) model time 0.3683 (0.3810) loss 2.2736 (2.5754) grad_norm 2.6462 (3.4343) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:52:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][520/625] eta 0:00:40 lr 0.000026 wd 0.0500 time 0.3778 (0.3829) data time 0.0011 (0.0026) model time 0.3768 (0.3809) loss 2.0832 (2.5785) grad_norm 2.6670 (3.4281) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][530/625] eta 0:00:36 lr 0.000026 wd 0.0500 time 0.3941 (0.3827) data time 0.0013 (0.0026) model time 0.3929 (0.3808) loss 2.1859 (2.5767) grad_norm 4.1231 (3.4237) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][540/625] eta 0:00:32 lr 0.000026 wd 0.0500 time 0.3734 (0.3826) data time 0.0008 (0.0026) model time 0.3726 (0.3806) loss 1.7035 (2.5750) grad_norm 2.6179 (3.4118) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][550/625] eta 0:00:28 lr 0.000026 wd 0.0500 time 0.3776 (0.3825) data time 0.0011 (0.0026) model time 0.3765 (0.3805) loss 2.2989 (2.5748) grad_norm 2.7535 (3.4047) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][560/625] eta 0:00:24 lr 0.000026 wd 0.0500 time 0.3669 (0.3823) data time 0.0009 (0.0025) model time 0.3660 (0.3803) loss 2.7890 (2.5781) grad_norm 3.1243 (3.3979) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][570/625] eta 0:00:21 lr 0.000026 wd 0.0500 time 0.3997 (0.3823) data time 0.0011 (0.0025) model time 0.3986 (0.3803) loss 2.8420 (2.5757) grad_norm 3.9042 (3.4123) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][580/625] eta 0:00:17 lr 0.000026 wd 0.0500 time 0.5833 (0.3825) data time 0.0009 (0.0025) model time 0.5824 (0.3806) loss 2.8273 (2.5806) grad_norm 3.0382 (3.4519) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][590/625] eta 0:00:13 lr 0.000026 wd 0.0500 time 0.3762 (0.3833) data time 0.0009 (0.0025) model time 0.3752 (0.3815) loss 1.7100 (2.5809) grad_norm 3.5513 (3.4421) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][600/625] eta 0:00:09 lr 0.000026 wd 0.0500 time 0.3705 (0.3832) data time 0.0012 (0.0024) model time 0.3693 (0.3813) loss 3.0361 (2.5828) grad_norm 3.0309 (3.4378) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][610/625] eta 0:00:05 lr 0.000026 wd 0.0500 time 0.3599 (0.3830) data time 0.0005 (0.0024) model time 0.3594 (0.3812) loss 3.0676 (2.5809) grad_norm 3.5146 (3.4329) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [289/300][620/625] eta 0:00:01 lr 0.000026 wd 0.0500 time 0.3593 (0.3827) data time 0.0007 (0.0024) model time 0.3586 (0.3808) loss 2.9737 (2.5842) grad_norm 2.4723 (3.4545) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:37 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 289 training takes 0:03:59 [2024-09-01 15:53:37 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:53:38 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:53:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.495 (0.495) Loss 0.5137 (0.5137) Acc@1 88.867 (88.867) Acc@5 98.486 (98.486) Mem 11224MB [2024-09-01 15:53:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.140) Loss 0.8345 (0.6070) Acc@1 80.566 (86.963) Acc@5 95.996 (97.838) Mem 11224MB [2024-09-01 15:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.118) Loss 0.9165 (0.7251) Acc@1 78.955 (84.040) Acc@5 95.166 (96.610) Mem 11224MB [2024-09-01 15:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.773 Acc@5 96.631 [2024-09-01 15:53:41 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 15:53:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.767 (0.767) Loss 0.5088 (0.5088) Acc@1 88.818 (88.818) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 15:53:43 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.170) Loss 0.8228 (0.6012) Acc@1 80.859 (87.087) Acc@5 95.801 (97.892) Mem 11224MB [2024-09-01 15:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.133) Loss 0.9053 (0.7181) Acc@1 78.613 (84.224) Acc@5 95.410 (96.717) Mem 11224MB [2024-09-01 15:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.921 Acc@5 96.733 [2024-09-01 15:53:44 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 15:53:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][0/625] eta 0:13:15 lr 0.000026 wd 0.0500 time 1.2735 (1.2735) data time 0.6605 (0.6605) model time 0.0000 (0.0000) loss 2.8148 (2.8148) grad_norm 3.8871 (3.8871) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][10/625] eta 0:04:43 lr 0.000026 wd 0.0500 time 0.3899 (0.4608) data time 0.0010 (0.0610) model time 0.0000 (0.0000) loss 1.9471 (2.7016) grad_norm 3.1968 (3.6639) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][20/625] eta 0:04:14 lr 0.000026 wd 0.0500 time 0.3781 (0.4200) data time 0.0008 (0.0325) model time 0.0000 (0.0000) loss 3.0673 (2.6549) grad_norm 2.6786 (3.2522) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:53:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][30/625] eta 0:04:01 lr 0.000026 wd 0.0500 time 0.3737 (0.4061) data time 0.0011 (0.0223) model time 0.0000 (0.0000) loss 3.2308 (2.5412) grad_norm 5.9638 (3.3806) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][40/625] eta 0:03:53 lr 0.000026 wd 0.0500 time 0.3718 (0.3987) data time 0.0008 (0.0171) model time 0.0000 (0.0000) loss 2.4201 (2.5731) grad_norm 1.8285 (3.3814) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][50/625] eta 0:03:46 lr 0.000026 wd 0.0500 time 0.3708 (0.3938) data time 0.0010 (0.0140) model time 0.0000 (0.0000) loss 2.4078 (2.5878) grad_norm 2.6374 (3.3375) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][60/625] eta 0:03:40 lr 0.000026 wd 0.0500 time 0.3717 (0.3904) data time 0.0011 (0.0118) model time 0.3707 (0.3721) loss 2.6856 (2.5793) grad_norm 2.1779 (3.3397) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][70/625] eta 0:03:35 lr 0.000026 wd 0.0500 time 0.3808 (0.3882) data time 0.0010 (0.0103) model time 0.3798 (0.3727) loss 2.7136 (2.6070) grad_norm 3.1667 (3.5008) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][80/625] eta 0:03:30 lr 0.000026 wd 0.0500 time 0.3755 (0.3863) data time 0.0010 (0.0092) model time 0.3746 (0.3726) loss 2.1029 (2.5935) grad_norm 3.0869 (3.4554) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][90/625] eta 0:03:26 lr 0.000026 wd 0.0500 time 0.3777 (0.3851) data time 0.0010 (0.0083) model time 0.3767 (0.3728) loss 3.0023 (2.5656) grad_norm 5.5508 (3.5187) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][100/625] eta 0:03:24 lr 0.000026 wd 0.0500 time 0.3787 (0.3896) data time 0.0012 (0.0076) model time 0.3775 (0.3842) loss 2.8714 (2.5784) grad_norm 2.5645 (3.4532) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][110/625] eta 0:03:19 lr 0.000026 wd 0.0500 time 0.3728 (0.3883) data time 0.0008 (0.0070) model time 0.3720 (0.3826) loss 2.0629 (2.5873) grad_norm 4.4006 (3.4363) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][120/625] eta 0:03:15 lr 0.000026 wd 0.0500 time 0.3637 (0.3869) data time 0.0008 (0.0065) model time 0.3629 (0.3808) loss 2.3376 (2.5880) grad_norm 2.6295 (3.4274) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][130/625] eta 0:03:11 lr 0.000026 wd 0.0500 time 0.3730 (0.3869) data time 0.0010 (0.0061) model time 0.3720 (0.3815) loss 2.5533 (2.5975) grad_norm 1.8810 (3.3633) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][140/625] eta 0:03:07 lr 0.000026 wd 0.0500 time 0.3798 (0.3861) data time 0.0008 (0.0057) model time 0.3790 (0.3807) loss 3.0967 (2.6110) grad_norm 2.3109 (3.3560) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][150/625] eta 0:03:03 lr 0.000026 wd 0.0500 time 0.3720 (0.3854) data time 0.0010 (0.0054) model time 0.3710 (0.3801) loss 2.8882 (2.5975) grad_norm 2.4519 (3.3277) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][160/625] eta 0:02:58 lr 0.000026 wd 0.0500 time 0.3740 (0.3847) data time 0.0010 (0.0051) model time 0.3730 (0.3795) loss 1.6860 (2.5812) grad_norm 3.0593 (3.3873) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][170/625] eta 0:02:54 lr 0.000026 wd 0.0500 time 0.3768 (0.3844) data time 0.0011 (0.0049) model time 0.3757 (0.3793) loss 2.7274 (2.5836) grad_norm 4.3652 (3.3956) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][180/625] eta 0:02:50 lr 0.000026 wd 0.0500 time 0.3765 (0.3840) data time 0.0008 (0.0047) model time 0.3757 (0.3791) loss 3.6504 (2.5907) grad_norm 2.0633 (3.4891) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:54:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][190/625] eta 0:02:46 lr 0.000026 wd 0.0500 time 0.3681 (0.3835) data time 0.0011 (0.0045) model time 0.3669 (0.3787) loss 2.7352 (2.5935) grad_norm 3.1369 (3.5186) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][200/625] eta 0:02:42 lr 0.000026 wd 0.0500 time 0.3706 (0.3830) data time 0.0010 (0.0043) model time 0.3696 (0.3783) loss 2.8322 (2.6025) grad_norm 3.3340 (3.4826) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][210/625] eta 0:02:38 lr 0.000026 wd 0.0500 time 0.3658 (0.3826) data time 0.0010 (0.0042) model time 0.3648 (0.3780) loss 2.8520 (2.6083) grad_norm 4.5218 (3.4637) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][220/625] eta 0:02:34 lr 0.000026 wd 0.0500 time 0.3697 (0.3822) data time 0.0010 (0.0040) model time 0.3687 (0.3777) loss 2.8221 (2.6097) grad_norm 3.2411 (3.4168) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][230/625] eta 0:02:30 lr 0.000026 wd 0.0500 time 0.3849 (0.3820) data time 0.0012 (0.0039) model time 0.3837 (0.3776) loss 2.8601 (2.6054) grad_norm 3.4127 (3.4200) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][240/625] eta 0:02:26 lr 0.000026 wd 0.0500 time 0.3745 (0.3817) data time 0.0009 (0.0038) model time 0.3735 (0.3774) loss 1.4128 (2.5896) grad_norm 5.5700 (3.4175) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][250/625] eta 0:02:23 lr 0.000026 wd 0.0500 time 0.3715 (0.3814) data time 0.0008 (0.0037) model time 0.3707 (0.3772) loss 1.5671 (2.5897) grad_norm 3.1943 (3.3988) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][260/625] eta 0:02:19 lr 0.000026 wd 0.0500 time 0.3774 (0.3811) data time 0.0008 (0.0036) model time 0.3766 (0.3770) loss 2.5058 (2.5907) grad_norm 1.9495 (3.3814) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][270/625] eta 0:02:15 lr 0.000026 wd 0.0500 time 0.3763 (0.3809) data time 0.0008 (0.0035) model time 0.3755 (0.3769) loss 2.3423 (2.5808) grad_norm 2.9645 (3.4191) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][280/625] eta 0:02:11 lr 0.000026 wd 0.0500 time 0.3784 (0.3808) data time 0.0009 (0.0034) model time 0.3775 (0.3768) loss 3.0171 (2.5809) grad_norm 4.3216 (3.4176) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 15:55:33 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 15:55:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 15:55:34 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 15:57:23 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 15:57:24 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 15:57:32 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 15:57:47 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 15:57:47 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 15:57:49 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 15:57:52 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 15:57:52 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 290) [2024-09-01 15:57:52 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 15:58:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][290/625] eta 0:17:02 lr 0.000026 wd 0.0500 time 0.3428 (3.0526) data time 0.0006 (0.1689) model time 0.3422 (2.8837) loss 3.1426 (2.9769) grad_norm 2.8890 (3.2501) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][300/625] eta 0:06:45 lr 0.000026 wd 0.0500 time 0.3447 (1.2485) data time 0.0009 (0.0569) model time 0.3438 (1.1916) loss 2.8077 (2.8241) grad_norm 2.8201 (3.4806) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][310/625] eta 0:04:42 lr 0.000026 wd 0.0500 time 0.3377 (0.8954) data time 0.0007 (0.0345) model time 0.3370 (0.8609) loss 2.9293 (2.8740) grad_norm 16.3357 (3.7650) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][320/625] eta 0:03:44 lr 0.000026 wd 0.0500 time 0.3387 (0.7371) data time 0.0009 (0.0249) model time 0.3379 (0.7122) loss 2.7045 (2.8303) grad_norm 2.4478 (3.4230) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][330/625] eta 0:03:13 lr 0.000026 wd 0.0500 time 0.3378 (0.6557) data time 0.0008 (0.0196) model time 0.3370 (0.6362) loss 2.8191 (2.8080) grad_norm 9.9322 (3.5884) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][340/625] eta 0:02:50 lr 0.000026 wd 0.0500 time 0.3413 (0.5989) data time 0.0006 (0.0161) model time 0.3407 (0.5828) loss 2.3424 (2.7769) grad_norm 4.6542 (3.5059) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][350/625] eta 0:02:33 lr 0.000026 wd 0.0500 time 0.3478 (0.5596) data time 0.0008 (0.0138) model time 0.3469 (0.5458) loss 2.6921 (2.7421) grad_norm 4.1143 (3.5412) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][360/625] eta 0:02:20 lr 0.000026 wd 0.0500 time 0.3345 (0.5304) data time 0.0008 (0.0120) model time 0.3337 (0.5184) loss 2.1134 (2.6974) grad_norm 2.8821 (3.5933) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][370/625] eta 0:02:09 lr 0.000026 wd 0.0500 time 0.3410 (0.5082) data time 0.0006 (0.0107) model time 0.3404 (0.4974) loss 2.6003 (2.6874) grad_norm 2.7174 (3.5672) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][380/625] eta 0:02:00 lr 0.000025 wd 0.0500 time 0.3408 (0.4909) data time 0.0010 (0.0097) model time 0.3398 (0.4813) loss 2.5173 (2.6718) grad_norm 2.4265 (3.4962) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][390/625] eta 0:01:52 lr 0.000025 wd 0.0500 time 0.3436 (0.4768) data time 0.0009 (0.0088) model time 0.3426 (0.4680) loss 2.4793 (2.6871) grad_norm 4.1364 (3.5857) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][400/625] eta 0:01:44 lr 0.000025 wd 0.0500 time 0.3451 (0.4652) data time 0.0006 (0.0082) model time 0.3445 (0.4571) loss 1.9738 (2.6766) grad_norm 2.7126 (3.5411) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][410/625] eta 0:01:37 lr 0.000025 wd 0.0500 time 0.3457 (0.4555) data time 0.0006 (0.0076) model time 0.3451 (0.4479) loss 2.6670 (2.6792) grad_norm 2.2075 (3.4943) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:58:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][420/625] eta 0:01:31 lr 0.000025 wd 0.0500 time 0.3447 (0.4473) data time 0.0006 (0.0071) model time 0.3441 (0.4403) loss 2.4414 (2.6812) grad_norm 3.7400 (3.5041) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][430/625] eta 0:01:25 lr 0.000025 wd 0.0500 time 0.3377 (0.4402) data time 0.0008 (0.0066) model time 0.3369 (0.4335) loss 2.6041 (2.6720) grad_norm 2.5751 (3.4713) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][440/625] eta 0:01:20 lr 0.000025 wd 0.0500 time 0.3559 (0.4340) data time 0.0007 (0.0063) model time 0.3552 (0.4277) loss 2.9045 (2.6694) grad_norm 3.0029 (3.4525) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][450/625] eta 0:01:14 lr 0.000025 wd 0.0500 time 0.3416 (0.4285) data time 0.0008 (0.0059) model time 0.3408 (0.4226) loss 2.7460 (2.6681) grad_norm 2.6528 (3.4156) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][460/625] eta 0:01:09 lr 0.000025 wd 0.0500 time 0.3498 (0.4238) data time 0.0006 (0.0056) model time 0.3492 (0.4181) loss 2.7532 (2.6608) grad_norm 3.0742 (3.3948) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][470/625] eta 0:01:05 lr 0.000025 wd 0.0500 time 0.3421 (0.4195) data time 0.0008 (0.0054) model time 0.3414 (0.4142) loss 2.8844 (2.6542) grad_norm 2.2972 (3.3942) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][480/625] eta 0:01:00 lr 0.000025 wd 0.0500 time 0.3433 (0.4157) data time 0.0009 (0.0052) model time 0.3424 (0.4105) loss 1.6576 (2.6461) grad_norm 2.7624 (3.8837) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][490/625] eta 0:00:55 lr 0.000025 wd 0.0500 time 0.3397 (0.4121) data time 0.0008 (0.0049) model time 0.3388 (0.4072) loss 2.2134 (2.6356) grad_norm 2.5137 (3.9569) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][500/625] eta 0:00:51 lr 0.000025 wd 0.0500 time 0.3506 (0.4090) data time 0.0008 (0.0048) model time 0.3498 (0.4043) loss 2.4347 (2.6304) grad_norm 3.5350 (4.1216) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][510/625] eta 0:00:46 lr 0.000025 wd 0.0500 time 0.5635 (0.4080) data time 0.0007 (0.0046) model time 0.5628 (0.4034) loss 2.6992 (2.6291) grad_norm 2.7650 (4.0857) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][520/625] eta 0:00:42 lr 0.000025 wd 0.0500 time 0.3439 (0.4060) data time 0.0009 (0.0044) model time 0.3431 (0.4016) loss 2.9848 (2.6214) grad_norm 2.9199 (4.1374) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][530/625] eta 0:00:38 lr 0.000025 wd 0.0500 time 0.3412 (0.4035) data time 0.0008 (0.0043) model time 0.3404 (0.3992) loss 2.7875 (2.6254) grad_norm 2.7453 (4.0897) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][540/625] eta 0:00:34 lr 0.000025 wd 0.0500 time 0.3437 (0.4012) data time 0.0007 (0.0041) model time 0.3429 (0.3970) loss 2.1882 (2.6143) grad_norm 10.2382 (4.0846) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][550/625] eta 0:00:29 lr 0.000025 wd 0.0500 time 0.3462 (0.3991) data time 0.0006 (0.0040) model time 0.3455 (0.3951) loss 1.9438 (2.6065) grad_norm 2.8718 (4.0602) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][560/625] eta 0:00:25 lr 0.000025 wd 0.0500 time 0.3529 (0.3972) data time 0.0009 (0.0039) model time 0.3519 (0.3933) loss 2.6988 (2.6028) grad_norm 2.0558 (4.0231) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][570/625] eta 0:00:21 lr 0.000025 wd 0.0500 time 0.3432 (0.3953) data time 0.0008 (0.0038) model time 0.3424 (0.3915) loss 2.9584 (2.6020) grad_norm 3.2263 (4.0135) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][580/625] eta 0:00:17 lr 0.000025 wd 0.0500 time 0.3534 (0.3936) data time 0.0008 (0.0037) model time 0.3526 (0.3898) loss 2.6919 (2.5995) grad_norm 2.5237 (3.9674) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][590/625] eta 0:00:13 lr 0.000025 wd 0.0500 time 0.3573 (0.3920) data time 0.0008 (0.0036) model time 0.3565 (0.3884) loss 2.2042 (2.5916) grad_norm 3.2580 (3.9419) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 15:59:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][600/625] eta 0:00:09 lr 0.000025 wd 0.0500 time 0.3482 (0.3905) data time 0.0008 (0.0035) model time 0.3474 (0.3870) loss 3.0625 (2.5935) grad_norm 3.1408 (3.9175) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 16:00:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][610/625] eta 0:00:05 lr 0.000025 wd 0.0500 time 0.3386 (0.3892) data time 0.0004 (0.0035) model time 0.3382 (0.3857) loss 3.0415 (2.6018) grad_norm 2.8888 (3.9041) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 16:00:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [290/300][620/625] eta 0:00:01 lr 0.000025 wd 0.0500 time 0.3353 (0.3876) data time 0.0004 (0.0034) model time 0.3349 (0.3842) loss 3.3482 (2.6002) grad_norm 4.0651 (3.8700) loss_scale 128.0000 (128.0000) mem 11219MB [2024-09-01 16:00:07 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 290 training takes 0:02:11 [2024-09-01 16:00:07 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:00:09 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:00:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.426 (0.426) Loss 0.5112 (0.5112) Acc@1 89.014 (89.014) Acc@5 98.633 (98.633) Mem 11219MB [2024-09-01 16:00:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.129) Loss 0.8389 (0.6082) Acc@1 80.518 (87.034) Acc@5 95.947 (97.860) Mem 11219MB [2024-09-01 16:00:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.110) Loss 0.9146 (0.7266) Acc@1 78.955 (84.152) Acc@5 95.312 (96.670) Mem 11219MB [2024-09-01 16:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.833 Acc@5 96.699 [2024-09-01 16:00:14 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:00:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.922 (0.922) Loss 0.5088 (0.5088) Acc@1 88.867 (88.867) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 16:00:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.174) Loss 0.8237 (0.6016) Acc@1 80.908 (87.114) Acc@5 95.801 (97.892) Mem 11219MB [2024-09-01 16:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.134) Loss 0.9062 (0.7186) Acc@1 78.662 (84.229) Acc@5 95.459 (96.717) Mem 11219MB [2024-09-01 16:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.933 Acc@5 96.733 [2024-09-01 16:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:00:17 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.93% [2024-09-01 16:00:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 16:00:19 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 16:00:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][0/625] eta 0:07:45 lr 0.000025 wd 0.0500 time 0.7440 (0.7440) data time 0.4035 (0.4035) model time 0.0000 (0.0000) loss 2.8547 (2.8547) grad_norm 3.0527 (3.0527) loss_scale 128.0000 (128.0000) mem 11227MB [2024-09-01 16:00:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][10/625] eta 0:04:15 lr 0.000025 wd 0.0500 time 0.3465 (0.4153) data time 0.0008 (0.0375) model time 0.0000 (0.0000) loss 2.1830 (2.6116) grad_norm 3.1365 (3.1397) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][20/625] eta 0:03:51 lr 0.000025 wd 0.0500 time 0.3451 (0.3822) data time 0.0006 (0.0202) model time 0.0000 (0.0000) loss 3.1495 (2.6226) grad_norm 2.9073 (3.2326) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][30/625] eta 0:03:43 lr 0.000025 wd 0.0500 time 0.3396 (0.3759) data time 0.0008 (0.0140) model time 0.0000 (0.0000) loss 2.7890 (2.5995) grad_norm 2.6406 (3.2701) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][40/625] eta 0:03:35 lr 0.000025 wd 0.0500 time 0.3486 (0.3681) data time 0.0006 (0.0107) model time 0.0000 (0.0000) loss 2.4085 (2.6100) grad_norm 2.9018 (3.2543) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][50/625] eta 0:03:29 lr 0.000025 wd 0.0500 time 0.3404 (0.3635) data time 0.0008 (0.0088) model time 0.0000 (0.0000) loss 1.7952 (2.5469) grad_norm 2.1808 (3.1769) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][60/625] eta 0:03:23 lr 0.000025 wd 0.0500 time 0.3492 (0.3602) data time 0.0007 (0.0075) model time 0.3485 (0.3428) loss 2.6906 (2.5893) grad_norm 2.0909 (3.1675) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][70/625] eta 0:03:18 lr 0.000025 wd 0.0500 time 0.3473 (0.3582) data time 0.0005 (0.0066) model time 0.3468 (0.3440) loss 2.5861 (2.6069) grad_norm 3.3655 (3.2141) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][80/625] eta 0:03:14 lr 0.000025 wd 0.0500 time 0.3458 (0.3567) data time 0.0006 (0.0059) model time 0.3451 (0.3443) loss 2.3347 (2.5891) grad_norm 2.4810 (3.1376) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][90/625] eta 0:03:10 lr 0.000025 wd 0.0500 time 0.3427 (0.3552) data time 0.0009 (0.0053) model time 0.3418 (0.3437) loss 2.9589 (2.6140) grad_norm 2.5577 (3.1063) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][100/625] eta 0:03:05 lr 0.000025 wd 0.0500 time 0.3441 (0.3541) data time 0.0008 (0.0049) model time 0.3432 (0.3436) loss 2.1208 (2.6237) grad_norm 3.3555 (3.0709) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:00:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][110/625] eta 0:03:01 lr 0.000025 wd 0.0500 time 0.3477 (0.3532) data time 0.0006 (0.0045) model time 0.3471 (0.3435) loss 3.0521 (2.6250) grad_norm 2.6716 (3.4561) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][120/625] eta 0:02:58 lr 0.000025 wd 0.0500 time 0.3395 (0.3525) data time 0.0006 (0.0042) model time 0.3388 (0.3436) loss 2.9018 (2.6015) grad_norm 2.4722 (3.4244) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][130/625] eta 0:02:55 lr 0.000025 wd 0.0500 time 0.3516 (0.3549) data time 0.0007 (0.0039) model time 0.3509 (0.3486) loss 2.5189 (2.5850) grad_norm 2.8185 (3.4108) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][140/625] eta 0:02:51 lr 0.000025 wd 0.0500 time 0.3406 (0.3542) data time 0.0008 (0.0037) model time 0.3398 (0.3481) loss 2.0288 (2.5821) grad_norm 2.3091 (3.4003) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][150/625] eta 0:02:48 lr 0.000025 wd 0.0500 time 0.3471 (0.3549) data time 0.0006 (0.0035) model time 0.3465 (0.3496) loss 2.5090 (2.5931) grad_norm 2.2599 (3.3856) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][160/625] eta 0:02:44 lr 0.000025 wd 0.0500 time 0.3514 (0.3543) data time 0.0007 (0.0034) model time 0.3507 (0.3492) loss 1.8407 (2.5840) grad_norm 2.9092 (3.3801) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][170/625] eta 0:02:41 lr 0.000025 wd 0.0500 time 0.3410 (0.3540) data time 0.0007 (0.0032) model time 0.3402 (0.3490) loss 2.8409 (2.5925) grad_norm 2.1460 (3.3697) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][180/625] eta 0:02:37 lr 0.000025 wd 0.0500 time 0.3467 (0.3535) data time 0.0006 (0.0031) model time 0.3460 (0.3488) loss 3.0277 (2.5986) grad_norm 2.1054 (3.3966) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][190/625] eta 0:02:33 lr 0.000025 wd 0.0500 time 0.3488 (0.3531) data time 0.0008 (0.0030) model time 0.3480 (0.3485) loss 2.7395 (2.5856) grad_norm 2.8354 (3.3902) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][200/625] eta 0:02:29 lr 0.000025 wd 0.0500 time 0.3411 (0.3527) data time 0.0006 (0.0029) model time 0.3405 (0.3482) loss 3.0570 (2.5871) grad_norm 2.2592 (3.3981) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][210/625] eta 0:02:26 lr 0.000025 wd 0.0500 time 0.3523 (0.3524) data time 0.0008 (0.0028) model time 0.3515 (0.3479) loss 1.8931 (2.5854) grad_norm 4.6090 (3.4366) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][220/625] eta 0:02:22 lr 0.000025 wd 0.0500 time 0.3477 (0.3521) data time 0.0008 (0.0027) model time 0.3469 (0.3478) loss 2.4837 (2.5918) grad_norm 2.6854 (3.4283) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][230/625] eta 0:02:18 lr 0.000025 wd 0.0500 time 0.3448 (0.3518) data time 0.0008 (0.0026) model time 0.3440 (0.3476) loss 2.8100 (2.5955) grad_norm 3.8310 (3.4161) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][240/625] eta 0:02:15 lr 0.000025 wd 0.0500 time 0.3401 (0.3515) data time 0.0008 (0.0025) model time 0.3394 (0.3474) loss 2.1530 (2.5962) grad_norm 4.5256 (3.4096) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][250/625] eta 0:02:11 lr 0.000025 wd 0.0500 time 0.3442 (0.3513) data time 0.0006 (0.0025) model time 0.3437 (0.3473) loss 3.0104 (2.6015) grad_norm 3.2698 (3.5505) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][260/625] eta 0:02:08 lr 0.000025 wd 0.0500 time 0.3449 (0.3511) data time 0.0008 (0.0024) model time 0.3441 (0.3472) loss 2.7490 (2.6025) grad_norm 2.0513 (3.5402) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][270/625] eta 0:02:04 lr 0.000025 wd 0.0500 time 0.3458 (0.3509) data time 0.0006 (0.0023) model time 0.3452 (0.3472) loss 2.3156 (2.5964) grad_norm 3.3487 (3.5189) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:01:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][280/625] eta 0:02:01 lr 0.000025 wd 0.0500 time 0.3489 (0.3508) data time 0.0008 (0.0023) model time 0.3481 (0.3471) loss 2.4496 (2.5960) grad_norm 3.1021 (3.5199) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][290/625] eta 0:01:57 lr 0.000025 wd 0.0500 time 0.3411 (0.3506) data time 0.0007 (0.0022) model time 0.3404 (0.3470) loss 2.9183 (2.5985) grad_norm 2.1691 (3.4884) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][300/625] eta 0:01:53 lr 0.000025 wd 0.0500 time 0.3464 (0.3505) data time 0.0006 (0.0022) model time 0.3458 (0.3469) loss 1.5851 (2.5991) grad_norm 5.2236 (3.4764) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][310/625] eta 0:01:50 lr 0.000025 wd 0.0500 time 0.3461 (0.3504) data time 0.0007 (0.0022) model time 0.3454 (0.3469) loss 2.8066 (2.5926) grad_norm 2.1418 (3.4632) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][320/625] eta 0:01:46 lr 0.000024 wd 0.0500 time 0.3406 (0.3502) data time 0.0007 (0.0021) model time 0.3399 (0.3468) loss 2.7531 (2.5901) grad_norm 2.0701 (3.4604) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][330/625] eta 0:01:43 lr 0.000024 wd 0.0500 time 0.3453 (0.3500) data time 0.0006 (0.0021) model time 0.3447 (0.3467) loss 3.1035 (2.5944) grad_norm 3.3360 (3.4394) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][340/625] eta 0:01:40 lr 0.000024 wd 0.0500 time 0.3505 (0.3512) data time 0.0008 (0.0020) model time 0.3497 (0.3481) loss 2.0699 (2.5967) grad_norm 3.6437 (3.4432) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][350/625] eta 0:01:36 lr 0.000024 wd 0.0500 time 0.3449 (0.3510) data time 0.0006 (0.0020) model time 0.3443 (0.3480) loss 1.7844 (2.5970) grad_norm 2.6966 (3.4430) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][360/625] eta 0:01:33 lr 0.000024 wd 0.0500 time 0.3586 (0.3514) data time 0.0006 (0.0020) model time 0.3580 (0.3485) loss 1.9509 (2.5907) grad_norm 2.5441 (3.4298) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][370/625] eta 0:01:29 lr 0.000024 wd 0.0500 time 0.3449 (0.3512) data time 0.0008 (0.0019) model time 0.3441 (0.3484) loss 3.1159 (2.5947) grad_norm 2.8751 (3.4124) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][380/625] eta 0:01:26 lr 0.000024 wd 0.0500 time 0.3398 (0.3510) data time 0.0007 (0.0019) model time 0.3390 (0.3482) loss 1.7579 (2.5867) grad_norm 3.7277 (3.4009) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][390/625] eta 0:01:22 lr 0.000024 wd 0.0500 time 0.3395 (0.3509) data time 0.0009 (0.0019) model time 0.3386 (0.3481) loss 3.0301 (2.5888) grad_norm 4.2280 (3.4027) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][400/625] eta 0:01:18 lr 0.000024 wd 0.0500 time 0.3467 (0.3508) data time 0.0008 (0.0019) model time 0.3460 (0.3481) loss 2.5294 (2.5944) grad_norm 3.9983 (3.4985) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][410/625] eta 0:01:15 lr 0.000024 wd 0.0500 time 0.3387 (0.3505) data time 0.0009 (0.0018) model time 0.3379 (0.3479) loss 2.3797 (2.5920) grad_norm 2.3978 (3.4757) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][420/625] eta 0:01:11 lr 0.000024 wd 0.0500 time 0.3420 (0.3505) data time 0.0007 (0.0018) model time 0.3412 (0.3478) loss 2.8479 (2.5913) grad_norm 1.9390 (3.4652) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][430/625] eta 0:01:08 lr 0.000024 wd 0.0500 time 0.3453 (0.3503) data time 0.0008 (0.0018) model time 0.3445 (0.3477) loss 2.8666 (2.5914) grad_norm 3.2555 (3.4855) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:02:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][440/625] eta 0:01:04 lr 0.000024 wd 0.0500 time 0.3467 (0.3503) data time 0.0008 (0.0018) model time 0.3460 (0.3477) loss 2.9688 (2.5937) grad_norm 2.6400 (3.4775) loss_scale 256.0000 (129.7415) mem 11224MB [2024-09-01 16:02:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][450/625] eta 0:01:01 lr 0.000024 wd 0.0500 time 0.3446 (0.3502) data time 0.0008 (0.0017) model time 0.3438 (0.3476) loss 2.5538 (2.5933) grad_norm 7.6765 (3.4822) loss_scale 256.0000 (132.5410) mem 11224MB [2024-09-01 16:02:59 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 16:02:59 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:02:59 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:04:50 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 16:04:51 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 16:05:04 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 16:05:13 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 16:05:13 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 16:05:15 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 16:05:18 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 16:05:18 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 291) [2024-09-01 16:05:18 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 16:05:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][460/625] eta 0:06:39 lr 0.000024 wd 0.0500 time 0.3371 (2.4235) data time 0.0007 (0.1304) model time 0.3363 (2.2931) loss 3.1646 (2.9832) grad_norm 2.5451 (3.1414) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:05:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][470/625] eta 0:02:53 lr 0.000024 wd 0.0500 time 0.3460 (1.1224) data time 0.0007 (0.0495) model time 0.3453 (1.0730) loss 2.7353 (2.8112) grad_norm 3.9721 (3.6159) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:05:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][480/625] eta 0:02:00 lr 0.000024 wd 0.0500 time 0.3411 (0.8310) data time 0.0007 (0.0308) model time 0.3404 (0.8003) loss 2.7204 (2.8495) grad_norm 2.3828 (3.3852) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:05:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][490/625] eta 0:01:33 lr 0.000024 wd 0.0500 time 0.3425 (0.6955) data time 0.0008 (0.0225) model time 0.3417 (0.6730) loss 2.9408 (2.8334) grad_norm 2.7729 (3.5562) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:05:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][500/625] eta 0:01:18 lr 0.000024 wd 0.0500 time 0.3411 (0.6248) data time 0.0008 (0.0178) model time 0.3402 (0.6070) loss 2.3311 (2.7801) grad_norm 3.8030 (3.5212) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:05:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][510/625] eta 0:01:06 lr 0.000024 wd 0.0500 time 0.3468 (0.5746) data time 0.0006 (0.0148) model time 0.3462 (0.5599) loss 3.0818 (2.7814) grad_norm 3.0363 (3.4643) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:05:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][520/625] eta 0:00:56 lr 0.000024 wd 0.0500 time 0.3480 (0.5397) data time 0.0006 (0.0127) model time 0.3474 (0.5270) loss 2.1857 (2.7458) grad_norm 4.3407 (3.4127) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][530/625] eta 0:00:48 lr 0.000024 wd 0.0500 time 0.3397 (0.5138) data time 0.0009 (0.0111) model time 0.3388 (0.5027) loss 2.7952 (2.7211) grad_norm 4.1860 (3.3941) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][540/625] eta 0:00:42 lr 0.000024 wd 0.0500 time 0.3393 (0.4943) data time 0.0008 (0.0099) model time 0.3385 (0.4843) loss 2.2065 (2.6894) grad_norm 2.8532 (3.3702) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][550/625] eta 0:00:35 lr 0.000024 wd 0.0500 time 0.3462 (0.4785) data time 0.0006 (0.0090) model time 0.3456 (0.4695) loss 2.6938 (2.6874) grad_norm 3.1658 (3.3357) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][560/625] eta 0:00:30 lr 0.000024 wd 0.0500 time 0.3542 (0.4659) data time 0.0008 (0.0082) model time 0.3534 (0.4577) loss 2.8154 (2.7045) grad_norm 2.5648 (3.2915) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][570/625] eta 0:00:25 lr 0.000024 wd 0.0500 time 0.3496 (0.4553) data time 0.0008 (0.0076) model time 0.3488 (0.4477) loss 3.3346 (2.6948) grad_norm 3.3556 (3.2583) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][580/625] eta 0:00:20 lr 0.000024 wd 0.0500 time 0.3453 (0.4463) data time 0.0006 (0.0071) model time 0.3447 (0.4393) loss 1.8770 (2.6949) grad_norm 2.6609 (3.8036) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][590/625] eta 0:00:15 lr 0.000024 wd 0.0500 time 0.3449 (0.4388) data time 0.0009 (0.0066) model time 0.3440 (0.4321) loss 2.1926 (2.6949) grad_norm 2.4417 (3.7326) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][600/625] eta 0:00:10 lr 0.000024 wd 0.0500 time 0.3419 (0.4322) data time 0.0007 (0.0062) model time 0.3412 (0.4259) loss 2.2307 (2.6818) grad_norm 3.5026 (3.6968) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][610/625] eta 0:00:06 lr 0.000024 wd 0.0500 time 0.3345 (0.4264) data time 0.0006 (0.0059) model time 0.3339 (0.4205) loss 2.6370 (2.6790) grad_norm 3.2339 (3.6485) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [291/300][620/625] eta 0:00:02 lr 0.000024 wd 0.0500 time 0.3429 (0.4211) data time 0.0006 (0.0056) model time 0.3423 (0.4155) loss 2.8471 (2.6834) grad_norm 3.8819 (3.6363) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:06:33 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 291 training takes 0:01:11 [2024-09-01 16:06:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:06:35 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:06:36 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.427 (0.427) Loss 0.5112 (0.5112) Acc@1 88.770 (88.770) Acc@5 98.682 (98.682) Mem 11219MB [2024-09-01 16:06:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.131) Loss 0.8306 (0.6072) Acc@1 81.006 (87.043) Acc@5 95.801 (97.834) Mem 11219MB [2024-09-01 16:06:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.112) Loss 0.9194 (0.7274) Acc@1 78.955 (84.070) Acc@5 95.312 (96.640) Mem 11219MB [2024-09-01 16:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.765 Acc@5 96.661 [2024-09-01 16:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:06:40 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.744 (0.744) Loss 0.5083 (0.5083) Acc@1 89.014 (89.014) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 16:06:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.162) Loss 0.8242 (0.6016) Acc@1 80.908 (87.109) Acc@5 95.752 (97.874) Mem 11219MB [2024-09-01 16:06:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.129) Loss 0.9072 (0.7188) Acc@1 78.662 (84.229) Acc@5 95.459 (96.703) Mem 11219MB [2024-09-01 16:06:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.933 Acc@5 96.723 [2024-09-01 16:06:43 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:06:43 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.93% [2024-09-01 16:06:43 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 16:06:44 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 16:06:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][0/625] eta 0:07:50 lr 0.000024 wd 0.0500 time 0.7527 (0.7527) data time 0.4124 (0.4124) model time 0.0000 (0.0000) loss 2.7692 (2.7692) grad_norm 3.6204 (3.6204) loss_scale 256.0000 (256.0000) mem 11227MB [2024-09-01 16:06:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][10/625] eta 0:04:03 lr 0.000024 wd 0.0500 time 0.3428 (0.3962) data time 0.0008 (0.0389) model time 0.0000 (0.0000) loss 2.7227 (2.5149) grad_norm 2.6753 (3.0753) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:06:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][20/625] eta 0:03:44 lr 0.000024 wd 0.0500 time 0.3413 (0.3717) data time 0.0009 (0.0208) model time 0.0000 (0.0000) loss 2.3815 (2.5703) grad_norm 3.8622 (3.3296) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:06:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][30/625] eta 0:03:35 lr 0.000024 wd 0.0500 time 0.3442 (0.3625) data time 0.0009 (0.0144) model time 0.0000 (0.0000) loss 2.6103 (2.5195) grad_norm 6.1526 (3.3207) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:06:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][40/625] eta 0:03:29 lr 0.000024 wd 0.0500 time 0.3545 (0.3581) data time 0.0008 (0.0111) model time 0.0000 (0.0000) loss 2.9685 (2.5456) grad_norm 2.4573 (3.2251) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][50/625] eta 0:03:24 lr 0.000024 wd 0.0500 time 0.3458 (0.3552) data time 0.0006 (0.0091) model time 0.0000 (0.0000) loss 2.7718 (2.5578) grad_norm 3.1131 (3.2381) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][60/625] eta 0:03:19 lr 0.000024 wd 0.0500 time 0.3466 (0.3533) data time 0.0007 (0.0078) model time 0.3459 (0.3423) loss 1.7098 (2.5504) grad_norm 2.4619 (3.2436) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][70/625] eta 0:03:18 lr 0.000024 wd 0.0500 time 0.3469 (0.3582) data time 0.0007 (0.0068) model time 0.3463 (0.3648) loss 2.6238 (2.5575) grad_norm 2.3255 (3.2349) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][80/625] eta 0:03:14 lr 0.000024 wd 0.0500 time 0.3477 (0.3565) data time 0.0006 (0.0061) model time 0.3471 (0.3577) loss 2.7378 (2.5453) grad_norm 2.7921 (3.1790) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][90/625] eta 0:03:10 lr 0.000024 wd 0.0500 time 0.3434 (0.3552) data time 0.0006 (0.0055) model time 0.3428 (0.3543) loss 2.8646 (2.5224) grad_norm 3.6529 (3.1855) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][100/625] eta 0:03:05 lr 0.000024 wd 0.0500 time 0.3422 (0.3541) data time 0.0006 (0.0050) model time 0.3416 (0.3520) loss 2.8274 (2.5156) grad_norm 2.1940 (3.1789) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][110/625] eta 0:03:01 lr 0.000024 wd 0.0500 time 0.3433 (0.3533) data time 0.0009 (0.0047) model time 0.3425 (0.3508) loss 2.6308 (2.5374) grad_norm 2.5226 (3.1666) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][120/625] eta 0:02:58 lr 0.000024 wd 0.0500 time 0.3493 (0.3525) data time 0.0006 (0.0044) model time 0.3487 (0.3496) loss 1.5056 (2.5253) grad_norm 5.2189 (3.2214) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:07:27 msvmambav3_small_224] (main_hfai_mnodes.py 383): INFO Suspend command received, saving checkpoint and exiting [2024-09-01 16:07:27 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:07:29 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:09:09 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 16:09:10 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 16:09:14 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 16:09:33 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 16:09:33 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 16:09:35 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 16:09:37 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 16:09:37 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 292) [2024-09-01 16:09:37 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 16:09:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][130/625] eta 0:13:54 lr 0.000024 wd 0.0500 time 0.3442 (1.6854) data time 0.0009 (0.0772) model time 0.3433 (1.6081) loss 2.6766 (2.9083) grad_norm 2.4854 (3.7166) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][140/625] eta 0:08:12 lr 0.000024 wd 0.0500 time 0.3418 (1.0151) data time 0.0006 (0.0390) model time 0.3412 (0.9761) loss 3.0271 (2.8164) grad_norm 3.2081 (3.2570) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][150/625] eta 0:06:19 lr 0.000024 wd 0.0500 time 0.3420 (0.7989) data time 0.0008 (0.0263) model time 0.3412 (0.7725) loss 2.8940 (2.8191) grad_norm 3.6380 (3.2390) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][160/625] eta 0:05:18 lr 0.000024 wd 0.0500 time 0.3482 (0.6857) data time 0.0007 (0.0200) model time 0.3475 (0.6657) loss 1.9661 (2.7457) grad_norm 3.3020 (3.3485) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][170/625] eta 0:04:43 lr 0.000024 wd 0.0500 time 0.3411 (0.6228) data time 0.0010 (0.0161) model time 0.3401 (0.6067) loss 2.2459 (2.7248) grad_norm 2.8039 (3.2804) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][180/625] eta 0:04:16 lr 0.000024 wd 0.0500 time 0.3458 (0.5767) data time 0.0007 (0.0136) model time 0.3451 (0.5631) loss 2.8373 (2.7157) grad_norm 2.9652 (3.3725) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][190/625] eta 0:03:56 lr 0.000024 wd 0.0500 time 0.3488 (0.5436) data time 0.0006 (0.0118) model time 0.3482 (0.5318) loss 2.0120 (2.6895) grad_norm 2.6124 (3.2890) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][200/625] eta 0:03:40 lr 0.000024 wd 0.0500 time 0.3478 (0.5189) data time 0.0008 (0.0104) model time 0.3470 (0.5084) loss 2.7654 (2.6735) grad_norm 5.4429 (3.3987) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][210/625] eta 0:03:27 lr 0.000024 wd 0.0500 time 0.3394 (0.4997) data time 0.0007 (0.0094) model time 0.3387 (0.4903) loss 2.9453 (2.6589) grad_norm 2.6726 (3.5372) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][220/625] eta 0:03:16 lr 0.000024 wd 0.0500 time 0.3416 (0.4843) data time 0.0009 (0.0085) model time 0.3408 (0.4757) loss 3.2260 (2.6686) grad_norm 2.2447 (3.4775) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][230/625] eta 0:03:06 lr 0.000024 wd 0.0500 time 0.3468 (0.4715) data time 0.0008 (0.0078) model time 0.3460 (0.4637) loss 2.5871 (2.6852) grad_norm 2.6285 (3.4316) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][240/625] eta 0:02:57 lr 0.000024 wd 0.0500 time 0.3464 (0.4610) data time 0.0006 (0.0073) model time 0.3457 (0.4537) loss 2.9170 (2.6848) grad_norm 3.5955 (3.3774) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][250/625] eta 0:02:49 lr 0.000024 wd 0.0500 time 0.3455 (0.4520) data time 0.0007 (0.0068) model time 0.3448 (0.4453) loss 2.2414 (2.6708) grad_norm 2.5002 (3.3515) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][260/625] eta 0:02:42 lr 0.000024 wd 0.0500 time 0.3424 (0.4444) data time 0.0007 (0.0063) model time 0.3417 (0.4381) loss 1.6250 (2.6539) grad_norm 3.8114 (3.3516) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][270/625] eta 0:02:35 lr 0.000024 wd 0.0500 time 0.3475 (0.4378) data time 0.0008 (0.0060) model time 0.3466 (0.4318) loss 2.5983 (2.6513) grad_norm 2.1292 (3.3240) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][280/625] eta 0:02:29 lr 0.000024 wd 0.0500 time 0.3513 (0.4321) data time 0.0010 (0.0057) model time 0.3503 (0.4264) loss 2.8349 (2.6491) grad_norm 3.2256 (3.3224) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][290/625] eta 0:02:23 lr 0.000024 wd 0.0500 time 0.3443 (0.4271) data time 0.0007 (0.0054) model time 0.3436 (0.4217) loss 1.8547 (2.6416) grad_norm 2.9016 (3.3060) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:10:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][300/625] eta 0:02:17 lr 0.000024 wd 0.0500 time 0.3437 (0.4226) data time 0.0007 (0.0051) model time 0.3430 (0.4175) loss 1.9553 (2.6291) grad_norm 2.2438 (3.2814) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][310/625] eta 0:02:11 lr 0.000024 wd 0.0500 time 0.3469 (0.4185) data time 0.0006 (0.0049) model time 0.3463 (0.4136) loss 2.2382 (2.6302) grad_norm 3.2488 (3.3107) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][320/625] eta 0:02:06 lr 0.000023 wd 0.0500 time 0.3484 (0.4149) data time 0.0008 (0.0047) model time 0.3476 (0.4102) loss 2.8223 (2.6165) grad_norm 3.5888 (3.2989) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][330/625] eta 0:02:01 lr 0.000023 wd 0.0500 time 0.3548 (0.4117) data time 0.0006 (0.0045) model time 0.3541 (0.4072) loss 2.6970 (2.6091) grad_norm 2.8448 (3.4577) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][340/625] eta 0:01:56 lr 0.000023 wd 0.0500 time 0.3479 (0.4088) data time 0.0008 (0.0044) model time 0.3470 (0.4044) loss 2.4085 (2.6084) grad_norm 2.5628 (3.4464) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][350/625] eta 0:01:52 lr 0.000023 wd 0.0500 time 0.3442 (0.4079) data time 0.0008 (0.0042) model time 0.3434 (0.4037) loss 3.3077 (2.6144) grad_norm 4.3295 (3.4587) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][360/625] eta 0:01:47 lr 0.000023 wd 0.0500 time 0.3438 (0.4060) data time 0.0008 (0.0041) model time 0.3430 (0.4019) loss 3.0337 (2.6107) grad_norm 2.3599 (3.4385) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][370/625] eta 0:01:42 lr 0.000023 wd 0.0500 time 0.3451 (0.4036) data time 0.0006 (0.0039) model time 0.3445 (0.3997) loss 1.7412 (2.6043) grad_norm 3.0190 (3.4817) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][380/625] eta 0:01:38 lr 0.000023 wd 0.0500 time 0.3443 (0.4014) data time 0.0010 (0.0038) model time 0.3433 (0.3976) loss 2.1256 (2.6004) grad_norm 6.4578 (3.5209) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][390/625] eta 0:01:33 lr 0.000023 wd 0.0500 time 0.3465 (0.3993) data time 0.0006 (0.0037) model time 0.3459 (0.3956) loss 3.2590 (2.5935) grad_norm 2.9125 (3.5062) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][400/625] eta 0:01:29 lr 0.000023 wd 0.0500 time 0.3468 (0.3974) data time 0.0008 (0.0036) model time 0.3460 (0.3938) loss 2.8476 (2.6000) grad_norm 2.5270 (3.5390) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][410/625] eta 0:01:25 lr 0.000023 wd 0.0500 time 0.3479 (0.3957) data time 0.0008 (0.0035) model time 0.3470 (0.3922) loss 2.0639 (2.5975) grad_norm 2.1270 (3.5164) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][420/625] eta 0:01:20 lr 0.000023 wd 0.0500 time 0.3473 (0.3941) data time 0.0007 (0.0034) model time 0.3466 (0.3907) loss 2.3331 (2.5881) grad_norm 2.1854 (3.5324) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][430/625] eta 0:01:16 lr 0.000023 wd 0.0500 time 0.3516 (0.3927) data time 0.0008 (0.0034) model time 0.3508 (0.3893) loss 2.6400 (2.5839) grad_norm 2.7085 (3.5127) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][440/625] eta 0:01:12 lr 0.000023 wd 0.0500 time 0.3470 (0.3913) data time 0.0009 (0.0033) model time 0.3461 (0.3880) loss 2.4634 (2.5897) grad_norm 3.4002 (3.5017) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][450/625] eta 0:01:08 lr 0.000023 wd 0.0500 time 0.3524 (0.3899) data time 0.0007 (0.0032) model time 0.3516 (0.3867) loss 2.9321 (2.5923) grad_norm 2.6612 (3.4943) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][460/625] eta 0:01:04 lr 0.000023 wd 0.0500 time 0.3492 (0.3887) data time 0.0008 (0.0031) model time 0.3484 (0.3855) loss 2.6629 (2.5952) grad_norm 2.9973 (3.4822) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:11:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][470/625] eta 0:01:00 lr 0.000023 wd 0.0500 time 0.3447 (0.3886) data time 0.0008 (0.0031) model time 0.3439 (0.3856) loss 2.4255 (2.5977) grad_norm 4.1042 (3.4695) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][480/625] eta 0:00:56 lr 0.000023 wd 0.0500 time 0.3473 (0.3874) data time 0.0006 (0.0030) model time 0.3467 (0.3844) loss 3.0775 (2.6009) grad_norm 3.6720 (3.4513) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][490/625] eta 0:00:52 lr 0.000023 wd 0.0500 time 0.3446 (0.3863) data time 0.0009 (0.0030) model time 0.3437 (0.3834) loss 2.5203 (2.5974) grad_norm 2.1278 (3.5208) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][500/625] eta 0:00:48 lr 0.000023 wd 0.0500 time 0.3501 (0.3853) data time 0.0007 (0.0029) model time 0.3494 (0.3824) loss 2.5765 (2.5969) grad_norm 2.3576 (3.5267) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][510/625] eta 0:00:44 lr 0.000023 wd 0.0500 time 0.3452 (0.3843) data time 0.0009 (0.0028) model time 0.3443 (0.3815) loss 1.9322 (2.5876) grad_norm 2.6467 (3.5146) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][520/625] eta 0:00:40 lr 0.000023 wd 0.0500 time 0.3445 (0.3834) data time 0.0008 (0.0028) model time 0.3437 (0.3806) loss 2.3797 (2.5901) grad_norm 4.8770 (3.5150) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][530/625] eta 0:00:36 lr 0.000023 wd 0.0500 time 0.3464 (0.3824) data time 0.0007 (0.0027) model time 0.3457 (0.3797) loss 2.4950 (2.5919) grad_norm 2.5744 (3.4916) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][540/625] eta 0:00:32 lr 0.000023 wd 0.0500 time 0.3446 (0.3820) data time 0.0007 (0.0027) model time 0.3439 (0.3793) loss 2.5923 (2.5897) grad_norm 2.9032 (3.4700) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][550/625] eta 0:00:28 lr 0.000023 wd 0.0500 time 0.3425 (0.3812) data time 0.0009 (0.0027) model time 0.3417 (0.3785) loss 3.2626 (2.5967) grad_norm 2.2540 (3.4829) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][560/625] eta 0:00:24 lr 0.000023 wd 0.0500 time 0.3413 (0.3804) data time 0.0009 (0.0026) model time 0.3404 (0.3777) loss 2.4171 (2.6006) grad_norm 3.9405 (3.4825) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][570/625] eta 0:00:20 lr 0.000023 wd 0.0500 time 0.3422 (0.3796) data time 0.0007 (0.0026) model time 0.3415 (0.3770) loss 2.6244 (2.6002) grad_norm 3.6989 (3.4738) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][580/625] eta 0:00:17 lr 0.000023 wd 0.0500 time 0.3404 (0.3788) data time 0.0008 (0.0026) model time 0.3397 (0.3762) loss 2.6423 (2.5964) grad_norm 1.9842 (3.4701) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][590/625] eta 0:00:13 lr 0.000023 wd 0.0500 time 0.3372 (0.3781) data time 0.0008 (0.0025) model time 0.3364 (0.3755) loss 2.6384 (2.5915) grad_norm 2.7151 (3.4791) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][600/625] eta 0:00:09 lr 0.000023 wd 0.0500 time 0.3415 (0.3774) data time 0.0010 (0.0025) model time 0.3405 (0.3749) loss 2.2786 (2.5893) grad_norm 2.4707 (3.4834) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][610/625] eta 0:00:05 lr 0.000023 wd 0.0500 time 0.3375 (0.3768) data time 0.0004 (0.0025) model time 0.3371 (0.3743) loss 2.1871 (2.5914) grad_norm 3.6623 (3.4747) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [292/300][620/625] eta 0:00:01 lr 0.000023 wd 0.0500 time 0.3328 (0.3760) data time 0.0005 (0.0024) model time 0.3323 (0.3736) loss 1.8358 (2.5914) grad_norm 2.7163 (3.5275) loss_scale 256.0000 (256.0000) mem 11219MB [2024-09-01 16:12:51 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 292 training takes 0:03:09 [2024-09-01 16:12:51 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:12:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:12:56 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.425 (0.425) Loss 0.5103 (0.5103) Acc@1 88.916 (88.916) Acc@5 98.633 (98.633) Mem 11219MB [2024-09-01 16:12:57 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.127) Loss 0.8350 (0.6046) Acc@1 80.420 (86.998) Acc@5 95.898 (97.825) Mem 11219MB [2024-09-01 16:12:58 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.109) Loss 0.9263 (0.7246) Acc@1 78.418 (84.061) Acc@5 95.215 (96.615) Mem 11219MB [2024-09-01 16:12:59 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.805 Acc@5 96.637 [2024-09-01 16:12:59 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:13:00 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.709 (0.709) Loss 0.5088 (0.5088) Acc@1 89.014 (89.014) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 16:13:01 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.094 (0.161) Loss 0.8242 (0.6017) Acc@1 80.811 (87.114) Acc@5 95.752 (97.856) Mem 11219MB [2024-09-01 16:13:02 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.128) Loss 0.9077 (0.7191) Acc@1 78.564 (84.222) Acc@5 95.410 (96.696) Mem 11219MB [2024-09-01 16:13:03 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.923 Acc@5 96.719 [2024-09-01 16:13:03 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:13:03 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.92% [2024-09-01 16:13:03 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 16:13:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 16:13:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][0/625] eta 0:07:43 lr 0.000023 wd 0.0500 time 0.7412 (0.7412) data time 0.3910 (0.3910) model time 0.0000 (0.0000) loss 2.6162 (2.6162) grad_norm 2.8990 (2.8990) loss_scale 256.0000 (256.0000) mem 11227MB [2024-09-01 16:13:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][10/625] eta 0:03:54 lr 0.000023 wd 0.0500 time 0.3459 (0.3818) data time 0.0009 (0.0364) model time 0.0000 (0.0000) loss 2.4514 (2.8721) grad_norm 6.0743 (3.3625) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][20/625] eta 0:03:40 lr 0.000023 wd 0.0500 time 0.3407 (0.3645) data time 0.0007 (0.0195) model time 0.0000 (0.0000) loss 3.1866 (2.6579) grad_norm 2.1916 (3.4024) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][30/625] eta 0:03:43 lr 0.000023 wd 0.0500 time 0.3505 (0.3765) data time 0.0006 (0.0135) model time 0.0000 (0.0000) loss 1.9013 (2.5785) grad_norm 2.5169 (3.3607) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][40/625] eta 0:03:35 lr 0.000023 wd 0.0500 time 0.3402 (0.3689) data time 0.0008 (0.0104) model time 0.0000 (0.0000) loss 2.4086 (2.6053) grad_norm 2.7827 (3.2819) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][50/625] eta 0:03:29 lr 0.000023 wd 0.0500 time 0.3419 (0.3645) data time 0.0009 (0.0086) model time 0.0000 (0.0000) loss 2.5340 (2.6225) grad_norm 2.6441 (4.0309) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][60/625] eta 0:03:24 lr 0.000023 wd 0.0500 time 0.3485 (0.3615) data time 0.0010 (0.0073) model time 0.3475 (0.3451) loss 3.0574 (2.6389) grad_norm 2.8037 (4.2164) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][70/625] eta 0:03:19 lr 0.000023 wd 0.0500 time 0.3514 (0.3596) data time 0.0008 (0.0064) model time 0.3506 (0.3462) loss 2.6734 (2.6273) grad_norm 3.1839 (4.0769) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][80/625] eta 0:03:15 lr 0.000023 wd 0.0500 time 0.3481 (0.3580) data time 0.0008 (0.0057) model time 0.3473 (0.3461) loss 2.7800 (2.6257) grad_norm 4.0513 (4.0431) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][90/625] eta 0:03:10 lr 0.000023 wd 0.0500 time 0.3476 (0.3566) data time 0.0007 (0.0052) model time 0.3468 (0.3457) loss 3.0214 (2.6326) grad_norm 3.9579 (4.0107) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][100/625] eta 0:03:06 lr 0.000023 wd 0.0500 time 0.3401 (0.3553) data time 0.0007 (0.0048) model time 0.3394 (0.3450) loss 2.9658 (2.6251) grad_norm 2.7225 (3.9234) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][110/625] eta 0:03:02 lr 0.000023 wd 0.0500 time 0.3497 (0.3543) data time 0.0008 (0.0044) model time 0.3489 (0.3447) loss 3.1408 (2.6317) grad_norm 3.1872 (3.8121) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][120/625] eta 0:02:58 lr 0.000023 wd 0.0500 time 0.3478 (0.3539) data time 0.0010 (0.0042) model time 0.3469 (0.3453) loss 2.8012 (2.6410) grad_norm 4.5297 (3.7480) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][130/625] eta 0:02:54 lr 0.000023 wd 0.0500 time 0.3432 (0.3535) data time 0.0009 (0.0039) model time 0.3423 (0.3455) loss 2.6089 (2.6354) grad_norm 3.4974 (3.7055) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:13:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][140/625] eta 0:02:51 lr 0.000023 wd 0.0500 time 0.3427 (0.3529) data time 0.0007 (0.0037) model time 0.3421 (0.3454) loss 2.1182 (2.6198) grad_norm 2.4007 (3.7720) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][150/625] eta 0:02:47 lr 0.000023 wd 0.0500 time 0.3471 (0.3525) data time 0.0007 (0.0035) model time 0.3464 (0.3454) loss 2.7489 (2.6191) grad_norm 4.2026 (3.7544) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][160/625] eta 0:02:43 lr 0.000023 wd 0.0500 time 0.3431 (0.3522) data time 0.0007 (0.0033) model time 0.3424 (0.3455) loss 2.8505 (2.6075) grad_norm 2.3840 (3.7113) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][170/625] eta 0:02:40 lr 0.000023 wd 0.0500 time 0.3405 (0.3518) data time 0.0007 (0.0032) model time 0.3398 (0.3455) loss 2.7981 (2.6259) grad_norm 4.2724 (3.7238) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][180/625] eta 0:02:36 lr 0.000023 wd 0.0500 time 0.3465 (0.3516) data time 0.0009 (0.0031) model time 0.3457 (0.3456) loss 2.6255 (2.6196) grad_norm 2.7801 (3.6974) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][190/625] eta 0:02:32 lr 0.000023 wd 0.0500 time 0.3471 (0.3513) data time 0.0008 (0.0030) model time 0.3463 (0.3456) loss 2.6579 (2.6101) grad_norm 4.3689 (3.6968) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][200/625] eta 0:02:29 lr 0.000023 wd 0.0500 time 0.3434 (0.3511) data time 0.0007 (0.0029) model time 0.3426 (0.3456) loss 2.9596 (2.6083) grad_norm 5.8409 (3.6864) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][210/625] eta 0:02:25 lr 0.000023 wd 0.0500 time 0.3604 (0.3510) data time 0.0009 (0.0028) model time 0.3595 (0.3458) loss 2.8380 (2.5909) grad_norm 3.8896 (3.7123) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][220/625] eta 0:02:22 lr 0.000023 wd 0.0500 time 0.3457 (0.3515) data time 0.0007 (0.0027) model time 0.3449 (0.3467) loss 2.5468 (2.5892) grad_norm 3.4965 (3.7141) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][230/625] eta 0:02:18 lr 0.000023 wd 0.0500 time 0.3503 (0.3515) data time 0.0007 (0.0026) model time 0.3496 (0.3468) loss 2.9386 (2.5925) grad_norm 2.7018 (3.6735) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][240/625] eta 0:02:16 lr 0.000023 wd 0.0500 time 0.3465 (0.3537) data time 0.0009 (0.0026) model time 0.3456 (0.3498) loss 2.6050 (2.5945) grad_norm 3.8149 (3.6706) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][250/625] eta 0:02:12 lr 0.000023 wd 0.0500 time 0.3412 (0.3533) data time 0.0008 (0.0025) model time 0.3404 (0.3495) loss 1.9626 (2.5882) grad_norm 2.8492 (3.6484) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][260/625] eta 0:02:08 lr 0.000023 wd 0.0500 time 0.3556 (0.3530) data time 0.0006 (0.0024) model time 0.3550 (0.3493) loss 2.9067 (2.5976) grad_norm 3.9768 (3.6366) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][270/625] eta 0:02:05 lr 0.000023 wd 0.0500 time 0.3451 (0.3528) data time 0.0008 (0.0024) model time 0.3442 (0.3492) loss 2.5322 (2.6006) grad_norm 3.8638 (3.6466) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][280/625] eta 0:02:01 lr 0.000023 wd 0.0500 time 0.3481 (0.3526) data time 0.0008 (0.0023) model time 0.3473 (0.3490) loss 1.7399 (2.5952) grad_norm 2.8565 (3.7836) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][290/625] eta 0:01:58 lr 0.000023 wd 0.0500 time 0.3431 (0.3523) data time 0.0006 (0.0023) model time 0.3424 (0.3488) loss 2.4046 (2.5990) grad_norm 4.1181 (3.7827) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][300/625] eta 0:01:54 lr 0.000023 wd 0.0500 time 0.3456 (0.3521) data time 0.0007 (0.0022) model time 0.3449 (0.3486) loss 3.2229 (2.5976) grad_norm 2.5301 (3.7581) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:14:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][310/625] eta 0:01:50 lr 0.000023 wd 0.0500 time 0.3461 (0.3519) data time 0.0008 (0.0022) model time 0.3454 (0.3485) loss 2.5049 (2.5892) grad_norm 9.2649 (3.7531) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][320/625] eta 0:01:47 lr 0.000023 wd 0.0500 time 0.3443 (0.3517) data time 0.0006 (0.0021) model time 0.3437 (0.3483) loss 1.7748 (2.5834) grad_norm 2.6224 (3.7325) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][330/625] eta 0:01:43 lr 0.000023 wd 0.0500 time 0.3498 (0.3515) data time 0.0008 (0.0021) model time 0.3490 (0.3482) loss 1.9563 (2.5781) grad_norm 3.3638 (3.7451) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][340/625] eta 0:01:40 lr 0.000023 wd 0.0500 time 0.3551 (0.3514) data time 0.0007 (0.0021) model time 0.3544 (0.3482) loss 1.5149 (2.5718) grad_norm 2.1881 (3.7205) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][350/625] eta 0:01:36 lr 0.000023 wd 0.0500 time 0.3456 (0.3513) data time 0.0008 (0.0020) model time 0.3448 (0.3481) loss 2.5978 (2.5747) grad_norm 5.5174 (3.7117) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][360/625] eta 0:01:33 lr 0.000023 wd 0.0500 time 0.3451 (0.3512) data time 0.0008 (0.0020) model time 0.3443 (0.3481) loss 2.2262 (2.5741) grad_norm 3.6460 (3.6935) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][370/625] eta 0:01:29 lr 0.000023 wd 0.0500 time 0.3388 (0.3510) data time 0.0006 (0.0020) model time 0.3382 (0.3479) loss 2.7296 (2.5772) grad_norm 2.9462 (3.6792) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][380/625] eta 0:01:25 lr 0.000023 wd 0.0500 time 0.3412 (0.3508) data time 0.0007 (0.0020) model time 0.3405 (0.3478) loss 2.2788 (2.5713) grad_norm 2.6442 (3.6661) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][390/625] eta 0:01:22 lr 0.000023 wd 0.0500 time 0.3474 (0.3507) data time 0.0008 (0.0019) model time 0.3465 (0.3477) loss 2.8524 (2.5722) grad_norm 2.7911 (3.6458) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][400/625] eta 0:01:18 lr 0.000023 wd 0.0500 time 0.3482 (0.3506) data time 0.0006 (0.0019) model time 0.3476 (0.3477) loss 2.8127 (2.5727) grad_norm 2.3149 (3.6209) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][410/625] eta 0:01:15 lr 0.000023 wd 0.0500 time 0.3426 (0.3509) data time 0.0007 (0.0019) model time 0.3419 (0.3480) loss 2.8658 (2.5706) grad_norm 4.1126 (3.6265) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][420/625] eta 0:01:12 lr 0.000022 wd 0.0500 time 0.3489 (0.3512) data time 0.0006 (0.0018) model time 0.3482 (0.3485) loss 3.0899 (2.5759) grad_norm 2.8721 (3.6800) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][430/625] eta 0:01:08 lr 0.000022 wd 0.0500 time 0.3443 (0.3524) data time 0.0006 (0.0018) model time 0.3437 (0.3499) loss 1.7120 (2.5782) grad_norm 2.9077 (3.6681) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][440/625] eta 0:01:05 lr 0.000022 wd 0.0500 time 0.3450 (0.3523) data time 0.0006 (0.0018) model time 0.3444 (0.3498) loss 2.9153 (2.5753) grad_norm 5.0021 (3.6753) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][450/625] eta 0:01:01 lr 0.000022 wd 0.0500 time 0.3467 (0.3522) data time 0.0008 (0.0018) model time 0.3458 (0.3497) loss 2.8241 (2.5706) grad_norm 3.9173 (3.6612) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][460/625] eta 0:00:58 lr 0.000022 wd 0.0500 time 0.3433 (0.3521) data time 0.0008 (0.0018) model time 0.3424 (0.3497) loss 2.2594 (2.5719) grad_norm 3.4132 (3.6396) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][470/625] eta 0:00:54 lr 0.000022 wd 0.0500 time 0.3456 (0.3520) data time 0.0006 (0.0017) model time 0.3450 (0.3495) loss 2.9704 (2.5788) grad_norm 2.2259 (3.6272) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:15:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][480/625] eta 0:00:51 lr 0.000022 wd 0.0500 time 0.3442 (0.3518) data time 0.0008 (0.0017) model time 0.3434 (0.3494) loss 3.1057 (2.5763) grad_norm 2.1342 (3.6052) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][490/625] eta 0:00:47 lr 0.000022 wd 0.0500 time 0.3434 (0.3517) data time 0.0006 (0.0017) model time 0.3428 (0.3493) loss 3.1122 (2.5776) grad_norm 1.8805 (3.5970) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][500/625] eta 0:00:43 lr 0.000022 wd 0.0500 time 0.3425 (0.3517) data time 0.0007 (0.0017) model time 0.3418 (0.3493) loss 1.7507 (2.5767) grad_norm 2.9766 (3.5911) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][510/625] eta 0:00:40 lr 0.000022 wd 0.0500 time 0.3490 (0.3516) data time 0.0008 (0.0017) model time 0.3482 (0.3492) loss 3.1843 (2.5831) grad_norm 3.4101 (3.5813) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][520/625] eta 0:00:36 lr 0.000022 wd 0.0500 time 0.3502 (0.3515) data time 0.0007 (0.0017) model time 0.3495 (0.3491) loss 1.7448 (2.5759) grad_norm 4.2528 (3.5969) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][530/625] eta 0:00:33 lr 0.000022 wd 0.0500 time 0.3442 (0.3517) data time 0.0006 (0.0017) model time 0.3436 (0.3495) loss 2.8931 (2.5745) grad_norm 4.4300 (3.5961) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][540/625] eta 0:00:29 lr 0.000022 wd 0.0500 time 0.3480 (0.3520) data time 0.0008 (0.0016) model time 0.3471 (0.3498) loss 2.1920 (2.5761) grad_norm 2.1082 (3.5832) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][550/625] eta 0:00:26 lr 0.000022 wd 0.0500 time 0.3447 (0.3527) data time 0.0007 (0.0016) model time 0.3440 (0.3505) loss 2.7395 (2.5766) grad_norm 2.2896 (3.5699) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][560/625] eta 0:00:22 lr 0.000022 wd 0.0500 time 0.3465 (0.3525) data time 0.0006 (0.0016) model time 0.3459 (0.3504) loss 2.8579 (2.5788) grad_norm 2.4700 (3.5561) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][570/625] eta 0:00:19 lr 0.000022 wd 0.0500 time 0.3500 (0.3524) data time 0.0008 (0.0016) model time 0.3492 (0.3503) loss 2.5648 (2.5746) grad_norm 4.7417 (3.5584) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][580/625] eta 0:00:15 lr 0.000022 wd 0.0500 time 0.3438 (0.3523) data time 0.0007 (0.0016) model time 0.3431 (0.3502) loss 1.4982 (2.5746) grad_norm 3.4274 (3.5798) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][590/625] eta 0:00:12 lr 0.000022 wd 0.0500 time 0.3471 (0.3522) data time 0.0006 (0.0016) model time 0.3465 (0.3501) loss 2.1639 (2.5742) grad_norm 3.1713 (3.5833) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][600/625] eta 0:00:08 lr 0.000022 wd 0.0500 time 0.3473 (0.3521) data time 0.0007 (0.0016) model time 0.3467 (0.3501) loss 2.3959 (2.5775) grad_norm 5.4916 (3.5764) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][610/625] eta 0:00:05 lr 0.000022 wd 0.0500 time 0.3349 (0.3521) data time 0.0004 (0.0016) model time 0.3345 (0.3500) loss 3.2598 (2.5790) grad_norm 3.5357 (3.5729) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [293/300][620/625] eta 0:00:01 lr 0.000022 wd 0.0500 time 0.3359 (0.3519) data time 0.0004 (0.0015) model time 0.3354 (0.3498) loss 2.9365 (2.5785) grad_norm 3.9832 (3.5652) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:47 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 293 training takes 0:03:39 [2024-09-01 16:16:47 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:16:48 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:16:49 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.438 (0.438) Loss 0.5142 (0.5142) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 16:16:50 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.130) Loss 0.8306 (0.6046) Acc@1 81.006 (87.043) Acc@5 95.947 (97.860) Mem 11224MB [2024-09-01 16:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.112) Loss 0.9194 (0.7246) Acc@1 79.053 (84.140) Acc@5 95.215 (96.633) Mem 11224MB [2024-09-01 16:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.801 Acc@5 96.661 [2024-09-01 16:16:51 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:16:52 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.792 (0.792) Loss 0.5088 (0.5088) Acc@1 88.867 (88.867) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 16:16:53 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.093 (0.166) Loss 0.8252 (0.6019) Acc@1 80.811 (87.092) Acc@5 95.801 (97.860) Mem 11224MB [2024-09-01 16:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.089 (0.130) Loss 0.9077 (0.7193) Acc@1 78.711 (84.229) Acc@5 95.312 (96.680) Mem 11224MB [2024-09-01 16:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.927 Acc@5 96.705 [2024-09-01 16:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:16:54 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.93% [2024-09-01 16:16:54 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 16:16:55 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 16:16:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][0/625] eta 0:07:22 lr 0.000022 wd 0.0500 time 0.7076 (0.7076) data time 0.3947 (0.3947) model time 0.0000 (0.0000) loss 2.5434 (2.5434) grad_norm 16.5709 (16.5709) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:16:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][10/625] eta 0:03:55 lr 0.000022 wd 0.0500 time 0.3489 (0.3827) data time 0.0009 (0.0368) model time 0.0000 (0.0000) loss 2.8344 (2.5156) grad_norm 9.9098 (6.3477) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][20/625] eta 0:03:41 lr 0.000022 wd 0.0500 time 0.3479 (0.3666) data time 0.0008 (0.0197) model time 0.0000 (0.0000) loss 2.1247 (2.4886) grad_norm 2.6388 (4.8677) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][30/625] eta 0:03:34 lr 0.000022 wd 0.0500 time 0.3482 (0.3604) data time 0.0006 (0.0136) model time 0.0000 (0.0000) loss 1.3838 (2.4955) grad_norm 3.0887 (4.4160) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][40/625] eta 0:03:29 lr 0.000022 wd 0.0500 time 0.3500 (0.3579) data time 0.0008 (0.0105) model time 0.0000 (0.0000) loss 2.8484 (2.5211) grad_norm 3.6018 (4.1476) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][50/625] eta 0:03:26 lr 0.000022 wd 0.0500 time 0.3523 (0.3589) data time 0.0007 (0.0086) model time 0.0000 (0.0000) loss 3.1829 (2.5817) grad_norm 3.6408 (3.8773) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][60/625] eta 0:03:21 lr 0.000022 wd 0.0500 time 0.3465 (0.3570) data time 0.0007 (0.0074) model time 0.3459 (0.3463) loss 2.8972 (2.6144) grad_norm 3.5895 (3.9235) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][70/625] eta 0:03:18 lr 0.000022 wd 0.0500 time 0.3534 (0.3586) data time 0.0007 (0.0065) model time 0.3528 (0.3568) loss 3.1973 (2.6265) grad_norm 3.7613 (3.8773) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][80/625] eta 0:03:17 lr 0.000022 wd 0.0500 time 0.3460 (0.3621) data time 0.0009 (0.0058) model time 0.3451 (0.3668) loss 2.6301 (2.6351) grad_norm 2.1031 (3.7852) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][90/625] eta 0:03:12 lr 0.000022 wd 0.0500 time 0.3474 (0.3603) data time 0.0009 (0.0052) model time 0.3466 (0.3612) loss 2.6633 (2.6169) grad_norm 3.5760 (4.1445) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][100/625] eta 0:03:08 lr 0.000022 wd 0.0500 time 0.3409 (0.3589) data time 0.0009 (0.0048) model time 0.3400 (0.3581) loss 2.8608 (2.6053) grad_norm 2.7533 (4.0502) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][110/625] eta 0:03:04 lr 0.000022 wd 0.0500 time 0.3563 (0.3579) data time 0.0006 (0.0044) model time 0.3557 (0.3561) loss 2.9319 (2.6052) grad_norm 3.2503 (3.9724) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][120/625] eta 0:03:00 lr 0.000022 wd 0.0500 time 0.3544 (0.3571) data time 0.0009 (0.0042) model time 0.3535 (0.3548) loss 1.6838 (2.5932) grad_norm 2.6871 (4.0667) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][130/625] eta 0:02:56 lr 0.000022 wd 0.0500 time 0.3466 (0.3563) data time 0.0009 (0.0039) model time 0.3457 (0.3538) loss 2.7800 (2.6009) grad_norm 3.7342 (4.4070) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][140/625] eta 0:02:52 lr 0.000022 wd 0.0500 time 0.3508 (0.3557) data time 0.0009 (0.0037) model time 0.3499 (0.3530) loss 2.6884 (2.6154) grad_norm 3.1821 (4.3610) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][150/625] eta 0:02:48 lr 0.000022 wd 0.0500 time 0.3533 (0.3554) data time 0.0007 (0.0035) model time 0.3527 (0.3526) loss 2.8524 (2.6240) grad_norm 2.4659 (4.2761) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][160/625] eta 0:02:45 lr 0.000022 wd 0.0500 time 0.3498 (0.3550) data time 0.0008 (0.0034) model time 0.3490 (0.3522) loss 2.6096 (2.6295) grad_norm 3.4868 (4.2319) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][170/625] eta 0:02:41 lr 0.000022 wd 0.0500 time 0.3492 (0.3544) data time 0.0010 (0.0032) model time 0.3481 (0.3516) loss 2.8229 (2.6315) grad_norm 2.0796 (4.1945) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:17:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][180/625] eta 0:02:37 lr 0.000022 wd 0.0500 time 0.3488 (0.3541) data time 0.0006 (0.0031) model time 0.3481 (0.3513) loss 3.0176 (2.6255) grad_norm 2.7643 (4.1439) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][190/625] eta 0:02:33 lr 0.000022 wd 0.0500 time 0.3551 (0.3540) data time 0.0007 (0.0030) model time 0.3544 (0.3513) loss 2.6293 (2.6083) grad_norm 2.9022 (4.0985) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][200/625] eta 0:02:30 lr 0.000022 wd 0.0500 time 0.3478 (0.3537) data time 0.0009 (0.0029) model time 0.3469 (0.3510) loss 3.1452 (2.6123) grad_norm 2.9421 (4.0804) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][210/625] eta 0:02:26 lr 0.000022 wd 0.0500 time 0.3494 (0.3534) data time 0.0008 (0.0028) model time 0.3486 (0.3507) loss 2.8828 (2.6161) grad_norm 4.0357 (4.0404) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][220/625] eta 0:02:23 lr 0.000022 wd 0.0500 time 0.3447 (0.3531) data time 0.0007 (0.0027) model time 0.3440 (0.3505) loss 3.2533 (2.6153) grad_norm 4.1196 (3.9889) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][230/625] eta 0:02:19 lr 0.000022 wd 0.0500 time 0.3486 (0.3529) data time 0.0009 (0.0026) model time 0.3477 (0.3503) loss 2.8335 (2.6184) grad_norm 2.5239 (3.9396) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][240/625] eta 0:02:15 lr 0.000022 wd 0.0500 time 0.3503 (0.3532) data time 0.0008 (0.0026) model time 0.3495 (0.3508) loss 2.6404 (2.6234) grad_norm 3.0051 (3.8955) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][250/625] eta 0:02:12 lr 0.000022 wd 0.0500 time 0.3488 (0.3530) data time 0.0009 (0.0025) model time 0.3479 (0.3506) loss 2.7439 (2.6248) grad_norm 3.1198 (3.8624) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][260/625] eta 0:02:08 lr 0.000022 wd 0.0500 time 0.3497 (0.3528) data time 0.0008 (0.0024) model time 0.3489 (0.3504) loss 2.7841 (2.6266) grad_norm 2.4551 (3.8394) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][270/625] eta 0:02:05 lr 0.000022 wd 0.0500 time 0.3524 (0.3526) data time 0.0008 (0.0024) model time 0.3516 (0.3502) loss 2.8368 (2.6223) grad_norm 2.6514 (3.8309) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][280/625] eta 0:02:01 lr 0.000022 wd 0.0500 time 0.5433 (0.3531) data time 0.0008 (0.0023) model time 0.5425 (0.3509) loss 3.1511 (2.6284) grad_norm 2.0116 (3.7933) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][290/625] eta 0:01:58 lr 0.000022 wd 0.0500 time 0.3427 (0.3543) data time 0.0007 (0.0023) model time 0.3419 (0.3524) loss 2.6736 (2.6255) grad_norm 2.6922 (3.7693) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][300/625] eta 0:01:55 lr 0.000022 wd 0.0500 time 0.3446 (0.3541) data time 0.0007 (0.0022) model time 0.3440 (0.3522) loss 2.8210 (2.6246) grad_norm 2.2730 (3.7457) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][310/625] eta 0:01:51 lr 0.000022 wd 0.0500 time 0.3471 (0.3539) data time 0.0009 (0.0022) model time 0.3462 (0.3520) loss 2.8262 (2.6232) grad_norm 2.6762 (3.7172) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][320/625] eta 0:01:47 lr 0.000022 wd 0.0500 time 0.3400 (0.3536) data time 0.0007 (0.0021) model time 0.3393 (0.3517) loss 2.2113 (2.6224) grad_norm 2.0033 (3.8356) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][330/625] eta 0:01:44 lr 0.000022 wd 0.0500 time 0.3501 (0.3534) data time 0.0007 (0.0021) model time 0.3494 (0.3515) loss 3.0324 (2.6261) grad_norm 2.1703 (3.8014) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][340/625] eta 0:01:40 lr 0.000022 wd 0.0500 time 0.3484 (0.3532) data time 0.0008 (0.0021) model time 0.3475 (0.3513) loss 2.8806 (2.6259) grad_norm 3.7416 (3.7889) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:18:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][350/625] eta 0:01:37 lr 0.000022 wd 0.0500 time 0.3519 (0.3531) data time 0.0007 (0.0020) model time 0.3513 (0.3512) loss 1.9300 (2.6156) grad_norm 5.2475 (3.7736) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][360/625] eta 0:01:33 lr 0.000022 wd 0.0500 time 0.3487 (0.3534) data time 0.0008 (0.0020) model time 0.3479 (0.3516) loss 1.7075 (2.6069) grad_norm 2.9340 (3.7435) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][370/625] eta 0:01:30 lr 0.000022 wd 0.0500 time 0.3463 (0.3533) data time 0.0008 (0.0020) model time 0.3455 (0.3515) loss 2.7809 (2.6039) grad_norm 4.0289 (3.7317) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][380/625] eta 0:01:26 lr 0.000022 wd 0.0500 time 0.3490 (0.3532) data time 0.0008 (0.0019) model time 0.3481 (0.3514) loss 1.8570 (2.6046) grad_norm 2.3638 (3.7143) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][390/625] eta 0:01:22 lr 0.000022 wd 0.0500 time 0.3474 (0.3531) data time 0.0007 (0.0019) model time 0.3467 (0.3513) loss 2.2685 (2.6058) grad_norm 4.1428 (3.7133) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][400/625] eta 0:01:19 lr 0.000022 wd 0.0500 time 0.3474 (0.3529) data time 0.0009 (0.0019) model time 0.3465 (0.3512) loss 2.2274 (2.6057) grad_norm 2.5681 (3.7337) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][410/625] eta 0:01:15 lr 0.000022 wd 0.0500 time 0.3519 (0.3528) data time 0.0006 (0.0019) model time 0.3513 (0.3510) loss 3.4335 (2.6079) grad_norm 4.9156 (3.7223) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][420/625] eta 0:01:12 lr 0.000022 wd 0.0500 time 0.3475 (0.3527) data time 0.0008 (0.0018) model time 0.3467 (0.3509) loss 2.9353 (2.6054) grad_norm 2.9224 (3.7157) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][430/625] eta 0:01:08 lr 0.000022 wd 0.0500 time 0.3458 (0.3526) data time 0.0008 (0.0018) model time 0.3449 (0.3508) loss 2.0505 (2.6040) grad_norm 2.1873 (3.7071) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][440/625] eta 0:01:05 lr 0.000022 wd 0.0500 time 0.3449 (0.3525) data time 0.0007 (0.0018) model time 0.3442 (0.3507) loss 2.2249 (2.5956) grad_norm 3.0364 (3.7085) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][450/625] eta 0:01:01 lr 0.000022 wd 0.0500 time 0.3482 (0.3524) data time 0.0009 (0.0018) model time 0.3474 (0.3507) loss 2.3158 (2.5950) grad_norm 4.9759 (3.7148) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][460/625] eta 0:00:58 lr 0.000022 wd 0.0500 time 0.3453 (0.3523) data time 0.0006 (0.0018) model time 0.3447 (0.3506) loss 2.6245 (2.5956) grad_norm 1.9370 (3.7122) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][470/625] eta 0:00:54 lr 0.000022 wd 0.0500 time 0.3461 (0.3542) data time 0.0007 (0.0017) model time 0.3454 (0.3527) loss 3.2838 (2.6017) grad_norm 3.4442 (3.7123) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][480/625] eta 0:00:51 lr 0.000022 wd 0.0500 time 0.3413 (0.3540) data time 0.0007 (0.0017) model time 0.3406 (0.3525) loss 2.8733 (2.6041) grad_norm 4.0416 (3.7250) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][490/625] eta 0:00:47 lr 0.000022 wd 0.0500 time 0.3448 (0.3538) data time 0.0009 (0.0017) model time 0.3439 (0.3523) loss 2.4052 (2.6029) grad_norm 2.8384 (3.7183) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][500/625] eta 0:00:44 lr 0.000022 wd 0.0500 time 0.3445 (0.3536) data time 0.0007 (0.0017) model time 0.3439 (0.3522) loss 2.8132 (2.6038) grad_norm 3.9721 (3.7057) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][510/625] eta 0:00:40 lr 0.000022 wd 0.0500 time 0.3459 (0.3535) data time 0.0009 (0.0017) model time 0.3450 (0.3521) loss 1.8501 (2.6050) grad_norm 2.3140 (3.6898) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:19:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][520/625] eta 0:00:37 lr 0.000022 wd 0.0500 time 0.3476 (0.3535) data time 0.0010 (0.0017) model time 0.3467 (0.3520) loss 2.7760 (2.5982) grad_norm 4.7611 (3.6729) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:20:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][530/625] eta 0:00:33 lr 0.000022 wd 0.0500 time 0.3478 (0.3534) data time 0.0008 (0.0016) model time 0.3470 (0.3519) loss 2.8581 (2.5924) grad_norm 2.7417 (3.6653) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:20:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][540/625] eta 0:00:30 lr 0.000022 wd 0.0500 time 0.3490 (0.3533) data time 0.0007 (0.0016) model time 0.3483 (0.3518) loss 2.8153 (2.5926) grad_norm 2.3615 (3.6633) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:20:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][550/625] eta 0:00:26 lr 0.000022 wd 0.0500 time 0.3503 (0.3531) data time 0.0006 (0.0016) model time 0.3497 (0.3517) loss 2.7468 (2.5919) grad_norm 3.9884 (3.6608) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:20:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][560/625] eta 0:00:22 lr 0.000022 wd 0.0500 time 0.3499 (0.3531) data time 0.0014 (0.0016) model time 0.3486 (0.3516) loss 2.2949 (2.5921) grad_norm 4.3205 (3.6614) loss_scale 512.0000 (256.4563) mem 11224MB [2024-09-01 16:20:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][570/625] eta 0:00:19 lr 0.000022 wd 0.0500 time 0.3479 (0.3533) data time 0.0009 (0.0016) model time 0.3470 (0.3519) loss 2.9181 (2.5954) grad_norm 3.6395 (3.6582) loss_scale 512.0000 (260.9317) mem 11224MB [2024-09-01 16:20:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][580/625] eta 0:00:15 lr 0.000022 wd 0.0500 time 0.3467 (0.3532) data time 0.0008 (0.0016) model time 0.3459 (0.3518) loss 2.1198 (2.5958) grad_norm 2.9267 (3.6570) loss_scale 512.0000 (265.2530) mem 11224MB [2024-09-01 16:20:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][590/625] eta 0:00:12 lr 0.000022 wd 0.0500 time 0.3535 (0.3542) data time 0.0008 (0.0016) model time 0.3527 (0.3528) loss 2.5132 (2.5948) grad_norm 3.5950 (3.6816) loss_scale 512.0000 (269.4281) mem 11224MB [2024-09-01 16:20:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][600/625] eta 0:00:08 lr 0.000022 wd 0.0500 time 0.3427 (0.3540) data time 0.0008 (0.0016) model time 0.3419 (0.3527) loss 2.7782 (2.5962) grad_norm 3.0888 (3.6798) loss_scale 512.0000 (273.4642) mem 11224MB [2024-09-01 16:20:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][610/625] eta 0:00:05 lr 0.000022 wd 0.0500 time 0.3404 (0.3539) data time 0.0005 (0.0016) model time 0.3399 (0.3525) loss 3.0201 (2.5964) grad_norm 1.9644 (3.6777) loss_scale 512.0000 (277.3682) mem 11224MB [2024-09-01 16:20:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [294/300][620/625] eta 0:00:01 lr 0.000022 wd 0.0500 time 0.3371 (0.3536) data time 0.0006 (0.0015) model time 0.3365 (0.3522) loss 1.8585 (2.5931) grad_norm 2.4062 (3.6757) loss_scale 512.0000 (281.1465) mem 11224MB [2024-09-01 16:20:36 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 294 training takes 0:03:40 [2024-09-01 16:20:36 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:20:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:20:37 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.456 (0.456) Loss 0.5107 (0.5107) Acc@1 89.258 (89.258) Acc@5 98.682 (98.682) Mem 11224MB [2024-09-01 16:20:38 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.129) Loss 0.8345 (0.6089) Acc@1 80.713 (86.954) Acc@5 95.947 (97.847) Mem 11224MB [2024-09-01 16:20:39 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.111) Loss 0.9165 (0.7272) Acc@1 78.613 (84.091) Acc@5 95.410 (96.626) Mem 11224MB [2024-09-01 16:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.791 Acc@5 96.649 [2024-09-01 16:20:40 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:20:41 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.801 (0.801) Loss 0.5093 (0.5093) Acc@1 88.867 (88.867) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 16:20:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.164) Loss 0.8252 (0.6021) Acc@1 80.811 (87.087) Acc@5 95.801 (97.869) Mem 11224MB [2024-09-01 16:20:42 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.129) Loss 0.9082 (0.7196) Acc@1 78.857 (84.226) Acc@5 95.312 (96.682) Mem 11224MB [2024-09-01 16:20:43 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.917 Acc@5 96.709 [2024-09-01 16:20:43 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:20:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][0/625] eta 0:12:09 lr 0.000022 wd 0.0500 time 1.1675 (1.1675) data time 0.5341 (0.5341) model time 0.0000 (0.0000) loss 1.9256 (1.9256) grad_norm 4.6341 (4.6341) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:20:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][10/625] eta 0:04:21 lr 0.000022 wd 0.0500 time 0.3465 (0.4257) data time 0.0009 (0.0493) model time 0.0000 (0.0000) loss 2.1766 (2.6648) grad_norm 2.3658 (3.0411) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:20:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][20/625] eta 0:03:55 lr 0.000022 wd 0.0500 time 0.3481 (0.3889) data time 0.0009 (0.0263) model time 0.0000 (0.0000) loss 3.0102 (2.6762) grad_norm 2.7788 (2.9640) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:20:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][30/625] eta 0:03:44 lr 0.000022 wd 0.0500 time 0.3514 (0.3766) data time 0.0008 (0.0181) model time 0.0000 (0.0000) loss 2.4431 (2.6425) grad_norm 2.1992 (2.9849) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:20:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][40/625] eta 0:03:36 lr 0.000022 wd 0.0500 time 0.3478 (0.3699) data time 0.0007 (0.0139) model time 0.0000 (0.0000) loss 2.8040 (2.6699) grad_norm 2.2324 (2.9971) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][50/625] eta 0:03:30 lr 0.000022 wd 0.0500 time 0.3572 (0.3663) data time 0.0006 (0.0113) model time 0.0000 (0.0000) loss 1.9947 (2.6370) grad_norm 2.5137 (3.0410) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][60/625] eta 0:03:25 lr 0.000021 wd 0.0500 time 0.3452 (0.3636) data time 0.0009 (0.0096) model time 0.3443 (0.3493) loss 2.7207 (2.6193) grad_norm 3.5194 (3.0873) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][70/625] eta 0:03:20 lr 0.000021 wd 0.0500 time 0.3468 (0.3617) data time 0.0009 (0.0084) model time 0.3459 (0.3491) loss 2.9081 (2.6157) grad_norm 3.4297 (3.1544) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][80/625] eta 0:03:16 lr 0.000021 wd 0.0500 time 0.3458 (0.3601) data time 0.0009 (0.0075) model time 0.3449 (0.3487) loss 2.7158 (2.6300) grad_norm 2.9418 (3.2033) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][90/625] eta 0:03:12 lr 0.000021 wd 0.0500 time 0.3484 (0.3589) data time 0.0006 (0.0068) model time 0.3477 (0.3486) loss 2.0293 (2.6068) grad_norm 5.7431 (3.3322) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][100/625] eta 0:03:10 lr 0.000021 wd 0.0500 time 0.3475 (0.3635) data time 0.0007 (0.0062) model time 0.3467 (0.3598) loss 2.3832 (2.5849) grad_norm 6.1720 (3.3840) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][110/625] eta 0:03:06 lr 0.000021 wd 0.0500 time 0.3504 (0.3623) data time 0.0006 (0.0057) model time 0.3498 (0.3579) loss 3.1464 (2.5849) grad_norm 2.4683 (3.3390) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][120/625] eta 0:03:02 lr 0.000021 wd 0.0500 time 0.3466 (0.3612) data time 0.0009 (0.0053) model time 0.3457 (0.3566) loss 2.6776 (2.5713) grad_norm 2.8735 (3.2921) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][130/625] eta 0:02:58 lr 0.000021 wd 0.0500 time 0.3438 (0.3602) data time 0.0007 (0.0050) model time 0.3431 (0.3555) loss 3.0036 (2.5831) grad_norm 3.9880 (3.3831) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][140/625] eta 0:02:54 lr 0.000021 wd 0.0500 time 0.3510 (0.3594) data time 0.0009 (0.0047) model time 0.3501 (0.3547) loss 2.6273 (2.5681) grad_norm 2.1293 (3.3474) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][150/625] eta 0:02:50 lr 0.000021 wd 0.0500 time 0.3523 (0.3587) data time 0.0009 (0.0044) model time 0.3514 (0.3540) loss 2.8860 (2.5767) grad_norm 2.5554 (3.3707) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][160/625] eta 0:02:46 lr 0.000021 wd 0.0500 time 0.3440 (0.3581) data time 0.0008 (0.0042) model time 0.3432 (0.3534) loss 1.5886 (2.5643) grad_norm 2.7632 (3.3651) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][170/625] eta 0:02:42 lr 0.000021 wd 0.0500 time 0.3465 (0.3575) data time 0.0006 (0.0040) model time 0.3459 (0.3528) loss 2.6548 (2.5563) grad_norm 4.0314 (3.3671) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][180/625] eta 0:02:38 lr 0.000021 wd 0.0500 time 0.3496 (0.3570) data time 0.0008 (0.0038) model time 0.3488 (0.3525) loss 2.7105 (2.5510) grad_norm 3.3156 (3.3506) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][190/625] eta 0:02:35 lr 0.000021 wd 0.0500 time 0.3493 (0.3567) data time 0.0006 (0.0037) model time 0.3486 (0.3523) loss 2.3543 (2.5442) grad_norm 2.7797 (3.3390) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][200/625] eta 0:02:31 lr 0.000021 wd 0.0500 time 0.3475 (0.3563) data time 0.0006 (0.0035) model time 0.3468 (0.3520) loss 2.8711 (2.5320) grad_norm 2.5399 (3.3012) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:21:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][210/625] eta 0:02:27 lr 0.000021 wd 0.0500 time 0.3486 (0.3559) data time 0.0006 (0.0034) model time 0.3480 (0.3517) loss 2.8554 (2.5334) grad_norm 5.7167 (3.3112) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][220/625] eta 0:02:24 lr 0.000021 wd 0.0500 time 0.3510 (0.3563) data time 0.0009 (0.0033) model time 0.3502 (0.3524) loss 2.7478 (2.5317) grad_norm 2.3483 (3.3032) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][230/625] eta 0:02:20 lr 0.000021 wd 0.0500 time 0.3461 (0.3560) data time 0.0007 (0.0032) model time 0.3454 (0.3522) loss 2.8276 (2.5226) grad_norm 2.8122 (3.3132) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][240/625] eta 0:02:16 lr 0.000021 wd 0.0500 time 0.3538 (0.3557) data time 0.0007 (0.0031) model time 0.3531 (0.3520) loss 1.8106 (2.5212) grad_norm 2.4879 (3.3069) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][250/625] eta 0:02:13 lr 0.000021 wd 0.0500 time 0.3521 (0.3554) data time 0.0006 (0.0030) model time 0.3515 (0.3517) loss 3.1501 (2.5277) grad_norm 1.9763 (3.2919) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][260/625] eta 0:02:09 lr 0.000021 wd 0.0500 time 0.3459 (0.3552) data time 0.0006 (0.0029) model time 0.3453 (0.3517) loss 2.3280 (2.5345) grad_norm 3.5282 (3.3161) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][270/625] eta 0:02:05 lr 0.000021 wd 0.0500 time 0.3483 (0.3549) data time 0.0008 (0.0029) model time 0.3475 (0.3514) loss 2.7780 (2.5399) grad_norm 2.8889 (3.3144) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][280/625] eta 0:02:02 lr 0.000021 wd 0.0500 time 0.3466 (0.3546) data time 0.0008 (0.0028) model time 0.3459 (0.3511) loss 1.5612 (2.5425) grad_norm 1.7768 (3.3125) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][290/625] eta 0:01:58 lr 0.000021 wd 0.0500 time 0.3486 (0.3544) data time 0.0008 (0.0027) model time 0.3477 (0.3510) loss 2.9508 (2.5491) grad_norm 2.4105 (3.3096) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][300/625] eta 0:01:55 lr 0.000021 wd 0.0500 time 0.3427 (0.3541) data time 0.0007 (0.0027) model time 0.3420 (0.3507) loss 2.2912 (2.5533) grad_norm 2.3331 (3.3061) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][310/625] eta 0:01:52 lr 0.000021 wd 0.0500 time 0.3510 (0.3559) data time 0.0006 (0.0026) model time 0.3503 (0.3530) loss 3.0452 (2.5487) grad_norm 2.5132 (3.3284) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][320/625] eta 0:01:48 lr 0.000021 wd 0.0500 time 0.3473 (0.3556) data time 0.0006 (0.0025) model time 0.3468 (0.3527) loss 3.0346 (2.5557) grad_norm 2.2278 (3.3230) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][330/625] eta 0:01:44 lr 0.000021 wd 0.0500 time 0.3494 (0.3553) data time 0.0007 (0.0025) model time 0.3487 (0.3525) loss 3.1436 (2.5563) grad_norm 1.9749 (3.3159) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][340/625] eta 0:01:41 lr 0.000021 wd 0.0500 time 0.3486 (0.3551) data time 0.0006 (0.0024) model time 0.3480 (0.3522) loss 2.8013 (2.5609) grad_norm 3.0471 (3.3476) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][350/625] eta 0:01:37 lr 0.000021 wd 0.0500 time 0.3519 (0.3548) data time 0.0008 (0.0024) model time 0.3511 (0.3520) loss 3.2675 (2.5596) grad_norm 3.5795 (3.3415) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][360/625] eta 0:01:33 lr 0.000021 wd 0.0500 time 0.3497 (0.3546) data time 0.0008 (0.0024) model time 0.3489 (0.3518) loss 2.5796 (2.5577) grad_norm 3.0393 (3.3322) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][370/625] eta 0:01:30 lr 0.000021 wd 0.0500 time 0.3500 (0.3544) data time 0.0006 (0.0023) model time 0.3494 (0.3516) loss 3.2255 (2.5624) grad_norm 3.4744 (3.3323) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:22:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][380/625] eta 0:01:26 lr 0.000021 wd 0.0500 time 0.3439 (0.3542) data time 0.0008 (0.0023) model time 0.3430 (0.3514) loss 3.0757 (2.5697) grad_norm 2.9825 (3.3378) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][390/625] eta 0:01:23 lr 0.000021 wd 0.0500 time 0.3403 (0.3540) data time 0.0007 (0.0023) model time 0.3396 (0.3512) loss 3.1041 (2.5709) grad_norm 3.8696 (3.3350) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][400/625] eta 0:01:19 lr 0.000021 wd 0.0500 time 0.3513 (0.3538) data time 0.0008 (0.0022) model time 0.3505 (0.3511) loss 2.8604 (2.5749) grad_norm 4.0168 (3.3341) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][410/625] eta 0:01:16 lr 0.000021 wd 0.0500 time 0.3477 (0.3536) data time 0.0007 (0.0022) model time 0.3471 (0.3509) loss 2.5553 (2.5762) grad_norm 3.4646 (3.3498) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][420/625] eta 0:01:12 lr 0.000021 wd 0.0500 time 0.3495 (0.3534) data time 0.0006 (0.0022) model time 0.3489 (0.3507) loss 1.9732 (2.5756) grad_norm 5.3171 (3.3461) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][430/625] eta 0:01:08 lr 0.000021 wd 0.0500 time 0.3468 (0.3537) data time 0.0010 (0.0021) model time 0.3458 (0.3511) loss 2.8220 (2.5786) grad_norm 1.7911 (3.3484) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][440/625] eta 0:01:05 lr 0.000021 wd 0.0500 time 0.3468 (0.3535) data time 0.0008 (0.0021) model time 0.3460 (0.3510) loss 2.5552 (2.5822) grad_norm 2.1645 (3.4066) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][450/625] eta 0:01:01 lr 0.000021 wd 0.0500 time 0.3429 (0.3534) data time 0.0008 (0.0021) model time 0.3421 (0.3508) loss 1.9641 (2.5770) grad_norm 3.5998 (3.4055) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][460/625] eta 0:00:58 lr 0.000021 wd 0.0500 time 0.3494 (0.3532) data time 0.0008 (0.0021) model time 0.3486 (0.3507) loss 2.6273 (2.5737) grad_norm 4.6449 (3.4011) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][470/625] eta 0:00:54 lr 0.000021 wd 0.0500 time 0.3445 (0.3531) data time 0.0007 (0.0020) model time 0.3438 (0.3506) loss 1.7248 (2.5682) grad_norm 3.4233 (3.4441) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][480/625] eta 0:00:51 lr 0.000021 wd 0.0500 time 0.3459 (0.3529) data time 0.0008 (0.0020) model time 0.3450 (0.3505) loss 2.8295 (2.5676) grad_norm 2.7467 (3.4325) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][490/625] eta 0:00:47 lr 0.000021 wd 0.0500 time 0.5341 (0.3536) data time 0.0009 (0.0020) model time 0.5332 (0.3513) loss 2.3676 (2.5680) grad_norm 2.2979 (3.4135) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][500/625] eta 0:00:44 lr 0.000021 wd 0.0500 time 0.3429 (0.3539) data time 0.0007 (0.0020) model time 0.3422 (0.3516) loss 2.1718 (2.5660) grad_norm 3.3361 (3.4154) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][510/625] eta 0:00:40 lr 0.000021 wd 0.0500 time 0.3451 (0.3538) data time 0.0007 (0.0019) model time 0.3444 (0.3515) loss 3.2594 (2.5679) grad_norm 2.3183 (3.4037) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][520/625] eta 0:00:37 lr 0.000021 wd 0.0500 time 0.3437 (0.3536) data time 0.0008 (0.0019) model time 0.3428 (0.3513) loss 2.7617 (2.5672) grad_norm 3.1013 (3.4082) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][530/625] eta 0:00:33 lr 0.000021 wd 0.0500 time 0.3383 (0.3534) data time 0.0006 (0.0019) model time 0.3376 (0.3512) loss 1.9169 (2.5639) grad_norm 2.1943 (3.4318) loss_scale 512.0000 (512.0000) mem 11224MB [2024-09-01 16:23:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][540/625] eta 0:00:30 lr 0.000021 wd 0.0500 time 0.3401 (0.3533) data time 0.0007 (0.0019) model time 0.3394 (0.3510) loss 2.0197 (2.5618) grad_norm 3.2087 (inf) loss_scale 256.0000 (509.1608) mem 11224MB [2024-09-01 16:23:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][550/625] eta 0:00:26 lr 0.000021 wd 0.0500 time 0.3498 (0.3532) data time 0.0007 (0.0019) model time 0.3492 (0.3509) loss 1.9362 (2.5606) grad_norm 12.3098 (inf) loss_scale 256.0000 (504.5662) mem 11224MB [2024-09-01 16:24:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][560/625] eta 0:00:22 lr 0.000021 wd 0.0500 time 0.3413 (0.3530) data time 0.0010 (0.0018) model time 0.3404 (0.3508) loss 2.6630 (2.5612) grad_norm 3.7439 (inf) loss_scale 256.0000 (500.1355) mem 11224MB [2024-09-01 16:24:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][570/625] eta 0:00:19 lr 0.000021 wd 0.0500 time 0.3467 (0.3529) data time 0.0008 (0.0018) model time 0.3459 (0.3507) loss 2.8212 (2.5660) grad_norm 3.3703 (inf) loss_scale 256.0000 (495.8599) mem 11224MB [2024-09-01 16:24:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][580/625] eta 0:00:15 lr 0.000021 wd 0.0500 time 0.3535 (0.3528) data time 0.0009 (0.0018) model time 0.3526 (0.3506) loss 3.0268 (2.5701) grad_norm 2.5783 (inf) loss_scale 256.0000 (491.7315) mem 11224MB [2024-09-01 16:24:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][590/625] eta 0:00:12 lr 0.000021 wd 0.0500 time 0.3478 (0.3527) data time 0.0008 (0.0018) model time 0.3469 (0.3505) loss 2.2650 (2.5680) grad_norm 2.8517 (inf) loss_scale 256.0000 (487.7428) mem 11224MB [2024-09-01 16:24:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][600/625] eta 0:00:08 lr 0.000021 wd 0.0500 time 0.3423 (0.3526) data time 0.0009 (0.0018) model time 0.3414 (0.3505) loss 2.1934 (2.5670) grad_norm 3.1408 (inf) loss_scale 256.0000 (483.8869) mem 11224MB [2024-09-01 16:24:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][610/625] eta 0:00:05 lr 0.000021 wd 0.0500 time 0.5359 (0.3534) data time 0.0006 (0.0018) model time 0.5353 (0.3513) loss 2.6649 (2.5657) grad_norm 3.0290 (inf) loss_scale 256.0000 (480.1571) mem 11224MB [2024-09-01 16:24:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [295/300][620/625] eta 0:00:01 lr 0.000021 wd 0.0500 time 0.3352 (0.3534) data time 0.0004 (0.0018) model time 0.3348 (0.3514) loss 1.7519 (2.5638) grad_norm 4.6956 (inf) loss_scale 256.0000 (476.5475) mem 11224MB [2024-09-01 16:24:24 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 295 training takes 0:03:40 [2024-09-01 16:24:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:24:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:24:25 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.456 (0.456) Loss 0.5098 (0.5098) Acc@1 89.258 (89.258) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 16:24:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.131) Loss 0.8315 (0.6055) Acc@1 80.664 (87.043) Acc@5 95.801 (97.843) Mem 11224MB [2024-09-01 16:24:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.9253 (0.7264) Acc@1 78.809 (84.117) Acc@5 95.264 (96.619) Mem 11224MB [2024-09-01 16:24:27 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.829 Acc@5 96.643 [2024-09-01 16:24:27 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:24:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.839 (0.839) Loss 0.5093 (0.5093) Acc@1 88.916 (88.916) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 16:24:29 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.165) Loss 0.8252 (0.6021) Acc@1 80.762 (87.083) Acc@5 95.801 (97.874) Mem 11224MB [2024-09-01 16:24:30 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.093 (0.130) Loss 0.9102 (0.7199) Acc@1 78.857 (84.217) Acc@5 95.264 (96.675) Mem 11224MB [2024-09-01 16:24:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.909 Acc@5 96.701 [2024-09-01 16:24:30 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:24:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][0/625] eta 0:12:04 lr 0.000021 wd 0.0500 time 1.1587 (1.1587) data time 0.7020 (0.7020) model time 0.0000 (0.0000) loss 2.7042 (2.7042) grad_norm 2.2025 (2.2025) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:24:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][10/625] eta 0:04:18 lr 0.000021 wd 0.0500 time 0.3452 (0.4205) data time 0.0008 (0.0646) model time 0.0000 (0.0000) loss 2.0481 (2.4622) grad_norm 2.9835 (3.0326) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:24:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][20/625] eta 0:03:52 lr 0.000021 wd 0.0500 time 0.3473 (0.3844) data time 0.0009 (0.0343) model time 0.0000 (0.0000) loss 2.2613 (2.4518) grad_norm 3.9870 (3.1407) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:24:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][30/625] eta 0:03:41 lr 0.000021 wd 0.0500 time 0.3475 (0.3722) data time 0.0008 (0.0235) model time 0.0000 (0.0000) loss 2.5386 (2.5973) grad_norm 9.5900 (3.2639) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:24:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][40/625] eta 0:03:34 lr 0.000021 wd 0.0500 time 0.3488 (0.3660) data time 0.0005 (0.0180) model time 0.0000 (0.0000) loss 2.9296 (2.5200) grad_norm 2.4486 (3.1493) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:24:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][50/625] eta 0:03:28 lr 0.000021 wd 0.0500 time 0.3479 (0.3624) data time 0.0009 (0.0146) model time 0.0000 (0.0000) loss 2.4687 (2.5383) grad_norm 2.4807 (3.1495) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:24:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][60/625] eta 0:03:23 lr 0.000021 wd 0.0500 time 0.3451 (0.3597) data time 0.0006 (0.0124) model time 0.3444 (0.3449) loss 2.5639 (2.5730) grad_norm 2.3966 (3.0736) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:24:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][70/625] eta 0:03:18 lr 0.000021 wd 0.0500 time 0.3493 (0.3578) data time 0.0007 (0.0108) model time 0.3486 (0.3453) loss 2.7430 (2.5975) grad_norm 3.7076 (3.0911) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:24:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][80/625] eta 0:03:14 lr 0.000021 wd 0.0500 time 0.3472 (0.3568) data time 0.0009 (0.0095) model time 0.3463 (0.3465) loss 2.4036 (2.6017) grad_norm 2.2736 (3.0623) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][90/625] eta 0:03:10 lr 0.000021 wd 0.0500 time 0.3438 (0.3557) data time 0.0007 (0.0086) model time 0.3432 (0.3463) loss 3.3598 (2.5937) grad_norm 3.0117 (3.0436) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][100/625] eta 0:03:06 lr 0.000021 wd 0.0500 time 0.3455 (0.3548) data time 0.0009 (0.0078) model time 0.3446 (0.3462) loss 2.2290 (2.5812) grad_norm 2.1405 (3.1014) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][110/625] eta 0:03:02 lr 0.000021 wd 0.0500 time 0.3453 (0.3539) data time 0.0007 (0.0072) model time 0.3446 (0.3458) loss 3.0795 (2.5938) grad_norm 3.0450 (3.2177) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][120/625] eta 0:02:58 lr 0.000021 wd 0.0500 time 0.3436 (0.3534) data time 0.0008 (0.0067) model time 0.3427 (0.3459) loss 2.8050 (2.5840) grad_norm 2.5343 (3.1990) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][130/625] eta 0:02:54 lr 0.000021 wd 0.0500 time 0.3485 (0.3529) data time 0.0009 (0.0063) model time 0.3475 (0.3459) loss 2.3652 (2.5845) grad_norm 2.5397 (3.2116) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][140/625] eta 0:02:50 lr 0.000021 wd 0.0500 time 0.3469 (0.3524) data time 0.0009 (0.0059) model time 0.3460 (0.3458) loss 2.5525 (2.5899) grad_norm 3.5208 (3.4907) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][150/625] eta 0:02:47 lr 0.000021 wd 0.0500 time 0.3485 (0.3520) data time 0.0009 (0.0056) model time 0.3476 (0.3458) loss 1.8865 (2.6008) grad_norm 4.6752 (3.4810) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][160/625] eta 0:02:44 lr 0.000021 wd 0.0500 time 0.3478 (0.3530) data time 0.0006 (0.0053) model time 0.3472 (0.3477) loss 2.7915 (2.5968) grad_norm 3.0258 (3.4463) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][170/625] eta 0:02:41 lr 0.000021 wd 0.0500 time 0.3445 (0.3547) data time 0.0009 (0.0050) model time 0.3436 (0.3505) loss 2.6660 (2.5967) grad_norm 2.1085 (3.4234) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][180/625] eta 0:02:38 lr 0.000021 wd 0.0500 time 0.3498 (0.3552) data time 0.0008 (0.0048) model time 0.3490 (0.3515) loss 3.2554 (2.6119) grad_norm 3.4533 (3.3939) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][190/625] eta 0:02:34 lr 0.000021 wd 0.0500 time 0.3393 (0.3547) data time 0.0008 (0.0046) model time 0.3385 (0.3510) loss 3.1384 (2.6206) grad_norm 2.6506 (3.3818) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][200/625] eta 0:02:30 lr 0.000021 wd 0.0500 time 0.3436 (0.3543) data time 0.0008 (0.0044) model time 0.3428 (0.3506) loss 2.7155 (2.6215) grad_norm 2.7644 (3.3456) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][210/625] eta 0:02:26 lr 0.000021 wd 0.0500 time 0.3420 (0.3539) data time 0.0006 (0.0042) model time 0.3414 (0.3503) loss 2.3642 (2.6175) grad_norm 4.5462 (3.3419) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][220/625] eta 0:02:23 lr 0.000021 wd 0.0500 time 0.3499 (0.3536) data time 0.0006 (0.0041) model time 0.3492 (0.3500) loss 2.9977 (2.6195) grad_norm 3.5847 (3.3284) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][230/625] eta 0:02:19 lr 0.000021 wd 0.0500 time 0.3494 (0.3533) data time 0.0006 (0.0039) model time 0.3488 (0.3498) loss 2.1880 (2.6089) grad_norm 39.2903 (3.4841) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][240/625] eta 0:02:15 lr 0.000021 wd 0.0500 time 0.3434 (0.3530) data time 0.0007 (0.0038) model time 0.3427 (0.3496) loss 2.7880 (2.6122) grad_norm 4.1772 (3.4739) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:25:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][250/625] eta 0:02:12 lr 0.000021 wd 0.0500 time 0.3429 (0.3528) data time 0.0008 (0.0037) model time 0.3421 (0.3494) loss 2.6274 (2.6107) grad_norm 2.9075 (3.4464) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][260/625] eta 0:02:08 lr 0.000021 wd 0.0500 time 0.3448 (0.3525) data time 0.0009 (0.0036) model time 0.3439 (0.3492) loss 2.7630 (2.6155) grad_norm 2.0353 (3.6011) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][270/625] eta 0:02:05 lr 0.000021 wd 0.0500 time 0.3424 (0.3522) data time 0.0007 (0.0035) model time 0.3417 (0.3489) loss 2.0784 (2.6206) grad_norm 3.1785 (3.5693) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][280/625] eta 0:02:01 lr 0.000021 wd 0.0500 time 0.3460 (0.3520) data time 0.0008 (0.0034) model time 0.3452 (0.3487) loss 2.7604 (2.6196) grad_norm 3.8917 (3.6814) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][290/625] eta 0:01:58 lr 0.000021 wd 0.0500 time 0.3422 (0.3525) data time 0.0008 (0.0033) model time 0.3414 (0.3495) loss 2.9143 (2.6204) grad_norm 2.4138 (3.6545) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][300/625] eta 0:01:54 lr 0.000021 wd 0.0500 time 0.3379 (0.3530) data time 0.0008 (0.0032) model time 0.3370 (0.3502) loss 3.1263 (2.6246) grad_norm 4.2923 (3.6393) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][310/625] eta 0:01:51 lr 0.000021 wd 0.0500 time 0.3453 (0.3528) data time 0.0007 (0.0032) model time 0.3446 (0.3500) loss 3.1218 (2.6255) grad_norm 3.5294 (3.6294) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][320/625] eta 0:01:47 lr 0.000021 wd 0.0500 time 0.3482 (0.3526) data time 0.0008 (0.0031) model time 0.3474 (0.3499) loss 2.9960 (2.6257) grad_norm 3.8972 (3.6125) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][330/625] eta 0:01:43 lr 0.000021 wd 0.0500 time 0.3454 (0.3524) data time 0.0008 (0.0030) model time 0.3446 (0.3497) loss 2.5080 (2.6293) grad_norm 2.9205 (3.5999) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][340/625] eta 0:01:40 lr 0.000021 wd 0.0500 time 0.3469 (0.3529) data time 0.0007 (0.0030) model time 0.3462 (0.3503) loss 1.9883 (2.6283) grad_norm 4.1686 (3.5815) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][350/625] eta 0:01:36 lr 0.000021 wd 0.0500 time 0.3457 (0.3527) data time 0.0007 (0.0029) model time 0.3450 (0.3501) loss 2.5955 (2.6302) grad_norm 5.4089 (3.6000) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][360/625] eta 0:01:33 lr 0.000021 wd 0.0500 time 0.3563 (0.3525) data time 0.0006 (0.0029) model time 0.3557 (0.3500) loss 2.3726 (2.6231) grad_norm 3.6677 (3.5981) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][370/625] eta 0:01:29 lr 0.000021 wd 0.0500 time 0.3479 (0.3524) data time 0.0009 (0.0028) model time 0.3470 (0.3499) loss 2.5934 (2.6200) grad_norm 3.0844 (3.6381) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][380/625] eta 0:01:26 lr 0.000021 wd 0.0500 time 0.3523 (0.3523) data time 0.0008 (0.0028) model time 0.3515 (0.3498) loss 2.5861 (2.6125) grad_norm 3.3807 (3.6794) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][390/625] eta 0:01:22 lr 0.000021 wd 0.0500 time 0.3511 (0.3522) data time 0.0006 (0.0027) model time 0.3506 (0.3497) loss 3.2272 (2.6186) grad_norm 3.5044 (3.6657) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][400/625] eta 0:01:19 lr 0.000021 wd 0.0500 time 0.3416 (0.3520) data time 0.0006 (0.0027) model time 0.3410 (0.3496) loss 1.9573 (2.6166) grad_norm 10.3010 (3.6626) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][410/625] eta 0:01:15 lr 0.000021 wd 0.0500 time 0.3491 (0.3519) data time 0.0009 (0.0026) model time 0.3482 (0.3495) loss 2.8520 (2.6137) grad_norm 4.4087 (3.6543) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:26:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][420/625] eta 0:01:12 lr 0.000021 wd 0.0500 time 0.3422 (0.3517) data time 0.0008 (0.0026) model time 0.3414 (0.3493) loss 2.6034 (2.6127) grad_norm 2.9234 (3.6474) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][430/625] eta 0:01:08 lr 0.000021 wd 0.0500 time 0.3530 (0.3516) data time 0.0006 (0.0025) model time 0.3523 (0.3493) loss 2.5981 (2.6117) grad_norm 2.8821 (3.6422) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][440/625] eta 0:01:05 lr 0.000021 wd 0.0500 time 0.3462 (0.3515) data time 0.0007 (0.0025) model time 0.3455 (0.3492) loss 3.2226 (2.6128) grad_norm 2.4447 (3.6308) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][450/625] eta 0:01:01 lr 0.000021 wd 0.0500 time 0.3500 (0.3514) data time 0.0008 (0.0025) model time 0.3492 (0.3491) loss 2.3889 (2.6102) grad_norm 3.1796 (3.6180) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][460/625] eta 0:00:58 lr 0.000021 wd 0.0500 time 0.3476 (0.3517) data time 0.0006 (0.0024) model time 0.3469 (0.3495) loss 3.1078 (2.6097) grad_norm 2.6405 (3.6046) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][470/625] eta 0:00:54 lr 0.000021 wd 0.0500 time 0.3469 (0.3516) data time 0.0009 (0.0024) model time 0.3460 (0.3494) loss 3.0998 (2.6063) grad_norm 4.4861 (3.5958) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][480/625] eta 0:00:50 lr 0.000021 wd 0.0500 time 0.3469 (0.3515) data time 0.0009 (0.0024) model time 0.3459 (0.3493) loss 2.0639 (2.5998) grad_norm 2.2381 (3.6521) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][490/625] eta 0:00:47 lr 0.000021 wd 0.0500 time 0.3457 (0.3514) data time 0.0009 (0.0023) model time 0.3448 (0.3493) loss 2.9026 (2.5969) grad_norm 4.7942 (3.6454) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][500/625] eta 0:00:43 lr 0.000021 wd 0.0500 time 0.3492 (0.3518) data time 0.0008 (0.0023) model time 0.3484 (0.3497) loss 2.6433 (2.5949) grad_norm 1.9295 (3.6280) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][510/625] eta 0:00:40 lr 0.000021 wd 0.0500 time 0.3512 (0.3522) data time 0.0007 (0.0023) model time 0.3505 (0.3501) loss 2.6874 (2.5903) grad_norm 2.0108 (3.6251) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][520/625] eta 0:00:36 lr 0.000021 wd 0.0500 time 0.3471 (0.3521) data time 0.0006 (0.0023) model time 0.3465 (0.3501) loss 2.7576 (2.5932) grad_norm 15.5577 (3.6523) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][530/625] eta 0:00:33 lr 0.000021 wd 0.0500 time 0.3434 (0.3520) data time 0.0006 (0.0022) model time 0.3427 (0.3500) loss 1.8113 (2.5886) grad_norm 2.2846 (3.6481) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][540/625] eta 0:00:29 lr 0.000021 wd 0.0500 time 0.3487 (0.3519) data time 0.0008 (0.0022) model time 0.3479 (0.3499) loss 1.9963 (2.5927) grad_norm 2.3263 (3.6372) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][550/625] eta 0:00:26 lr 0.000021 wd 0.0500 time 0.3450 (0.3518) data time 0.0007 (0.0022) model time 0.3443 (0.3499) loss 2.6341 (2.5954) grad_norm 2.4657 (3.6265) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][560/625] eta 0:00:22 lr 0.000021 wd 0.0500 time 0.3450 (0.3517) data time 0.0007 (0.0022) model time 0.3443 (0.3498) loss 2.2430 (2.5988) grad_norm 2.7302 (3.6144) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][570/625] eta 0:00:19 lr 0.000021 wd 0.0500 time 0.3498 (0.3517) data time 0.0009 (0.0021) model time 0.3490 (0.3497) loss 2.6632 (2.5953) grad_norm 3.5547 (3.6137) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][580/625] eta 0:00:15 lr 0.000021 wd 0.0500 time 0.3435 (0.3516) data time 0.0009 (0.0021) model time 0.3426 (0.3496) loss 2.5460 (2.5930) grad_norm 3.7677 (3.6261) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:27:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][590/625] eta 0:00:12 lr 0.000021 wd 0.0500 time 0.3393 (0.3515) data time 0.0008 (0.0021) model time 0.3385 (0.3495) loss 2.4080 (2.5920) grad_norm 2.9030 (3.6434) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:28:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][600/625] eta 0:00:08 lr 0.000021 wd 0.0500 time 0.3478 (0.3514) data time 0.0008 (0.0021) model time 0.3470 (0.3495) loss 2.6009 (2.5961) grad_norm 3.8943 (3.6573) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:28:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][610/625] eta 0:00:05 lr 0.000021 wd 0.0500 time 0.3366 (0.3513) data time 0.0004 (0.0021) model time 0.3362 (0.3494) loss 2.8624 (2.5943) grad_norm 2.6572 (3.6452) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:28:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [296/300][620/625] eta 0:00:01 lr 0.000021 wd 0.0500 time 0.3393 (0.3511) data time 0.0003 (0.0020) model time 0.3390 (0.3492) loss 3.4239 (2.5952) grad_norm 4.0750 (3.6339) loss_scale 256.0000 (256.0000) mem 11224MB [2024-09-01 16:28:10 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 296 training takes 0:03:39 [2024-09-01 16:28:10 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:28:11 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:28:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.451 (0.451) Loss 0.5127 (0.5127) Acc@1 89.111 (89.111) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 16:28:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.128) Loss 0.8408 (0.6089) Acc@1 80.713 (87.003) Acc@5 95.850 (97.834) Mem 11224MB [2024-09-01 16:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.110) Loss 0.9214 (0.7288) Acc@1 78.711 (84.061) Acc@5 95.166 (96.626) Mem 11224MB [2024-09-01 16:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.785 Acc@5 96.641 [2024-09-01 16:28:13 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:28:14 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.791 (0.791) Loss 0.5088 (0.5088) Acc@1 89.014 (89.014) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 16:28:15 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.166) Loss 0.8262 (0.6025) Acc@1 80.762 (87.105) Acc@5 95.801 (97.869) Mem 11224MB [2024-09-01 16:28:16 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.130) Loss 0.9111 (0.7204) Acc@1 78.809 (84.222) Acc@5 95.312 (96.670) Mem 11224MB [2024-09-01 16:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.907 Acc@5 96.697 [2024-09-01 16:28:17 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:29:57 msvmambav3_small_224] (main_hfai_mnodes.py 533): INFO Full config saved to ./exclude/output_msv2/msvmambav3_small_224/20240819232309/config.json [2024-09-01 16:29:58 msvmambav3_small_224] (main_hfai_mnodes.py 129): INFO Creating model:vssm/msvmambav3_small_224 [2024-09-01 16:30:11 msvmambav3_small_224] (optimizer.py 18): INFO ==============> building optimizer adamw.................... [2024-09-01 16:30:21 msvmambav3_small_224] (main_hfai_mnodes.py 194): INFO auto resuming from ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth [2024-09-01 16:30:21 msvmambav3_small_224] (utils.py 21): INFO ==============> Resuming form ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth.................... [2024-09-01 16:30:24 msvmambav3_small_224] (utils.py 30): INFO resuming model: [2024-09-01 16:30:26 msvmambav3_small_224] (utils.py 37): INFO resuming model_ema: [2024-09-01 16:30:26 msvmambav3_small_224] (utils.py 61): INFO => loaded successfully './exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth' (epoch 296) [2024-09-01 16:30:26 msvmambav3_small_224] (main_hfai_mnodes.py 234): INFO Start training [2024-09-01 16:30:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][0/625] eta 2:08:57 lr 0.000021 wd 0.0500 time 12.3797 (12.3797) data time 0.6528 (0.6528) model time 0.0000 (0.0000) loss 3.0419 (3.0419) grad_norm 2.5883 (2.5883) loss_scale 256.0000 (256.0000) mem 23875MB [2024-09-01 16:30:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][10/625] eta 0:14:51 lr 0.000021 wd 0.0500 time 0.3476 (1.4491) data time 0.0007 (0.0601) model time 0.0000 (0.0000) loss 2.4082 (2.8057) grad_norm 2.0483 (inf) loss_scale 128.0000 (186.1818) mem 11219MB [2024-09-01 16:30:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][20/625] eta 0:09:18 lr 0.000021 wd 0.0500 time 0.3427 (0.9234) data time 0.0009 (0.0319) model time 0.0000 (0.0000) loss 2.8885 (2.7773) grad_norm 2.1931 (inf) loss_scale 128.0000 (158.4762) mem 11219MB [2024-09-01 16:30:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][30/625] eta 0:07:23 lr 0.000021 wd 0.0500 time 0.3477 (0.7449) data time 0.0006 (0.0219) model time 0.0000 (0.0000) loss 1.7639 (2.8027) grad_norm 2.9673 (inf) loss_scale 128.0000 (148.6452) mem 11219MB [2024-09-01 16:30:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][40/625] eta 0:06:22 lr 0.000021 wd 0.0500 time 0.6187 (0.6541) data time 0.0009 (0.0168) model time 0.0000 (0.0000) loss 2.5731 (2.7575) grad_norm 2.6994 (inf) loss_scale 128.0000 (143.6098) mem 11219MB [2024-09-01 16:31:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][50/625] eta 0:05:41 lr 0.000021 wd 0.0500 time 0.3455 (0.5934) data time 0.0007 (0.0137) model time 0.0000 (0.0000) loss 3.0339 (2.7378) grad_norm 2.3504 (inf) loss_scale 128.0000 (140.5490) mem 11219MB [2024-09-01 16:31:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][60/625] eta 0:05:12 lr 0.000021 wd 0.0500 time 0.3441 (0.5527) data time 0.0010 (0.0116) model time 0.3431 (0.3446) loss 2.4115 (2.7078) grad_norm 2.6777 (inf) loss_scale 128.0000 (138.4918) mem 11219MB [2024-09-01 16:31:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][70/625] eta 0:04:50 lr 0.000021 wd 0.0500 time 0.3430 (0.5233) data time 0.0009 (0.0101) model time 0.3421 (0.3439) loss 2.4111 (2.6721) grad_norm 6.6820 (inf) loss_scale 128.0000 (137.0141) mem 11219MB [2024-09-01 16:31:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][80/625] eta 0:04:33 lr 0.000021 wd 0.0500 time 0.3422 (0.5013) data time 0.0008 (0.0089) model time 0.3414 (0.3439) loss 2.3135 (2.6583) grad_norm 3.0190 (inf) loss_scale 128.0000 (135.9012) mem 11219MB [2024-09-01 16:31:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][90/625] eta 0:04:19 lr 0.000021 wd 0.0500 time 0.3425 (0.4843) data time 0.0007 (0.0080) model time 0.3419 (0.3443) loss 2.8199 (2.6465) grad_norm 6.1526 (inf) loss_scale 128.0000 (135.0330) mem 11219MB [2024-09-01 16:31:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][100/625] eta 0:04:07 lr 0.000021 wd 0.0500 time 0.3455 (0.4706) data time 0.0006 (0.0073) model time 0.3448 (0.3445) loss 2.6865 (2.6562) grad_norm 5.3347 (inf) loss_scale 128.0000 (134.3366) mem 11219MB [2024-09-01 16:31:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][110/625] eta 0:03:56 lr 0.000020 wd 0.0500 time 0.3449 (0.4593) data time 0.0009 (0.0067) model time 0.3440 (0.3445) loss 2.2221 (2.6557) grad_norm 4.1633 (inf) loss_scale 128.0000 (133.7658) mem 11219MB [2024-09-01 16:31:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][120/625] eta 0:03:47 lr 0.000020 wd 0.0500 time 0.3454 (0.4500) data time 0.0007 (0.0063) model time 0.3447 (0.3446) loss 1.9355 (2.6457) grad_norm 3.5135 (inf) loss_scale 128.0000 (133.2893) mem 11219MB [2024-09-01 16:31:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][130/625] eta 0:03:38 lr 0.000020 wd 0.0500 time 0.3454 (0.4422) data time 0.0008 (0.0058) model time 0.3446 (0.3449) loss 2.7454 (2.6420) grad_norm 3.1867 (inf) loss_scale 128.0000 (132.8855) mem 11219MB [2024-09-01 16:31:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][140/625] eta 0:03:31 lr 0.000020 wd 0.0500 time 0.3405 (0.4353) data time 0.0008 (0.0055) model time 0.3398 (0.3449) loss 2.6778 (2.6278) grad_norm 2.7454 (inf) loss_scale 128.0000 (132.5390) mem 11219MB [2024-09-01 16:31:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][150/625] eta 0:03:23 lr 0.000020 wd 0.0500 time 0.3512 (0.4294) data time 0.0009 (0.0052) model time 0.3503 (0.3449) loss 1.9770 (2.6243) grad_norm 3.2367 (inf) loss_scale 128.0000 (132.2384) mem 11219MB [2024-09-01 16:31:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][160/625] eta 0:03:17 lr 0.000020 wd 0.0500 time 0.3444 (0.4243) data time 0.0008 (0.0049) model time 0.3436 (0.3450) loss 2.7266 (2.6225) grad_norm 2.5324 (inf) loss_scale 128.0000 (131.9752) mem 11219MB [2024-09-01 16:31:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][170/625] eta 0:03:10 lr 0.000020 wd 0.0500 time 0.3434 (0.4197) data time 0.0009 (0.0047) model time 0.3424 (0.3451) loss 2.5351 (2.6177) grad_norm 5.1823 (inf) loss_scale 128.0000 (131.7427) mem 11219MB [2024-09-01 16:31:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][180/625] eta 0:03:05 lr 0.000020 wd 0.0500 time 0.3504 (0.4157) data time 0.0008 (0.0045) model time 0.3496 (0.3452) loss 2.6987 (2.6011) grad_norm 2.8550 (inf) loss_scale 128.0000 (131.5359) mem 11219MB [2024-09-01 16:31:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][190/625] eta 0:02:59 lr 0.000020 wd 0.0500 time 0.3491 (0.4122) data time 0.0008 (0.0043) model time 0.3483 (0.3453) loss 2.3342 (2.5992) grad_norm 2.5390 (inf) loss_scale 128.0000 (131.3508) mem 11219MB [2024-09-01 16:31:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][200/625] eta 0:02:53 lr 0.000020 wd 0.0500 time 0.3445 (0.4088) data time 0.0008 (0.0041) model time 0.3436 (0.3452) loss 2.4994 (2.5908) grad_norm 2.8149 (inf) loss_scale 128.0000 (131.1841) mem 11219MB [2024-09-01 16:31:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][210/625] eta 0:02:48 lr 0.000020 wd 0.0500 time 0.3455 (0.4059) data time 0.0008 (0.0040) model time 0.3447 (0.3453) loss 3.1493 (2.5912) grad_norm 3.6319 (inf) loss_scale 128.0000 (131.0332) mem 11219MB [2024-09-01 16:32:00 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][220/625] eta 0:02:43 lr 0.000020 wd 0.0500 time 0.3497 (0.4032) data time 0.0008 (0.0038) model time 0.3489 (0.3453) loss 3.0630 (2.5929) grad_norm 2.1862 (inf) loss_scale 128.0000 (130.8959) mem 11219MB [2024-09-01 16:32:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][230/625] eta 0:02:38 lr 0.000020 wd 0.0500 time 0.3484 (0.4025) data time 0.0007 (0.0037) model time 0.3477 (0.3475) loss 1.6797 (2.5888) grad_norm 2.2605 (inf) loss_scale 128.0000 (130.7706) mem 11219MB [2024-09-01 16:32:07 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][240/625] eta 0:02:34 lr 0.000020 wd 0.0500 time 0.3448 (0.4009) data time 0.0006 (0.0036) model time 0.3442 (0.3483) loss 2.7380 (2.5875) grad_norm 2.0648 (inf) loss_scale 128.0000 (130.6556) mem 11219MB [2024-09-01 16:32:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][250/625] eta 0:02:29 lr 0.000020 wd 0.0500 time 0.3504 (0.3987) data time 0.0007 (0.0035) model time 0.3497 (0.3482) loss 2.6650 (2.5769) grad_norm 5.1051 (inf) loss_scale 128.0000 (130.5498) mem 11219MB [2024-09-01 16:32:14 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][260/625] eta 0:02:24 lr 0.000020 wd 0.0500 time 0.3468 (0.3967) data time 0.0007 (0.0034) model time 0.3461 (0.3480) loss 2.3780 (2.5679) grad_norm 2.1720 (inf) loss_scale 128.0000 (130.4521) mem 11219MB [2024-09-01 16:32:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][270/625] eta 0:02:20 lr 0.000020 wd 0.0500 time 0.3437 (0.3948) data time 0.0008 (0.0033) model time 0.3429 (0.3479) loss 2.8825 (2.5661) grad_norm 2.3304 (inf) loss_scale 128.0000 (130.3616) mem 11219MB [2024-09-01 16:32:21 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][280/625] eta 0:02:15 lr 0.000020 wd 0.0500 time 0.3457 (0.3930) data time 0.0009 (0.0032) model time 0.3448 (0.3477) loss 2.5405 (2.5691) grad_norm 2.3264 (inf) loss_scale 128.0000 (130.2776) mem 11219MB [2024-09-01 16:32:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][290/625] eta 0:02:11 lr 0.000020 wd 0.0500 time 0.3503 (0.3915) data time 0.0006 (0.0031) model time 0.3497 (0.3477) loss 1.5679 (2.5615) grad_norm 2.7217 (inf) loss_scale 128.0000 (130.1993) mem 11219MB [2024-09-01 16:32:28 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][300/625] eta 0:02:06 lr 0.000020 wd 0.0500 time 0.3454 (0.3899) data time 0.0009 (0.0030) model time 0.3446 (0.3475) loss 2.4985 (2.5573) grad_norm 3.7714 (inf) loss_scale 128.0000 (130.1262) mem 11219MB [2024-09-01 16:32:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][310/625] eta 0:02:02 lr 0.000020 wd 0.0500 time 0.3450 (0.3885) data time 0.0009 (0.0030) model time 0.3442 (0.3474) loss 2.9827 (2.5556) grad_norm 3.4643 (inf) loss_scale 128.0000 (130.0579) mem 11219MB [2024-09-01 16:32:35 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][320/625] eta 0:01:58 lr 0.000020 wd 0.0500 time 0.3465 (0.3871) data time 0.0007 (0.0029) model time 0.3459 (0.3473) loss 3.1330 (2.5649) grad_norm 4.0594 (inf) loss_scale 128.0000 (129.9938) mem 11219MB [2024-09-01 16:32:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][330/625] eta 0:01:53 lr 0.000020 wd 0.0500 time 0.3413 (0.3858) data time 0.0007 (0.0028) model time 0.3406 (0.3471) loss 1.8165 (2.5654) grad_norm 3.8251 (inf) loss_scale 128.0000 (129.9335) mem 11219MB [2024-09-01 16:32:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][340/625] eta 0:01:49 lr 0.000020 wd 0.0500 time 0.3446 (0.3846) data time 0.0009 (0.0028) model time 0.3438 (0.3470) loss 2.9724 (2.5684) grad_norm 2.6637 (inf) loss_scale 128.0000 (129.8768) mem 11219MB [2024-09-01 16:32:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][350/625] eta 0:01:45 lr 0.000020 wd 0.0500 time 0.3440 (0.3846) data time 0.0006 (0.0027) model time 0.3433 (0.3483) loss 2.8286 (2.5678) grad_norm 2.9486 (inf) loss_scale 128.0000 (129.8234) mem 11219MB [2024-09-01 16:32:49 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][360/625] eta 0:01:41 lr 0.000020 wd 0.0500 time 0.3455 (0.3836) data time 0.0008 (0.0027) model time 0.3447 (0.3482) loss 2.5615 (2.5674) grad_norm 3.1934 (inf) loss_scale 128.0000 (129.7729) mem 11219MB [2024-09-01 16:32:53 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][370/625] eta 0:01:37 lr 0.000020 wd 0.0500 time 0.3454 (0.3825) data time 0.0010 (0.0026) model time 0.3444 (0.3480) loss 1.9728 (2.5653) grad_norm 24.3990 (inf) loss_scale 128.0000 (129.7251) mem 11219MB [2024-09-01 16:32:56 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][380/625] eta 0:01:33 lr 0.000020 wd 0.0500 time 0.3499 (0.3815) data time 0.0007 (0.0026) model time 0.3492 (0.3479) loss 1.7153 (2.5655) grad_norm 2.8049 (inf) loss_scale 128.0000 (129.6798) mem 11219MB [2024-09-01 16:32:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][390/625] eta 0:01:29 lr 0.000020 wd 0.0500 time 0.3449 (0.3806) data time 0.0006 (0.0025) model time 0.3443 (0.3479) loss 3.2687 (2.5636) grad_norm 2.5708 (inf) loss_scale 128.0000 (129.6368) mem 11219MB [2024-09-01 16:33:03 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][400/625] eta 0:01:25 lr 0.000020 wd 0.0500 time 0.3443 (0.3798) data time 0.0008 (0.0025) model time 0.3434 (0.3478) loss 2.8885 (2.5681) grad_norm 3.2106 (inf) loss_scale 128.0000 (129.5960) mem 11219MB [2024-09-01 16:33:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][410/625] eta 0:01:21 lr 0.000020 wd 0.0500 time 0.3423 (0.3790) data time 0.0009 (0.0025) model time 0.3414 (0.3478) loss 3.1682 (2.5723) grad_norm 2.3959 (inf) loss_scale 128.0000 (129.5572) mem 11219MB [2024-09-01 16:33:10 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][420/625] eta 0:01:17 lr 0.000020 wd 0.0500 time 0.3420 (0.3786) data time 0.0008 (0.0024) model time 0.3412 (0.3482) loss 3.2035 (2.5720) grad_norm 2.0436 (inf) loss_scale 128.0000 (129.5202) mem 11219MB [2024-09-01 16:33:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][430/625] eta 0:01:13 lr 0.000020 wd 0.0500 time 0.3418 (0.3779) data time 0.0006 (0.0024) model time 0.3412 (0.3481) loss 2.6663 (2.5749) grad_norm 2.7889 (inf) loss_scale 128.0000 (129.4849) mem 11219MB [2024-09-01 16:33:17 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][440/625] eta 0:01:09 lr 0.000020 wd 0.0500 time 0.3425 (0.3771) data time 0.0007 (0.0023) model time 0.3418 (0.3480) loss 2.9834 (2.5783) grad_norm 2.8086 (inf) loss_scale 128.0000 (129.4512) mem 11219MB [2024-09-01 16:33:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][450/625] eta 0:01:05 lr 0.000020 wd 0.0500 time 0.3409 (0.3764) data time 0.0008 (0.0023) model time 0.3401 (0.3479) loss 2.2719 (2.5759) grad_norm 5.2537 (inf) loss_scale 128.0000 (129.4191) mem 11219MB [2024-09-01 16:33:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][460/625] eta 0:01:02 lr 0.000020 wd 0.0500 time 0.3502 (0.3758) data time 0.0008 (0.0023) model time 0.3494 (0.3479) loss 2.8643 (2.5726) grad_norm 3.4586 (inf) loss_scale 128.0000 (129.3883) mem 11219MB [2024-09-01 16:33:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][470/625] eta 0:00:58 lr 0.000020 wd 0.0500 time 0.3427 (0.3751) data time 0.0009 (0.0023) model time 0.3417 (0.3477) loss 2.5296 (2.5650) grad_norm 2.0180 (inf) loss_scale 128.0000 (129.3588) mem 11219MB [2024-09-01 16:33:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][480/625] eta 0:00:54 lr 0.000020 wd 0.0500 time 0.3405 (0.3746) data time 0.0006 (0.0022) model time 0.3399 (0.3477) loss 3.1954 (2.5631) grad_norm 3.1562 (inf) loss_scale 128.0000 (129.3306) mem 11219MB [2024-09-01 16:33:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][490/625] eta 0:00:50 lr 0.000020 wd 0.0500 time 0.3451 (0.3740) data time 0.0008 (0.0022) model time 0.3443 (0.3477) loss 2.7093 (2.5685) grad_norm 2.6925 (inf) loss_scale 128.0000 (129.3035) mem 11219MB [2024-09-01 16:33:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][500/625] eta 0:00:46 lr 0.000020 wd 0.0500 time 0.3432 (0.3734) data time 0.0009 (0.0022) model time 0.3423 (0.3476) loss 3.1034 (2.5682) grad_norm 2.4896 (inf) loss_scale 128.0000 (129.2774) mem 11219MB [2024-09-01 16:33:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][510/625] eta 0:00:42 lr 0.000020 wd 0.0500 time 0.3447 (0.3729) data time 0.0007 (0.0021) model time 0.3440 (0.3475) loss 3.1790 (2.5727) grad_norm 2.2429 (inf) loss_scale 128.0000 (129.2524) mem 11219MB [2024-09-01 16:33:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][520/625] eta 0:00:39 lr 0.000020 wd 0.0500 time 0.3490 (0.3723) data time 0.0006 (0.0021) model time 0.3484 (0.3475) loss 2.3999 (2.5730) grad_norm 2.6076 (inf) loss_scale 128.0000 (129.2284) mem 11219MB [2024-09-01 16:33:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][530/625] eta 0:00:35 lr 0.000020 wd 0.0500 time 0.3403 (0.3718) data time 0.0009 (0.0021) model time 0.3394 (0.3474) loss 2.8140 (2.5682) grad_norm 3.3089 (inf) loss_scale 128.0000 (129.2053) mem 11219MB [2024-09-01 16:33:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][540/625] eta 0:00:31 lr 0.000020 wd 0.0500 time 0.3453 (0.3717) data time 0.0009 (0.0021) model time 0.3444 (0.3477) loss 2.6168 (2.5663) grad_norm 3.8969 (inf) loss_scale 128.0000 (129.1830) mem 11219MB [2024-09-01 16:33:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][550/625] eta 0:00:27 lr 0.000020 wd 0.0500 time 0.3446 (0.3712) data time 0.0008 (0.0020) model time 0.3438 (0.3477) loss 3.2943 (2.5654) grad_norm 3.0215 (inf) loss_scale 128.0000 (129.1615) mem 11219MB [2024-09-01 16:33:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][560/625] eta 0:00:24 lr 0.000020 wd 0.0500 time 0.3437 (0.3715) data time 0.0010 (0.0020) model time 0.3427 (0.3484) loss 2.9064 (2.5689) grad_norm 3.2547 (inf) loss_scale 128.0000 (129.1408) mem 11219MB [2024-09-01 16:34:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][570/625] eta 0:00:20 lr 0.000020 wd 0.0500 time 0.3536 (0.3710) data time 0.0009 (0.0020) model time 0.3528 (0.3484) loss 2.6055 (2.5690) grad_norm 2.0011 (inf) loss_scale 128.0000 (129.1208) mem 11219MB [2024-09-01 16:34:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][580/625] eta 0:00:16 lr 0.000020 wd 0.0500 time 0.3424 (0.3706) data time 0.0007 (0.0020) model time 0.3417 (0.3483) loss 2.9656 (2.5713) grad_norm 4.3799 (inf) loss_scale 128.0000 (129.1015) mem 11219MB [2024-09-01 16:34:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][590/625] eta 0:00:12 lr 0.000020 wd 0.0500 time 0.3457 (0.3701) data time 0.0009 (0.0020) model time 0.3449 (0.3482) loss 2.6591 (2.5720) grad_norm 3.9888 (inf) loss_scale 128.0000 (129.0829) mem 11219MB [2024-09-01 16:34:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][600/625] eta 0:00:09 lr 0.000020 wd 0.0500 time 0.3451 (0.3697) data time 0.0008 (0.0020) model time 0.3442 (0.3481) loss 1.7093 (2.5695) grad_norm 3.6491 (inf) loss_scale 128.0000 (129.0649) mem 11219MB [2024-09-01 16:34:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][610/625] eta 0:00:05 lr 0.000020 wd 0.0500 time 0.3376 (0.3693) data time 0.0006 (0.0019) model time 0.3370 (0.3480) loss 2.5247 (2.5699) grad_norm 2.9810 (inf) loss_scale 128.0000 (129.0475) mem 11219MB [2024-09-01 16:34:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [297/300][620/625] eta 0:00:01 lr 0.000020 wd 0.0500 time 0.3345 (0.3688) data time 0.0004 (0.0019) model time 0.3341 (0.3478) loss 2.8604 (2.5694) grad_norm 1.9750 (inf) loss_scale 128.0000 (129.0306) mem 11219MB [2024-09-01 16:34:21 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 297 training takes 0:03:50 [2024-09-01 16:34:21 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:34:26 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:34:26 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.448 (0.448) Loss 0.5205 (0.5205) Acc@1 89.062 (89.062) Acc@5 98.486 (98.486) Mem 11219MB [2024-09-01 16:34:27 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.130) Loss 0.8350 (0.6080) Acc@1 80.518 (87.061) Acc@5 95.850 (97.865) Mem 11219MB [2024-09-01 16:34:28 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.112) Loss 0.9199 (0.7271) Acc@1 79.102 (84.117) Acc@5 95.312 (96.640) Mem 11219MB [2024-09-01 16:34:30 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.813 Acc@5 96.649 [2024-09-01 16:34:30 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:34:31 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.811 (0.811) Loss 0.5093 (0.5093) Acc@1 89.062 (89.062) Acc@5 98.584 (98.584) Mem 11219MB [2024-09-01 16:34:32 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.089 (0.166) Loss 0.8257 (0.6028) Acc@1 80.762 (87.105) Acc@5 95.801 (97.856) Mem 11219MB [2024-09-01 16:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.094 (0.130) Loss 0.9126 (0.7208) Acc@1 78.662 (84.222) Acc@5 95.215 (96.668) Mem 11219MB [2024-09-01 16:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.911 Acc@5 96.693 [2024-09-01 16:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:34:33 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.91% [2024-09-01 16:34:33 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 16:34:37 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 16:34:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][0/625] eta 0:07:14 lr 0.000020 wd 0.0500 time 0.6944 (0.6944) data time 0.3401 (0.3401) model time 0.0000 (0.0000) loss 2.3295 (2.3295) grad_norm 2.7135 (2.7135) loss_scale 128.0000 (128.0000) mem 11228MB [2024-09-01 16:34:42 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][10/625] eta 0:03:52 lr 0.000020 wd 0.0500 time 0.3431 (0.3785) data time 0.0007 (0.0317) model time 0.0000 (0.0000) loss 2.5132 (2.5297) grad_norm 3.8150 (2.9339) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:34:45 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][20/625] eta 0:03:39 lr 0.000020 wd 0.0500 time 0.3444 (0.3630) data time 0.0009 (0.0170) model time 0.0000 (0.0000) loss 2.9299 (2.4344) grad_norm 3.3189 (2.9325) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:34:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][30/625] eta 0:03:32 lr 0.000020 wd 0.0500 time 0.3442 (0.3568) data time 0.0009 (0.0118) model time 0.0000 (0.0000) loss 1.8827 (2.4728) grad_norm 3.9087 (3.1843) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:34:52 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][40/625] eta 0:03:26 lr 0.000020 wd 0.0500 time 0.3480 (0.3538) data time 0.0006 (0.0091) model time 0.0000 (0.0000) loss 3.0984 (2.4530) grad_norm 2.6349 (3.0875) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:34:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][50/625] eta 0:03:22 lr 0.000020 wd 0.0500 time 0.3459 (0.3521) data time 0.0007 (0.0075) model time 0.0000 (0.0000) loss 2.1567 (2.5070) grad_norm 14.0489 (3.3552) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:34:59 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][60/625] eta 0:03:18 lr 0.000020 wd 0.0500 time 0.3457 (0.3506) data time 0.0007 (0.0064) model time 0.3450 (0.3423) loss 2.3987 (2.5061) grad_norm 4.5205 (3.3916) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][70/625] eta 0:03:15 lr 0.000020 wd 0.0500 time 0.3537 (0.3527) data time 0.0007 (0.0056) model time 0.3530 (0.3534) loss 2.1705 (2.4888) grad_norm 2.4679 (3.4000) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][80/625] eta 0:03:11 lr 0.000020 wd 0.0500 time 0.3446 (0.3516) data time 0.0007 (0.0050) model time 0.3439 (0.3499) loss 3.1501 (2.4987) grad_norm 6.5077 (3.4090) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][90/625] eta 0:03:07 lr 0.000020 wd 0.0500 time 0.3496 (0.3509) data time 0.0009 (0.0046) model time 0.3487 (0.3485) loss 2.5845 (2.4682) grad_norm 3.1355 (3.4137) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][100/625] eta 0:03:03 lr 0.000020 wd 0.0500 time 0.3539 (0.3503) data time 0.0007 (0.0042) model time 0.3532 (0.3476) loss 2.5297 (2.4740) grad_norm 7.6666 (3.5571) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][110/625] eta 0:03:00 lr 0.000020 wd 0.0500 time 0.3472 (0.3500) data time 0.0007 (0.0039) model time 0.3466 (0.3474) loss 3.2780 (2.5016) grad_norm 2.2730 (3.7931) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][120/625] eta 0:02:56 lr 0.000020 wd 0.0500 time 0.3449 (0.3497) data time 0.0006 (0.0037) model time 0.3442 (0.3470) loss 2.8595 (2.5155) grad_norm 2.9314 (3.7419) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:24 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][130/625] eta 0:02:55 lr 0.000020 wd 0.0500 time 0.3465 (0.3541) data time 0.0006 (0.0034) model time 0.3459 (0.3546) loss 3.1572 (2.5069) grad_norm 4.3542 (3.8394) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][140/625] eta 0:02:51 lr 0.000020 wd 0.0500 time 0.3435 (0.3537) data time 0.0010 (0.0033) model time 0.3425 (0.3538) loss 2.8126 (2.5233) grad_norm 2.2470 (3.8628) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:31 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][150/625] eta 0:02:47 lr 0.000020 wd 0.0500 time 0.3403 (0.3530) data time 0.0009 (0.0031) model time 0.3394 (0.3526) loss 2.5897 (2.5310) grad_norm 3.1600 (3.8101) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][160/625] eta 0:02:43 lr 0.000020 wd 0.0500 time 0.3481 (0.3525) data time 0.0010 (0.0030) model time 0.3471 (0.3518) loss 2.7143 (2.5331) grad_norm 2.6250 (3.8202) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:38 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][170/625] eta 0:02:40 lr 0.000020 wd 0.0500 time 0.3525 (0.3520) data time 0.0007 (0.0028) model time 0.3518 (0.3512) loss 1.7699 (2.5335) grad_norm 3.2577 (3.7887) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][180/625] eta 0:02:36 lr 0.000020 wd 0.0500 time 0.3500 (0.3516) data time 0.0007 (0.0027) model time 0.3493 (0.3505) loss 1.4287 (2.5264) grad_norm 5.2363 (3.8826) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][190/625] eta 0:02:32 lr 0.000020 wd 0.0500 time 0.3459 (0.3512) data time 0.0007 (0.0026) model time 0.3452 (0.3500) loss 3.3488 (2.5239) grad_norm 2.8340 (3.8288) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][200/625] eta 0:02:29 lr 0.000020 wd 0.0500 time 0.3421 (0.3509) data time 0.0009 (0.0025) model time 0.3413 (0.3496) loss 1.7032 (2.5167) grad_norm 2.7539 (3.7736) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][210/625] eta 0:02:25 lr 0.000020 wd 0.0500 time 0.3437 (0.3506) data time 0.0007 (0.0025) model time 0.3430 (0.3493) loss 3.2155 (2.5195) grad_norm 2.0948 (3.7455) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][220/625] eta 0:02:21 lr 0.000020 wd 0.0500 time 0.3449 (0.3504) data time 0.0006 (0.0024) model time 0.3443 (0.3490) loss 3.2966 (2.5158) grad_norm 13.0833 (3.7537) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:35:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][230/625] eta 0:02:18 lr 0.000020 wd 0.0500 time 0.3444 (0.3502) data time 0.0010 (0.0023) model time 0.3434 (0.3488) loss 2.7293 (2.5193) grad_norm 3.3187 (3.7481) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][240/625] eta 0:02:14 lr 0.000020 wd 0.0500 time 0.3456 (0.3500) data time 0.0009 (0.0023) model time 0.3447 (0.3485) loss 2.6761 (2.5186) grad_norm 3.0057 (3.7518) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][250/625] eta 0:02:11 lr 0.000020 wd 0.0500 time 0.3422 (0.3505) data time 0.0009 (0.0022) model time 0.3414 (0.3493) loss 2.5007 (2.5225) grad_norm 3.4488 (3.7576) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][260/625] eta 0:02:07 lr 0.000020 wd 0.0500 time 0.3453 (0.3503) data time 0.0006 (0.0022) model time 0.3446 (0.3490) loss 3.0211 (2.5179) grad_norm 2.2556 (3.7269) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][270/625] eta 0:02:04 lr 0.000020 wd 0.0500 time 0.3556 (0.3501) data time 0.0010 (0.0021) model time 0.3547 (0.3488) loss 2.8928 (2.5184) grad_norm 2.2455 (3.7276) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][280/625] eta 0:02:00 lr 0.000020 wd 0.0500 time 0.3544 (0.3500) data time 0.0009 (0.0021) model time 0.3535 (0.3487) loss 3.0057 (2.5240) grad_norm 2.6455 (3.7075) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][290/625] eta 0:01:57 lr 0.000020 wd 0.0500 time 0.3436 (0.3498) data time 0.0006 (0.0020) model time 0.3430 (0.3485) loss 3.0581 (2.5271) grad_norm 2.9370 (3.6924) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][300/625] eta 0:01:53 lr 0.000020 wd 0.0500 time 0.3431 (0.3497) data time 0.0009 (0.0020) model time 0.3422 (0.3483) loss 2.7153 (2.5358) grad_norm 3.4697 (3.6840) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][310/625] eta 0:01:50 lr 0.000020 wd 0.0500 time 0.5509 (0.3513) data time 0.0006 (0.0019) model time 0.5502 (0.3503) loss 1.6476 (2.5369) grad_norm 3.9808 (3.6642) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][320/625] eta 0:01:47 lr 0.000020 wd 0.0500 time 0.3416 (0.3522) data time 0.0008 (0.0019) model time 0.3408 (0.3514) loss 3.1252 (2.5366) grad_norm 2.4508 (3.6410) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][330/625] eta 0:01:43 lr 0.000020 wd 0.0500 time 0.3490 (0.3520) data time 0.0007 (0.0019) model time 0.3483 (0.3511) loss 2.6477 (2.5278) grad_norm 3.0073 (3.6244) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][340/625] eta 0:01:40 lr 0.000020 wd 0.0500 time 0.3470 (0.3518) data time 0.0008 (0.0018) model time 0.3462 (0.3509) loss 3.0663 (2.5314) grad_norm 7.4655 (3.6252) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][350/625] eta 0:01:36 lr 0.000020 wd 0.0500 time 0.3464 (0.3516) data time 0.0009 (0.0018) model time 0.3455 (0.3507) loss 2.8681 (2.5405) grad_norm 4.7129 (3.6326) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][360/625] eta 0:01:33 lr 0.000020 wd 0.0500 time 0.3462 (0.3515) data time 0.0009 (0.0018) model time 0.3453 (0.3505) loss 1.7061 (2.5380) grad_norm 3.0815 (3.6129) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][370/625] eta 0:01:29 lr 0.000020 wd 0.0500 time 0.3426 (0.3519) data time 0.0008 (0.0018) model time 0.3418 (0.3510) loss 2.5678 (2.5398) grad_norm 2.3333 (3.6299) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][380/625] eta 0:01:26 lr 0.000020 wd 0.0500 time 0.3484 (0.3518) data time 0.0009 (0.0018) model time 0.3476 (0.3509) loss 3.0582 (2.5394) grad_norm 4.1300 (3.6405) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][390/625] eta 0:01:22 lr 0.000020 wd 0.0500 time 0.3425 (0.3516) data time 0.0007 (0.0017) model time 0.3418 (0.3507) loss 2.1709 (2.5431) grad_norm 2.5828 (3.6169) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:36:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][400/625] eta 0:01:19 lr 0.000020 wd 0.0500 time 0.3523 (0.3515) data time 0.0008 (0.0017) model time 0.3514 (0.3506) loss 2.4671 (2.5365) grad_norm 5.5533 (3.6213) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][410/625] eta 0:01:15 lr 0.000020 wd 0.0500 time 0.3533 (0.3515) data time 0.0007 (0.0017) model time 0.3526 (0.3505) loss 3.0092 (2.5365) grad_norm 2.4445 (3.6096) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][420/625] eta 0:01:12 lr 0.000020 wd 0.0500 time 0.3564 (0.3515) data time 0.0009 (0.0017) model time 0.3555 (0.3505) loss 2.7453 (2.5393) grad_norm 2.6829 (3.5940) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][430/625] eta 0:01:08 lr 0.000020 wd 0.0500 time 0.3505 (0.3524) data time 0.0008 (0.0016) model time 0.3496 (0.3515) loss 2.5358 (2.5413) grad_norm 3.0561 (3.5913) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:13 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][440/625] eta 0:01:05 lr 0.000020 wd 0.0500 time 0.3487 (0.3527) data time 0.0006 (0.0016) model time 0.3480 (0.3519) loss 1.9655 (2.5426) grad_norm 2.4979 (3.5935) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][450/625] eta 0:01:01 lr 0.000020 wd 0.0500 time 0.3524 (0.3526) data time 0.0009 (0.0016) model time 0.3516 (0.3518) loss 2.3641 (2.5388) grad_norm 2.2196 (3.6247) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:20 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][460/625] eta 0:00:58 lr 0.000020 wd 0.0500 time 0.3428 (0.3524) data time 0.0007 (0.0016) model time 0.3421 (0.3516) loss 2.6855 (2.5403) grad_norm 3.4487 (3.6187) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:23 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][470/625] eta 0:00:54 lr 0.000020 wd 0.0500 time 0.3542 (0.3524) data time 0.0007 (0.0016) model time 0.3535 (0.3515) loss 2.7675 (2.5410) grad_norm 15.9388 (3.6431) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:27 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][480/625] eta 0:00:51 lr 0.000020 wd 0.0500 time 0.3470 (0.3523) data time 0.0009 (0.0016) model time 0.3461 (0.3515) loss 2.8574 (2.5442) grad_norm 3.8061 (3.6493) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][490/625] eta 0:00:47 lr 0.000020 wd 0.0500 time 0.3562 (0.3523) data time 0.0008 (0.0016) model time 0.3553 (0.3514) loss 2.6464 (2.5471) grad_norm 3.3255 (3.6417) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:34 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][500/625] eta 0:00:44 lr 0.000020 wd 0.0500 time 0.3508 (0.3523) data time 0.0009 (0.0015) model time 0.3499 (0.3514) loss 3.2395 (2.5484) grad_norm 3.8686 (3.6377) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][510/625] eta 0:00:40 lr 0.000020 wd 0.0500 time 0.3446 (0.3522) data time 0.0009 (0.0015) model time 0.3437 (0.3514) loss 2.9819 (2.5500) grad_norm 2.1068 (3.6264) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:41 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][520/625] eta 0:00:36 lr 0.000020 wd 0.0500 time 0.3481 (0.3522) data time 0.0007 (0.0015) model time 0.3474 (0.3513) loss 1.9414 (2.5494) grad_norm 1.9275 (3.6182) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][530/625] eta 0:00:33 lr 0.000020 wd 0.0500 time 0.3600 (0.3521) data time 0.0007 (0.0015) model time 0.3594 (0.3513) loss 3.1092 (2.5527) grad_norm 3.0929 (3.6174) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:48 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][540/625] eta 0:00:29 lr 0.000020 wd 0.0500 time 0.3485 (0.3521) data time 0.0007 (0.0015) model time 0.3478 (0.3513) loss 3.1397 (2.5487) grad_norm 3.8206 (3.6148) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][550/625] eta 0:00:26 lr 0.000020 wd 0.0500 time 0.3503 (0.3521) data time 0.0008 (0.0015) model time 0.3495 (0.3512) loss 2.6892 (2.5529) grad_norm 4.0274 (3.6073) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:55 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][560/625] eta 0:00:22 lr 0.000020 wd 0.0500 time 0.3531 (0.3521) data time 0.0008 (0.0015) model time 0.3522 (0.3512) loss 2.6772 (2.5581) grad_norm 5.7467 (3.6045) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:37:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][570/625] eta 0:00:19 lr 0.000020 wd 0.0500 time 0.3563 (0.3521) data time 0.0008 (0.0015) model time 0.3555 (0.3512) loss 2.5841 (2.5585) grad_norm 2.4606 (3.5983) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:02 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][580/625] eta 0:00:15 lr 0.000020 wd 0.0500 time 0.3478 (0.3523) data time 0.0007 (0.0015) model time 0.3471 (0.3514) loss 2.5243 (2.5624) grad_norm 3.5683 (3.5984) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:06 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][590/625] eta 0:00:12 lr 0.000020 wd 0.0500 time 0.3582 (0.3523) data time 0.0007 (0.0014) model time 0.3576 (0.3514) loss 2.0536 (2.5624) grad_norm 2.2782 (3.6073) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:09 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][600/625] eta 0:00:08 lr 0.000020 wd 0.0500 time 0.3460 (0.3522) data time 0.0006 (0.0014) model time 0.3454 (0.3514) loss 2.0651 (2.5605) grad_norm 3.0046 (3.6169) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][610/625] eta 0:00:05 lr 0.000020 wd 0.0500 time 0.3335 (0.3521) data time 0.0004 (0.0014) model time 0.3331 (0.3512) loss 2.0578 (2.5589) grad_norm 2.5962 (3.6079) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:16 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [298/300][620/625] eta 0:00:01 lr 0.000020 wd 0.0500 time 0.3334 (0.3518) data time 0.0006 (0.0014) model time 0.3328 (0.3509) loss 2.7181 (2.5593) grad_norm 5.2069 (3.6013) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:17 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 298 training takes 0:03:39 [2024-09-01 16:38:17 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:38:18 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:38:19 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.452 (0.452) Loss 0.5132 (0.5132) Acc@1 89.111 (89.111) Acc@5 98.633 (98.633) Mem 11224MB [2024-09-01 16:38:20 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.091 (0.131) Loss 0.8374 (0.6079) Acc@1 80.713 (87.096) Acc@5 96.094 (97.869) Mem 11224MB [2024-09-01 16:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.112) Loss 0.9219 (0.7275) Acc@1 79.053 (84.124) Acc@5 95.410 (96.654) Mem 11224MB [2024-09-01 16:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.829 Acc@5 96.673 [2024-09-01 16:38:21 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.8% [2024-09-01 16:38:22 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.875 (0.875) Loss 0.5093 (0.5093) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 16:38:23 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.170) Loss 0.8262 (0.6028) Acc@1 80.762 (87.140) Acc@5 95.801 (97.860) Mem 11224MB [2024-09-01 16:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.090 (0.132) Loss 0.9121 (0.7211) Acc@1 78.662 (84.249) Acc@5 95.264 (96.675) Mem 11224MB [2024-09-01 16:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.945 Acc@5 96.699 [2024-09-01 16:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:38:24 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.95% [2024-09-01 16:38:24 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 16:38:25 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 16:38:26 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][0/625] eta 0:07:15 lr 0.000020 wd 0.0500 time 0.6963 (0.6963) data time 0.3767 (0.3767) model time 0.0000 (0.0000) loss 2.5897 (2.5897) grad_norm 3.5463 (3.5463) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:30 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][10/625] eta 0:04:25 lr 0.000020 wd 0.0500 time 0.5420 (0.4321) data time 0.0008 (0.0351) model time 0.0000 (0.0000) loss 2.4549 (2.6783) grad_norm 3.4332 (3.2496) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][20/625] eta 0:03:56 lr 0.000020 wd 0.0500 time 0.3414 (0.3901) data time 0.0006 (0.0187) model time 0.0000 (0.0000) loss 2.1151 (2.6801) grad_norm 3.1604 (3.4890) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][30/625] eta 0:03:43 lr 0.000020 wd 0.0500 time 0.3421 (0.3751) data time 0.0006 (0.0129) model time 0.0000 (0.0000) loss 2.8412 (2.6872) grad_norm 4.0849 (3.5356) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][40/625] eta 0:03:34 lr 0.000020 wd 0.0500 time 0.3423 (0.3674) data time 0.0007 (0.0100) model time 0.0000 (0.0000) loss 2.8888 (2.6703) grad_norm 2.5276 (3.4113) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][50/625] eta 0:03:29 lr 0.000020 wd 0.0500 time 0.3503 (0.3635) data time 0.0007 (0.0082) model time 0.0000 (0.0000) loss 2.9877 (2.6685) grad_norm 3.0628 (3.4060) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][60/625] eta 0:03:24 lr 0.000020 wd 0.0500 time 0.3508 (0.3612) data time 0.0006 (0.0070) model time 0.3502 (0.3484) loss 1.5418 (2.6230) grad_norm 3.2803 (3.5953) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][70/625] eta 0:03:19 lr 0.000020 wd 0.0500 time 0.3468 (0.3591) data time 0.0008 (0.0061) model time 0.3460 (0.3469) loss 1.8838 (2.5850) grad_norm 6.8818 (3.5754) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][80/625] eta 0:03:16 lr 0.000020 wd 0.0500 time 0.3109 (0.3599) data time 0.0008 (0.0055) model time 0.3101 (0.3530) loss 2.9585 (2.5953) grad_norm 4.7050 (3.5910) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:38:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][90/625] eta 0:03:11 lr 0.000020 wd 0.0500 time 0.3485 (0.3585) data time 0.0006 (0.0050) model time 0.3479 (0.3512) loss 3.0090 (2.6059) grad_norm 4.7004 (3.5222) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][100/625] eta 0:03:07 lr 0.000020 wd 0.0500 time 0.3459 (0.3573) data time 0.0008 (0.0045) model time 0.3451 (0.3501) loss 2.5634 (2.6115) grad_norm 3.5277 (3.5127) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][110/625] eta 0:03:03 lr 0.000020 wd 0.0500 time 0.3470 (0.3566) data time 0.0008 (0.0042) model time 0.3461 (0.3500) loss 2.7706 (2.6254) grad_norm 3.4349 (3.6680) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][120/625] eta 0:02:59 lr 0.000020 wd 0.0500 time 0.3491 (0.3559) data time 0.0008 (0.0039) model time 0.3483 (0.3495) loss 2.6324 (2.6434) grad_norm 2.7073 (3.6028) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:12 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][130/625] eta 0:02:55 lr 0.000020 wd 0.0500 time 0.3458 (0.3553) data time 0.0009 (0.0037) model time 0.3450 (0.3492) loss 3.0125 (2.6503) grad_norm 4.0374 (3.5907) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][140/625] eta 0:02:52 lr 0.000020 wd 0.0500 time 0.3487 (0.3547) data time 0.0007 (0.0035) model time 0.3480 (0.3490) loss 2.5701 (2.6482) grad_norm 2.3934 (3.5753) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:19 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][150/625] eta 0:02:48 lr 0.000020 wd 0.0500 time 0.3450 (0.3543) data time 0.0009 (0.0033) model time 0.3441 (0.3488) loss 3.0605 (2.6419) grad_norm 2.9012 (3.6663) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][160/625] eta 0:02:44 lr 0.000020 wd 0.0500 time 0.3432 (0.3537) data time 0.0007 (0.0032) model time 0.3425 (0.3484) loss 2.4431 (2.6534) grad_norm 4.0708 (3.6461) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][170/625] eta 0:02:40 lr 0.000020 wd 0.0500 time 0.3463 (0.3533) data time 0.0009 (0.0030) model time 0.3454 (0.3481) loss 2.7581 (2.6524) grad_norm 3.1616 (3.6758) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][180/625] eta 0:02:37 lr 0.000020 wd 0.0500 time 0.3721 (0.3530) data time 0.0007 (0.0029) model time 0.3715 (0.3481) loss 2.9158 (2.6498) grad_norm 3.4038 (3.6318) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][190/625] eta 0:02:33 lr 0.000020 wd 0.0500 time 0.3458 (0.3527) data time 0.0006 (0.0028) model time 0.3452 (0.3480) loss 2.4297 (2.6444) grad_norm 31.3775 (3.7503) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][200/625] eta 0:02:30 lr 0.000020 wd 0.0500 time 0.3458 (0.3534) data time 0.0007 (0.0027) model time 0.3451 (0.3491) loss 2.8282 (2.6524) grad_norm 4.0122 (3.7190) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][210/625] eta 0:02:26 lr 0.000020 wd 0.0500 time 0.3531 (0.3532) data time 0.0008 (0.0026) model time 0.3522 (0.3491) loss 2.7211 (2.6566) grad_norm 4.1922 (3.7013) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:44 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][220/625] eta 0:02:24 lr 0.000020 wd 0.0500 time 0.5777 (0.3557) data time 0.0008 (0.0025) model time 0.5769 (0.3526) loss 1.5316 (2.6555) grad_norm 4.0552 (3.8635) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][230/625] eta 0:02:20 lr 0.000020 wd 0.0500 time 0.3510 (0.3553) data time 0.0006 (0.0025) model time 0.3504 (0.3522) loss 2.5928 (2.6435) grad_norm 3.5469 (3.8531) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][240/625] eta 0:02:16 lr 0.000020 wd 0.0500 time 0.3450 (0.3549) data time 0.0008 (0.0024) model time 0.3442 (0.3517) loss 2.5018 (2.6236) grad_norm 4.4222 (3.8119) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][250/625] eta 0:02:12 lr 0.000020 wd 0.0500 time 0.3400 (0.3544) data time 0.0009 (0.0023) model time 0.3391 (0.3513) loss 2.5936 (2.6210) grad_norm 3.1219 (3.7901) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:39:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][260/625] eta 0:02:09 lr 0.000020 wd 0.0500 time 0.3462 (0.3541) data time 0.0010 (0.0023) model time 0.3453 (0.3510) loss 2.6539 (2.6249) grad_norm 2.5895 (3.7619) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][270/625] eta 0:02:05 lr 0.000020 wd 0.0500 time 0.3540 (0.3539) data time 0.0007 (0.0022) model time 0.3533 (0.3508) loss 2.7152 (2.6289) grad_norm 4.0900 (3.7616) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][280/625] eta 0:02:01 lr 0.000020 wd 0.0500 time 0.3412 (0.3536) data time 0.0006 (0.0022) model time 0.3406 (0.3506) loss 1.6120 (2.6238) grad_norm 2.4768 (3.9019) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][290/625] eta 0:01:58 lr 0.000020 wd 0.0500 time 0.3476 (0.3534) data time 0.0006 (0.0021) model time 0.3470 (0.3504) loss 1.7307 (2.6237) grad_norm 2.5300 (3.8695) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][300/625] eta 0:01:54 lr 0.000020 wd 0.0500 time 0.3524 (0.3531) data time 0.0009 (0.0021) model time 0.3516 (0.3502) loss 2.5866 (2.6211) grad_norm 2.5788 (3.8915) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][310/625] eta 0:01:51 lr 0.000020 wd 0.0500 time 0.3486 (0.3529) data time 0.0007 (0.0020) model time 0.3480 (0.3500) loss 1.8342 (2.6214) grad_norm 1.8740 (3.8698) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][320/625] eta 0:01:47 lr 0.000020 wd 0.0500 time 0.3514 (0.3528) data time 0.0006 (0.0020) model time 0.3508 (0.3499) loss 2.6077 (2.6148) grad_norm 3.2115 (3.8655) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][330/625] eta 0:01:44 lr 0.000020 wd 0.0500 time 0.3472 (0.3526) data time 0.0006 (0.0020) model time 0.3466 (0.3498) loss 1.8365 (2.6099) grad_norm 4.7249 (3.8484) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][340/625] eta 0:01:40 lr 0.000020 wd 0.0500 time 0.3455 (0.3524) data time 0.0008 (0.0019) model time 0.3446 (0.3496) loss 2.5280 (2.6146) grad_norm 2.8397 (3.8741) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][350/625] eta 0:01:36 lr 0.000020 wd 0.0500 time 0.3512 (0.3522) data time 0.0006 (0.0019) model time 0.3506 (0.3495) loss 2.8573 (2.6201) grad_norm 2.9379 (3.8876) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:32 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][360/625] eta 0:01:33 lr 0.000020 wd 0.0500 time 0.3454 (0.3521) data time 0.0009 (0.0019) model time 0.3446 (0.3494) loss 2.2845 (2.6228) grad_norm 3.9025 (3.8746) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:36 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][370/625] eta 0:01:29 lr 0.000020 wd 0.0500 time 0.3438 (0.3520) data time 0.0009 (0.0019) model time 0.3430 (0.3493) loss 2.0195 (2.6205) grad_norm 3.0351 (3.9124) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:39 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][380/625] eta 0:01:26 lr 0.000020 wd 0.0500 time 0.3467 (0.3518) data time 0.0009 (0.0018) model time 0.3458 (0.3492) loss 3.2461 (2.6231) grad_norm 2.8698 (3.9257) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][390/625] eta 0:01:22 lr 0.000020 wd 0.0500 time 0.3507 (0.3517) data time 0.0006 (0.0018) model time 0.3501 (0.3491) loss 2.8623 (2.6221) grad_norm 3.1447 (3.9102) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:46 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][400/625] eta 0:01:19 lr 0.000020 wd 0.0500 time 0.5474 (0.3526) data time 0.0009 (0.0018) model time 0.5465 (0.3502) loss 2.0238 (2.6120) grad_norm 3.6860 (3.8962) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:51 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][410/625] eta 0:01:16 lr 0.000020 wd 0.0500 time 0.3470 (0.3543) data time 0.0009 (0.0018) model time 0.3461 (0.3522) loss 2.2076 (2.6065) grad_norm 2.6868 (3.9097) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][420/625] eta 0:01:12 lr 0.000020 wd 0.0500 time 0.3465 (0.3540) data time 0.0009 (0.0017) model time 0.3456 (0.3519) loss 2.5432 (2.6066) grad_norm 3.8023 (3.8987) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:40:58 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][430/625] eta 0:01:08 lr 0.000020 wd 0.0500 time 0.3469 (0.3538) data time 0.0008 (0.0017) model time 0.3461 (0.3517) loss 3.1178 (2.6069) grad_norm 2.6951 (3.8921) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][440/625] eta 0:01:05 lr 0.000020 wd 0.0500 time 0.3530 (0.3536) data time 0.0007 (0.0017) model time 0.3524 (0.3515) loss 2.3137 (2.6086) grad_norm 3.4536 (3.9391) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:05 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][450/625] eta 0:01:01 lr 0.000020 wd 0.0500 time 0.3465 (0.3535) data time 0.0006 (0.0017) model time 0.3458 (0.3514) loss 3.2259 (2.6136) grad_norm 2.2348 (3.9210) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:08 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][460/625] eta 0:00:58 lr 0.000020 wd 0.0500 time 0.3447 (0.3534) data time 0.0007 (0.0017) model time 0.3440 (0.3513) loss 2.4789 (2.6115) grad_norm 2.7108 (3.9008) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:11 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][470/625] eta 0:00:54 lr 0.000020 wd 0.0500 time 0.3533 (0.3532) data time 0.0006 (0.0016) model time 0.3526 (0.3512) loss 1.9695 (2.6086) grad_norm 2.3144 (3.8926) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:15 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][480/625] eta 0:00:51 lr 0.000020 wd 0.0500 time 0.3388 (0.3531) data time 0.0008 (0.0016) model time 0.3380 (0.3510) loss 2.7413 (2.6108) grad_norm 5.6415 (3.9438) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:18 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][490/625] eta 0:00:47 lr 0.000020 wd 0.0500 time 0.3494 (0.3529) data time 0.0009 (0.0016) model time 0.3486 (0.3509) loss 2.2319 (2.6084) grad_norm 3.4314 (3.9267) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:22 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][500/625] eta 0:00:44 lr 0.000020 wd 0.0500 time 0.3504 (0.3528) data time 0.0008 (0.0016) model time 0.3496 (0.3508) loss 2.6417 (2.6047) grad_norm 3.5209 (3.9143) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:25 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][510/625] eta 0:00:40 lr 0.000020 wd 0.0500 time 0.3462 (0.3527) data time 0.0007 (0.0016) model time 0.3455 (0.3507) loss 2.9856 (2.6046) grad_norm 3.8460 (3.9003) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:29 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][520/625] eta 0:00:37 lr 0.000020 wd 0.0500 time 0.3418 (0.3533) data time 0.0009 (0.0016) model time 0.3410 (0.3514) loss 2.2031 (2.6042) grad_norm 2.8886 (4.0479) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:33 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][530/625] eta 0:00:33 lr 0.000020 wd 0.0500 time 0.3453 (0.3540) data time 0.0007 (0.0015) model time 0.3446 (0.3521) loss 2.9823 (2.6041) grad_norm 3.4164 (4.0390) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:37 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][540/625] eta 0:00:30 lr 0.000020 wd 0.0500 time 0.3459 (0.3539) data time 0.0007 (0.0015) model time 0.3452 (0.3520) loss 2.7449 (2.6029) grad_norm 2.6646 (4.0244) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:40 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][550/625] eta 0:00:26 lr 0.000020 wd 0.0500 time 0.3524 (0.3537) data time 0.0008 (0.0015) model time 0.3516 (0.3518) loss 2.4045 (2.6054) grad_norm 2.8083 (4.0052) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:43 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][560/625] eta 0:00:22 lr 0.000020 wd 0.0500 time 0.3498 (0.3536) data time 0.0007 (0.0015) model time 0.3491 (0.3517) loss 1.9823 (2.6035) grad_norm 3.0092 (3.9870) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:47 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][570/625] eta 0:00:19 lr 0.000020 wd 0.0500 time 0.3457 (0.3535) data time 0.0008 (0.0015) model time 0.3449 (0.3516) loss 2.7389 (2.6041) grad_norm 3.6363 (3.9720) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:50 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][580/625] eta 0:00:15 lr 0.000020 wd 0.0500 time 0.3474 (0.3533) data time 0.0008 (0.0015) model time 0.3466 (0.3515) loss 2.7480 (2.6078) grad_norm 2.5076 (3.9551) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:54 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][590/625] eta 0:00:12 lr 0.000020 wd 0.0500 time 0.5655 (0.3536) data time 0.0006 (0.0015) model time 0.5649 (0.3518) loss 2.0473 (2.6062) grad_norm 4.4472 (3.9487) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:41:57 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][600/625] eta 0:00:08 lr 0.000020 wd 0.0500 time 0.3447 (0.3534) data time 0.0007 (0.0015) model time 0.3440 (0.3516) loss 1.8643 (2.6044) grad_norm 2.2811 (3.9405) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:42:01 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][610/625] eta 0:00:05 lr 0.000020 wd 0.0500 time 0.3329 (0.3533) data time 0.0004 (0.0015) model time 0.3325 (0.3515) loss 2.6252 (2.6048) grad_norm 13.3550 (3.9444) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:42:04 msvmambav3_small_224] (main_hfai_mnodes.py 371): INFO Train: [299/300][620/625] eta 0:00:01 lr 0.000020 wd 0.0500 time 0.3348 (0.3530) data time 0.0005 (0.0014) model time 0.3343 (0.3512) loss 2.9208 (2.6026) grad_norm 2.2795 (4.0079) loss_scale 128.0000 (128.0000) mem 11224MB [2024-09-01 16:42:06 msvmambav3_small_224] (main_hfai_mnodes.py 398): INFO EPOCH 299 training takes 0:03:40 [2024-09-01 16:42:06 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saving...... [2024-09-01 16:42:07 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/latest_ckpt.pth saved !!! [2024-09-01 16:42:07 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.464 (0.464) Loss 0.5151 (0.5151) Acc@1 89.014 (89.014) Acc@5 98.535 (98.535) Mem 11224MB [2024-09-01 16:42:08 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.132) Loss 0.8311 (0.6077) Acc@1 81.055 (87.056) Acc@5 96.143 (97.852) Mem 11224MB [2024-09-01 16:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.091 (0.112) Loss 0.9199 (0.7268) Acc@1 79.150 (84.173) Acc@5 95.264 (96.659) Mem 11224MB [2024-09-01 16:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.859 Acc@5 96.673 [2024-09-01 16:42:09 msvmambav3_small_224] (main_hfai_mnodes.py 261): INFO Accuracy of the network on the 50000 test images: 83.9% [2024-09-01 16:42:10 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [0/25] Time 0.760 (0.760) Loss 0.5098 (0.5098) Acc@1 89.111 (89.111) Acc@5 98.584 (98.584) Mem 11224MB [2024-09-01 16:42:11 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [10/25] Time 0.090 (0.165) Loss 0.8257 (0.6030) Acc@1 80.811 (87.149) Acc@5 95.850 (97.869) Mem 11224MB [2024-09-01 16:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 442): INFO Test: [20/25] Time 0.092 (0.129) Loss 0.9126 (0.7214) Acc@1 78.955 (84.261) Acc@5 95.215 (96.666) Mem 11224MB [2024-09-01 16:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 449): INFO * Acc@1 83.957 Acc@5 96.687 [2024-09-01 16:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 276): INFO Accuracy of the network on the 50000 test images: 84.0% [2024-09-01 16:42:12 msvmambav3_small_224] (main_hfai_mnodes.py 285): INFO New max accuracy ema: 83.96% [2024-09-01 16:42:12 msvmambav3_small_224] (utils.py 118): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saving...... [2024-09-01 16:42:14 msvmambav3_small_224] (utils.py 120): INFO ./exclude/output_msv2/msvmambav3_small_224/20240819232309/best_ckpt_ema.pth saved !!! [2024-09-01 16:42:14 msvmambav3_small_224] (main_hfai_mnodes.py 295): INFO Training time 0:11:47